A stock heterogeneous data and existing data joint Gaussian method, device, storage medium and equipment

Abstract

The application belongs to the technical field of computer vision, and discloses a stock heterogeneous data and existing data joint Gaussian method, device, storage medium and equipment. The application renders a stock tilt model through a virtual camera, generates a two-dimensional training image with accurate interior and exterior orientation elements, and efficiently constructs an initial Gaussian splash model based on the two-dimensional training image, so as to realize lightweight reuse of stock heterogeneous three-dimensional achievements. Through feature extraction and same-name point matching, a newly collected panoramic image and a virtual image are aligned in a world coordinate system, and then a new Gaussian point set reconstructed based on measured data is fused with an existing model. The method realizes rapid construction, incremental updating and accurate maintenance of a Gaussian splash model on the basis of stock data, significantly reduces data collection and calculation costs, and improves model updating efficiency and actual application adaptability.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a joint Gaussian method, apparatus, storage medium and device for existing heterogeneous data and existing data. Background Technology

[0002] 3D mesh models acquired using oblique photogrammetry have been widely applied in urban planning, environmental monitoring, intelligent transportation, and emergency management, providing crucial 3D spatial support for various refined decision-making and intelligent services. However, 3D mesh models suffer from drawbacks such as complex geometric topology, low rendering efficiency for large scenes, and difficulty in accurately representing weak textures or highly reflective areas. Gaussian splashing models are gradually emerging as a new generation of 3D reconstruction paradigms that combine high-quality real-time rendering capabilities with efficient explicit scene representation. Currently, the mainstream technical approaches to Gaussian splashing mainly fall into two categories: one is vision-based reconstruction methods, which generate Gaussian splashing models through multi-view image matching, feature extraction, and sparse point cloud reconstruction; the other is multimodal reconstruction methods that integrate multi-source sensor data such as laser point clouds and images. These methods typically initialize with point clouds and then iteratively optimize images to generate more detailed Gaussian splashing models. While these methods often outperform single-vision methods in terms of accuracy and detail, they suffer from large data volumes and high computational complexity.

[0003] The aforementioned methods share a common drawback: they all fail to directly and efficiently integrate existing tilted 3D model results with newly acquired data to generate a Gaussian splash model that meets the requirements. Both purely visual methods and multimodal fusion methods rely on the entire original data for overall reconstruction or global optimization, lacking an effective mechanism for reusing existing tilted model results. This necessitates large-scale data acquisition and overall calculations when scene updates are required, resulting in low update efficiency, redundant resource investment, and an inability to revitalize existing 3D data assets, failing to meet the dual demands of data timeliness and cost-effective updates in business applications. Summary of the Invention

[0004] To this end, embodiments of this application provide a joint Gaussian method, apparatus, storage medium, and device for existing heterogeneous data and existing data, which enables rapid construction, incremental updates, and accurate maintenance of Gaussian splashing models based on existing data, significantly reducing data acquisition and computation costs, and improving model update efficiency and practical application adaptability.

[0005] Firstly, this application provides a joint Gaussian method for existing heterogeneous data and existing data.

[0006] This application is achieved through the following technical solution: A joint Gaussian method for existing heterogeneous data and current data includes: Import the existing tilt model stored in a grid format; Simulate a virtual camera to take pictures from multiple angles around the existing tilt model, and generate a set of camera poses by making the camera face the center of the existing tilt model; Render the existing tilt model from each camera pose to generate a rendered image; The grid of the existing tilt model is used as the initial Gaussian points of the Gaussian model, and the Gaussian model is trained using the rendered image to optimize the Gaussian point parameters. Panoramic images are captured by a panoramic camera, and each panoramic image is converted into a cube map through spherical coordinate mapping. Sparse reconstruction based on cube mapping; Extract local feature points from the rendered image, construct image pairing based on the initial pose of the panoramic camera and the aerial rendered image, obtain corresponding point pairs in the image pairing, and when a preset number of corresponding point pairs are obtained, solve the exterior orientation elements of the model rendered image camera in the world coordinate system, and solve the rotation matrix and translation vector from the model coordinate system to the world coordinate system. Render a Gaussian model at the same location in three-dimensional space to generate a simulated image, and calculate the structural similarity between the simulated image and the measured panoramic image. When the structural similarity is lower than the preset similarity threshold, the Gaussian sphere will be deleted. New Gaussian elements are constructed based on position, covariance, color, and weight. The new Gaussian elements are compared with the original Gaussian set one by one by Mahalanobis distance. All Gaussian elements whose Mahalanobis distance is less than a preset distance threshold are collected and merged to obtain a new Gaussian model.

[0007] In a preferred example of this application, the step of generating a rendered image from a stock tilt model rendered from each camera pose can be further configured as follows: Using image rasterization technology, a two-dimensional color rendered image is generated from the existing model of the camera pose rendering of each virtual camera: The intrinsic parameter matrix of the virtual camera is World point coordinates Projected onto image point coordinates Represented as: , in, It is a projection function.

[0008] In a preferred embodiment of this application, the step of using the mesh of the existing tilt model as the initial Gaussian point of the Gaussian model includes: The vertex coordinates of each grid in the stock tilt model are used as the Gaussian point positions of the Gaussian model, the color is mapped from the texture, and the covariance matrix is ​​calculated based on the vertex neighborhood.

[0009] In a preferred example of this application, sparse reconstruction based on a cube map can be further configured, including: Local feature points of the cube map are extracted using a feature detection algorithm; Feature matching is performed on all cube maps using sequence matching. Global optimization is performed based on geometric constraints.

[0010] In a preferred example of this application, the method can be further configured to compare the new Gaussian elements with the existing Gaussian set one by one using Mahalanobis distance, and collect and merge all Gaussian elements whose Mahalanobis distance is less than a preset distance threshold, including: For each new Gaussian element and the existing Gaussian set Compare the Mahalanobis distances one by one ; When the Mahal distance If the distance is less than a preset threshold, the new Gaussian element is considered to meet the candidate merging condition, and an index set of all Gaussian elements that meet the candidate merging condition is collected. ; When the index set When there is only one candidate Gaussian element, the nearest neighbor merging principle is used to merge the new Gaussian element with the original Gaussian set; When the index set When there are multiple candidate Gaussian elements, the multi-weighted merging principle is used to merge the new Gaussian elements with the original Gaussian set.

[0011] In a preferred example of this application, the merged Gaussian model may further include: New simulated images are generated by rendering based on the new Gaussian model, and the structural similarity between the new simulated images and the measured panoramic images is calculated. For regions with structural similarity less than a preset similarity threshold, an iterative Gaussian compaction process is performed until the structural similarity meets the requirements.

[0012] In a preferred example of this application, the preset similarity threshold can be further set to 0.5.

[0013] Secondly, this application provides a joint Gaussian device for existing heterogeneous data and existing data.

[0014] This application is achieved through the following technical solution: A joint Gaussian device for existing heterogeneous data and current data, used to execute the joint Gaussian method for existing heterogeneous data and current data as described in the first aspect above, comprising: The Gaussian model optimization module is used to acquire panoramic images through a panoramic camera, convert each panoramic image into a cube map through spherical coordinate mapping, perform sparse reconstruction based on the cube map, extract local feature points of the rendered image, construct image pairing based on the initial pose value of the panoramic camera and the aerial rendered image, obtain corresponding point pairs in the image pairing, when a preset number of corresponding point pairs are obtained, solve the exterior orientation elements of the model rendered image camera in the world coordinate system, solve the rotation matrix and translation vector from the model coordinate system to the world coordinate system, render the Gaussian model at the same position in 3D space to generate a simulated image, calculate the structural similarity between the simulated image and the measured panoramic image, delete the Gaussian sphere when the structural similarity is lower than a preset similarity threshold, construct new Gaussian elements based on position, covariance, color and weight, compare the new Gaussian elements with the original Gaussian set one by one by Mahalanobis distance, collect all Gaussian elements whose Mahalanobis distance is less than a preset distance threshold and merge them to obtain a new Gaussian model.

[0015] Thirdly, this application is achieved through the following technical solution: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described joint Gaussian methods for existing heterogeneous data and existing data.

[0016] Fourthly, this application provides a computer-readable storage medium.

[0017] This application is achieved through the following technical solution: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described joint Gaussian methods for existing heterogeneous data and existing data.

[0018] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following: This application proposes an efficient Gaussian splash modeling method for fusing existing and newly acquired data. First, an existing tilt model is rendered using a virtual camera to generate a 2D training image with precise interior and exterior orientation elements. This image is then used to efficiently construct an initial Gaussian splash model, enabling lightweight reuse of existing heterogeneous 3D results. Second, through feature extraction and corresponding point matching, the pose of the newly acquired panoramic image and the virtual image is aligned in the world coordinate system. Then, the newly reconstructed Gaussian point set based on measured data is fused with the existing model. This fusion employs an adaptive strategy of weighted merging of Gaussian points in consistent regions and adding new Gaussian points in dissimilar regions. This maintains the spatiotemporal continuity and consistency of the model while supporting dynamic scene expansion and updates. This method enables rapid construction, incremental updates, and precise maintenance of the Gaussian splash model based on existing data, significantly reducing data acquisition and computation costs, and improving model update efficiency and practical application adaptability. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the joint Gaussian method for existing heterogeneous data and existing data provided in an embodiment of this application. Detailed Implementation

[0020] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application. To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this application, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0022] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0023] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0024] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0025] Figure 1 This is a first exemplary embodiment of the present application providing a joint Gaussian method for existing heterogeneous data and existing data, which includes: S1: Import the stock tilt model stored in grid form.

[0026] In actual implementation, a 3D stock tilt model is imported. This model is usually stored in a mesh format, such as OBJ, PLY, or OSGB, and includes at least the model's vertex coordinates, face indices, texture coordinates, and texture image.

[0027] S2: Simulate a virtual camera to take pictures from multiple angles around the existing tilt model, so that the camera is oriented toward the center of the existing tilt model to generate a set of camera poses.

[0028] Specifically, assuming the stock skew model is as follows: ,in It is a set of vertices. , It is a collection of dough pieces. , It is a texture mapping relationship.

[0029] A virtual camera is used to simulate the original data acquisition path, such as the flight trajectory of a drone. This is achieved by sampling the camera position above the surface of an existing tilted model and aligning the camera towards the model's center or along the surface normal, generating a set of virtual camera poses. These poses include both the camera position and orientation. The camera pose generation needs to cover the entire existing tilted model to ensure the diversity of the rendered images. The camera pose is represented by an extrinsic parameter matrix, such as the first... Each pose is represented as ,in Represented as a rotation matrix, Let be the translation vector. Camera pose generation can be achieved using spherical sampling, where the radius of the model's bounding sphere is . The camera position is represented as: , in, and For sampling angle, The virtual camera points towards the center of the model.

[0030] S3: Render the stock tilt model from each camera pose to generate a rendered image.

[0031] In some embodiments, generating a rendered image from a stock tilt model rendered from each camera pose includes: Using computer image rasterization technology, two-dimensional color rendered images are generated from the existing camera pose rendering model of each virtual camera: The intrinsic parameter matrix of the virtual camera is World point coordinates Projected onto image point coordinates Represented as: , in, It is a projection function.

[0032] The rendering process utilizes GPU acceleration to ensure efficiency, and the rendered images are used as training data for training the Gaussian model.

[0033] S4: Use the grid of the existing tilt model as the initial Gaussian points of the Gaussian model, and train the Gaussian model using the rendered image to optimize the Gaussian point parameters.

[0034] In some embodiments, using the mesh of the stock tilt model as the initial Gaussian points of the Gaussian model includes: The vertex coordinates of each grid in the stock tilt model are used as the Gaussian point positions of the Gaussian model, the color is mapped from the texture, and the covariance matrix is ​​calculated based on the vertex neighborhood.

[0035] Specifically, the parameters of the 3D Gaussian points are initialized from the existing tilted model, including position, color, transparency, and covariance. The position of the Gaussian point is initialized to the vertex coordinates of each mesh, the color is obtained from the texture mapping, and the covariance matrix is ​​calculated based on the vertex neighborhood.

[0036] Assuming the vertex of the stock tilt model Corresponding Gaussian point Its parameters are initialized as follows: Location ,color Obtained from texture sampling, transparency Initialized to fixed values, covariance matrix The calculation method is expressed as follows: , in, yes The set of k nearest neighbors.

[0037] S5: Captures panoramic images using a panoramic camera, and converts each panoramic image into a cube map through spherical coordinate mapping.

[0038] Specifically, the panoramic image is a 360-degree image, stitched together from simultaneous shots taken by two fisheye cameras. During acquisition, GPS and IMU data are recorded to provide initial camera pose values, and the acquired data can be represented as a panoramic image. ,in For timestamps.

[0039] The panoramic image is converted into a cubemap, consisting of six face images: front, back, left, right, top, and bottom. Each face image is treated as a perspective image, simplifying pose estimation and rendering.

[0040] The transformation can be achieved through spherical coordinate mapping, assuming the panoramic image is in spherical coordinates. This is mapped to a cube face. For a cube face... front, back, left, right, up, down Image points The mapping relationship with 3D direction vectors is expressed as follows: normalize The specific determination depends on the type of surface.

[0041] S6: Sparse reconstruction based on cube maps; In some embodiments, the sparse reconstruction process includes: Local feature points of the cube map are extracted using a feature detection algorithm; Feature matching is performed on all cube maps using sequence matching. Global optimization is performed based on geometric constraints.

[0042] Specifically, the sparse reconstruction process includes feature extraction, feature matching, and global optimization. First, the SIFT feature detection algorithm is used to extract local feature points from the six cubemaps (also called cube maps) decomposed from the panoramic image. Then, sequence matching is used to perform feature matching on all cubemaps. Since the camera positions of the panoramic images originate from the same projection center, the camera positions of each panoramic image are strictly consistent. The pose (rotation matrix) of the panoramic image also has fixed, known geometric relationships.

[0043] Global optimization utilizes the aforementioned geometric relationships. Assume the pose of the panoramic camera in the world coordinate system is... and For a specific cube map, let's take the left side as an example to illustrate, where the camera's extrinsic parameter matrix in the world coordinate system is... The global pose can be compared with the fixed relative pose of the surface. The combination yields: , , in, It is a known fixed rotation matrix, representing a 90-degree rotation from the front to the left.

[0044] Based on geometric constraints, the sparse reconstruction optimization problem is formulated as follows: Assumption For the first A 3D spatial point, For the first The spatial point at the th 2D pixel coordinates observed on a cube map To base on the camera model Projected onto image The sparse reconstruction optimization problem for functions on the above is formulated as follows: , in, Indicates the first There are M panoramic shooting locations. Indicates the first There are 6 cubemap faces in total. It is the first A fixed rotation matrix of each cubemap face relative to the global pose of the panoramic camera. It is the first A fixed intrinsic parameter matrix for each cubemap surface. It is in the The position of the first A set of 3D points visible on a surface, of which and It is known a priori.

[0045] S7: Extract local feature points from the rendered image, construct image pairing based on the initial pose value of the panoramic camera and the aerial rendered image, obtain corresponding point pairs in the image pairing, and when a preset number of corresponding point pairs are obtained, solve the exterior orientation elements of the model rendered image camera in the world coordinate system, and solve the rotation matrix and translation vector from the model coordinate system to the world coordinate system.

[0046] The purpose of this step is to perform associative visual localization.

[0047] Specifically, the SIFT feature detection algorithm is used to extract local feature points from the rendered image. Image pairing is constructed based on the initial pose values ​​of the panoramic camera and the aerial rendered image. The Random Consistency Sampling (RANSAC) algorithm is used to obtain a set of corresponding point pairs in the image pairing. After obtaining at least four pairs of corresponding points, the PnP algorithm is used to solve for the exterior orientation elements of the camera in the world coordinate system of the model rendering image. The rotation matrix from the model coordinate system to the world coordinate system is then solved by minimizing the projection point error. Translation vector .

[0048] , in, This represents the projection operator from homogeneous coordinates to pixel coordinates.

[0049] S8: Render the Gaussian model at the same location in 3D space to generate a simulated image, and calculate the structural similarity between the simulated image and the measured panoramic image.

[0050] Specifically, the calculation of structural similarity is expressed as follows: , In the formula, It is the average brightness of the photo. This refers to contrast ratio (standard deviation). It is covariance. It is a constant used to prevent the denominator from being zero.

[0051] S9: When the structural similarity is lower than the preset similarity threshold, the Gaussian sphere will be deleted.

[0052] The preset similarity threshold is set to 0.5.

[0053] Specifically, when the structural similarity is below 0.5, the Gaussian point is considered to be significantly inconsistent with the measured effect, representing model error or scene change. In this case, the Gaussian point is deleted.

[0054] S10: Construct new Gaussian units based on position, covariance, color, and weight. Compare the new Gaussian units with the original Gaussian set one by one using Mahalanobis distance. Collect all Gaussian units whose Mahalanobis distance is less than a preset distance threshold and merge them to obtain a new Gaussian model.

[0055] In some embodiments, the new Gaussian elements are compared one by one with the existing Gaussian set by Mahalanobis distance, and all Gaussian elements whose Mahalanobis distance is less than a preset distance threshold are collected and merged, including: For each new Gaussian element and the existing Gaussian set Compare the Mahalanobis distances one by one ; When the Mahal distance If the distance is less than a preset distance threshold of 3.0, the new Gaussian element is considered to meet the candidate merging condition, and the index set of all Gaussian elements that meet the candidate merging condition is collected. ; When the index set When there is only one candidate Gaussian element, the nearest neighbor merging principle is used to merge the new Gaussian element with the original Gaussian set; When the index set When there are multiple candidate Gaussian elements, the multi-weighted merging principle is used to merge the new Gaussian elements with the original Gaussian set.

[0056] In actual implementation, for each measured image pixel Using depth estimation Construct three-dimensional points in the world coordinate system; based on the depth error variance and pixel coordinate precision The covariance is calculated through error propagation.

[0057] In some preferred embodiments, after obtaining a new Gaussian model through merging, the method further includes: New simulated images are generated by rendering based on the new Gaussian model, and the structural similarity between the new simulated images and the measured panoramic images is calculated. For regions with structural similarity less than a preset similarity threshold, an iterative Gaussian compaction process is performed until the structural similarity meets the requirements.

[0058] This application proposes a general method for 3D model conversion and incremental update, capable of uniformly converting common existing 3D model formats (such as OBJ, PLY, or OSGB) into Gaussian splash models to support subsequent dynamic updates and fusion. This method renders existing 3D models using a virtual camera, generating multi-view 2D color images and accurately recording the interior and exterior orientation parameters of each image. These rendered images and their pose information are used as training data to efficiently generate Gaussian splash models, thereby achieving standardized representation and preprocessing of existing 3D data.

[0059] Further, the Gaussian model is fused and optimized based on cross-temporal data. Through feature extraction and corresponding point matching, the newly acquired panoramic images and virtual rendered images are visually located, and their poses are uniformly registered in the world coordinate system. Subsequently, the measured data is transformed into a new Gaussian point set through depth reconstruction and compared with the original Gaussian model in detail. For Gaussian points that are spatially close and have the same appearance or structure, a weighted update method is used for fusion to maintain the continuity and temporal consistency of the model; for newly appearing or significantly different regions, Gaussian points are adaptively added to accurately reflect scene changes, thereby achieving dynamic expansion and optimization of the model.

[0060] Another embodiment of this application provides a joint Gaussian device for existing heterogeneous data and existing data, used to perform the above method, the device comprising: The Gaussian model conversion module is used to import existing tilt models stored in a mesh format; simulate a virtual camera to take multi-angle shots around the existing tilt model, so that the camera faces the center of the existing tilt model to generate a set of camera poses; render the existing tilt model from each camera pose to generate a rendered image; use the mesh of the existing tilt model as the initial Gaussian points of the Gaussian model, and use the rendered image to train the Gaussian model and optimize the Gaussian point parameters. The Gaussian model optimization module is used to acquire panoramic images through a panoramic camera, convert each panoramic image into a cube map through spherical coordinate mapping, perform sparse reconstruction based on the cube map, extract local feature points of the rendered image, construct image pairing based on the initial pose value of the panoramic camera and the aerial rendered image, obtain corresponding point pairs in the image pairing, when a preset number of corresponding point pairs are obtained, solve the exterior orientation elements of the model rendered image camera in the world coordinate system, solve the rotation matrix and translation vector from the model coordinate system to the world coordinate system, render the Gaussian model at the same position in 3D space to generate a simulated image, calculate the structural similarity between the simulated image and the measured panoramic image, delete the Gaussian sphere when the structural similarity is lower than a preset similarity threshold, construct new Gaussian elements based on position, covariance, color and weight, compare the new Gaussian elements with the original Gaussian set one by one by Mahalanobis distance, collect all Gaussian elements whose Mahalanobis distance is less than a preset distance threshold and merge them to obtain a new Gaussian model.

[0061] The specific limitations of the joint Gaussian device for existing heterogeneous data and current data provided in this embodiment can be found in the embodiments of the joint Gaussian method for existing heterogeneous data and current data described above, and will not be repeated here. Each module in the aforementioned joint Gaussian device for existing heterogeneous data and current data can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0062] This application provides a computer device that may include a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it causes the processor to perform the steps of the joint Gaussian method for existing heterogeneous data and existing data as described in any of the above embodiments.

[0063] The working process, working details and technical effects of the computer device provided in this embodiment can be found in the embodiment of the joint Gaussian method for existing heterogeneous data and existing data in the above text, and will not be repeated here.

[0064] This application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the joint Gaussian method for existing heterogeneous data and existing data as described in any of the above embodiments. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0065] The working process, working details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the embodiment of the joint Gaussian method for existing heterogeneous data and existing data described above, and will not be repeated here.

[0066] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0067] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system described in this application can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A joint Gaussian method for existing heterogeneous data and current data, characterized in that, include: Import the existing tilt model stored in a grid format; Simulate a virtual camera to take pictures from multiple angles around the existing tilt model, and generate a set of camera poses by making the camera face the center of the existing tilt model; Render the existing tilt model from each camera pose to generate a rendered image; The grid of the existing tilt model is used as the initial Gaussian points of the Gaussian model, and the Gaussian model is trained using the rendered image to optimize the Gaussian point parameters. Panoramic images are captured by a panoramic camera, and each panoramic image is converted into a cube map through spherical coordinate mapping. Sparse reconstruction based on cube mapping; Extract local feature points from the rendered image, construct image pairing based on the initial pose of the panoramic camera and the aerial rendered image, obtain corresponding point pairs in the image pairing, and when a preset number of corresponding point pairs are obtained, solve the exterior orientation elements of the model rendered image camera in the world coordinate system, and solve the rotation matrix and translation vector from the model coordinate system to the world coordinate system. Render a Gaussian model at the same location in three-dimensional space to generate a simulated image, and calculate the structural similarity between the simulated image and the measured panoramic image. When the structural similarity is lower than the preset similarity threshold, the Gaussian sphere will be deleted. New Gaussian elements are constructed based on position, covariance, color, and weight. The new Gaussian elements are compared with the original Gaussian set one by one by Mahalanobis distance. All Gaussian elements whose Mahalanobis distance is less than a preset distance threshold are collected and merged to obtain a new Gaussian model.

2. The joint Gaussian method for existing heterogeneous data and current data according to claim 1, characterized in that, The step of rendering a stock tilt model from each camera pose to generate a rendered image includes: Using image rasterization technology, a two-dimensional color rendered image is generated from the existing model of the camera pose rendering of each virtual camera: The intrinsic parameter matrix of the virtual camera is World point coordinates Projected onto image point coordinates Represented as: ， in, It is a projection function.

3. The joint Gaussian method for existing heterogeneous data and current data according to claim 1, characterized in that, The step of using the mesh of the existing tilt model as the initial Gaussian points of the Gaussian model includes: The vertex coordinates of each grid in the stock tilt model are used as the Gaussian point positions of the Gaussian model, the color is mapped from the texture, and the covariance matrix is ​​calculated based on the vertex neighborhood.

4. The joint Gaussian method for existing heterogeneous data and current data according to claim 1, characterized in that, Sparse reconstruction based on cube maps includes: Local feature points of the cube map are extracted using a feature detection algorithm; Feature matching is performed on all cube maps using sequence matching. Global optimization is performed based on geometric constraints.

5. The joint Gaussian method for existing heterogeneous data and current data according to claim 1, characterized in that, The new Gaussian elements are compared one by one with the existing Gaussian set by Mahalanobis distance. All Gaussian elements whose Mahalanobis distance is less than a preset distance threshold are collected and merged, including: For each new Gaussian element and the existing Gaussian set Compare the Mahalanobis distances one by one ; When the Mahal distance If the distance is less than a preset threshold, the new Gaussian element is considered to meet the candidate merging condition, and an index set of all Gaussian elements that meet the candidate merging condition is collected. ; When the index set When there is only one candidate Gaussian element, the nearest neighbor merging principle is used to merge the new Gaussian element with the original Gaussian set; When the index set When there are multiple candidate Gaussian elements, the multi-weighted merging principle is used to merge the new Gaussian elements with the original Gaussian set.

6. The joint Gaussian method for existing heterogeneous data and current data according to claim 1, characterized in that, After merging, the resulting new Gaussian model also includes: New simulated images are generated by rendering based on the new Gaussian model, and the structural similarity between the new simulated images and the measured panoramic images is calculated. For regions with structural similarity less than a preset similarity threshold, an iterative Gaussian compaction process is performed until the structural similarity meets the requirements.

7. The joint Gaussian method for existing heterogeneous data and current data according to any one of claims 1 to 6, characterized in that, The preset similarity threshold is set to 0.

5.

8. A joint Gaussian device for existing heterogeneous data and current data, characterized in that, For performing the method as described in any one of claims 1 to 7, comprising: The Gaussian model conversion module is used to import existing tilt models stored in a mesh format; simulate a virtual camera to take multi-angle shots around the existing tilt model, so that the camera faces the center of the existing tilt model to generate a set of camera poses; render the existing tilt model from each camera pose to generate a rendered image; use the mesh of the existing tilt model as the initial Gaussian points of the Gaussian model, and use the rendered image to train the Gaussian model and optimize the Gaussian point parameters. The Gaussian model optimization module is used to acquire panoramic images through a panoramic camera, convert each panoramic image into a cube map through spherical coordinate mapping, perform sparse reconstruction based on the cube map, extract local feature points of the rendered image, construct image pairing based on the initial pose value of the panoramic camera and the aerial rendered image, obtain corresponding point pairs in the image pairing, when a preset number of corresponding point pairs are obtained, solve the exterior orientation elements of the model rendered image camera in the world coordinate system, solve the rotation matrix and translation vector from the model coordinate system to the world coordinate system, render the Gaussian model at the same position in 3D space to generate a simulated image, calculate the structural similarity between the simulated image and the measured panoramic image, delete the Gaussian sphere when the structural similarity is lower than a preset similarity threshold, construct new Gaussian elements based on position, covariance, color and weight, compare the new Gaussian elements with the original Gaussian set one by one by Mahalanobis distance, collect all Gaussian elements whose Mahalanobis distance is less than a preset distance threshold and merge them to obtain a new Gaussian model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

Images