A three-dimensional model real-time generation system based on multi-source data synchronous acquisition equipment

Abstract

The application provides a three-dimensional model real-time generation system based on a multi-source data synchronous acquisition device, and relates to the technical field of three-dimensional modeling.The system comprises a multi-source data synchronous acquisition device, a control device, a processor and a memory storing a computer program.The multi-source data synchronous acquisition device comprises a device main body, and is integrated with a laser radar, an image acquisition device, an RTK measurement device and a Beidou satellite positioning device.In the application, an initial Gauss ellipsoid set can be initialized and generated based on scene images and depth point clouds corresponding to a plurality of acquisition angles of a scene to be modeled, and an initial Gauss ellipsoid corresponding to each key single object corresponding to a corresponding cluster of a target Gauss ellipsoid is replaced, so that the updating of the initial Gauss ellipsoid set is realized, the data calculation amount in the subsequent optimization process of the initial Gauss ellipsoid set after all updates is effectively reduced, the calculation load is reduced, and the generation efficiency of the three-dimensional model is improved.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and in particular to a real-time 3D model generation system based on a multi-source data synchronous acquisition device. Background Technology

[0002] In recent years, the efficient and high-quality reconstruction of 3D scenes has received widespread attention in fields such as digital twins and virtual reality. 3D Gaussian sputtering is an efficient method for displaying 3D scene representations. This method first acquires multi-view images of the scene to be modeled, processes the multi-view images using motion recovery structures to generate sparse point clouds, initializes several 3D Gaussian ellipsoids based on the sparse point clouds, and optimizes the Gaussian ellipsoids to finally obtain a 3D model of the scene to be modeled. This method has been widely used in the field of 3D modeling technology.

[0003] However, the above method also has the following technical problems:

[0004] When building a large-scale 3D model of a scene to be modeled, the number of initialized 3D Gaussian ellipsoids increases significantly as the scale of the scene increases. This leads to a large amount of computing resources being consumed in the subsequent optimization of the initialized 3D Gaussian ellipsoids, which in turn reduces the efficiency of building the 3D model of the scene to be modeled. Summary of the Invention

[0005] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0006] A real-time 3D model generation system based on a multi-source data synchronous acquisition device, comprising: a multi-source data synchronous acquisition device, a control unit, a processor, and a memory storing a computer program; the multi-source data synchronous acquisition device includes a main body integrating a lidar, an image acquisition device, an RTK measurement device, and a BeiDou satellite positioning device; when the computer program is executed by the processor, the following steps are implemented:

[0007] S1. Based on several scene images and depth point clouds with different acquisition perspectives corresponding to the scene to be modeled, an initial Gaussian ellipsoid set is generated; the scene images and depth point clouds are obtained by data acquisition through a multi-source data synchronous acquisition device driven by a control device.

[0008] S2. Cluster the target feature vectors corresponding to all key individual targets to obtain several clusters; key individual targets are obtained based on all scene images.

[0009] S3. If the number of target feature vectors in a cluster is not less than a preset threshold, then based on the scene sub-image list set corresponding to the cluster and the Gaussian ellipsoid prediction model, obtain several target Gaussian ellipsoids corresponding to the cluster; the scene sub-image list set includes the scene sub-image list corresponding to each key single target of the cluster; the scene sub-image list includes each scene sub-image corresponding to the key single target.

[0010] S4. Replace the initial Gaussian ellipsoid corresponding to each key single target in the initial Gaussian ellipsoid set with the target Gaussian ellipsoid corresponding to the cluster, so as to update the initial Gaussian ellipsoid set.

[0011] S5. Optimize the initial Gaussian ellipsoid set after all updates are completed to obtain the 3D model corresponding to the scene to be modeled.

[0012] The present invention has at least the following beneficial effects:

[0013] This invention provides a real-time 3D model generation system based on a multi-source data synchronous acquisition device. The system includes: a multi-source data synchronous acquisition device, a control device, a processor, and a memory storing a computer program. The multi-source data synchronous acquisition device includes a main body that integrates a lidar, an image acquisition device, an RTK measurement device, and a BeiDou satellite positioning device. When the computer program is executed by the processor, it can initialize and generate an initial Gaussian ellipsoid set based on several scene images and depth point clouds with different acquisition perspectives corresponding to the scene to be modeled. It then performs clustering processing on the target feature vectors corresponding to all key individual targets to obtain several clusters. If the number of target feature vectors in a cluster is not less than a preset threshold, it obtains several target Gaussian ellipsoids corresponding to that cluster based on the scene sub-image list set and the Gaussian ellipsoid prediction model corresponding to that cluster. The initial Gaussian ellipsoids corresponding to each key individual target in the initial Gaussian ellipsoid set are replaced with the target Gaussian ellipsoids corresponding to that cluster in the initial Gaussian ellipsoid set, thereby updating the initial Gaussian ellipsoid set. Finally, the updated initial Gaussian ellipsoid set is optimized to obtain the 3D model corresponding to the scene to be modeled. As can be seen, this invention introduces a Gaussian ellipsoid prediction model and a target Gaussian ellipsoid replacement mechanism. When the number of target feature vectors in a cluster is not less than a preset threshold, the target Gaussian ellipsoid replaces the initial Gaussian ellipsoid corresponding to each key single target in the corresponding cluster, thereby updating the initial Gaussian ellipsoid set. This effectively reduces the amount of data computation in the subsequent optimization process of the initial Gaussian ellipsoid set after all updates have been completed, reduces the computational load, and thus improves the generation efficiency of the three-dimensional model. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 The flowchart shows the execution of a computer program for a real-time 3D model generation system based on a multi-source data synchronous acquisition device, as provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar tasks and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0018] Embodiments of the present invention provide a real-time 3D model generation system based on a multi-source data synchronous acquisition device. The system includes: a multi-source data synchronous acquisition device, a control unit, a processor, and a memory storing a computer program. The multi-source data synchronous acquisition device includes a main body integrating a lidar, an image acquisition device, an RTK measurement device, and a BeiDou satellite positioning device. When the computer program is executed by the processor, the following steps are implemented: Figure 1 As shown:

[0019] S1. Based on several scene images and depth point clouds with different acquisition perspectives corresponding to the scene to be modeled, an initial Gaussian ellipsoid set is generated; the scene images and depth point clouds are obtained by data acquisition through a multi-source data synchronous acquisition device driven by a control device.

[0020] Specifically, the scene image is obtained by an image acquisition device capturing images of the scene to be modeled.

[0021] Furthermore, the image acquisition device is a panoramic camera.

[0022] Specifically, based on the centimeter-level positioning information output by the RTK measurement equipment, the 3D point cloud of the scene to be modeled obtained by the lidar is spatiotemporally aligned and coordinate transformed to obtain the depth point cloud; the centimeter-level positioning information is geographic coordinates based on the Earth coordinate system.

[0023] Specifically, the RTK measurement equipment outputs centimeter-level positioning information based on the signals provided by the BeiDou satellite positioning equipment through differential positioning technology.

[0024] Furthermore, each point in the deep point cloud has corresponding geographic coordinates based on the Earth coordinate system.

[0025] Specifically, the initial Gaussian ellipsoid set includes several initial Gaussian ellipsoids. The initial Gaussian ellipsoid can be understood as a Gaussian ellipsoid obtained by initializing the center, covariance, color and transparency of the Gaussian ellipsoid based on the several scene images and the depth point cloud through spatial alignment and feature fusion.

[0026] Through the above steps, an initial Gaussian ellipsoid set is generated based on scene images and depth point clouds with different acquisition perspectives corresponding to the scene to be modeled. This avoids the scale drift or positional deviation caused by the 3D Gaussian ellipsoid generated by sparse point clouds generated by motion recovery structure based on multi-view images. This significantly improves the accuracy of the 3D model generated based on the initial Gaussian ellipsoid set.

[0027] S2. Cluster the target feature vectors corresponding to all key individual targets to obtain several clusters; key individual targets are obtained based on all scene images.

[0028] Specifically, the clustering method is a clustering algorithm that does not require a pre-defined number of clusters, such as density-based clustering (DBSCAN).

[0029] Specifically, the method includes the following steps to obtain key single targets:

[0030] S21. Perform single-target identification on the several scene images with different acquisition perspectives to determine at least one single target.

[0031] Specifically, each individual target corresponds to several scene sub-images, wherein the scene sub-image corresponding to the individual target is a local region image in the scene image that includes the individual target.

[0032] Specifically, the single target recognition is achieved through instance segmentation, which can be understood as: processing the several scene images through instance segmentation technology to separate each individual independent object instance, and taking the separated individual independent object instance as a single target.

[0033] Furthermore, a single target can be understood as a single object, such as a table, a chair, or a standard electrical appliance.

[0034] Specifically, a scene sub-image corresponding to a single target is extracted from the scene image; the scene sub-image corresponding to the single target includes only the single target and does not include other single targets besides the single target.

[0035] Specifically, the number of scene sub-images corresponding to a single target is less than or equal to the number of scene images corresponding to the scene to be modeled.

[0036] S22. Traverse each individual target. If the current individual target does not meet the preset replacement conditions, then the current individual target is taken as the key individual target.

[0037] Specifically, the preset replacement conditions are as follows: in the preset Gaussian ellipsoid template library, there exists at least one preset Gaussian ellipsoid set whose feature similarity to the single target is greater than a preset similarity threshold; the preset Gaussian ellipsoid template library includes several preset Gaussian ellipsoid sets; each preset Gaussian ellipsoid set includes several preset Gaussian ellipsoids.

[0038] Specifically, the preset similarity threshold ranges from 0.7 to 0.9.

[0039] Specifically, the feature similarity between the preset Gaussian ellipsoid set and the single target is obtained through the following steps S100-S200:

[0040] S100. Based on several scene sub-images corresponding to a single target, obtain the target feature vector corresponding to the single target.

[0041] Specifically, several scene sub-images corresponding to a single target are input into an image feature extraction model to obtain the target feature vector corresponding to the single target.

[0042] Specifically, if the image feature extraction model is a neural network model capable of extracting image features, such as ResNet or Swing Transformer, it will not be elaborated further here.

[0043] S200. The vector similarity between the target feature vector corresponding to the single target and the object feature vector corresponding to the preset Gaussian ellipsoid set is taken as the feature similarity between the preset Gaussian ellipsoid set and the single target; wherein, the greater the vector similarity, the more similar the corresponding object feature vector is to the corresponding target feature vector.

[0044] Specifically, the vector similarity between the target feature vector corresponding to a single target and the object feature vector corresponding to the preset Gaussian ellipsoid set is not less than 0 and not greater than 1.

[0045] In one specific embodiment, the vector distance between the target feature vector corresponding to a single target and the object feature vector corresponding to a preset Gaussian ellipsoid set is converted into a value between 0 and 1, and the value is used as the similarity between the target feature vector corresponding to the single target and the object feature vector corresponding to the preset Gaussian ellipsoid set. The smaller the vector distance, the closer the converted value is to 1; the larger the vector distance, the closer the converted value is to 0. Those skilled in the art will understand that any existing method for converting vector distance into a value between 0 and 1, such that the smaller the vector distance, the closer the converted value is to 1, and the larger the vector distance, the closer the converted value is to 0, falls within the protection scope of this invention and will not be elaborated further here.

[0046] Specifically, each preset Gaussian ellipsoid set corresponds to a preset object; wherein, the preset Gaussian ellipsoid set and its corresponding object feature vector are both obtained based on several object images corresponding to the preset object; the acquisition perspectives of any two object images corresponding to the preset object are different.

[0047] Specifically, several object images corresponding to a preset object corresponding to a preset Gaussian ellipsoid set are input into an image feature extraction model to obtain the object feature vector corresponding to the preset Gaussian ellipsoid set; the object feature vector corresponding to the preset Gaussian ellipsoid set has the same vector dimension as the target feature vector corresponding to a single target.

[0048] Specifically, the preset object is an object predetermined by those skilled in the art based on actual needs. It can be understood that when it is known that the scene to be modeled contains multiple identical or similar objects, any one of the multiple identical or similar objects can be used as the preset object and modeled in advance using 3D Gaussian sputtering technology. By reusing the three-dimensional model corresponding to the preset object, efficient modeling of the scene to be modeled can be achieved.

[0049] Specifically, the object image corresponding to the preset object is pre-captured by those skilled in the art using an image acquisition device. Each object image contains only its corresponding preset object and no other objects.

[0050] Through the above steps, based on the vector similarity between the target feature vector corresponding to a single target and the object feature vector corresponding to a preset Gaussian ellipsoid set, the feature similarity between the preset Gaussian ellipsoid set and the single target is determined. The greater the feature similarity, the more similar the features of the preset Gaussian ellipsoid set are to the features of the single target, indicating that the real object corresponding to the preset Gaussian ellipsoid set is more similar to the single target. Therefore, when there is no preset Gaussian ellipsoid set in the preset Gaussian ellipsoid template library with a feature similarity greater than a preset similarity threshold to the single target, it means that the preset Gaussian ellipsoid in the preset Gaussian ellipsoid set cannot be used to replace the initial Gaussian ellipsoid. The initial Gaussian ellipsoids corresponding to the individual targets in the Gaussian ellipsoid set need to be replaced by other methods. Therefore, if the current individual target does not meet the preset replacement conditions, the current individual target is taken as the key individual target. The target feature vectors corresponding to all key individual targets are clustered to obtain several clusters. The initial Gaussian ellipsoids in the initial Gaussian ellipsoid set are then replaced to update the initial Gaussian ellipsoid set. This effectively reduces the amount of data computation in the subsequent optimization process of the initial Gaussian ellipsoid set after all updates are completed, reduces the computational load, and thus improves the generation efficiency of the 3D model.

[0051] S3. If the number of target feature vectors in a cluster is not less than a preset threshold, then based on the scene sub-image list set corresponding to the cluster and the Gaussian ellipsoid prediction model, obtain several target Gaussian ellipsoids corresponding to the cluster; the scene sub-image list set includes the scene sub-image list corresponding to each key single target of the cluster; the scene sub-image list includes each scene sub-image corresponding to the key single target.

[0052] Specifically, the preset quantity threshold is a value pre-set by those skilled in the art based on actual needs, such as 10 or 20, which will not be elaborated here.

[0053] Specifically, the target Gaussian ellipsoid is generated based on the Gaussian ellipsoid attribute information set output by the Gaussian ellipsoid prediction model.

[0054] Furthermore, the target Gaussian ellipsoid corresponds one-to-one with the Gaussian ellipsoid attribute information in the Gaussian ellipsoid attribute information set output by the Gaussian ellipsoid prediction model.

[0055] Specifically, the Gaussian ellipsoid prediction model is a model obtained by supervised training of a target neural network model using a specific training sample set; the specific training sample set includes several specific training samples; each specific training sample includes an image set and a Gaussian ellipsoid attribute information set corresponding to the image set; the Gaussian ellipsoid attribute information set includes several Gaussian ellipsoid attribute information, including: Gaussian ellipsoid position data, scaling and rotation degree data, and opacity.

[0056] Specifically, the image set includes several image lists, and each image list includes several images.

[0057] Specifically, the Gaussian ellipsoid attribute information does not include color data.

[0058] Specifically, color attributes exhibit significant viewpoint dependence and illumination sensitivity. If used as a model prediction target, they can easily introduce noise and affect the model's convergence stability. Therefore, in this application, color data is not included in the Gaussian ellipsoid attribute information of specific training samples. Color information can be dynamically adjusted based on multi-view image observation data during subsequent 3D Gaussian sputtering differentiable rendering optimization, thereby balancing modeling efficiency and rendering quality.

[0059] Specifically, the target neural network model includes a CNN feature extraction module, a feature fusion module, and an MLP multilayer perceptron; wherein, the CNN feature extraction module is used to extract the feature vector of each image in each image list in the image set; the feature fusion module is used to fuse the feature vectors of all images in each image list into a feature vector of each image list; the MLP perceptron takes the comprehensive feature vector formed by concatenating the feature vectors of all image lists as input and outputs a Gaussian ellipsoidal attribute information set.

[0060] Specifically, the CNN feature extraction module includes multiple convolutional layers; the feature fusion module adopts a multi-layer fully connected network; those skilled in the art can select an appropriate CNN architecture according to actual needs, and determine the number of convolutional layers, the size of the convolutional kernels, the number of layers and neurons in the fully connected network, and the number and width of the MLP perceptron. This embodiment does not make specific limitations. For example, the number of convolutional layers and the size of the convolutional kernels in the CNN feature extraction module can be determined based on the number and resolution of images in the input image set, which will not be elaborated here.

[0061] Furthermore, the parameters of each module in the initial target neural network model can be set using standard initialization methods in existing technologies, which will not be elaborated here.

[0062] In some other embodiments, the Gaussian ellipsoid prediction model is obtained based on a specific training sample set and a pre-trained multimodal large language model. Specific prompt words are designed and the pre-trained multimodal large language model is fine-tuned using specific training samples to obtain the Gaussian ellipsoid prediction model, so that the Gaussian ellipsoid prediction model can output a set of Gaussian ellipsoid attribute information based on the input image set.

[0063] S4. Replace the initial Gaussian ellipsoid corresponding to each key single target in the initial Gaussian ellipsoid set with the target Gaussian ellipsoid corresponding to the cluster, so as to update the initial Gaussian ellipsoid set.

[0064] Specifically, when replacing the initial Gaussian ellipsoid corresponding to the key single target in the initial Gaussian ellipsoid set of the target Gaussian ellipsoid set corresponding to the cluster with the target Gaussian ellipsoid set corresponding to the cluster, at least one target Gaussian ellipsoid is used to replace the initial Gaussian ellipsoid corresponding to the key single target in the cluster corresponding to the target feature vector, and a rigid body transformation is performed on the used target Gaussian ellipsoid, the rigid body transformation including rotation and translation.

[0065] In one specific embodiment, after step S1 and before step S5, the following steps are further included:

[0066] S11. Traverse each individual target. If the current individual target meets the preset replacement condition, obtain the preset Gaussian ellipsoid set corresponding to the current individual target, and replace the initial Gaussian ellipsoid corresponding to the current individual target in the initial Gaussian ellipsoid set with the preset Gaussian ellipsoid set, so as to update the initial Gaussian ellipsoid set.

[0067] Specifically, when replacing the initial Gaussian ellipsoid corresponding to the current single target in the initial Gaussian ellipsoid set with a preset Gaussian ellipsoid from the preset Gaussian ellipsoid set, at least one preset Gaussian ellipsoid is used to replace the initial Gaussian ellipsoid, and a rigid body transformation is performed on the used preset Gaussian ellipsoid. The rigid body transformation includes rotation and translation. As those skilled in the art know, any method for obtaining rigid body transformation parameters in the prior art is within the protection scope of this invention, and will not be elaborated here.

[0068] Specifically, when there is at least one preset Gaussian ellipsoid set in the preset Gaussian ellipsoid template library whose feature similarity to the single target is greater than a preset similarity threshold, the preset Gaussian ellipsoid set with the largest feature similarity to the single target in the preset Gaussian ellipsoid template library is taken as the preset Gaussian ellipsoid set corresponding to the single target.

[0069] Through the above steps, when there is at least one set of preset Gaussian ellipsoids in the preset Gaussian ellipsoid template library whose feature similarity to the single target is greater than a preset similarity threshold, it means that the preset Gaussian ellipsoids in these preset Gaussian ellipsoid sets whose feature similarity is greater than the preset similarity threshold can be used to replace the initial Gaussian ellipsoids in the initial Gaussian ellipsoid set corresponding to the single target. At this time, the preset Gaussian ellipsoid set with the highest feature similarity to the single target in the preset Gaussian ellipsoid template library is used as the preset Gaussian ellipsoid set corresponding to the single target. This can filter out the optimal preset Gaussian ellipsoid set for replacing the initial Gaussian ellipsoids in the initial Gaussian ellipsoid set corresponding to the single target. Using the preset Gaussian ellipsoids in the preset Gaussian ellipsoid set to replace the initial Gaussian ellipsoids in the initial Gaussian ellipsoid set corresponding to the single target can reduce the amount of data calculation in the subsequent optimization process of the initial Gaussian ellipsoid sets after all updates are completed, reduce the computational load, and improve the generation efficiency of the 3D model.

[0070] Specifically, the initial Gaussian ellipsoid for each individual target is obtained through the following steps:

[0071] S01. For each individual target, based on the scene images with different acquisition perspectives, the depth point cloud, and the scene sub-images corresponding to the individual target, obtain the minimum stereo bounding box corresponding to the individual target.

[0072] In a specific embodiment, based on the position information of each scene sub-image corresponding to the single target in the scene image, the local depth point cloud corresponding to the single target is extracted from the depth point cloud, and based on the local depth point cloud corresponding to the single target, the minimum stereo bounding box corresponding to the single target is determined. The minimum stereo bounding box is an AABB bounding box, that is, a hexahedral bounding box.

[0073] S02. Traverse each initial Gaussian ellipsoid in the initial Gaussian ellipsoid set. If the percentage of the overlap volume between the initial Gaussian ellipsoid and the smallest bounding box corresponding to the single target is not less than a preset ratio, then mark the initial Gaussian ellipsoid as the initial Gaussian ellipsoid corresponding to the single target.

[0074] Specifically, the overlap volume ratio is the ratio of the intersection volume of the initial Gaussian ellipsoid and the minimum bounding box corresponding to the single target to the volume of the initial Gaussian ellipsoid itself; the ratio can be approximately obtained by methods such as Monte Carlo sampling and gridded voxel approximation.

[0075] Specifically, the preset ratio ranges from 0.7 to 0.9.

[0076] Through the above steps, for each individual target, based on the several scene images, the depth point cloud, and the several scene sub-images corresponding to the individual target, the minimum bounding box corresponding to the individual target is obtained. When the percentage of the overlap volume between the initial Gaussian ellipsoid and the minimum bounding box corresponding to the individual target is not less than a preset ratio, it indicates that most of the volume of the initial Gaussian ellipsoid is within the minimum bounding box. The initial Gaussian ellipsoid is likely the initial Gaussian ellipsoid corresponding to the individual target. Therefore, by traversing each initial Gaussian ellipsoid in the set of initial Gaussian ellipsoids, if the percentage of the overlap volume between the initial Gaussian ellipsoid and the minimum bounding box corresponding to the individual target is not less than a preset ratio, the initial Gaussian ellipsoid is marked as the initial Gaussian ellipsoid corresponding to the individual target, which helps to improve the accuracy of obtaining the initial Gaussian ellipsoid corresponding to the individual target.

[0077] S5. Optimize the initial Gaussian ellipsoid set after all updates are completed to obtain the 3D model corresponding to the scene to be modeled.

[0078] Specifically, the optimization process calculates the gradient of the loss function based on the differentiable rendering mechanism, and uses a gradient descent-type optimization algorithm to jointly optimize the center position, covariance matrix, color, and opacity of the Gaussian ellipsoid. This can be understood as an iterative process in which differentiable rendering, reconstruction loss calculation, backpropagation gradient calculation, and Gaussian ellipsoid parameters are updated in each iteration until convergence.

[0079] Through the above steps, for each cluster, if the number of target feature vectors in the cluster is not less than a preset threshold, then the scene sub-image list set corresponding to the cluster is obtained; based on the scene sub-image list set corresponding to the cluster and the Gaussian ellipsoid prediction model, the target Gaussian ellipsoid set corresponding to the cluster is obtained; using the target Gaussian ellipsoid set corresponding to the cluster, the initial Gaussian ellipsoid corresponding to the key single target in the initial Gaussian ellipsoid set is replaced, so as to update the initial Gaussian ellipsoid set. Clustering is performed on the target feature vectors corresponding to all key individual targets to obtain several clusters. Target feature vectors within the same cluster are similar, and consequently, the key individual targets corresponding to the target feature vectors within the same cluster are similar. When the number of target feature vectors in a cluster is less than a preset threshold, generating a target Gaussian ellipsoid set and performing replacement optimization may result in the overall computational resources consumed by the key individual targets corresponding to this cluster being higher than the computational resources consumed by optimizing the original Gaussian ellipsoid, thus reducing the generation efficiency of the 3D model. Therefore, when the number of target feature vectors in a cluster is not less than the preset threshold, the base... Based on the scene sub-image list set and Gaussian ellipsoid prediction model corresponding to the cluster, the target Gaussian ellipsoid set corresponding to the cluster is obtained. Then, the target Gaussian ellipsoids from this set are used to replace the initial Gaussian ellipsoids corresponding to the key individual targets in the initial Gaussian ellipsoid set for each target feature vector within the cluster. This ensures efficient utilization of computing resources while avoiding full optimization of the initial Gaussian ellipsoids corresponding to a large number of key individual targets. This effectively reduces the amount of data computation during subsequent optimization of the updated initial Gaussian ellipsoid sets, lowers the computational load, and thus improves the generation efficiency of the 3D model.

[0080] Specifically, specific training samples are obtained through the following steps:

[0081] S10. Perform clustering on the image feature vectors of all historically modeled objects to obtain the first vector cluster set A = {A1, ..., A2}. i A n}, A i ={A i1 A ij A im(i)},A i Let A be the i-th first vector cluster, 1≤i≤n, where n is the number of first vector clusters; ij For A i The j-th image feature vector, 1≤j≤m(i), where m(i) is A iThe number of image feature vectors; historical modeling objects are determined from objects whose 3D models have been constructed using the 3D Gaussian sputtering method before the current time point.

[0082] Specifically, the clustering method is a clustering algorithm that does not require a pre-defined number of clusters, such as density-based clustering (DBSCAN).

[0083] Specifically, the historical modeling objects can be pre-determined by those skilled in the art from objects whose three-dimensional models have been constructed using the 3D Gaussian sputtering method before the current point in time, based on actual needs, and will not be elaborated further here.

[0084] Specifically, different historical modeling objects can be categorized into different scenarios; the richer the scenarios corresponding to the historical modeling objects in a specific training sample set, the stronger the generalization ability of the Gaussian ellipsoid prediction model obtained based on the specific training sample set.

[0085] Specifically, the image feature vector of the historical modeled object is obtained based on several object images corresponding to the historical modeled object. The acquisition perspectives of any two object images corresponding to the historical modeled object are different, and each object image only includes the historical modeled object and does not include other objects.

[0086] Specifically, the images of several objects corresponding to the historically modeled objects are input into the image feature extraction model to obtain the image feature vectors of the historically modeled objects.

[0087] S20, Obtain A ij The corresponding list of Gaussian ellipsoid eigenvectors B ij B ij Including A ij Corresponding historical modeling object A 0 ij Correspondingly, the Zth was completed. ij Gaussian ellipsoid feature vectors corresponding to the Gaussian ellipsoid attribute information set of several Gaussian ellipsoids optimized in each iteration; X ij ≤Z ij ≤Y ij X ij For A 0 ij The corresponding preset minimum number of iterations, Y ij For A 0 ij The corresponding preset maximum number of iterations.

[0088] Specifically, X ij The following conditions must be met:

[0089] X ij =ceil(R ij×α), ceil() is the floor function, R ij For in A 0 ij During the construction of the corresponding 3D model, A 0 ij The total number of iterations for the corresponding Gaussian ellipsoids, where α is the adjustment parameter corresponding to the preset minimum number of iterations, 0.1 < α < 0.3.

[0090] Specifically, in A 0 ij During the construction of the corresponding 3D model, A 0 ij The total number of iterations for optimizing the corresponding Gaussian ellipsoids can be understood as: A 0 ij The corresponding Gaussian ellipsoids from initialization to A 0 ij The cumulative number of optimization iterations before the corresponding 3D model converges.

[0091] Specifically, A 0 ij Correspondingly, the Zth was completed. ij The Gaussian ellipsoid feature vectors of the Gaussian ellipsoid attribute information set corresponding to several Gaussian ellipsoids optimized in each iteration can be understood as: for A 0 ij Correspondingly, the Zth was completed. ij The vector is obtained by feature encoding the Gaussian ellipsoid attribute information of each Gaussian ellipsoid in several Gaussian ellipsoids in the iteration optimization.

[0092] Specifically, A 0 ij Correspondingly, the Zth was completed. ij The iteration number corresponding to several Gaussian ellipsoids optimized in this iteration is Z. ij .

[0093] Specifically, Y ij The following conditions must be met:

[0094] Y ij =floor(R ij ×β), floor() is the floor function, β is the adjustment parameter corresponding to the preset maximum number of iterations, 0.7<β<0.9.

[0095] Through the above steps, based on the preset minimum and maximum number of iterations, a Gaussian ellipsoid feature vector list is constructed by selecting the feature vectors corresponding to the data in the intermediate stable stage of the Gaussian ellipsoid optimization process. The data in the intermediate stable stage is free from the initial random state and has not overfitted specific noise. Therefore, constructing a Gaussian ellipsoid feature vector list based on the feature vectors corresponding to the data in the intermediate stable stage of the Gaussian ellipsoid optimization process makes the features represented by the feature vectors in the Gaussian ellipsoid feature vector list more representative and robust.

[0096] Specifically, for B ij The Gaussian ellipsoid attribute information set corresponding to each Gaussian ellipsoid feature vector in the set includes the Gaussian ellipsoid attribute information corresponding to each of the several Gaussian ellipsoids.

[0097] S30, To B i1 B ij B im(i) All Gaussian ellipsoidal eigenvectors are clustered to obtain A. i The corresponding several second vector clusters.

[0098] Specifically, the second vector cluster includes several Gaussian ellipsoidal eigenvectors.

[0099] Specifically, the clustering method is a clustering algorithm that does not require a pre-defined number of clusters, such as density-based clustering (DBSCAN).

[0100] S40, If A i If only one of the corresponding second vector clusters meets the preset judgment condition, then the second vector cluster that meets the preset judgment condition is taken as A. i The corresponding target vector cluster C i The preset judgment condition is: the second vector cluster includes the Gaussian ellipsoidal feature vectors in the list of Gaussian ellipsoidal feature vectors corresponding to each image feature vector in the corresponding first vector cluster. For example: if A i In the corresponding second vector cluster, there exists a Gaussian ellipsoidal eigenvector belonging to B. i1 There exists a Gaussian ellipsoid eigenvector belonging to B. ij There exists a Gaussian ellipsoid eigenvector belonging to B. im(i) If so, then the second vector cluster is determined to meet the preset judgment conditions.

[0101] Specifically, a second vector cluster must contain Gaussian ellipsoidal feature vectors from the list of Gaussian ellipsoidal feature vectors corresponding to each image feature vector in its corresponding first vector cluster in order to be recognized as a target vector cluster. This ensures that the extracted features are universally present in similar objects, rather than only appearing in individual samples, thereby improving the universality and reliability of the features learned by the model.

[0102] S50, according to C i Specific training samples are obtained by using several object images of historical modeling objects corresponding to the Gaussian ellipsoid feature vectors and the Gaussian ellipsoid attribute information set.

[0103] Through the above steps, the image feature vectors of all historical modeled objects are clustered to obtain a first vector cluster set, so that the historical modeled objects corresponding to the image feature vectors in the same vector cluster are similar. A list of Gaussian ellipsoidal feature vectors corresponding to each image feature vector in each first vector cluster is obtained, and all Gaussian ellipsoidal feature vectors in the list of Gaussian ellipsoidal feature vectors corresponding to all image feature vectors in each first vector cluster are clustered to obtain several second vector clusters corresponding to each first vector cluster. If only one second vector cluster among the several second vector clusters corresponding to the first vector cluster meets a preset judgment condition, then the second vector cluster meeting the preset judgment condition is taken as the target vector cluster corresponding to the first vector cluster. Specific training samples are obtained based on several object images and Gaussian ellipsoidal attribute information sets of the historical modeled objects corresponding to the Gaussian ellipsoidal feature vectors in the target vector cluster. This achieves automated generation of specific training samples without manual annotation, and the specific training samples come from historical modeled objects in multiple different scenes, significantly improving the generalization ability, structural accuracy, and modeling efficiency of the Gaussian ellipsoidal prediction model, especially suitable for fast, high-quality 3D reconstruction tasks of repetitive objects in large-scale scenes.

[0104] Specifically, step S50 includes the following sub-steps:

[0105] S51, C i China and B ij The distance C in the corresponding Gaussian ellipsoidal eigenvectors i The eigenvector of the Gaussian ellipsoid closest to the cluster center is used as A ij The corresponding key feature vector D ij .

[0106] Specifically, with B ij The corresponding Gaussian ellipsoid eigenvectors can be understood as belonging to B. ij The Gaussian ellipsoid eigenvectors.

[0107] Specifically, the eigenvectors of the Gaussian ellipsoid are compared with C. iThe vector distance between the eigenvectors corresponding to the cluster centers is used as the eigenvector of the Gaussian ellipsoid and C. i The distance between cluster centers; C i The eigenvector corresponding to the cluster center is C. i The mean vector of all Gaussian ellipsoidal eigenvectors.

[0108] S52, Obtain D ij The corresponding list of object images E ij E ij Includes A i Except for A ij The list of object images corresponding to the historical modeled object for each image feature vector other than the one in the image; the list of object images corresponding to the historical modeled object includes several object images corresponding to the historical modeled object.

[0109] S53, according to D ij The corresponding Gaussian ellipsoid attribute information set and E ij Obtain specific training samples, wherein the image set in the specific training samples is E. ij The Gaussian ellipsoid attribute information set corresponding to the image set is D. ij The corresponding Gaussian ellipsoid attribute information set.

[0110] Through the above steps, the Gaussian ellipsoid feature vector closest to the center of the target vector cluster among several Gaussian ellipsoid feature vectors belonging to each Gaussian ellipsoid feature vector list in the target vector cluster is taken as the key feature vector; ensuring that the key feature vector has representativeness and stability; based on the list of object images corresponding to the historical modeled objects in the first vector cluster corresponding to the key feature vector (excluding the image feature vector corresponding to the key feature vector), a set of object image lists corresponding to the key feature vector is constructed, and specific training samples are constructed based on the key feature vector and its corresponding set of object image lists, realizing the diversification of specific training samples, thereby realizing high-quality supervised pairing from multi-view input to standardized structure output, which is beneficial to improving the efficiency and convergence stability of model training.

[0111] Specifically, after step S40 and before step S50, the following steps are also included:

[0112] S41. If A i If at least two of the corresponding second vector clusters meet the preset judgment conditions, then each second vector cluster that meets the preset judgment conditions is designated as A. i The corresponding third vector cluster, to obtain A i The corresponding third vector cluster F i ={F i1 F igF ih(i)}, F ig For A i The corresponding g-th third vector cluster, 1≤g≤h(i), where h(i) is A i The number of corresponding third vector clusters.

[0113] Specifically, the third vector cluster includes several Gaussian ellipsoidal eigenvectors.

[0114] S42, F ig China and B ij The distance F in the corresponding Gaussian ellipsoid eigenvectors ig The eigenvector of the Gaussian ellipsoid closest to the cluster center is taken as F ig With B ij The specified feature vector G between g ij .

[0115] Specifically, the eigenvectors of the Gaussian ellipsoid are compared with F... ig The vector distance between the eigenvectors corresponding to the cluster centers is used as the eigenvector of the Gaussian ellipsoid and F. ig The distance between cluster centers; F ig The eigenvector corresponding to the cluster center is F ig The mean vector of all Gaussian ellipsoidal eigenvectors.

[0116] S43, according to G g ij Get F ig The corresponding average number of iterations H ig H ig The following conditions must be met:

[0117] H ig =(Σ m(i) j=1 G 0g ij ) / m(i), G 0g ij To be with G g ij The number of iterations for optimizing several Gaussian ellipsoids corresponding to the set of Gaussian ellipsoid attribute information.

[0118] S44, H i1 H ig H ih(i) The third vector cluster corresponding to the maximum value in C is used as C i .

[0119] Through the above steps, when at least two of the several second vector clusters corresponding to the first vector cluster meet the preset judgment conditions, the second vector clusters that meet the preset judgment conditions are taken as the third vector clusters corresponding to the first vector cluster, and the average iteration number corresponding to the third vector cluster is obtained. The third vector cluster with the largest average iteration number among all the third vector clusters corresponding to the first vector cluster is taken as the target vector cluster for the first vector cluster. The higher the iteration degree, the closer the Gaussian ellipsoid is to the final shape, and the more stable and representative the feature vectors obtained based on the attribute information of the Gaussian ellipsoid are. Taking the third vector cluster with the largest average iteration number as the target vector cluster makes it easier to construct specific training samples based on the stable and representative Gaussian ellipsoid feature vectors, which is beneficial to improving the reliability and representativeness of specific training samples.

[0120] Specifically, the Gaussian ellipsoids in all embodiments of this application are 3D Gaussian ellipsoids.

[0121] This invention provides a real-time 3D model generation system based on a multi-source data synchronous acquisition device. The system includes: a multi-source data synchronous acquisition device, a control device, a processor, and a memory storing a computer program. The multi-source data synchronous acquisition device includes a main body that integrates a lidar, an image acquisition device, an RTK measurement device, and a BeiDou satellite positioning device. When the computer program is executed by the processor, it can initialize and generate an initial Gaussian ellipsoid set based on several scene images and depth point clouds with different acquisition perspectives corresponding to the scene to be modeled. It then performs clustering processing on the target feature vectors corresponding to all key individual targets to obtain several clusters. If the number of target feature vectors in a cluster is not less than a preset threshold, it obtains several target Gaussian ellipsoids corresponding to that cluster based on the scene sub-image list set and the Gaussian ellipsoid prediction model corresponding to that cluster. The initial Gaussian ellipsoids corresponding to each key individual target in the initial Gaussian ellipsoid set are replaced with the target Gaussian ellipsoids corresponding to that cluster in the initial Gaussian ellipsoid set, thereby updating the initial Gaussian ellipsoid set. Finally, the updated initial Gaussian ellipsoid set is optimized to obtain the 3D model corresponding to the scene to be modeled. As can be seen, this invention introduces a Gaussian ellipsoid prediction model and a target Gaussian ellipsoid replacement mechanism. When the number of target feature vectors in a cluster is not less than a preset threshold, the target Gaussian ellipsoid replaces the initial Gaussian ellipsoid corresponding to each key single target in the corresponding cluster, thereby updating the initial Gaussian ellipsoid set. This effectively reduces the amount of data computation in the subsequent optimization process of the initial Gaussian ellipsoid set after all updates have been completed, reduces the computational load, and thus improves the generation efficiency of the three-dimensional model.

[0122] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A three-dimensional model real-time generation system based on multi-source data synchronous acquisition equipment, characterized in that, The system includes: a multi-source data synchronous acquisition device, a control unit, a processor, and a memory storing a computer program; the multi-source data synchronous acquisition device includes a main body that integrates a lidar, an image acquisition device, an RTK measurement device, and a BeiDou satellite positioning device; when the computer program is executed by the processor, the following steps are implemented: S1. Based on several scene images and depth point clouds with different acquisition perspectives corresponding to the scene to be modeled, an initial Gaussian ellipsoid set is generated; the scene images and depth point clouds are obtained by data acquisition through a multi-source data synchronous acquisition device driven by a control device. S2. Cluster the target feature vectors corresponding to all key individual targets to obtain several clusters; key individual targets are obtained based on all scene images. S3. If the number of target feature vectors in a cluster is not less than a preset threshold, then based on the scene sub-image list set corresponding to the cluster and the Gaussian ellipsoid prediction model, obtain several target Gaussian ellipsoids corresponding to the cluster; the scene sub-image list set includes the scene sub-image list corresponding to each key single target of the cluster; the scene sub-image list includes each scene sub-image corresponding to the key single target. S4. Replace the initial Gaussian ellipsoid corresponding to each key single target in the initial Gaussian ellipsoid set with the target Gaussian ellipsoid corresponding to the cluster, so as to update the initial Gaussian ellipsoid set. S5. Optimize the initial Gaussian ellipsoid set after all updates are completed to obtain the 3D model corresponding to the scene to be modeled. 2.The real-time three-dimensional model generation system based on multi-source data synchronous acquisition device of claim 1, wherein, The target Gaussian ellipsoid is generated based on the Gaussian ellipsoid attribute information set output by the Gaussian ellipsoid prediction model. 3.The real-time three-dimensional model generation system based on multi-source data synchronous acquisition device of claim 2, wherein, The Gaussian ellipsoid prediction model is a model obtained by supervised training of a target neural network model using a specific training sample set; the specific training sample set includes several specific training samples; each specific training sample includes an image set and a Gaussian ellipsoid attribute information set corresponding to the image set; the Gaussian ellipsoid attribute information set includes several Gaussian ellipsoid attribute information.

4. The real-time generation system for three-dimensional models based on multi-source data synchronous acquisition equipment according to claim 3, characterized in that, Obtain specific training samples through the following steps: S10. Perform clustering on the image feature vectors of all historically modeled objects to obtain the first vector cluster set A = {A1, ..., A2}. i A n }, A i ={A i1 A ij A im(i) },A i Let A be the i-th first vector cluster, 1≤i≤n, where n is the number of first vector clusters; ij For A i The j-th image feature vector, 1≤j≤m(i), where m(i) is A i The number of image feature vectors; historical modeling objects are determined from objects whose 3D models have been constructed using the 3D Gaussian sputtering method before the current time point; S20, Obtain A ij The corresponding list of Gaussian ellipsoid eigenvectors B ij B ij Including A ij Corresponding historical modeling object A 0 ij Correspondingly, the Zth step was completed. ij Gaussian ellipsoid feature vectors corresponding to the Gaussian ellipsoid attribute information set of several Gaussian ellipsoids optimized in each iteration; X ij ≤Z ij ≤Y ij X ij For A 0 ij The corresponding preset minimum number of iterations, Y ij For A 0 ij The corresponding preset maximum number of iterations; S30, To B i1 B ij B im(i) All Gaussian ellipsoidal eigenvectors are clustered to obtain A. i The corresponding several second vector clusters; S40, If A i If only one of the corresponding second vector clusters meets the preset judgment condition, then the second vector cluster that meets the preset judgment condition is taken as A. i The corresponding target vector cluster C i The preset judgment condition is: the second vector cluster includes the Gaussian ellipsoid feature vector in the list of Gaussian ellipsoid feature vectors corresponding to each image feature vector in the corresponding first vector cluster; S50、According to C i The Gaussian ellipsoid feature vector in the historical modeling object corresponds to a set of several object images and Gaussian ellipsoid attribute information of the object, and obtains a specific training sample.

5. The real-time three-dimensional model generation system based on multi-source data synchronous acquisition equipment according to claim 4, characterized in that, Step S50 includes the following sub-steps: S51, C i China and B ij The distance C in the corresponding Gaussian ellipsoidal eigenvectors i The eigenvector of the Gaussian ellipsoid closest to the cluster center is used as A ij The corresponding key feature vector D ij ; S52, Obtain D ij The corresponding list of object images E ij E ij Includes A i Except for A ij For each image feature vector other than the one in the image, there is a list of object images corresponding to the historical modeled object; the list of object images corresponding to the historical modeled object includes several object images corresponding to the historical modeled object. S53、According to D ij corresponding Gaussian ellipsoid attribute information set E ij obtain a specific training sample, wherein the image set in the specific training sample is E ij corresponding Gaussian ellipsoid attribute information set D ij corresponding Gaussian ellipsoid attribute information set. 6.The real-time three-dimensional model generation system based on multi-source data synchronous acquisition device of claim 4, wherein, After step S40 and before step S50, the following steps are also included: S41. If A i If at least two of the corresponding second vector clusters meet the preset judgment conditions, then each second vector cluster that meets the preset judgment conditions is designated as A. i The corresponding third vector cluster, to obtain A i The corresponding third vector cluster set F i ={F i1 F ig F ih(i) }, F ig For A i The corresponding g-th third vector cluster, 1≤g≤h(i), where h(i) is A i The number of corresponding third vector clusters; S42, the F ig corresponding to B ij cluster center closest to F ig corresponding to B ig specified feature vector G ij between B g ij ; S43, according to G g ij Get F ig The corresponding average number of iterations H ig H ig Meets the following conditions: H ig =(Σ m(i) j=1 G 0g ij ) / m(i), G 0g ij To be with G g ij The number of iterations for optimizing several Gaussian ellipsoids corresponding to the corresponding Gaussian ellipsoid attribute information set; S44, H i1 ,..., H ig ,..., H ih(i) ,..., H i .

7. The real-time three-dimensional model generation system based on multi-source data synchronous acquisition equipment according to claim 4, characterized in that, X ij meets the following conditions: X ij =ceil(R ij ×α), ceil() is the floor function, R ij For in A 0 ij During the construction of the corresponding 3D model, A 0 ij The total number of iterations for the corresponding Gaussian ellipsoids, where α is the adjustment parameter corresponding to the preset minimum number of iterations, 0.1 < α < 0.

3.

8. The real-time three-dimensional model generation system based on multi-source data synchronous acquisition equipment according to claim 7, characterized in that, Y ij meets the following conditions: Y ij =floor(R ij ×β), floor() is the floor function, β is the adjustment parameter corresponding to the preset maximum number of iterations, 0.7<β<0.

9. 9.The real-time three-dimensional model generation system based on multi-source data synchronous acquisition device of claim 4, wherein, For B ij corresponding to each Gaussian ellipsoid feature vector in the Gaussian ellipsoid attribute information set, the Gaussian ellipsoid attribute information set comprising Gaussian ellipsoid attribute information corresponding to each Gaussian ellipsoid in the Gaussian ellipsoid corresponding to the Gaussian ellipsoid feature vector. 10.The real-time three-dimensional model generation system based on multi-source data synchronous acquisition device of claim 4, wherein, The image feature vector of the historical modeling object is obtained based on several object images corresponding to the historical modeling object. The acquisition perspectives of any two object images corresponding to the historical modeling object are different, and each object image only includes the historical modeling object.

Images