Image reconstruction method and device based on 3D Gaussian model, electronic equipment and medium

Abstract

The application discloses a kind of based on 3D Gaussian model's image reconstruction method, device, electronic equipment and medium, it is related to image reconstruction and other technical fields, obtain the point cloud data and camera pose data of image to be handled, and based on point cloud data obtains initial Gaussian distribution;Depth prediction is carried out to the image to be handled to obtain depth image, and depth image is carried out semantic clustering processing, and obtain instance semantic image;Based on depth image and instance semantic image, target loss function is constructed, and based on target loss function, initial Gaussian distribution is constrained to obtain target Gaussian distribution;Projection is carried out based on target Gaussian distribution and camera pose data to obtain projection image, and based on the evaluation result of projection image, Gaussian sphere model is adjusted;Image rendering is carried out based on adjusted Gaussian sphere model, and target image is obtained.The application fuses the integrity method of initial pose guide, depth estimation optimization and Gaussian modeling control, improves the reconstruction precision, stability and controllability of image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image reconstruction, and particularly relates to a 3D Gaussian model-based image reconstruction method and device, electronic equipment and medium. BACKGROUND

[0002] In the operation and maintenance of a power system, a tower is an important component of a power transmission line, and its structural safety and state monitoring are crucial. With the advancement of intelligent inspection of distribution networks, more and more operation modes are shifting to image / video-driven automatic analysis and three-dimensional reconstruction. Traditional three-dimensional reconstruction methods, such as a processing flow based on SfM (Structure-from-Motion) and MVS (Multi-View Stereo), rely on dense, multi-angle, and uniformly distributed image inputs, and can obtain a relatively high-precision reconstruction result in a specific experimental environment. However, in an actual inspection scene, due to limitations of shooting conditions (such as flight path restrictions, safety distance requirements, and occlusion problems), image data is often sparse view, that is, the number of shooting angles is small and the coverage is incomplete, which seriously affects the performance of traditional algorithms. Therefore, there is an urgent need for a three-dimensional reconstruction method that can adapt to sparse view conditions, has strong robustness, and is suitable for the structural characteristics of a tower. SUMMARY

[0003] To this end, the purpose of the embodiments of the present application is to propose a 3D Gaussian model-based image reconstruction method, device, electronic equipment, medium, and computer program product, which integrates an overall method of initial pose guidance, depth estimation optimization, and Gaussian modeling control, and improves the reconstruction accuracy, stability, and controllability in a sparse view scene.

[0004] The embodiments of the present application provide a 3D Gaussian model-based image reconstruction method, which comprises the following steps: obtaining point cloud data and camera pose data of a to-be-processed image, and obtaining an initial Gaussian distribution based on the point cloud data; performing depth prediction on the to-be-processed image to obtain a depth image, and performing semantic clustering processing on the depth image to obtain an instance semantic image; constructing a target loss function based on the depth image and the instance semantic image, and constraining the initial Gaussian distribution based on the target loss function to obtain a target Gaussian distribution; projecting based on the target Gaussian distribution and the camera pose data to obtain a projection image, and adjusting a Gaussian sphere model based on an evaluation result of the projection image; performing image rendering based on the adjusted Gaussian sphere model to obtain a target image.

[0005] Exemplarily, the semantic clustering processing on the depth image obtains an instance semantic image, including: constructing a weighted spatial feature vector based on valid depth points and corresponding depth values in the depth image; performing clustering processing based on the weighted spatial feature vector to obtain a plurality of depth semantic regions; performing connected domain extraction processing on each of the depth semantic regions to obtain a plurality of instance semantic regions; and obtaining an instance semantic image based on the instance semantic regions and corresponding instance encodings.

[0006] Exemplarily, the weighted spatial feature vector is constructed by the following formula:

[0007]

[0008] wherein, is the weighted spatial feature vector, is a spatial weight factor, is a pixel position of the valid depth point, is a depth value of the valid depth point.

[0009] Exemplarily, the clustering processing based on the weighted spatial feature vector obtains a plurality of depth semantic regions, including: randomly selecting K valid depth points as centroids, calculating the Euclidean distance between each valid depth point and the centroid based on the weighted spatial feature vector; assigning the valid depth points to clusters based on the Euclidean distance, and updating the centroid based on the mean of the valid depth points in the cluster, and repeating iteration until a termination condition is met; and obtaining a plurality of depth semantic regions based on a plurality of clusters, wherein the clustering labels of the valid depth points in the same depth semantic region are the same.

[0010] Exemplarily, the connected domain extraction processing on each of the depth semantic regions obtains a plurality of instance semantic regions, including: performing connected domain extraction processing on each of the depth semantic regions to obtain a plurality of sub-regions; and determining the sub-region as the instance semantic region if the area of the sub-region is greater than or equal to a preset area.

[0011] Exemplarily, the target loss function is constructed based on the depth image and the instance semantic image, including: constructing a first regular loss function based on a full-image absolute loss of the depth image; constructing a second regular loss function based on an intra-instance relative depth consistency loss in the instance semantic image; constructing a third regular loss function based on an inter-instance relative depth structure loss in the instance semantic image; and determining the target loss function based on at least one of the first regular loss function, the second regular loss function, and the third regular loss function.

[0012] For example, the depth image includes a first depth image obtained based on a 3D Gaussian model and a second depth image obtained based on monocular depth estimation, and the first regularized loss function is constructed using the following formula:

[0013]

[0014] in, Let the first regularization loss function be... Let (i,j) be the set of valid pixels, where (i,j) is the pixel in the i-th row and j-th column. Let (i,j) be the depth value of the first depth image pixel. Let be the depth value of pixel (i,j) in the second depth image.

[0015] For example, the second regularized loss function is constructed using the following formula:

[0016]

[0017] or,

[0018]

[0019] in, Let be the second regularization loss function. Let (i,j) and (p,q) be the semantic regions of the k-th instance. Pixel pairs within the region, Let (i,j) be the depth value of the first depth image pixel. Let be the depth value of the first depth image pixel (p, q). Let (i,j) be the depth value of the second depth image pixel. Let be the depth value of the second depth image pixel (p, q). This is a smoothing control factor.

[0020] For example, the third regularized loss function is constructed using the following formula:

[0021]

[0022] in, The third regularized loss function, For any two distinct instances, For example average depth, For example average depth, The minimum depth difference threshold. .

[0023] Illustratively, the determining the target loss function based on at least one of the first regularization loss function, the second regularization loss function, and the third regularization loss function comprises: performing weighting processing on the first regularization loss function, the second regularization loss function, and the third regularization loss function to obtain the target loss function.

[0024] Illustratively, the adjusting the Gaussian sphere model comprises at least one of the following: adjusting a number of Gaussian spheres in the Gaussian sphere model; adjusting a covariance matrix of the Gaussian sphere model; resetting parameters of a Gaussian sphere in the Gaussian sphere model.

[0025] Illustratively, the adjusting the number of Gaussian spheres in the Gaussian sphere model comprises: in a case where the number of Gaussian spheres is greater than a preset number threshold, sorting a current Gaussian sphere based on a transparency corresponding to the current Gaussian sphere; and retaining a preset number threshold of Gaussian spheres in front of the transparency sorting.

[0026] Illustratively, the adjusting the covariance matrix of the Gaussian sphere model comprises at least one of the following: in a case where a current eigenvalue of the covariance matrix is greater than a preset maximum eigenvalue threshold, determining the current eigenvalue as the preset maximum eigenvalue threshold; in a case where the current eigenvalue of the covariance matrix is less than a preset minimum eigenvalue threshold, determining the current eigenvalue as the preset minimum eigenvalue threshold; and in a case where a ratio of a maximum eigenvalue and a minimum eigenvalue of the covariance matrix is greater than a preset ratio, initializing the covariance matrix.

[0027] Illustratively, the adjusting the Gaussian sphere model based on the evaluation result of the projection image comprises: determining a peak signal-to-noise ratio based on the projection image and a validation set; and in a case where the peak signal-to-noise ratio decreases continuously for a preset number of times of training periods, resetting parameters of a Gaussian sphere in the Gaussian sphere model.

[0028] Illustratively, the resetting the parameters of the Gaussian sphere in the Gaussian sphere model comprises: resetting at least one of a transparency parameter, a color parameter, and a covariance matrix parameter of the Gaussian sphere.

[0029] Illustratively, before the image rendering based on the adjusted Gaussian sphere model, the method further comprises: establishing a corresponding compensation factor based on each Gaussian sphere; performing processing on the compensation factor based on a preset probability to obtain a mask vector of the Gaussian sphere; and performing processing on a current transparency tensor of the Gaussian sphere based on the mask vector to obtain a target transparency tensor.

[0030] Another embodiment of the present application provides an image reconstruction device based on a 3D Gaussian model, the device comprising: an acquisition module configured to acquire point cloud data and camera pose data of an image to be processed, and obtain an initial Gaussian distribution based on the point cloud data; a clustering module configured to perform depth prediction on the image to be processed to obtain a depth image, and perform semantic clustering processing on the depth image to obtain an instance semantic image; a loss function construction module configured to construct a target loss function based on the depth image and the instance semantic image, and constrain the initial Gaussian distribution based on the target loss function to obtain a target Gaussian distribution; a projection and adjustment module configured to perform projection based on the target Gaussian distribution and the camera pose data to obtain a projection image, and adjust the Gaussian sphere model based on an evaluation result of the projection image; and a rendering module configured to perform image rendering based on the adjusted Gaussian sphere model to obtain a target image.

[0031] Another embodiment of the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the method of any of the above embodiments when executing the computer program.

[0032] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method of any of the above embodiments.

[0033] Another embodiment of the present application provides a computer program product comprising instructions, the instructions being executed by a processor of a computer device to enable the computer device to perform the steps of the method of any of the above embodiments.

[0034] In the above embodiments, the image reconstruction method comprises: acquiring point cloud data and camera pose data of an image to be processed, and obtaining an initial Gaussian distribution based on the point cloud data; performing depth prediction on the image to be processed to obtain a depth image, and performing semantic clustering processing on the depth image to obtain an instance semantic image; constructing a target loss function based on the depth image and the instance semantic image, and constraining the initial Gaussian distribution based on the target loss function to obtain a target Gaussian distribution; performing projection based on the target Gaussian distribution and the camera pose data to obtain a projection image, and adjusting the Gaussian sphere model based on an evaluation result of the projection image; and performing image rendering based on the adjusted Gaussian sphere model to obtain a target image. The present application integrates an overall method of initial pose guidance, depth estimation optimization and Gaussian modeling control, and improves the reconstruction accuracy, stability and controllability of the image. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 A flowchart of the image reconstruction method based on a 3D Gaussian model provided by the embodiments of the present application;

[0036] Figure 2 A flowchart of a semantic clustering process of a depth image is provided for the embodiments of the present application;

[0037] Figure 3 A flowchart of a clustering process based on a weighted spatial feature vector is provided for the embodiments of the present application;

[0038] Figure 4 A flowchart of a connected component extraction process of a depth semantic region is provided for the embodiments of the present application;

[0039] Figure 5 A schematic diagram of an example semantic image of a tower is provided for the embodiments of the present application;

[0040] Figure 6 A flowchart of constructing a target loss function is provided for the embodiments of the present application;

[0041] Figure 7 A flowchart of adjusting the number of Gaussian spheres in a Gaussian sphere model is provided for the embodiments of the present application;

[0042] Figure 8 A flowchart of adjusting a Gaussian sphere model is provided for the embodiments of the present application;

[0043] Figure 9 A flowchart of a transparency random drop mechanism is provided for the embodiments of the present application;

[0044] Figure 10 A flowchart of a tower image reconstruction method is provided for the embodiments of the present application;

[0045] Figure 11 A schematic diagram of a tower image after three-dimensional reconstruction is provided for the embodiments of the present application;

[0046] Figure 12 A structural schematic diagram of a three-dimensional Gaussian splash model is provided for the embodiments of the present application;

[0047] Figure 13 A schematic diagram of an image reconstruction device based on a 3D Gaussian model is provided for the embodiments of the present application;

[0048] Figure 14 A block diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0049] Embodiments of the present application are described in detail below with reference to examples illustrated in the accompanying drawings, in which the same or similar components are denoted by the same or similar reference numerals throughout, and embodiments described below are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0050] Figure 1 is a flowchart of an image reconstruction method according to an embodiment of the present application.

[0051] As an example, as shown in Figure 1 The image reconstruction method based on the 3D Gaussian model includes the following steps.

[0052] S101, obtaining point cloud data and camera pose data of a to-be-processed image, and obtaining an initial Gaussian distribution based on the point cloud data.

[0053] S102, performing depth prediction on the to-be-processed image to obtain a depth image, and performing semantic clustering processing on the depth image to obtain an instance semantic image.

[0054] S103, constructing a target loss function based on the depth image and the instance semantic image, and constraining the initial Gaussian distribution based on the target loss function to obtain a target Gaussian distribution.

[0055] S104, projecting based on the target Gaussian distribution and the camera pose data to obtain a projection image, and adjusting the Gaussian sphere model based on an evaluation result of the projection image.

[0056] S105, performing image rendering based on the adjusted Gaussian sphere model to obtain a target image.

[0057] As an example, first, the point cloud data and the camera pose data of the to-be-processed image are obtained. The to-be-processed image can be a tower image captured by a UAV. Of course, the image reconstruction method of the present application is not limited to the tower image reconstruction scenario, but is also applicable to image reconstruction in other limited image view scenarios. A VGGT (Visual Geometry Grounded Transformer) model can be used to extract features of the to-be-processed image, estimate an initial sparse point cloud and a camera pose, and obtain the required point cloud data and camera pose data. The point cloud data and the camera pose data can be converted into a COLMAP (open source library) compatible format to assist stable initialization of subsequent reconstruction tasks. The initial Gaussian distribution is obtained based on the point cloud data, for example, the position and shape of the Gaussian distribution are initialized based on the point cloud data.

[0058] Exemplarily, the depth prediction is performed on the to-be-processed image to obtain a depth image. For example, the depth prediction is performed on the to-be-processed image by using a DepthAnythingV2 monocular depth estimation model to obtain a depth map, and then the depth map is normalized to obtain the required depth image. The depth image can be an inverse depth image, in which a normal depth image has a larger value at a farther place, and the inverse depth image is opposite. Of course, the depth image can also be a normal depth image, and the form of the depth image is not limited in the present application, and the inverse depth image is converted for subsequent calculation. The depth image can be saved in a PNG format and used for subsequent geometric information analysis.

[0059] Exemplarily, the depth image is subjected to semantic clustering processing to obtain an instance semantic image. To achieve effective separation and semantic mask generation of multiple structural components or object regions in the to-be-processed image, a multi-instance segmentation method based on spatial coordinate and depth feature fusion is proposed, multiple instance regions are segmented, and finally an instance semantic image is obtained, realizing structured semantic segmentation of the depth map.

[0060] Exemplarily, a target loss function is constructed based on the depth image and the instance semantic image, and a depth regularization loss function based on instances is designed to improve depth consistency and structure expression capability. The initial Gaussian distribution is constrained based on the target loss function to obtain a target Gaussian distribution. Projection is performed based on the target Gaussian distribution and camera pose data to obtain a projection image, and the Gaussian sphere model is adjusted based on the evaluation result of the projection image. The evaluation result of the projection image can be obtained based on the real image, for example, the Gaussian sphere model is adjusted according to the difference between the projection image and the real image. It can be understood that the adjustment of the Gaussian sphere model is a dynamic process, which can be adjusted multiple times until the quality of the projection image meets the requirements. Finally, image rendering is performed based on the adjusted Gaussian sphere model to obtain a target image, and the three-dimensional Gaussian splashing technology can be used to realize image rendering.

[0061] The image reconstruction method based on the 3D Gaussian model can complete the three-dimensional reconstruction of the tower with complete structure and accurate geometry under limited image view angle, and the method has low resource occupation and can adapt to the efficient deployment requirements in edge devices or actual engineering scenes. The method effectively improves the reconstruction robustness and accuracy in complex environments by simultaneously fusing structure prior information, image feature guidance and lightweight depth estimation mechanism.

[0062] As an example, as shown in Figure 2 The semantic clustering processing is performed on the depth image to obtain an instance semantic image, including:

[0063] S201, constructing a weighted spatial feature vector based on effective depth points and corresponding depth values in the depth image.

[0064] S202, clustering processing is performed based on the weighted spatial feature vector, to obtain a plurality of depth semantic regions.

[0065] S203, connected domain extraction processing is performed on each depth semantic region, to obtain a plurality of instance semantic regions.

[0066] S204, an instance semantic image is obtained based on the instance semantic region and the corresponding instance code.

[0067] Exemplarily, first, a weighted spatial feature vector is constructed based on the effective depth points in the depth image and the corresponding depth values. The points with depth values greater than or equal to zero in the depth image are effective depth points. The present application proposes a spatial feature vector to enhance the influence of spatial information on clustering.

[0068] As an example, the weighted spatial feature vector is constructed by the following formula:

[0069]

[0070] wherein, is the weighted spatial feature vector, is a spatial weight factor, is the pixel position of the effective depth point, is the depth value of the effective depth point.

[0071] Exemplarily, a depth image is obtained according to the depth prediction on the image to be processed. The depth image can be denoted as , D is a set of depth values, the size of the depth image is dimension, each pixel point in the depth image has a corresponding depth value , all effective pixel positions and the corresponding depth values are extracted, and the weighted spatial feature vector is constructed, is a spatial weight factor, used to control the influence degree of spatial position on clustering. In the embodiment, the value of .

[0072] Exemplarily, after the weighted spatial feature vector is constructed, clustering processing is performed based on the weighted spatial feature vector, to obtain a plurality of depth semantic regions. The clustering processing can adopt K-means clustering algorithm (K-Means). Of course, the clustering algorithm is not limited to this one, and the present application takes the K-means clustering algorithm as an example for illustration.

[0073] As an example, as shown in Figure 3 , based on the weighted spatial feature vector, clustering processing is performed to obtain a plurality of depth semantic regions, including:

[0074] ​S301, randomly select K valid depth points as centroids, and calculate the Euclidean distance between each valid depth point and the centroid based on the weighted spatial feature vector.

[0075] S302, assign the valid depth points to the cluster based on the Euclidean distance, and update the centroid based on the mean of the valid depth points in the cluster, and repeat the iteration until the termination condition is met.

[0076] S303, obtain multiple depth semantic regions based on multiple clusters, wherein the clustering labels of the valid depth points in the same depth semantic region are the same.

[0077] For example, first, randomly select K valid depth points as centroids, and the value of K can be configured according to requirements. Calculate the Euclidean distance between other valid depth points and the centroid, and it should be noted that the Euclidean distance is calculated based on the weighted spatial feature vector of the centroid and the weighted spatial feature vector of the depth point, that is, the absolute distance between the two weighted spatial feature vectors is taken as the Euclidean distance. According to the distance of the Euclidean distance, the valid depth points are assigned to different clusters, and it can be understood that a cluster is a category. Update the centroid according to the mean of all valid depth points in the cluster, and then recalculate the Euclidean distance. Repeat the iteration until the termination condition is met, which can be set as the centroid tends to be stable, or the number of iterations. Each cluster is a depth semantic region, and the clustering labels of the valid depth points in the same depth semantic region are the same.

[0078] For example, the above clustering process can be expressed by the following formula:

[0079]

[0080] The clustering label (Y) is remapped to the label map of the whole image , and the invalid region can be marked as . It can be understood that each valid depth point has a clustering label, and the labels of the valid depth points in the same depth semantic region are the same. Some regions are not classified into a category during clustering processing, and these regions are invalid regions. Invalid regions are usually background regions in the image.

[0081] For example, after obtaining multiple depth semantic regions, perform connected component extraction processing on each depth semantic region to obtain multiple instance semantic regions. For each depth semantic region, perform a connected component separation operation to further divide the clustering result into multiple structurally coherent and sufficiently large instance regions.

[0082] As an example, as shown in Figure 4 , perform connected component extraction processing on each depth semantic region to obtain multiple instance semantic regions, including:

[0083] ​S401, a connected domain extraction process is performed on each depth semantic region to obtain a plurality of sub-regions.

[0084] S402, if the area of the sub-region is greater than or equal to a preset area, the sub-region is determined as an instance semantic region.

[0085] Exemplarily, a binary mask is generated for each depth semantic region, and a connected domain separation operation is performed on each depth semantic region It can be understood that the depth semantic region includes one or more connected domains, and after the connected domain extraction process is performed on each depth semantic region, a plurality of sub-regions are obtained. The sub-region is a small region in the depth semantic region. If the area of the sub-region is greater than or equal to a preset area, the sub-region is determined as an instance semantic region. The preset area can be determined according to the size of the depth image, for example, it can be 1% of the size of the depth image, that is , which is used to filter small-area pseudo-instances.

[0086] Exemplarily, if the area of the sub-region is greater than or equal to the preset area, it indicates that the sub-region is a valid instance, and the sub-region is determined as an instance semantic region and is assigned an independent instance number Then, according to each instance semantic region and the corresponding instance code, a required instance semantic image is obtained. A final instance mask image is constructed, each connected region is coded as a unique instance. As Figure 5 shown in the schematic diagram of the instance semantic image of the tower, different instance regions can be distinguished by different colors.

[0087] To realize effective separation and semantic mask generation of multiple structural components or object regions in a to-be-processed image, a multi-instance segmentation method based on spatial coordinate and depth feature fusion is proposed. On the basis of the depth map, combined with spatial weighted K-Means clustering and connected domain analysis, multiple instance regions with consistent structure and close space are automatically extracted.

[0088] As an example, as Figure 6 shown, a target loss function is constructed based on the depth image and the instance semantic image, including:

[0089] S601, a first regular loss function is constructed based on the absolute loss of the whole image of the depth image.

[0090] S602, a second regular loss function is constructed based on the relative depth consistency loss in each instance in the instance semantic image.

[0091] S603, a third regular loss function is constructed based on the relative depth structure loss between instances in the instance semantic image.

[0092] S604, determining the target loss function based on at least one of the first regular loss function, the second regular loss function, and the third regular loss function.

[0093] Exemplarily, the target loss function is constructed based on the depth image and the instance semantic image, to further improve the structural consistency and geometric expression ability of the depth map under the sparse view, a depth map regularization method combining semantic instance information is proposed, which utilizes the depth image obtained by depth prediction and the instance semantic image obtained by the above steps, and constructs the target loss function by defining three levels of regular term losses. Including but not limited to: constructing the first regular loss function based on the absolute loss of the whole depth image; constructing the second regular loss function based on the relative depth consistency loss within each instance in the instance semantic image; constructing the third regular loss function based on the relative depth structure loss between instances in the instance semantic image. The present application proposes to construct the regular term loss according to the three levels, but is not limited to the three levels, and can also include other types of regular terms. The target loss function is determined based on at least one of the first regular loss function, the second regular loss function, and the third regular loss function. It can be understood that one of the first regular loss function, the second regular loss function, and the third regular loss function can be determined as the target loss function, or two of the first regular loss function, the second regular loss function, and the third regular loss function can be determined as the target loss function, for example, the target loss function is determined based on the first regular loss function and the second regular loss function, or the target loss function is determined based on the second regular loss function and the third regular loss function, or the target loss function is determined based on the first regular loss function and the third regular loss function. The target loss function can also be determined based on the first regular loss function, the second regular loss function, and the third regular loss function.

[0094] As an example, the target loss function is determined based on at least one of the first regular loss function, the second regular loss function, and the third regular loss function, including: weighting the first regular loss function, the second regular loss function, and the third regular loss function to obtain the target loss function.

[0095] Exemplarily, the first regular loss function can be denoted as , the second regular loss function can be denoted as , and the third regular loss function can be denoted as , the first regular loss function, the second regular loss function, and the third regular loss function are weighted to obtain the target loss function , is a weighting parameter, .

[0096] It should be noted that the target loss function represents the instance-aware depth regularization term loss, and of course, the 3D Gaussian model itself also has a color reconstruction loss. Finally, the instance-aware depth regularization term loss and the color reconstruction loss are combined during training.

[0097] As an example, the color reconstruction loss function can be denoted as , which is used to measure the similarity between the rendered image and the real image at the pixel level and the perceptual level, and is composed of two sub-losses: pixel absolute difference and structural similarity loss . The pixel absolute difference is used to measure the absolute difference between the pixel value of the rendered image and the real image, and is defined as:

[0098]

[0099] wherein, represents the pixel value of the 3D Gaussian model rendered image ， represents the pixel value of the corresponding real image.

[0100] The structural similarity loss captures the similarity of the local structure, brightness and contrast of the image, and introduces a perceptual level of supervision signal, and is defined as:

[0101]

[0102] The final color reconstruction loss is the weighted sum of the two: . Wherein, 0.8, 0.2.

[0103] The final loss function is the weighted sum of the color reconstruction loss function and the target loss function . Namely:

[0104]

[0105] wherein, is the weight of the target loss function .

[0106] The first regularization loss function, the second regularization loss function and the third regularization loss function are described in detail below.

[0107] As an example, the depth image includes a first depth image based on the 3D Gaussian model and a second depth image based on monocular depth estimation, and the first regularization loss function is constructed by the following formula:

[0108]

[0109] wherein, is a first regular loss function, is a set of valid pixels, (i,j) is a pixel point in the i-th row and the j-th column, is a depth value of a first depth image pixel point (i,j), is a depth value of a second depth image pixel point (i,j).

[0110] Exemplarily, the depth image comprises a first depth image obtained based on a 3D Gaussian model and a second depth image obtained based on monocular depth estimation, denoted as , the first depth image obtained based on the 3D Gaussian model, and , the second depth image obtained based on the monocular depth estimation. The first regular loss function is constructed according to a pixel-level absolute depth loss , which can be used to ensure that the overall accuracy of depth estimation converges to the pseudo GT and provides global constraints.

[0111] As an example, the second regular loss function is constructed by the following formula:

[0112]

[0113] or,

[0114]

[0115] wherein, is a second regular loss function, is a k-th instance semantic region, (i,j) and (p,q) are pixel point pairs in the region, is a depth value of a first depth image pixel point (i,j), is a depth value of a first depth image pixel point (p,q), is a depth value of a second depth image pixel point (i,j), is a depth value of a second depth image pixel point (p,q), is a smooth control factor.

[0116] Exemplarily, for each instance , it is considered whether the depth difference between any two points in the region is consistent with the pseudo label. The second regular loss function is constructed according to the relative depth consistency loss within the instance . When the pseudo label is used, the second regular loss function . When the pseudo label is not used, the relative gradient smoothing term is adopted, and the second regular loss function ​Encourage instances where the depth variation is stable but not completely flat, avoiding the "depth collapse" caused by direct mean regression, and better reflecting the true relative geometric relationships of the structure.

[0117] As an example, the third regularized loss function is constructed using the following formula:

[0118]

[0119] in, The third regularized loss function, For any two distinct instances, For example average depth, For example average depth, The minimum depth difference threshold. .

[0120] For example, to prevent the depth of multiple adjacent instances from being overly smoothed (e.g., between towers and background, or between components), a relative depth structure loss between instances is introduced. Let... For any two distinct instances, the third regularization loss function This is used to encourage adjacent instances not to be too close, to preserve structural differences, and to enhance the clarity of 3D representation.

[0121] Constrained training is performed based on the initial Gaussian distribution of the constructed loss function to obtain the target Gaussian distribution. Projection is then performed based on the target Gaussian distribution and the obtained camera pose data, projecting the 3D Gaussian distribution onto the 2D image to obtain the projected image. To improve the image reconstruction quality and structural stability under sparse views, this application introduces a 3DGS model performance evaluation and structural control mechanism. The reconstruction effect of the current 3DGS model is monitored through the evaluation results of the projected image, and a series of structural control and Gaussian sphere model reset operations are performed accordingly to suppress redundancy, Gaussian drift, and structural degradation problems.

[0122] As an example, adjustments to the Gaussian sphere model may include at least one of the following: adjusting the number of Gaussian spheres in the Gaussian sphere model; adjusting the covariance matrix of the Gaussian sphere model; or resetting the parameters of the Gaussian spheres in the Gaussian sphere model.

[0123] For example, during the 3DGS model reconstruction process, to ensure the model's expressive power, the system dynamically generates new Gaussian points to fit the training images. Without restrictions, the number of Gaussian points will continue to grow. Therefore, adjusting the Gaussian sphere model includes adjusting the number of Gaussian spheres within the model. In 3DGS, the Gaussian covariance matrix... The shape of the point cloud projection in image space is controlled. If it is too elongated, it will lead to rendering distortion or numerical instability. Therefore, adjusting the Gaussian sphere model also includes adjusting its covariance matrix. Adjusting the Gaussian sphere model also includes resetting the parameters of the Gaussian sphere within the model; for example, performing a reset operation on a Gaussian model that does not meet quality requirements.

[0124] It should be noted that the above adjustments to the Gaussian sphere model can be made multiple times. The Gaussian sphere model is continuously adjusted based on the projection effect of the projected image until the projection quality of the projected image is optimal.

[0125] As an example, such as Figure 7 As shown, the number of Gaussian spheres in the Gaussian sphere model is adjusted, including:

[0126] S701, when the number of Gaussian spheres exceeds a preset threshold, sort the current Gaussian spheres based on their transparency.

[0127] S702, retains a preset threshold number of Gaussian spheres before transparency sorting.

[0128] For example, 3DGS models dynamically generate new Gaussian points (also known as Gaussian spheres) to fit training images. However, without restrictions, the number of Gaussian points will continue to grow, leading to a surge in GPU and memory overhead, structural overfitting, reduced model generalization ability, increased rendering costs and training time, and affecting the controllability of reconstruction results and post-processing efficiency, among other problems. This application employs a point cloud filtering mechanism based on transparency ranking to limit the number of Gaussian points within a controllable range, helping the model maintain the compactness and accuracy of structural representation in sparse views.

[0129] For example, let the current set of Gaussian spheres be... Each Gaussian sphere corresponds to a transparency value. As can be understood, transparency values ​​are attributes of Gaussian spheres, such as their color or distance, indicating their importance. When the total number of Gaussian spheres exceeds a set threshold... At that time, sort all Gaussian points by transparency from largest to smallest, and retain the top ones. Select one Gaussian point and discard the remaining Gaussian points with lower transparency.

[0130] As an example, the covariance matrix of the Gaussian sphere model is adjusted by including at least one of the following:

[0131] If the current eigenvalue of the covariance matrix is ​​greater than the preset maximum eigenvalue threshold, then the current eigenvalue is determined to be the preset maximum eigenvalue threshold.

[0132] If the current eigenvalue of the covariance matrix is ​​less than the preset minimum eigenvalue threshold, then the current eigenvalue is determined to be the preset minimum eigenvalue threshold.

[0133] The covariance matrix is ​​initialized if the ratio of the largest eigenvalue to the smallest eigenvalue of the covariance matrix is ​​greater than a preset ratio.

[0134] For example, in 3DGS, the Gaussian covariance matrix is ​​used to control the projection shape of the point cloud in image space. It can be understood that one Gaussian point corresponds to one covariance matrix; if this matrix is ​​too elongated (i.e., the covariance matrix has extremely small or extremely large eigenvalues), it can lead to rendering distortion or numerical instability. This application limits the scale range of the Gaussian covariance matrix to enhance stability.

[0135] For example, let the eigenvalues ​​of the Gaussian covariance matrix be... , Represents the Gaussian covariance matrix of the i-th Gaussian sphere The j-th eigenvalue. This application sets a minimum eigenvalue threshold and a maximum eigenvalue threshold. If the current eigenvalue of the covariance matrix is ​​greater than the preset maximum eigenvalue threshold, the current eigenvalue is determined to be the preset maximum eigenvalue threshold; if the current eigenvalue of the covariance matrix is ​​less than the preset minimum eigenvalue threshold, the current eigenvalue is determined to be the preset minimum eigenvalue threshold. It can be understood that this application sets a minimum eigenvalue threshold and a maximum eigenvalue threshold to limit the eigenvalue from not exceeding the maximum eigenvalue threshold and from being less than the minimum eigenvalue threshold.

[0136] For example, even with restrictions on the eigenvalues ​​of the covariance matrix, the entire matrix may sometimes degenerate into a "flat" or "highly directional" shape, requiring further processing. The covariance matrix is ​​initialized when the ratio of its largest to smallest eigenvalue is greater than a preset ratio. This application introduces a condition number criterion, namely the preset ratio. If the ratio of the largest eigenvalue to the smallest eigenvalue in a certain covariance matrix is... Greater than the preset ratio This indicates that the covariance matrix may degenerate into a "flat" or "highly directional" shape. In this case, the covariance matrix should be initialized or remodulated.

[0137] This application adjusts the eigenvalues ​​of the covariance matrix based on the Gaussian sphere model to ensure the shape of the covariance matrix and avoid "flat" or "highly directional" shapes.

[0138] As an example, such as Figure 8 As shown, the Gaussian sphere model is adjusted based on the evaluation results of the projected image, including:

[0139] S801, determine a peak signal-to-noise ratio based on the projection image and the verification set.

[0140] S802, in the case where the peak signal-to-noise ratio decreases continuously for a preset number of training periods, reset the parameters of the Gaussian ball in the Gaussian ball model.

[0141] Exemplarily, the peak signal-to-noise ratio is determined according to the difference between the projection image and the verification set image. If the peak signal-to-noise ratio decreases continuously for a preset number of training periods, it indicates that the parameters of some Gaussian points have deviated from the optimal structure, and the structure of these Gaussian points is reset. The preset number of times can be three, for example, if the peak signal-to-noise ratio decreases significantly on the verification set for three consecutive training stages, the parameters of the Gaussian ball in the Gaussian ball model are reset.

[0142] The application resets the Gaussian model that does not meet the quality requirements to ensure the quality of the Gaussian model and improve the quality of the reconstructed image.

[0143] As an example, resetting the parameters of the Gaussian ball in the Gaussian ball model includes resetting at least one of the transparency parameter, the color parameter, and the covariance matrix parameter of the Gaussian ball.

[0144] Exemplarily, when the parameters of the Gaussian ball in the Gaussian ball model are reset, the three-dimensional space coordinates can be retained . The reinitialization includes but is not limited to the following attributes: transparency ; color (e.g., mean color or Gaussian noise); covariance (matrix, usually a scaling of the identity matrix).

[0145] As an example, as Figure 9 shown, before image rendering based on the adjusted Gaussian ball model, the image reconstruction method further includes:

[0146] S901, establish a corresponding compensation factor based on each Gaussian ball.

[0147] S902, process the compensation factor based on a preset probability to obtain a mask vector of the Gaussian ball.

[0148] S903, process the current transparency tensor of the Gaussian ball based on the mask vector to obtain a target transparency tensor.

[0149] Exemplarily, in the three-dimensional reconstruction process of the Gaussian model, in order to reduce the influence of Gaussian points with low transparency on the rendering result, a transparency random discarding mechanism based on a compensation factor is proposed.

[0150] Exemplarily, a corresponding compensation factor is established based on each Gaussian sphere. It can be understood that the compensation factor is the discard probability of the Gaussian sphere, and a compensation factor is constructed for each Gaussian point, indicating whether the Gaussian point is retained for training in the current stage. The compensation factor is initialized in the form of a tensor as follows: . represents the total number of Gaussian points, is a one-dimensional vector. It can be understood that 1 indicates that the Gaussian point is retained for training, and 0 indicates that the Gaussian point is not retained. Based on the preset probability, the compensation factor is processed to obtain a mask vector of the Gaussian sphere. The Drop operation is applied to use the preset probability to control the Dropout operation acting on the above compensation factor to obtain a binary mask vector of whether to retain the current Gaussian point:

[0151]

[0152] The compensation mask processed by Drop is applied to the current transparency tensor to obtain a new target transparency tensor:

[0153]

[0154] The transparency random discard mechanism based on the compensation factor of the present application dynamically controls whether the Gaussian point participates in the rendering process by introducing a random Drop strategy without relying on explicit threshold screening, thereby improving the sparsity and representation efficiency of modeling.

[0155] After adjusting the Gaussian sphere model, image rendering is performed based on the adjusted Gaussian sphere model to obtain a target image. A three-dimensional Gaussian sputtering algorithm can be used for rendering processing to obtain the final target image required, thereby realizing high-fidelity restoration and visual expression of structural information under sparse view conditions.

[0156] Figure 10 is a flowchart of a tower image reconstruction method according to an embodiment of the present application.

[0157] As Figure 10As shown, taking the tower image reconstruction as an example, the VGGT model is used to predict the input image, generate the initial point cloud and camera pose, and convert the results into COLMAP compatible format to assist the stable initialization of the subsequent reconstruction task. Then the DepthAnything V2 monocular depth estimation model is used to predict the inverse depth map converted from the normalized input image. The K-Means algorithm of the experimental space perception is used to cluster the inverse depth map, and the mask is generated according to the connected region instance division. The depth map regularization method using semantic instance information is used to supervise the training of the model, and the transparency is discarded at any time when rendering the image. Supervise the model training, when the peak signal-to-noise ratio (PSNR) of the validation set continuously decreases, limit the number of Gaussian models and reset the model parameters to alleviate overfitting. Using 3DGS technology on the above model technology, after multiple rounds of training and adjustment, the reconstructed three-dimensional tower model is output. As shown in Figure 11 The schematic diagram of the tower image after three-dimensional reconstruction.

[0158] Figure 12 The structural schematic diagram of the three-dimensional Gaussian splash model of an embodiment of the present application.

[0159] As shown in Figure 12 The camera pose and initial sparse point cloud information generated by the VGGT model are used as the initial input of three-dimensional reconstruction to guide the establishment of the geometric framework of the scene structure. The inverse depth map output by the monocular depth estimation model Depth-Anything-V2 and the semantic instance mask map generated by spatial clustering are used to constrain the initialization position and distribution area of the Gaussian sphere in the three-dimensional space. During the training of the Gaussian model, the depth regularization strategy based on instance clustering is used to constrain the depth distribution of the Gaussian sphere, including absolute depth error, relative depth consistency error within the instance, and relative depth distinction loss between instances. Based on the PSNR index evaluation results of the validation set, the Gaussian sphere model is dynamically adjusted, including but not limited to limiting the number of Gaussian spheres, limiting the scale of the covariance matrix, and performing Gaussian sphere reset operation, to improve the modeling quality and rendering consistency. By introducing the random Drop strategy to dynamically control whether the Gaussian point participates in the rendering process, the generalization ability of the modeling is improved. Finally, based on the above multi-source information fusion and regularization optimization results, the three-dimensional Gaussian splash algorithm is used to generate the three-dimensional model of the target tower, realizing the high-fidelity restoration and visual expression of the structural information under the condition of sparse views.

[0160] The present application also proposes an image reconstruction device based on a 3D Gaussian model.

[0161] As an example, as Figure 13As shown, the image reconstruction device based on the 3D Gaussian model comprises: an acquisition module 1301 configured to acquire point cloud data and camera pose data of a to-be-processed image, and obtain an initial Gaussian distribution based on the point cloud data; a clustering module 1302 configured to perform depth prediction on the to-be-processed image to obtain a depth image, and perform semantic clustering processing on the depth image to obtain an instance semantic image; a loss function construction module 1303 configured to construct a target loss function based on the depth image and the instance semantic image, and constrain the initial Gaussian distribution based on the target loss function to obtain a target Gaussian distribution; a projection and adjustment module 1304 configured to perform projection based on the target Gaussian distribution and the camera pose data to obtain a projection image, and adjust the Gaussian sphere model based on an evaluation result of the projection image; and a rendering module 1305 configured to perform image rendering based on the adjusted Gaussian sphere model to obtain a target image.

[0162] The present application also provides a computer readable storage medium.

[0163] In this embodiment, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the image reconstruction method based on the 3D Gaussian model.

[0164] Figure 14 A block diagram of an electronic device according to an embodiment of the present application is provided.

[0165] An electronic device according to an embodiment of the present application includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the image reconstruction method based on the 3D Gaussian model.

[0166] As shown in Figure 14 To facilitate understanding, an embodiment of the present application shows a specific electronic device.

[0167] The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0168] As shown in Figure 14As shown, the device includes a computing unit 1401 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. Various programs and data required for operation of the electronic device can also be stored in the RAM 1403. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other through a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0169] A plurality of components in the electronic device are connected to the I / O interface 1405, including an input unit 1406 such as a keyboard, a mouse, etc., an output unit 1407 such as various types of displays, a speaker, etc., a storage unit 1408 such as a magnetic disk, an optical disk, etc., and a communication unit 1409 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1409 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0170] The computing unit 1401 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1401 performs various methods described above, such as the image reconstruction method based on a 3D Gaussian model. For example, in some embodiments, the image reconstruction method based on a 3D Gaussian model can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device via the ROM 1402 and / or the communication unit 1409. When the computer program is loaded into the RAM 1403 and executed by the computing unit 1401, the image reconstruction method based on a 3D Gaussian model described above can be performed. Alternatively, in other embodiments, the computing unit 1401 can be configured to perform the image reconstruction method based on a 3D Gaussian model by any other appropriate means, e.g., by means of firmware.

[0171] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this application, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0172] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0173] In the description of this application, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this application, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0174] In the description of the present application, it needs to be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the purpose of facilitating the description of the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0175] In addition, the terms "first", "second", and the like used in the embodiments of the present application are only for the purpose of description, and cannot be understood as indicating or implying relative importance, or implicitly indicating the number of technical features referred to in the embodiments. Therefore, the features defined with the terms "first", "second" and the like in the embodiments of the present application can be explicitly or implicitly indicated to include at least one of the features. In the description of the present application, the meaning of the word "plurality" is at least two or two or more, such as two, three, four, etc., unless otherwise specifically limited in the embodiments.

[0176] In the present application, unless otherwise specifically defined or limited in the embodiments, the terms "mounting", "connecting", "connecting" and "fixing" and the like appearing in the embodiments should be understood broadly, for example, the connection can be a fixed connection, or a detachable connection, or integrated, which can be understood, or can be a mechanical connection, an electrical connection, etc. Of course, it can also be directly connected, or indirectly connected through an intermediate medium, or it can be the internal communication of two elements, or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific implementation situation.

[0177] In the present application, unless otherwise specifically defined or limited, the first feature "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature "above", "above" and "above" the second feature can be that the first feature is directly above or obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "below" and "below" the second feature can be that the first feature is directly below or obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0178] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary, and are not to be interpreted as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for image reconstruction based on 3D Gaussian model, characterized in that, The method comprises: obtaining point cloud data and camera pose data of an image to be processed, and obtaining an initial Gaussian distribution based on the point cloud data; performing depth prediction on the image to be processed to obtain a depth image, and performing semantic clustering processing on the depth image to obtain an instance semantic image; constructing a target loss function based on the depth image and the instance semantic image, and constraining the initial Gaussian distribution based on the target loss function to obtain a target Gaussian distribution; projecting based on the target Gaussian distribution and the camera pose data to obtain a projection image, and adjusting a Gaussian sphere model based on an evaluation result of the projection image; performing image rendering based on the adjusted Gaussian sphere model to obtain a target image; the semantic clustering processing on the depth image to obtain an instance semantic image comprises: constructing a weighted spatial feature vector based on valid depth points in the depth image and corresponding depth values; performing clustering processing based on the weighted spatial feature vector to obtain a plurality of depth semantic regions; performing connected domain extraction processing on each of the depth semantic regions to obtain a plurality of instance semantic regions; obtaining an instance semantic image based on the instance semantic regions and corresponding instance encodings; the weighted spatial feature vector is constructed by the following formula: wherein, is the weighted spatial feature vector, is a spatial weight factor, is a pixel position of the valid depth point, is a depth value of the valid depth point.

2. The image reconstruction method of claim 1, wherein, the clustering processing based on the weighted spatial feature vector to obtain a plurality of depth semantic regions comprises: randomly selecting M valid depth points as centroids, and calculating the Euclidean distance between each valid depth point and the centroid based on the weighted spatial feature vector; based on the Euclidean distance, the valid depth points are assigned to clusters, and the centroids are updated based on the mean of the valid depth points in the clusters, and the iteration is repeated until the termination condition is met; a plurality of depth semantic regions are obtained based on a plurality of clusters, wherein the clustering labels of the valid depth points in the same depth semantic region are the same.

3. The image reconstruction method of claim 1, wherein, the connected domain extraction processing on each of the depth semantic regions to obtain a plurality of instance semantic regions comprises: performing connected domain extraction processing on each of the depth semantic regions to obtain a plurality of sub-regions; if the area of the sub-region is greater than or equal to a preset area, the sub-region is determined as the instance semantic region.

4. The image reconstruction method of claim 1, wherein, the target loss function is constructed based on the depth image and the instance semantic image, comprising: constructing a first regular loss function based on the full-image absolute loss of the depth image; constructing a second regular loss function based on the relative depth consistency loss within each instance in the instance semantic image; constructing a third regular loss function based on the relative depth structure loss between instances in the instance semantic image; determining the target loss function based on at least one of the first regular loss function, the second regular loss function, and the third regular loss function.

5. The image reconstruction method of claim 4, wherein, the depth image comprises a first depth image obtained based on a 3D Gaussian model and a second depth image obtained based on monocular depth estimation, and the first regular loss function is constructed by the following formula: wherein, is the first regular loss function, is the set of valid pixels, (i,j) is the pixel at the i-th row and j-th column, is the depth value of the first depth image pixel (i,j), is the depth value of the second depth image pixel (i,j).

6. The image reconstruction method of claim 5, wherein, the second regular loss function is constructed by the following formula: or, wherein, is the second regular loss function, is the k-th instance semantic region, (i,j) and (p,q) are pixel pairs within the region, is the depth value of the first depth image pixel point (i,j), is the depth value of the first depth image pixel point (p,q), is the depth value of the second depth image pixel point (i,j), is the depth value of the second depth image pixel point (p,q), is a smoothness control factor.

7. The image reconstruction method of claim 4, wherein, the third regular loss function is constructed by the following formula: wherein, is the third regular loss function, is any two different instances, is an instance average depth, is an instance average depth, is a minimum depth difference threshold, .

8. The image reconstruction method of claim 4, wherein, The method further comprises: The target loss function is determined based on at least one of the first regularization loss function, the second regularization loss function, and the third regularization loss function.

9. The image reconstruction method of claim 1, wherein, The first regularization loss function, the second regularization loss function, and the third regularization loss function are weighted to obtain the target loss function. The adjustment of the Gaussian sphere model comprises at least one of: The number of Gaussian spheres in the Gaussian sphere model is adjusted. The covariance matrix of the Gaussian sphere model is adjusted.

10. The image reconstruction method of claim 9, wherein, The parameters of the Gaussian spheres in the Gaussian sphere model are reset. The adjustment of the number of Gaussian spheres in the Gaussian sphere model comprises: In a case where the number of Gaussian spheres is greater than a preset number threshold, the current Gaussian sphere is sorted based on the transparency of the current Gaussian sphere.

11. The image reconstruction method of claim 9, wherein, A preset number threshold of Gaussian spheres before the transparency sorting is retained. The adjustment of the covariance matrix of the Gaussian sphere model comprises at least one of: In a case where a current eigenvalue of the covariance matrix is greater than a preset maximum eigenvalue threshold, the current eigenvalue is determined as the preset maximum eigenvalue threshold. In a case where the current eigenvalue of the covariance matrix is less than a preset minimum eigenvalue threshold, the current eigenvalue is determined as the preset minimum eigenvalue threshold.

12. The image reconstruction method of claim 1, wherein, In a case where a ratio of a maximum eigenvalue and a minimum eigenvalue of the covariance matrix is greater than a preset ratio, the covariance matrix is initialized. The adjustment of the Gaussian sphere model based on the evaluation result of the projection image comprises: A peak signal-to-noise ratio is determined based on the projection image and a validation set.

13. The image reconstruction method of claim 9 or 12, characterized in that, In a case where the peak signal-to-noise ratio decreases for a preset number of consecutive training periods, the parameters of the Gaussian spheres in the Gaussian sphere model are reset. The reset of the parameters of the Gaussian spheres in the Gaussian sphere model comprises:

14. The image reconstruction method of claim 1, wherein, At least one of the transparency parameters, the color parameters, and the covariance matrix parameters of the Gaussian spheres is reset. Before the image rendering based on the adjusted Gaussian sphere model, the method further comprises: A corresponding compensation factor is established based on each Gaussian sphere. The compensation factor is processed based on a preset probability to obtain a mask vector of the Gaussian sphere.

15. An image reconstruction apparatus based on a 3D Gaussian model, characterized by, A target transparency tensor is obtained by processing a current transparency tensor of the Gaussian sphere based on the mask vector. The device comprises: An acquisition module is configured to acquire point cloud data and camera pose data of an image to be processed, and obtain an initial Gaussian distribution based on the point cloud data; A clustering module is configured to perform depth prediction on the image to be processed to obtain a depth image, and perform semantic clustering processing on the depth image to obtain an instance semantic image; A loss function construction module is configured to construct a target loss function based on the depth image and the instance semantic image, and constrain the initial Gaussian distribution based on the target loss function to obtain a target Gaussian distribution; A projection and adjustment module is configured to perform projection based on the target Gaussian distribution and the camera pose data to obtain a projection image, and adjust a Gaussian sphere model based on an evaluation result of the projection image; A rendering module is configured to perform image rendering based on the adjusted Gaussian sphere model to obtain a target image. The clustering module is further configured to: construct a weighted spatial feature vector based on the valid depth points and corresponding depth values in the depth image; perform clustering processing based on the weighted spatial feature vector to obtain a plurality of depth semantic regions; perform connected domain extraction processing on each of the depth semantic regions to obtain a plurality of instance semantic regions; obtain an instance semantic image based on the instance semantic regions and corresponding instance encodings; The weighted spatial feature vector is constructed by the following formula: wherein, is the weighted spatial feature vector, is a spatial weight factor, is a pixel position of the valid depth point, is a depth value of the valid depth point.

16. An electronic device, comprising: A computer program product, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, A computer program product, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any one of claims 1-14.

Images