A semantic segmentation method and system for scene-level synthetic point cloud enhancement based on diffusion model

Abstract

This invention discloses a scene-level synthetic point cloud enhancement semantic segmentation method and system based on a diffusion model, aiming to solve the problem of insufficient point cloud training samples in deep learning. The method includes the following steps: First, a high-quality panoramic image is generated based on text using a diffusion model; next, a deep learning visual transformer is used to predict the depth of the panoramic image, and semantic segmentation annotation is performed using the SAM intelligent annotation method; then, a cylindrical projection method is used to map the depth information and semantic labels onto the panoramic image, generating a 3D point cloud containing spatial coordinates, RGB color values, and semantic labels. The generated synthetic point cloud is then combined with the real point cloud to train the deep learning model, thereby improving the accuracy of semantic segmentation. Finally, the trained model is used to predict the semantic segmentation of the target scene, generate semantic labels, and import the segmentation results into BIM software to complete 3D model reconstruction, contributing to the intelligent operation and maintenance and monitoring of buildings.

Description

Technical Field

[0001] This invention belongs to the field of civil engineering and artificial intelligence interaction technology, specifically relating to a semantic segmentation method and system for scene-level synthetic point cloud enhancement based on a diffusion model. Background Technology

[0002] Point cloud-based 3D model reconstruction can obtain a building's BIM (Building Information Model), thereby effectively promoting building operation and maintenance. The first step in 3D model reconstruction is semantic segmentation. Deep learning-based semantic segmentation significantly reduces reliance on manual design features and achieves better performance by autonomously learning and extracting complex data features from a large amount of labeled training data. However, deep learning typically requires a large amount of sample data for training. Compared to images, point cloud data acquisition relies on multi-station scanning laser scanners, which is very time-consuming and labor-intensive. Therefore, there is an urgent need for improved point cloud generation methods. Summary of the Invention

[0003] This invention addresses the problems existing in the prior art by providing a semantic segmentation method and system based on a diffusion model for scene-level synthetic point cloud enhancement, which can improve the semantic segmentation accuracy and efficiency of point cloud data in underground parking garages.

[0004] To address the above technical problems, this invention provides a semantic segmentation method for scene-level synthetic point cloud enhancement based on a diffusion model, comprising the following steps:

[0005] S1. Scan the real point cloud scene to be reconstructed to obtain the real point cloud and perform preprocessing. The real 3D point cloud includes the XYZ coordinates and RGB color information of each point.

[0006] S2. Construct and train the diffusion model, specifically: a two-branch method based on panoramic and perspective branches, with the panoramic branch as the input of the diffusion model; taking the noisy panoramic latent map as input, the perspective branch generates noisy perspective latent maps corresponding to different parts of the scene; the two branches achieve feature fusion of panoramic and perspective images through the isometric perspective projection attention (EPPA) mechanism; a low-rank adaptive LoRA layer is used to adaptively adjust the model weights; and within the two branches, a U-Net structure with skip connections is used. The diffusion model finally outputs a high-fidelity panoramic image.

[0007] S3. Obtain a high-fidelity panoramic image using a diffusion model, perform semantic segmentation and annotation on the panoramic image, and generate semantic tags for the panoramic image.

[0008] S4. Construct and train a depth prediction model based on the Vision Transformer (ViT). Specifically, take a panoramic image as input and the corresponding depth map as output to train the depth prediction model. During training, the input panoramic image is divided into small grids. Each small grid is flattened and linearly transformed into an embedding using the learned weight matrix and bias parameters. The embedded labels are processed by a self-attention mechanism. After several self-attention layers, the final depth prediction is obtained by mapping the output depth map of the regression head transformer.

[0009] S5. Obtain a depth map using a depth prediction model, and use a cylindrical projection method to map the depth map and semantic labels onto the panoramic image to generate synthetic 3D point cloud data containing spatial coordinates, RGB color values ​​and semantic labels.

[0010] S6. Add the synthesized 3D point cloud to the real 3D point cloud to obtain hybrid point cloud data. At the same time, construct a deep learning semantic segmentation network. Train the deep learning semantic segmentation network with the hybrid point cloud data as input and the semantic segmentation result as output to obtain a deep learning semantic segmentation model. Use the deep learning semantic segmentation model to segment the real point cloud scene to be reconstructed, and construct a BIM model based on the segmentation result.

[0011] Furthermore, in step S2 above, the following loss function L is minimized when training the diffusion model:

[0012] L=Eε (x) ,t,∈,y[||∈-∈ θ (z t ,t,τ(y))||2] (1)

[0013] Where, Eε (x) Let t, ∈, and y represent the expectations of the input image x, time step t, noise ∈, and condition variable y, respectively. θ (z t ,t,τ(y)) is a denoising function, which predicts noise in the latent space using a U-Net neural network, [||∈-∈ θ (z t ,t,τ(y))||2 denotes the L2 norm, used to calculate the predicted noise ∈ θ (z t The Euclidean distance between t, τ(y) and the actual noise ∈.

[0014] Furthermore, in step S3 mentioned above, the Roboflow platform is used for semantic annotation, utilizing an intelligent annotation method based on the SegmentAnything Model.

[0015] Furthermore, the aforementioned step S4 includes the following sub-steps:

[0016] S4.1 Divide the input image I into a small grid I_patch. Each small grid is flattened and linearly transformed into an embedding Em using the learned weight matrix and bias parameters, as shown in the following equation.

[0017] Em = Flatten(I) _patch )·W E +b E (2)

[0018] Among them, W E and b E These are the learned embedding weights and bias parameters.

[0019] S4.2 The embedded tags are processed through a self-attention mechanism to evaluate the correlation between each small grid and other small grids, and dynamically adjust the correlation of each small grid to capture local and global spatial dependencies.

[0020] S4.3 After processing through several self-attention layers, the final depth prediction is obtained by outputting the depth map from the regression head mapping transformer:

[0021] D = Em·W D +b D (3)

[0022] Where D is the predicted depth map, W D and b D These are the learning parameters in the regression head.

[0023] Furthermore, the aforementioned step S5 includes the following sub-steps:

[0024] S5.1 Calculate the spherical coordinates corresponding to each pixel (u,v) in a panoramic image of size H×W, as follows:

[0025]

[0026] Where θ is the azimuth angle and φ is the polar angle;

[0027] S5.2 Calculate the 3D coordinates (X, Y, Z) of each pixel in the world coordinate system, using the following formula:

[0028] X u，v =D u，v ·cos(φ)·cos(θ) (5)

[0029] Y u，v =D u，v ·cos(φ)·sin(θ) (6)

[0030] Z u，v =D u，vsin(φ) (7)

[0031] Among them, D u,v It is the depth value of the pixel;

[0032] S5.3 Each pixel (u,v) of the panoramic image I is assigned a semantic label L according to a predefined scene category. u,v Tag L u,v Each point in the generated synthetic point cloud is associated with a corresponding 3D point and has a corresponding label. Finally, the generated point cloud P consists of spatial coordinates (X,Y,Z), RGB color values, and semantic labels, as shown below:

[0033] P = (X u，v Y u，v Z u，v RGB u，v .L u，v )∣(u,v)∈I (8)

[0034] Where RGB represents the pixel color extracted from the panoramic image I, and L... u,v It is a semantic tag.

[0035] Furthermore, in step S6 above, a deep learning semantic segmentation network is constructed based on PointNet or PointNet++ network, and the deep learning semantic segmentation model is evaluated using three key indicators: overall accuracy (OA), intersection-to-union ratio (IoU), and average intersection-to-union ratio (mIoU).

[0036] Another aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described in the present invention.

[0037] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of any of the methods described in the present invention.

[0038] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0039] By employing deep learning methods, this invention significantly improves the semantic segmentation accuracy and efficiency of point cloud data for underground parking garages, while substantially reducing manual intervention and operation time. The trained model can quickly and efficiently process large-scale point cloud data, generating accurate 3D models to support the construction, monitoring, and maintenance of underground parking garages. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention.

[0041] Figure 2 This is a flowchart of the two-branch method of the present invention.

[0042] Figure 3 This is an example image with a panoramic view annotation.

[0043] Figure 4 Here are example images of depth prediction for panoramic images. In the image, (a) is an RGB panoramic image and (b) is the predicted depth map.

[0044] Figure 5 This is an example diagram illustrating the process of generating a synthetic point cloud. Detailed Implementation

[0045] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0046] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0047] refer to Figure 1 This invention provides a semantic segmentation method for scene-level synthetic point cloud enhancement based on a diffusion model, comprising the following steps:

[0048] S1. Scan the real point cloud scene to be reconstructed to obtain the real point cloud and perform preprocessing. The real 3D point cloud includes the XYZ coordinates and RGB color information of each point.

[0049] S2. Construct and train the diffusion model, specifically: a two-branch method based on panoramic and perspective branches, with the panoramic branch as the input of the diffusion model; taking the noisy panoramic latent map as input, the perspective branch generates noisy perspective latent maps corresponding to different parts of the scene; the two branches achieve feature fusion of panoramic and perspective images through the isometric perspective projection attention (EPPA) mechanism; a low-rank adaptive LoRA layer is used to adaptively adjust the model weights; and within the two branches, a U-Net structure with skip connections is used. The diffusion model finally outputs a high-fidelity panoramic image.

[0050] S3. Obtain a high-fidelity panoramic image using a diffusion model, perform semantic segmentation and annotation on the panoramic image, and generate semantic tags for the panoramic image.

[0051] S4. Construct and train a depth prediction model based on the Vision Transformer (ViT). Specifically, take a panoramic image as input and the corresponding depth map as output to train the depth prediction model. During training, the input panoramic image is divided into small grids. Each small grid is flattened and linearly transformed into an embedding using the learned weight matrix and bias parameters. The embedded labels are processed by a self-attention mechanism. After several self-attention layers, the final depth prediction is obtained by mapping the output depth map of the regression head transformer.

[0052] S5. Obtain a depth map using a depth prediction model, and use a cylindrical projection method to map the depth map and semantic labels onto the panoramic image to generate synthetic 3D point cloud data containing spatial coordinates, RGB color values ​​and semantic labels.

[0053] S6. Add the synthesized 3D point cloud to the real 3D point cloud to obtain hybrid point cloud data. At the same time, construct a deep learning semantic segmentation network. Train the deep learning semantic segmentation network with the hybrid point cloud data as input and the semantic segmentation result as output to obtain a deep learning semantic segmentation model. Use the deep learning semantic segmentation model to segment the real point cloud scene to be reconstructed, and construct a BIM model based on the segmentation result.

[0054] In a preferred embodiment of the present invention, step S1 uses a laser scanner such as a Leica RTC360 to acquire point cloud data of the building interior. The acquired point cloud data is preprocessed, including noise removal and outlier removal. Furthermore, the point cloud data is downsampled using a voxel downsampling method to simplify it to a reasonable size while preserving key geometric features, facilitating subsequent semantic segmentation and model reconstruction. The preprocessed point cloud data is stored in a standard format such as .pcd or .las for subsequent processing.

[0055] In a preferred embodiment of the present invention, step S2 involves training a diffusion model. During the model training phase, real panoramic images and their corresponding text descriptions are used. This study utilizes the Matterport3D dataset, which contains 10,800 panoramic images and 2,295 room layout annotations. However, the original Matterport3D dataset does not include text descriptions. To address this issue, the BLIP-2 model, an advanced visual-language model, is used to generate image titles and understand visual-language relationships. Each image is prompted with "a 360-degree view of," generating a concise text description related to the panoramic image for conditional text generation.

[0056] For example, to generate panoramic images of building interiors, descriptive languages ​​such as "An office with a window and a bookcase" or "A study room with a bright window" can be used to generate synthetic point cloud scenes.

[0057] In step S2, the goal of the diffusion model is to reconstruct a high-fidelity image by progressively optimizing the noise latent variables. The diffusion model is trained by minimizing the following loss function L:

[0058] L=Eε (x) ,t,∈,y[||∈-∈ θ (z t ,t,τ(y))||2] (1)

[0059] Where, Eε (x) Let t, ∈, and y represent the expectations of the input image x, time step t, noise ∈, and condition variable y, respectively. θ (z, t, τ(y)) is a denoising function that predicts noise in the latent space using a U-Net neural network. This function is based on the latent variable z. t The model uses conditional encoding τ(y) to predict noise, and through this denoising process, it can progressively correct z in the latent space. t This allows for the eventual reconstruction of the image. θ (z t ,t,τ(y))||2 denotes the L2 norm, used to calculate the predicted noise ∈ θ (z t The Euclidean distance between t, τ(y) and the actual noise ∈ is such that the closer this value is to zero, the better the denoising effect of the model.

[0060] Traditional diffusion-based image generation methods are limited to generating regular images and cannot generate panoramic images. To overcome this limitation, this invention employs a two-branch method, including a panoramic branch and a perspective branch, such as... Figure 2 As shown. The panorama branch is responsible for generating the panoramic view, while the perspective branch generates the perspective projection, facilitating cross-format information exchange.

[0061] In this model architecture, the generation process begins with a noisy panoramic latent map, initially created based on the input text prompts. This latent map serves as the foundation for generating the panoramic view, capturing a wide field of view. Simultaneously, a perspective branch generates noisy perspective latent maps corresponding to different parts of the scene. These perspective latent maps are then projected onto the panoramic map, creating a seamlessly integrated panoramic representation.

[0062] To facilitate the integration of information between the two branches, an isometric perspective projection attention (EPPA) mechanism is employed. EPPA achieves efficient cross-branch attention by aligning and merging features from the panoramic image and the perspective latent image, thereby enhancing spatial consistency and providing a unified scene representation. Furthermore, a low-rank adaptation (LoRA) layer is used, which adaptively adjusts the model's weights to optimize its performance in the panoramic generation task.

[0063] Within each branch, a U-Net structure with skip connections is used to ensure high-resolution output and preserve details, ultimately generating high-quality panoramic images. By leveraging these design choices, the two-branch approach achieves panoramic image generation while maintaining spatial consistency and visual fidelity across the entire field of view.

[0064] In a preferred embodiment of the present invention, step S3 performs semantic segmentation and annotation on the panoramic image. Since point clouds consist of a set of points in three-dimensional space, directly annotating these points may be time-consuming and labor-intensive. One advantage of generating point clouds from panoramic images is that semantic annotation can be performed on the image, and then these annotations can be projected onto the three-dimensional point cloud. Compared with three-dimensional point clouds, annotation of two-dimensional images is relatively simple and has more mature technologies to support it. The present invention performs annotation on the Roboflow image annotation platform, which adopts an intelligent annotation method based on the Segment Anything Model (SAM). This method allows users to automatically select similar pixels near manually clicked areas, thereby significantly improving annotation efficiency. In addition, only the required semantic categories need to be annotated, and any unannotated areas are automatically set to the "messy" category. Furthermore, custom algorithms can be customized according to annotation preferences. For example, if a window is embedded in a wall, and the pixels in the window area are simultaneously labeled as both wall and window categories, a custom algorithm can be designed to retain only the window category. Annotating panoramic images in this way significantly improves the efficiency of point cloud annotation. Figure 3 An example of completed annotation is shown.

[0065] Since the synthetic images generated by the diffusion model lack depth information, this invention utilizes a deep learning method—Visual Transformer (ViT)—to directly predict depth from image data. As a preferred embodiment of this invention, step S4 includes the following sub-steps:

[0066] S4.1, ViT's depth prediction process begins by segmenting the input image into small patches, each representing a local region in the image. By converting these patches into labels, the model can process the entire image while preserving its spatial structure. Specifically, the input image I is divided into a small grid I_patch, and each grid patch is flattened and linearly transformed into an embedding Em using the learned weight matrix and bias parameters, as shown in the following equation.

[0067] Em = Flatten(I) _patch )·W E +b E (2)

[0068] Among them, W E and b E These are the learned embedding weights and bias parameters.

[0069] S4.2 The embedded tags are processed through a self-attention mechanism to evaluate the correlation between each small grid and other small grids, and dynamically adjust the correlation of each small grid to capture local and global spatial dependencies.

[0070] S4.3 After processing through several self-attention layers, the final depth prediction is obtained by outputting the depth map from the regression head mapping transformer:

[0071] D = Em·W D +b D (3)

[0072] Where D is the predicted depth map, W D and b D These are the learning parameters in the regression head.

[0073] The depth prediction model based on the Vision Transformer (ViT) is built using a pre-trained model from Omnidata. The Omnidata pre-trained model has been trained extensively on various datasets for depth prediction tasks, enhancing its generalization ability. Furthermore, the predicted depth values ​​have been adjusted. For example, in indoor panoramic scenes, the maximum depth range is limited to the typical depth range of indoor environments to avoid predicted values ​​exceeding the actual range. An example of depth prediction is shown below. Figure 4 As shown in the figure, (a) shows the RGB panoramic image, and (b) shows the predicted depth map.

[0074] In a preferred embodiment of the present invention, step S5 includes the following sub-steps:

[0075] S5.1 After obtaining depth information and semantic labels from the panoramic image, this study uses a spherical projection method to generate a point cloud scene. For a panoramic image I of size H×W and its corresponding depth map D, the two-dimensional pixel values ​​are converted into three-dimensional coordinates through spherical projection. For each pixel (u,v), where u and v represent the horizontal and vertical indices respectively, the spherical coordinates are calculated as follows:

[0076]

[0077] Where θ is the azimuth angle and φ is the polar angle;

[0078] S5.2 Calculate the 3D coordinates (X, Y, Z) of each pixel in the world coordinate system, using the following formula:

[0079] X u，v =D u，v·cos (φ)·cos(θ) (5)

[0080] Y u，v =D u，v ·cos(φ)·sin(θ) (6)

[0081] Z u，v =D u，v sin(φ) (7)

[0082] Among them, D u,v It is the depth value of the pixel;

[0083] S5.3 Each pixel (u,v) of the panoramic image I is assigned a semantic label L according to a predefined scene category. u,v Tag L u,v Each point in the generated synthetic point cloud is associated with a corresponding 3D point and has a corresponding label. Finally, the generated point cloud P consists of spatial coordinates (X,Y,Z), RGB color values, and semantic labels, as shown below:

[0084] P = (X u，v Y u，v Z u，v RGB u，v .L u，v )∣(u,v)∈I (8)

[0085] Where RGB represents the pixel color extracted from the panoramic image I, and L... u,v It is a semantic tag.

[0086] Using this method, the present invention is able to convert the generated panoramic image into an accurate 3D point cloud and assign the correct semantic category to each point.

[0087] An example of projecting a panoramic image into a 3D point cloud, from textual description to generating the panoramic image and finally the composite point cloud, is shown below. Figure 5 As shown.

[0088] Step S6 specifically involves adding the generated synthetic point cloud to a real-world dataset for training. This invention chose PointNet++ as the primary model due to its fundamental and widely adopted architecture. Furthermore, other deep learning models (such as PointNet and PointNeXt) were used to further evaluate the effectiveness of the proposed method. To evaluate the performance of deep learning-based semantic segmentation, this study employed three key metrics: overall accuracy (OA), intersection-to-union ratio (IoU), and mean intersection-to-union ratio (mIoU).

[0089] Model Optimization and Evaluation. After training, the model is tuned using a validation set, adjusting the network structure and hyperparameters to further optimize model performance. The optimal model configuration is selected through cross-validation and multiple experiments. The evaluation process includes calculating the model's mean intersection-over-union (mIoU) and overall accuracy (OA) on the test set to ensure the model's accuracy and robustness meet the requirements of practical applications. The optimized model is used for point cloud semantic segmentation tasks in real-world scenarios. Practical Application. The trained and optimized deep learning model is applied to actual scanned point cloud data processing. First, point cloud data of a new scene is collected using a laser scanner and preprocessed using the same methods as in step S1. Then, the preprocessed point cloud data is input into the trained model for automatic semantic segmentation, identifying and classifying various objects in the point cloud scene. The segmentation results are used to reconstruct a 3D model in BIM software such as Revit, generating a 3D digital twin model of the point cloud scene, which can be used for subsequent facility management, monitoring, and maintenance.

[0090] The lab requirements for training include: a high-performance GPU (such as an NVIDIA RTX 4090), a Linux operating system, and the use of the Python programming language and the PyTorch deep learning framework.

[0091] Another aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described in the present invention.

[0092] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described in the present invention.

[0093] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A semantic segmentation method for scene-level synthetic point cloud enhancement based on a diffusion model, characterized in that, Includes the following steps: S1. Scan the real point cloud scene to be reconstructed to obtain the real point cloud and perform preprocessing. The real 3D point cloud includes the XYZ coordinates and RGB color information of each point. S2. Construct and train the diffusion model, specifically: a two-branch method based on panoramic and perspective branches, with the panoramic branch as the input of the diffusion model; taking the noisy panoramic latent map as input, the perspective branch generates noisy perspective latent maps corresponding to different parts of the scene; the two branches achieve feature fusion of panoramic and perspective images through the isometric perspective projection attention (EPPA) mechanism; a low-rank adaptive LoRA layer is used to adaptively adjust the model weights; and within the two branches, a U-Net structure with skip connections is used. The diffusion model finally outputs a high-fidelity panoramic image. S3. Obtain a high-fidelity panoramic image using a diffusion model, perform semantic segmentation and annotation on the panoramic image, and generate semantic tags for the panoramic image. S4. Construct and train a depth prediction model based on the Vision Transformer (ViT). Specifically, take a panoramic image as input and the corresponding depth map as output to train the depth prediction model. During training, the input panoramic image is divided into small grids. Each small grid is flattened and linearly transformed into an embedding using the learned weight matrix and bias parameters. The embedded labels are processed by a self-attention mechanism. After several self-attention layers, the final depth prediction is obtained by mapping the output depth map of the regression head transformer. S5. Obtain a depth map using a depth prediction model, and use a cylindrical projection method to map the depth map and semantic labels onto the panoramic image to generate synthetic 3D point cloud data containing spatial coordinates, RGB color values ​​and semantic labels. S6. Add the synthesized 3D point cloud to the real 3D point cloud to obtain hybrid point cloud data. At the same time, construct a deep learning semantic segmentation network. Train the deep learning semantic segmentation network with the hybrid point cloud data as input and the semantic segmentation result as output to obtain a deep learning semantic segmentation model. Use the deep learning semantic segmentation model to segment the real point cloud scene to be reconstructed, and construct a BIM model based on the segmentation result.

2. The semantic segmentation method for scene-level synthetic point cloud enhancement based on a diffusion model according to claim 1, characterized in that, In step S2, the following loss function L is minimized when training the diffusion model: L=Eε (x) ,t,∈,y[||∈-∈ θ (z t ,t,τ(y))||2] (1) Where, Eε (x) Let t, ∈, and y represent the expectations of the input image x, time step t, noise ∈, and condition variable y, respectively. θ (z t ,t,τ(y)) is a denoising function, which predicts noise in the latent space using a U-Net neural network, [||∈-∈ θ (z t ,t,τ(y))||2 denotes the L2 norm, used to calculate the predicted noise ∈ θ (z t The Euclidean distance between t, τ(y) and the actual noise ∈.

3. The semantic segmentation method for scene-level synthetic point cloud enhancement based on a diffusion model according to claim 1, characterized in that, In step S3, the Roboflow platform is used for semantic annotation, utilizing an intelligent annotation method based on Segment AnythingModel.

4. The semantic segmentation method for scene-level synthetic point cloud enhancement based on a diffusion model according to claim 1, characterized in that, Step S4 includes the following sub-steps: S4.1 Divide the input image I into a small grid I_patch. Each small grid is flattened and linearly transformed into an embedding Em using the learned weight matrix and bias parameters, as shown in the following equation. Em=Flatten(I _patch )·W E +b E (2) Among them, W E and b E These are the learned embedding weights and bias parameters. S4.2 The embedded tags are processed through a self-attention mechanism to evaluate the correlation between each small grid and other small grids, and dynamically adjust the correlation of each small grid to capture local and global spatial dependencies. S4.3 After processing through several self-attention layers, the final depth prediction is obtained by outputting the depth map from the regression head mapping transformer: D=Em·W D +b D (3) Where D is the predicted depth map, W D and b D These are the learning parameters in the regression head.

5. The semantic segmentation method for scene-level synthetic point cloud enhancement based on a diffusion model according to claim 1, characterized in that, Step S5 includes the following sub-steps: S5.1 Calculate the spherical coordinates corresponding to each pixel (u,v) in a panoramic image of size H×W, as follows: Where θ is the azimuth angle and φ is the polar angle; S5.2 Calculate the 3D coordinates (X, Y, Z) of each pixel in the world coordinate system, using the following formula: X u，v =D u，v ·cos (φ)·cos (θ) (5) Y u，v =D u，v ·cos (φ)·sin (θ) (6) WITH u，v =D u，v sin (φ) (7) Among them, D u,v It is the depth value of the pixel; S5.3 Each pixel (u,v) of the panoramic image I is assigned a semantic label L according to a predefined scene category. u,v Tag L u,v Each point in the generated synthetic point cloud is associated with a corresponding 3D point and has a corresponding label. Finally, the generated point cloud P consists of spatial coordinates (X,Y,Z), RGB color values, and semantic labels, as shown below: P=(X u，v , Y u，v , Z u，v ,RGB u，v . L u，v )∣(u,v)∈I (8) Where RGB represents the pixel color extracted from the panoramic image I, and L... u,v It is a semantic tag.

6. The semantic segmentation method for scene-level synthetic point cloud enhancement based on a diffusion model according to claim 1, characterized in that, In step S6z, a deep learning semantic segmentation network is constructed based on PointNet or PointNet++ network. At the same time, three key indicators, namely overall accuracy (OA), intersection-to-union ratio (IoU), and average intersection-to-union ratio (mIoU), are used to evaluate the deep learning semantic segmentation model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images