Construction method and system from depth map to point cloud data based on large model

Abstract

The invention discloses a construction method and system from a depth map to point cloud data based on a large model, and the method comprises the steps: carrying out the preprocessing of an input depth map through a pre-trained large model, and extracting the features of the depth map; fusing the extracted depth map features to generate fused features; and based on the fused features, generating final point cloud data. According to the method, the geometric and semantic understanding capability of the depth map data is remarkably enhanced, and the accuracy and adaptability of complex scene point cloud generation are improved. The large model, multi-modal fusion and context enhancement module are utilized, the generated point cloud has higher trueness and diversity, meanwhile, the generation efficiency and the automation level are greatly improved, and a large amount of manpower and time cost is saved for practical application. Besides, the quality of the generated point cloud is verified through various evaluation indexes, and the method can be directly applied to the fields of 3D modeling, automatic driving, virtual reality, robot navigation and the like, and has a wide and profound promotion effect on industrial and scientific research application.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a method and system for constructing point cloud data from a depth map based on a large model. Background Art

[0002] Depth maps, as an important two-dimensional data representation in computer vision, can intuitively record the distance information from each pixel in the scene to the camera. Therefore, they have wide applications in fields such as autonomous driving, robot navigation, and augmented reality. However, depth maps are essentially two-dimensional data, and their structural limitations make them difficult to use directly for analysis and manipulation of three-dimensional space. In contrast, point cloud data is a three-dimensional data representation that directly models the shape, position, and topology of target objects in the scene, providing richer information for three-dimensional reconstruction, environmental perception, and scene understanding. Therefore, generating high-quality point cloud data from depth maps has become an important research direction in three-dimensional modeling and reconstruction technology.

[0003] Traditional methods for converting depth maps to point cloud data primarily rely on mathematical methods such as geometric modeling and projection mapping. These methods typically assume that the depth map has ideal sampling quality and uniform distribution. However, in practical applications, depth maps are often affected by factors such as noise, occlusion, and resolution limitations, making traditional methods prone to error accumulation and reduced accuracy when generating point cloud data. Furthermore, traditional methods often require manual adjustment of multiple parameters and complex preprocessing of the depth map data, which not only increases development costs but also limits their applicability in diverse scenarios. Summary of the Invention

[0004] To address the technical issues outlined above, this paper provides a method for constructing point cloud data from depth maps based on a large model. This method aims to efficiently and accurately generate point cloud data from depth maps, while improving the integrity and detail restoration capabilities of the point cloud. This method effectively processes depth information in complex scenes while leveraging the multimodal understanding and generation capabilities of large models, resulting in a more efficient and intelligent depth map-to-point cloud conversion.

[0005] To achieve the above objectives, the present invention provides a method for constructing point cloud data from a depth map based on a large model, the steps comprising:

[0006] Use the pre-trained large model to pre-process the input depth map and extract the features of the depth map;

[0007] Fuse the extracted depth map features to generate fused features;

[0008] Based on the fused features, the final point cloud data is generated.

[0009] Preferably, the step of performing the pretreatment comprises:

[0010] First, the pixel values ​​of the input depth map are normalized:

[0011] Afterwards, the depth map is denoised using the non-local means method:

[0012] Finally, it is input into the ViT network to extract high-dimensional image features F img :

[0013] F img =ViT(D norm )

[0014] Among them, D norm Represents the processed depth map;

[0015] At the same time, the semantic feature F is generated by combining the processed depth image with the text prompt text .

[0016] Preferably, the method for performing normalization processing includes:

[0017]

[0018] Among them, d i，j Indicates the depth value; d′ i，j represents the normalized value; d max and d min are the maximum and minimum values ​​of the depth map, respectively.

[0019] Preferably, the step of performing noise reduction includes:

[0020]

[0021] Among them, d denosied (i) represents the depth map after denoising; w(i, j) represents the similarity weight between pixel i and pixel j; Ω represents the neighborhood window of pixel i; h represents the smoothing control parameter; d(i) and d(j) represent the depth values ​​of pixel i and pixel j in the depth map, respectively.

[0022] Preferably, the method for generating the fused features includes: transforming the image features F img and semantic features F text Embedded into a unified feature space:

[0023] Q,K,V=Linear(F img ),Linear(F text )

[0024] Where Q, K, and V are the query, key, and value matrices in the multimodal Transformer, respectively; Linear represents a linear transformation;

[0025] By multiplying the attention weight matrix with the value matrix V, the fused feature representation F can be generated fusion :

[0026] F fusion =A·V

[0027] Among them, A represents the attention weight matrix.

[0028] Preferably, the method for generating the point cloud data includes: using a generative adversarial network to achieve mapping of fusion features to three-dimensional point cloud data; and generating high-precision point cloud data through adversarial training of a generator and a discriminator.

[0029] The present invention also provides a system for constructing point cloud data from a depth map based on a large model, the system being used to implement the above method, comprising: a preprocessing module, a fusion module and a generation module;

[0030] The preprocessing module is used to preprocess the input depth map using a pre-trained large model to extract features of the depth map;

[0031] The fusion module is used to fuse the extracted depth map features to generate fused features;

[0032] The generation module is used to generate final point cloud data based on the fused features.

[0033] Preferably, the workflow of the preprocessing module includes:

[0034] First, the pixel values ​​of the input depth map are normalized:

[0035] Afterwards, the depth map is denoised using the non-local means method:

[0036] Finally, it is input into the ViT network to extract high-dimensional image features F img :

[0037] F img =ViT(D norm )

[0038] Among them, D norm Represents the processed depth map;

[0039] At the same time, the semantic feature F is generated by combining the processed depth image with the text prompt text .

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This invention significantly enhances the geometric and semantic understanding of depth map data, improving the accuracy and adaptability of point cloud generation for complex scenes. By utilizing large models, multimodal fusion, and context-enhanced modules, the generated point cloud exhibits greater realism and diversity, while significantly improving generation efficiency and automation, saving significant manpower and time costs for practical applications. Furthermore, the quality of the generated point cloud is verified using a variety of evaluation metrics and can be directly applied to fields such as 3D modeling, autonomous driving, virtual reality, and robotic navigation, having a broad and far-reaching impact on industrial and scientific research applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is a schematic diagram of the pre-processing process of an embodiment of the present invention;

[0044] Figure 2 This is a flow chart of the feature fusion stage in an embodiment of the present invention;

[0045] Figure 3 Schematic diagram of the point cloud generation process in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] Example 1

[0049] This embodiment discloses a method for constructing point cloud data from a depth map based on a large model, comprising the following steps:

[0050] S1. Use the pre-trained large model to preprocess the input depth map and extract the features of the depth map.

[0051] In this embodiment, the depth map obtained is assumed to be in an autonomous driving scenario. A pre-trained large model is used to extract global features from the input depth map. The spatial structural features and local detail features in the depth map are mined through a deep network structure to generate a multi-scale feature representation. Specifically, the pre-trained large model includes a convolutional neural network (CNN), a Transformer structure, or a combination of the two, which is used to simultaneously extract global and local features of the depth map.

[0052] like Figure 1 As shown, first, the geometric and semantic characteristics of the input data are optimized through normalization, denoising and completion operations.

[0053] For the input depth map, the pixel values ​​are first normalized to adjust the depth values ​​to the range of [0, 1]. The formula is:

[0054]

[0055] Among them, d i，j Indicates the depth value; d′ i,j represents the normalized value; d max and d min are the maximum and minimum values ​​of the depth map, respectively.

[0056] Subsequently, the depth map is denoised using the non-local means method (NLM), which is formulated as follows:

[0057]

[0058] Among them, d denosied (i) represents the denoised depth map; w(i, j) represents the similarity weight between pixels i and j; Ω represents the neighborhood window of pixel i; h represents the smoothing control parameter; d(i) and d(j) represent the depth values ​​of pixels i and j in the depth map, respectively, representing the physical distance from the source to the pixel. This approach effectively eliminates outliers in the depth map caused by sensor noise.

[0059] Based on preprocessing, the depth map data is structured and analyzed, including the calculation of the gradient distribution and edge characteristics of the data. The Sobel operator or Laplacian operator is used to extract the edge information of the depth map, forming a preliminary geometric understanding and providing richer input for subsequent multimodal models. The gradient distribution and edge characteristics of the depth map are calculated, and the Sobel operator is used to extract the edge:

[0060] g x =d*S x , g y =d*S y ,

[0061] Among them, S x 、S y Represents the Sobel kernel; * represents the convolution operation; g represents the gradient magnitude; d represents the input depth map data; g x Represents the gradient of the depth map in the horizontal direction (x direction); g y Represents the gradient of the depth map in the vertical direction (y direction).

[0062] The depth map after the above operation is input into the ViT network to extract high-dimensional image features F img :

[0063] F img =ViT(D norm )

[0064] Among them, D norm Represents the processed depth map.

[0065] The ViT module captures the local and global geometric properties of the depth map through a multi-layer stacked self-attention mechanism, thereby generating fine-grained geometric feature representations.

[0066] At the same time, text prompts are used to provide semantic guidance for subsequent point cloud data generation, such as "detecting the edges of complex surfaces" or "generating regular symmetrical shapes." The text prompts and the normalized depth image are processed by the multimodal large model (MLLM) to generate semantic features:

[0067] F text =LLM(Prompt)

[0068] The introduction of semantic features not only enriches the descriptive capabilities of depth maps, but also compensates for the shortcomings of geometric features in expressing the semantic information of objects. For example, for blurred or occluded areas in the depth map, text hints can generate corresponding semantic clues.

[0069] After the preprocessing stage, the image feature F of the depth map img and semantic features F text They are extracted independently, laying the foundation for subsequent feature fusion.

[0070] S2. Fuse the extracted depth map features to generate fused features.

[0071] like Figure 2 As shown in the figure, the multimodal Transformer module is used to efficiently integrate image features and semantic features, and deep fusion of multimodal information is achieved through dynamic weight adjustment.

[0072] First, the image feature F of the depth map imgand semantic features F text Embedded into a unified feature space through linear transformation:

[0073] Q,K,V=Linear(F img ),Linear(F text )

[0074] Where Q, K, and V are the query, key, and value matrices in the multimodal Transformer, respectively; Linear represents a linear transformation.

[0075] The self-attention mechanism calculates the attention weight using the following formula:

[0076]

[0077] Among them, A represents the attention weight matrix, d k is the dimension of the feature vector, Softmax is used to normalize the weights to ensure numerical stability; T represents the matrix transpose operation.

[0078] By multiplying the attention weight matrix with the value matrix V, the fused feature representation F can be generated fusion :

[0079] F fusion =A·V

[0080] The fused feature combines the geometric information in the depth map with the semantic information in the text prompt. For example, when the text prompt says "pay attention to the boundary features of the curved area", the fused feature can significantly highlight the details of the curved boundary in the depth map.

[0081] In order to further improve the expressive power of fusion features, a dynamic weighting mechanism is introduced to adjust the features of different regions:

[0082] F dynamic =softmax(W·F fusion )·F fusion

[0083] Among them, W is a learnable weighted parameter matrix, which can significantly improve the local expressiveness of the fusion feature by dynamically adjusting the weight of each feature area; F dynamic Represents the final fusion feature.

[0084] S3. Generate the final point cloud data based on the fused features.

[0085] like Figure 3 As shown in Figure 1, a generative adversarial network (GAN) is used to map fusion features to 3D point cloud data. This stage generates high-precision point cloud data through adversarial training of the generator and the discriminator.

[0086] Fusion feature F dynamic It is input to the generator G and combined with the random noise vector Z~N(0,1) to generate the initial point cloud data P gen :

[0087] P gen =G(F dynamic , Z)

[0088] In order to improve the authenticity of point cloud data, the discriminator D generates point cloud P gen and the real point cloud P real For classification. The objective function of the discriminator is L D :

[0089]

[0090] in, and They represent the expected logarithmic value of the discriminator output of the real point cloud data and the logarithmic expectation of the probability that the generated point cloud data is misclassified as real data.

[0091] The generator optimizes the point cloud generation quality by minimizing the following adversarial loss:

[0092]

[0093] Among them, L G Represents the objective function during generator training.

[0094] The generated initial point cloud is optimized through the post-processing module, including adjusting the distribution density of points, repairing discontinuities in the geometric structure, and smoothing edge areas. The position p of each point in the optimized point cloud is i Updated by the following formula:

[0095]

[0096] Among them, p′ i represents the updated position; α represents the optimization step size; L smooth represents the point cloud smoothing loss function.

[0097] The resulting point cloud data is visualized as a 3D model and applied in the field of autonomous driving.

[0098] Example 2

[0099] As a feasible implementation, after completing the prompt in step S2, the large model can be used to verify the prompt's understanding of the scene, generating a corresponding preliminary scene description. This is then compared and analyzed with the original depth map content to assess the accuracy and comprehensiveness of the prompt. If the description is found to be insufficient, the prompt is iteratively optimized to ensure that the generated scene description is consistent with the actual situation.

[0100] The depth map data and the optimized Prompt are input into the multimodal large model to extract the significant features in the scene (such as edges, instance objects, etc.) and generate structured description information, including target location, boundary information and mutual relationships.

[0101] Using the context enhancement module (Con text The Transformer Encoding Layer builds a dependency graph between instances, providing stronger support for semantic feature extraction.

[0102] The description information generated by the multimodal large model is input into the semantic feature extraction network to extract the key semantic features of the scene. The extraction process combines the saliency distribution in the scene and the instance semantic embedding to form a complete semantic feature representation F text .

[0103] The image feature F of the depth map img and semantic features F text Concatenate along the channel dimension:

[0104]

[0105] Among them, F fusion Represents the fused features; Represents the representative operation on the channel. The multimodal Transformer model is used to fuse features. During the fusion process, the multi-head attention mechanism is used to calculate the matching relationship between the image and the semantics, generating more expressive fusion features. The degree of matching between the two is calculated using the following attention score calculation formula:

[0106]

[0107] Among them, A represents the attention weight matrix, d k is the dimension of the feature vector, Softmax is used to normalize the weights to ensure numerical stability; T represents the matrix transpose operation.

[0108] Adjust the fused features to give higher weights to specific areas. For example, calculate the attention distribution of the fused features and reweight them:

[0109] F dynamic =softmax(W·F fusion )·F fusion

[0110] Among them, W is a learnable weighted parameter matrix, which can significantly improve the local expressiveness of the fusion feature by dynamically adjusting the weight of each feature area; F dynamic Represents the final fusion feature.

[0111] After step S3 is generated, the point cloud data can also be post-processed to optimize its distribution and quality. The local density of the point cloud is adjusted by the K-neighbor method:

[0112]

[0113] Among them, N i For point p i The field of i It is a measure of local density. Points in low-density areas are resampled, using downsampling techniques to optimize the point cloud representation. The geometric characteristics of the point cloud are analyzed, including the calculation of curvature information, which is used to describe the shape characteristics of the point cloud surface. This post-processing process can significantly improve the clarity and physical meaning of the point cloud, providing more reliable data support for subsequent applications.

[0114] Visualize and evaluate point cloud data. After generating point cloud data, evaluate its quality, including its realism and diversity. Use point cloud rendering tools to create 3D renderings of the point cloud data, showcasing its shape and spatial distribution. These evaluation methods comprehensively reflect the quality of point cloud data generation, ensuring that the generated results meet the needs of practical applications.

[0115] Example 3

[0116] This embodiment also provides a construction system for depth map to point cloud data based on a large model, including: a preprocessing module, a fusion module and a generation module; the preprocessing module is used to use the pre-trained large model to preprocess the input depth map and extract the features of the depth map; the fusion module is used to fuse the extracted depth map features to generate fused features; the generation module is used to generate the final point cloud data based on the fused features.

[0117] The following will describe in detail how the present invention solves technical problems in real life in conjunction with this embodiment.

[0118] First, the pre-training module uses the pre-trained large model to pre-process the input depth map and extract the features of the depth map.

[0119] In this embodiment, the depth map obtained is assumed to be in an autonomous driving scenario. A pre-trained large model is used to extract global features from the input depth map. The spatial structural features and local detail features in the depth map are mined through a deep network structure to generate a multi-scale feature representation. Specifically, the pre-trained large model includes a convolutional neural network (CNN), a Transformer structure, or a combination of the two, which is used to simultaneously extract global and local features of the depth map.

[0120] like Figure 1 As shown, first, the geometric and semantic characteristics of the input data are optimized through normalization, denoising and completion operations.

[0121] For the input depth map, the pixel values ​​are first normalized to adjust the depth values ​​to the range of [0, 1]. The formula is:

[0122]

[0123] Among them, d i，j Indicates the depth value; d′ i，j represents the normalized value; d max and d min are the maximum and minimum values ​​of the depth map, respectively.

[0124] Subsequently, the depth map is denoised using the non-local means method (NLM), which is formulated as follows:

[0125]

[0126] Among them, d denosid (i) represents the denoised depth map; w(i, j) represents the similarity weight between pixels i and j; Ω represents the neighborhood window of pixel i; h represents the smoothing control parameter; d(i) and d(j) represent the depth values ​​of pixels i and j in the depth map, respectively, representing the physical distance from the source to the pixel. This approach effectively eliminates outliers in the depth map caused by sensor noise.

[0127] Based on preprocessing, the depth map data is structured and analyzed, including the calculation of the gradient distribution and edge characteristics of the data. The Sobel operator or Laplacian operator is used to extract the edge information of the depth map, forming a preliminary geometric understanding and providing richer input for subsequent multimodal models. The gradient distribution and edge characteristics of the depth map are calculated, and the Sobel operator is used to extract the edge:

[0128] g x =d*S x , g y =d*S y ,

[0129] Among them, S x 、S y Represents the Sobel kernel; * represents the convolution operation; g represents the gradient magnitude; d represents the input depth map data; g x Represents the gradient of the depth map in the horizontal direction (x direction); g y Represents the gradient of the depth map in the vertical direction (y direction).

[0130] The depth map after the above operation is input into the ViT network to extract high-dimensional image features F img :

[0131] F img =ViT(D norm )

[0132] Among them, D norm Represents the processed depth map.

[0133] The ViT module captures the local and global geometric properties of the depth map through a multi-layer stacked self-attention mechanism, thereby generating fine-grained geometric feature representations.

[0134] At the same time, text prompts are used to provide semantic guidance for subsequent point cloud data generation, such as "detecting the edges of complex surfaces" or "generating regular symmetrical shapes." The text prompts and the normalized depth image are processed by the multimodal large model (MLLM) to generate semantic features:

[0135] F text =LLM(Prompt)

[0136] The introduction of semantic features not only enriches the descriptive capabilities of depth maps, but also compensates for the shortcomings of geometric features in expressing the semantic information of objects. For example, for blurred or occluded areas in the depth map, text hints can generate corresponding semantic clues.

[0137] After the preprocessing stage, the image feature F of the depth map img and semantic features F text They are extracted independently, laying the foundation for subsequent feature fusion.

[0138] The fusion module then fuses the extracted depth map features to generate fused features.

[0139] like Figure 2 As shown in the figure, the multimodal Transformer module is used to efficiently integrate image features and semantic features, and deep fusion of multimodal information is achieved through dynamic weight adjustment.

[0140] First, the image feature F of the depth mapimg and semantic features F text Embedded into a unified feature space through linear transformation:

[0141] Q,K,V=Linear(F img ),Linear(F text )

[0142] Where Q, K, and V are the query, key, and value matrices in the multimodal Transformer, respectively; Linear represents a linear transformation.

[0143] The self-attention mechanism calculates the attention weight using the following formula:

[0144]

[0145] Among them, A represents the attention weight matrix, d k is the dimension of the feature vector, Softmax is used to normalize the weights to ensure numerical stability; T represents the matrix transpose operation.

[0146] By multiplying the attention weight matrix with the value matrix V, the fused feature representation F can be generated fusion :

[0147] F fusion =A·V

[0148] The fused feature combines the geometric information in the depth map with the semantic information in the text prompt. For example, when the text prompt says "pay attention to the boundary features of the curved area", the fused feature can significantly highlight the details of the curved boundary in the depth map.

[0149] In order to further improve the expressive power of fusion features, a dynamic weighting mechanism is introduced to adjust the features of different regions:

[0150] F dynamic =softmax(W·F fusion )·F fusion

[0151] Among them, W is a learnable weighted parameter matrix, which can significantly improve the local expressiveness of the fusion feature by dynamically adjusting the weight of each feature area; F dynamic Represents the final fusion feature.

[0152] The final generation module generates the final point cloud data based on the fused features.

[0153] A generative adversarial network (GAN) is used to map the fused features to 3D point cloud data. This stage generates high-precision point cloud data through adversarial training of the generator and the discriminator.

[0154] Fusion feature F dynamic It is input to the generator G and combined with the random noise vector Z~N(0,1) to generate the initial point cloud data P gen :

[0155] P gen =G(F dynamic , Z)

[0156] In order to improve the authenticity of point cloud data, the discriminator D generates point cloud P gen and the real point cloud P real For classification. The objective function of the discriminator is L D :

[0157]

[0158] in, and They represent the expected logarithmic value of the discriminator output of the real point cloud data and the logarithmic expectation of the probability that the generated point cloud data is misclassified as real data.

[0159] The generator optimizes the point cloud generation quality by minimizing the following adversarial loss:

[0160]

[0161] Among them, L G Represents the objective function during generator training.

[0162] The generated initial point cloud is optimized through the post-processing module, including adjusting the distribution density of points, repairing discontinuities in the geometric structure, and smoothing edge areas. The position p of each point in the optimized point cloud is i Updated by the following formula:

[0163]

[0164] Among them, p′ i represents the updated position; α represents the optimization step size; L smooth represents the point cloud smoothing loss function.

[0165] The resulting point cloud data is visualized as a 3D model and applied in the field of autonomous driving.

[0166] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for constructing a depth map to point cloud data based on a large model, characterized in that the steps include: Use the pre-trained large model to pre-process the input depth map and extract the features of the depth map; Fuse the extracted depth map features to generate fused features; Based on the fused features, the final point cloud data is generated.

2. The method for constructing a depth map to point cloud data based on a large model according to claim 1, characterized in that: The steps of performing the pretreatment include: First, the pixel values ​​of the input depth map are normalized: Afterwards, the depth map is denoised using the non-local means method: Finally, it is input into the ViT network to extract high-dimensional image features F img : F img =ViT(D norm ) Among them, D norm Represents the processed depth map; At the same time, the semantic feature F is generated by combining the processed depth image with the text prompt text .

3. The method for constructing a depth map to point cloud data based on a large model according to claim 2, characterized in that: The methods for normalization include: Among them, d i，j Indicates the depth value; d′ i，j represents the normalized value; d max and d min are the maximum and minimum values ​​of the depth map, respectively.

4. The method for constructing a depth map to point cloud data based on a large model according to claim 2, characterized in that: The steps to noise reduction include: Among them, d denosied (i) represents the depth map after denoising; w(i, j) represents the similarity weight between pixel i and pixel j; Ω represents the neighborhood window of pixel i; h represents the smoothing control parameter; d(i) and d(j) represent the depth values ​​of pixel i and pixel j in the depth map, respectively.

5. The method for constructing a depth map to point cloud data based on a large model according to claim 1, characterized in that: The method for generating the fused features includes: transforming the image features F img and semantic features F text Embedded into a unified feature space: Q,K,V=Linear(F img ),Linear(F text ) Where Q, K, and V are the query, key, and value matrices in the multimodal Transformer, respectively; Linear represents a linear transformation; By multiplying the attention weight matrix with the value matrix V, the fused feature representation F can be generated fusion : F fusion =A·V Among them, A represents the attention weight matrix.

6. The method for constructing a depth map to point cloud data based on a large model according to claim 1, characterized in that: The method for generating the point cloud data includes: using a generative adversarial network to realize the mapping of fusion features to three-dimensional point cloud data; and generating high-precision point cloud data through adversarial training of a generator and a discriminator.

7. A system for constructing point cloud data from a depth map based on a large model, the system being used to implement the method according to any one of claims 1 to 6, characterized in that: include: Preprocessing module, fusion module and generation module; The preprocessing module is used to preprocess the input depth map using a pre-trained large model to extract features of the depth map; The fusion module is used to fuse the extracted depth map features to generate fused features; The generation module is used to generate final point cloud data based on the fused features.

8. The system for constructing a large model-based depth map to point cloud data according to claim 7, characterized in that: The workflow of the preprocessing module includes: First, the pixel values ​​of the input depth map are normalized: Afterwards, the depth map is denoised using the non-local means method: Finally, it is input into the ViT network to extract high-dimensional image features F img : F img =ViT(D norm ) Among them, D norm Represents the processed depth map; At the same time, the semantic feature F is generated by combining the processed depth image with the text prompt text .

Images