Point cloud processing method and electronic device

Abstract

The application discloses a point cloud processing method and an electronic device, and relates to the technical field of computers. In the method, an input point cloud is input into a pre-trained language model for inference to obtain an output point cloud corresponding to the input point cloud and reconstructed through semantic completion and densification based on the pre-trained language model. Since a prior knowledge fusion mechanism based on the pre-trained language model is introduced in the point cloud completion process, the model can be used to deeply understand and infer the general knowledge of three-dimensional objects and scenes, so that the reasonable geometric structure and semantic information of the occluded or unswept area can be accurately inferred and completed based on the sparse input point cloud, and the technical problems of insufficient point cloud completion precision and low reliability of the completion result caused by the lack of complete prior knowledge of objects in the related art are solved, and the accuracy, completeness and semantic richness of scene completion are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a point cloud processing method and electronic device. Background Technology

[0002] With the widespread adoption of applications such as robot navigation, autonomous driving, and augmented reality, the need for complete and detailed semantic understanding of 3D scenes is becoming increasingly urgent. Related technologies typically employ methods based on 3D convolutional neural networks or point cloud deep learning networks to directly complete the geometric structure and predict semantic labels from sparse point cloud data.

[0003] However, the accuracy of existing methods is still insufficient. When dealing with large-scale complex scenes, severe occlusion, or novel objects, their ability to understand the complete geometric priors of 3D objects and the semantic context of complex scenes is limited, which seriously affects the accuracy, completeness, and generalization performance of scene completion. Summary of the Invention

[0004] This application provides a point cloud processing method and electronic device to at least solve the problems of insufficient accuracy of sparse point cloud completion, lack of geometric prior knowledge, and limited semantic understanding ability in complex scenes in related technologies.

[0005] This application provides a point cloud processing method, including:

[0006] Obtain the input point cloud;

[0007] The input point cloud is fed into a pre-trained language model to perform semantic completion based on the pre-trained language model, obtaining the corresponding output point cloud. The point cloud density of the output point cloud is higher than that of the input point cloud. The pre-trained language model is used to perform the following operations:

[0008] Encode the input point cloud to obtain the point cloud features corresponding to the input point cloud;

[0009] Based on prior knowledge, feature completion is performed on the point cloud features to obtain the prior features corresponding to the input point cloud.

[0010] Decode the point cloud features and prior features to obtain the panoramic semantic point cloud corresponding to the input point cloud;

[0011] The panoramic semantic point cloud is extended to obtain the output point cloud corresponding to the input point cloud.

[0012] This application also provides a video behavior positioning device, including:

[0013] The acquisition module is used to acquire the input point cloud;

[0014] The processing module is used to input the input point cloud into a pre-trained language model to perform semantic completion on the input point cloud based on the pre-trained language model, and obtain the corresponding output point cloud. The point cloud density of the output point cloud is higher than that of the input point cloud. The pre-trained language model is used to perform the following operations:

[0015] Encode the input point cloud to obtain the point cloud features corresponding to the input point cloud;

[0016] Based on prior knowledge, feature completion is performed on the point cloud features to obtain the prior features corresponding to the input point cloud.

[0017] Decode the point cloud features and prior features to obtain the panoramic semantic point cloud corresponding to the input point cloud;

[0018] The panoramic semantic point cloud is extended to obtain the output point cloud corresponding to the input point cloud.

[0019] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the point cloud processing methods described above.

[0020] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the point cloud processing methods described above.

[0021] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the point cloud processing methods described above.

[0022] This application introduces a prior knowledge fusion mechanism based on a pre-trained language model during point cloud completion. This mechanism leverages the model's deep understanding and reasoning of general knowledge about 3D objects and scenes, enabling accurate inference and completion of reasonable geometric structures and semantic information for occluded or unscanned regions based on sparse input point clouds. This addresses the technical problems of insufficient point cloud completion accuracy and low reliability due to a lack of complete prior knowledge of objects in related technologies, thus improving the accuracy of scene completion. Furthermore, this scheme integrates point cloud encoding, prior knowledge injection, feature fusion, and dense generation into a coherent reasoning process using a unified pre-trained language model framework. This avoids the feature inconsistencies and error accumulation problems caused by multi-stage independent optimization in traditional methods. While ensuring the accuracy of 3D geometric reconstruction, it assigns precise semantic category labels to each point, thereby improving system completion accuracy while ensuring fine-grained and reliable semantic segmentation. Ultimately, this achieves efficient and high-precision 3D semantic scene reconstruction in complex and severely occluded scenes. Attached Figure Description

[0023] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram of a point cloud processing process provided in an embodiment of this application;

[0025] Figure 2 A flowchart illustrating the point cloud processing method provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram illustrating the principle of the pre-trained language model provided in the embodiments of this application;

[0027] Figure 4 A flowchart illustrating the point cloud feature generation process provided in this application embodiment;

[0028] Figure 5 A schematic diagram illustrating the principle of the point cloud feature generation process provided in the embodiments of this application;

[0029] Figure 6 A flowchart illustrating the prior feature generation process provided in this application embodiment;

[0030] Figure 7 A schematic diagram illustrating the principle of the prior feature generation process provided in this application embodiment;

[0031] Figure 8 The schematic diagram of the block division process provided in the embodiments of this application;

[0032] Figure 9 A flowchart illustrating the panoramic semantic point cloud generation process provided in this application embodiment;

[0033] Figure 10 This application provides a schematic diagram illustrating the principle of panoramic semantic point cloud generation process in its embodiments.

[0034] Figure 11 A schematic diagram illustrating the principle of the point cloud voxel heterogeneous map construction process provided in this application embodiment;

[0035] Figure 12 This application provides a flowchart illustrating the output point cloud generation process in its embodiments;

[0036] Figure 13 This application provides a schematic diagram illustrating the principle of the output point cloud generation process in its embodiments;

[0037] Figure 14 A flowchart illustrating the model training process provided in this application embodiment;

[0038] Figure 15 This is a schematic diagram of the point cloud processing device provided in the embodiments of this application;

[0039] Figure 16 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0041] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0042] First, let me explain the terms used in this application:

[0043] Point cloud is a three-dimensional spatial data representation consisting of a large number of discrete points. Each point contains three-dimensional coordinates (x, y, z) and may also include attributes such as color, normal vector, and reflection intensity.

[0044] Large Language Model (LLM / Llama);

[0045] A multi-layer perceptron (MLP) is an artificial neural network consisting of an input layer, at least one hidden layer, and an output layer. It is the most basic and commonly used deep learning model structure, used for feature transformation and nonlinear function fitting.

[0046] Attention architecture (Transformer), a deep learning model architecture mainly based on self-attention mechanism;

[0047] A Convolutional Neural Network (CNN) is an artificial neural network specifically designed to process data with a grid structure (such as images, videos, and voxels). Its core principle is to extract local features by sliding convolutional kernels across the input data and to achieve feature dimensionality reduction and translation invariance using pooling operations.

[0048] PointNet is a deep learning network architecture that directly processes unordered point cloud data. Its core idea is to extract the independent features of each point and aggregate them into global features through a shared multilayer perceptron and symmetric functions (such as max pooling), thereby achieving tasks such as point cloud classification and segmentation.

[0049] PointNet++ is an improved network based on PointNet, introducing a hierarchical feature learning mechanism. It constructs a hierarchical structure by recursively applying farthest point sampling and grouping operations on the point cloud, and extracts local features using PointNet at different scales, thereby better capturing the local geometric structure and multi-scale contextual information of the point cloud.

[0050] A voxel, analogous to a pixel in three-dimensional space, represents a cubic unit in a regular three-dimensional mesh. Voxelization is the process of discretizing a continuous three-dimensional space (such as a point cloud) into a regular mesh, where each voxel can store attributes such as density and color.

[0051] Leaky-ReLU (Leaky Rectified Linear Unit) is an improved linear rectified activation function. Leaky-ReLU retains a small gradient in the negative interval, which helps alleviate the problem of neuron "death" and improves the training stability of deep networks.

[0052] Red-Green-Blue (RGB) is a color model that uses the intensity values ​​of the red, green, and blue color channels (usually 0-255) to represent colors. It is the most basic and universal color representation format in digital images, computer graphics, and visual data.

[0053] In embodied artificial intelligence scenarios such as robot navigation, autonomous driving, and industrial automation, semantic scene completion (SSC) of 3D point cloud data is a key technology for achieving environmental perception and decision-making. For example, in the field of autonomous driving, LiDAR scans the environment by emitting laser beams, generating sparse and partial point cloud data. However, this data often cannot fully describe complex scenes (such as occluded areas or missing parts of dynamic objects). In this case, the system needs to use semantic scene completion technology to reconstruct the sparse point cloud into a dense 3D semantic point cloud, thereby helping the autonomous driving system accurately identify key targets such as roads, vehicles, pedestrians, and traffic signs, and predict the distribution of potential objects in unscanned areas. Similarly, in industrial robot grasping tasks, the robot needs to understand the complete shape of the workpiece (such as identifying the back or internal structure of the workpiece) through point cloud data in order to plan the optimal grasping path. However, the sparse point clouds generated by existing technologies have problems such as missing information and semantic ambiguity, which limits the reliability of subsequent tasks (such as navigation and obstacle avoidance, object recognition). Furthermore, point cloud semantic completion technology is widely used in fields such as smart homes, virtual reality (VR), and augmented reality (AR) for scenarios like 3D scene reconstruction and digital twin modeling. For example, in AR glasses, the system needs to use point cloud data to complete occluded objects (such as furniture behind a wall) in the user's field of vision in real time to provide a more immersive interactive experience. Therefore, how to solve the problem of missing information in sparse point clouds through efficient and high-precision point cloud semantic scene completion technology, and to give the completion results clear semantic labels (such as "car," "person," "tree," etc.), has become one of the core challenges for embodied intelligent systems to achieve environmental perception and decision-making.

[0054] In summary, the point cloud processing methods in related technologies mainly suffer from the following technical bottlenecks:

[0055] The relevant technologies mainly rely on traditional point cloud processing algorithms or deep learning-based single-modal models to achieve semantic scene completion. For example, traditional methods use techniques such as geometric interpolation and voxel mesh filling to densify sparse point clouds. However, these methods can only restore the geometric shape of the point cloud and cannot assign semantic labels to the completed results. Furthermore, their completion effect is poor for complex scenes (such as occluded areas). Existing deep learning-based technologies attempt to extract point cloud features using convolutional neural networks (CNNs) or point cloud-specific networks (such as PointNet and PointNet++) and assign category labels to the completed point cloud using semantic segmentation models. However, these methods have the following limitations:

[0056] Lack of prior knowledge: The model relies solely on the local geometric features of the input point cloud for completion, and cannot utilize semantic knowledge from natural scenes (such as "vehicles usually have symmetrical structures" and "the morphological distribution patterns of trees") to assist in reasoning, resulting in deviations in semantic consistency of the completion results.

[0057] Insufficient multimodal fusion: It fails to effectively integrate external knowledge sources such as language models, making it difficult to handle cross-modal information (such as the semantic relationship between text descriptions and 3D point clouds), thus limiting the model's ability to understand complex scenes.

[0058] Limited densification effect: In the process of point cloud densification, simple copying or interpolation strategies are usually used, resulting in insufficient density or unnatural distribution of the completed point cloud, which affects the accuracy of subsequent tasks.

[0059] Trade-off between computational efficiency and accuracy: To reduce computational complexity, point clouds are often downsampled or voxelized, but such operations can lead to information loss and further reduce completion accuracy.

[0060] This application proposes a point cloud processing method and an electronic device. The method includes: acquiring an input point cloud; inputting the input point cloud into a pre-trained language model for inference, so as to acquire an output point cloud corresponding to the input point cloud based on the pre-trained language model, wherein the point cloud density of the output point cloud is higher than that of the input point cloud.

[0061] In this embodiment, a prior knowledge fusion mechanism based on a pre-trained language model is introduced during point cloud completion. This mechanism leverages the model's deep understanding and reasoning of general knowledge about 3D objects and scenes, enabling accurate inference and completion of reasonable geometric structures and semantic information for occluded or unscanned areas based on sparse input point clouds. This addresses the technical problems of insufficient point cloud completion accuracy and low reliability of completion results due to a lack of complete prior knowledge of objects in related technologies, thereby improving the accuracy of scene completion. Furthermore, this scheme integrates point cloud encoding, prior knowledge injection, feature fusion, and dense generation into a coherent reasoning process through a unified pre-trained language model framework. This avoids the feature inconsistency and error accumulation problems caused by multi-stage independent optimization in traditional methods. While ensuring the accuracy of 3D geometric structure reconstruction, it assigns precise semantic category labels to each point, thereby improving the system's completion accuracy while ensuring fine-grained and reliable semantic segmentation. Ultimately, this achieves efficient and high-precision 3D semantic scene reconstruction in complex and severely occluded scenes.

[0062] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] This section describes the specific application environment architecture or hardware architecture upon which the point cloud processing method depends. (References) Figure 1 , Figure 1 This is a schematic diagram of a point cloud processing process provided in an embodiment of this application, such as... Figure 1 As shown, it includes input point cloud, electronic device and output point cloud.

[0064] In this embodiment, after acquiring the input point cloud, the electronic device performs the "point cloud semantic scene completion method" provided in this application on the input point cloud. Specifically, this includes: inputting the point cloud into a pre-trained language model, and performing inference on the input point cloud based on the pre-trained language model to obtain an output point cloud. Specifically, the pre-trained language model performs the following operations: encoding the input point cloud to extract point cloud features; completing and enhancing the features based on the prior knowledge embedded in the model; then decoding to generate a panoramic semantic point cloud with both complete geometric structure and semantic information; finally, densifying and expanding the semantic point cloud to output a point cloud that is significantly superior to the input point cloud in terms of both the number of points and the completeness of spatial coverage.

[0065] In some embodiments, the electronic device can be a terminal or a server. For example, the terminal can be a personal digital assistant (PDA) device, a handheld device with wireless communication capabilities (such as a smartphone or tablet), a computing device (such as a personal computer (PC)), an in-vehicle device, a wearable device (such as a smartwatch or smart bracelet), a smart home device (such as a smart display device), etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic services such as cloud computing and cloud storage.

[0066] Below, in conjunction with the above Figure 1 The application scenarios illustrated are used to provide a detailed explanation of the technical solutions of this application through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0067] Figure 2 This is a flowchart illustrating the point cloud processing method provided in an embodiment of this application. Specifically, an embodiment of this application provides a video behavior localization method, which is executed by an electronic device. Figure 2 The method is described in detail below:

[0068] S201: Obtain the input point cloud.

[0069] The input point cloud can be a sparse set of points representing a 3D scene acquired by any acquisition device. For example, the input point cloud can come from various sources such as LiDAR scanning, depth camera reconstruction, and multi-view stereo vision. The input point cloud typically contains the 3D spatial coordinates of a series of points, and may further include optical attribute information such as the color and reflectance intensity of each point.

[0070] S202: Input the input point cloud into a pre-trained language model to perform semantic completion on the input point cloud based on the pre-trained language model, and obtain the output point cloud corresponding to the input point cloud. The point cloud density of the output point cloud is higher than that of the input point cloud.

[0071] Compared to the input point cloud, the output point cloud significantly improves the spatial density of points and the completeness of scene spatial coverage, and includes semantic category labels for each point. For example, the original sparse LiDAR point cloud can only capture the front outline of a building, while the completed output point cloud can completely reconstruct the back and roof structure of the building, and label the points as semantic categories such as "wall", "window", and "roof".

[0072] In some embodiments, the pre-trained language model is a large-scale language model trained on large-scale 3D-text data and general corpus, which internally encodes prior knowledge of the geometric shape of common 3D objects (such as vehicles, pedestrians, buildings, trees, etc.) and the ability to understand scene semantics.

[0073] In some embodiments, the pre-trained language model is, for example, a large language model based on the Transformer architecture or a multimodal large model adapted for a 3D task.

[0074] Specifically, after obtaining the input point cloud, the electronic device inputs it into a pre-trained language model, which then performs the following steps S1-S4 to semantically complete the input point cloud, and finally outputs the output point cloud by the electronic device.

[0075] The pre-trained language model is used to perform the following operations:

[0076] S1. Encode the input point cloud to obtain the point cloud features corresponding to the input point cloud.

[0077] Specifically, the encoding process transforms the original disordered point cloud into a structured, multi-layered machine-readable feature representation, such as extracting its global geometric context and local detail information.

[0078] S2. Based on prior knowledge, feature completion is performed on the point cloud features to obtain the prior features corresponding to the input point cloud.

[0079] Specifically, this step utilizes the general 3D knowledge embedded in the pre-trained language model to infer and complete the feature regions missing in the input point cloud due to occlusion or scanning limitations, generating enhanced features rich in prior information about complete objects and scenes.

[0080] S3. Decode the point cloud features and prior features to obtain the panoramic semantic point cloud corresponding to the input point cloud.

[0081] Specifically, this step reconstructs a more complete 3D point set in space by fusing the original point cloud features with enhanced prior features, and predicts the semantic category to which each point belongs.

[0082] S4. Expand the panoramic semantic point cloud to obtain the output point cloud corresponding to the input point cloud.

[0083] Specifically, this step upsamples or densifies the completed semantic point cloud to further improve the spatial density and uniformity of its distribution, generating a high-quality dense semantic point cloud suitable for downstream tasks such as robot navigation and high-precision map construction.

[0084] In this embodiment, because a prior knowledge fusion mechanism based on a pre-trained language model is introduced in the point cloud completion process, the model can be used to deeply understand and reason about the general knowledge of 3D objects and scenes. This allows for accurate inference and completion of reasonable geometric structures and semantic information of occluded or unscanned areas based on sparse input point clouds. This solves the technical problems of insufficient point cloud completion accuracy and low reliability of completion results due to lack of complete prior knowledge of objects in related technologies, thereby improving the accuracy, completeness and semantic richness of scene completion.

[0085] Figure 3 This is a schematic diagram illustrating the principle of the pre-trained language model provided in an embodiment of this application. Figure 3 As shown, the pre-trained language model includes a point cloud encoding module, a prior information extraction module, a point cloud decoding module, and a point cloud dense super-resolution module.

[0086] First, the input point cloud (sparse, partial point cloud) is fed into the point cloud encoding module, which encodes the input point cloud to obtain the corresponding point cloud features. The point cloud encoding module converts the raw, unordered point cloud into a structured, machine-readable feature representation. Specifically, the point cloud encoding module is a preprocessing module used to process the point cloud into features for use by subsequent modules. The input to the point cloud encoding module is a point cloud of size [n, d] (where n represents the number of "points" in the point cloud; d represents the dimension used to describe the point data, typically d=6, i.e., the three-dimensional coordinates and RGB color values ​​of the point in space, which are then decomposed into pc and dc respectively). sp and PC opThis refers to the point cloud coordinate information and point cloud optical information (this embodiment only uses d=6 as an example). The output of the point cloud encoding module is of size [n, c tem ] coding sequence features feat tem and size [c vo The coding voxel features of [l,h,w] feature vo , where c tem and c vo Let l, h, and w represent the dimensions of the two features, respectively, where l, h, and w represent the lengths of the voxel features in three-dimensional space.

[0087] Next, the point cloud features are input into the prior information extraction module. This module, based on the general 3D knowledge embedded in the pre-trained language model, performs knowledge enhancement and completion on the point cloud features to obtain prior features. The purpose of the prior information extraction module is to rely on the pre-trained language model's knowledge reserves and reasoning ability regarding the complete shapes of common objects and scene layouts to inject reasonable prior information into the incomplete point cloud features, thereby supporting subsequent geometric and semantic completion. Specifically, the prior information extraction module can use LLM to complete the background knowledge for the point cloud 3D features (areas not scanned need to be determined using background knowledge to judge their true shape).

[0088] Next, the prior features and the original point cloud features are input together into the point cloud decoding module, which fuses and decodes the two types of features to extract the panoramic semantic point cloud. The point cloud decoding module is used to convert the voxel-based features (the output of the previous module, with a size of [c]) into voxel-based features. vo Prior features of [l,h,w] vp ) and the original features in point cloud form (output of the first module, size [n, c) tem ] coding sequence features feat tem By combining these features, point cloud-level decoding is achieved, which fills in missing point cloud information and assigns values ​​to these points (which category they belong to, such as trees, people, vehicles, etc.). Based on the enhanced features, a more complete 3D point set in space is reconstructed, and the corresponding semantic category label for each point is predicted, realizing the transformation from "local observation" to "global semantic understanding".

[0089] Finally, the panoramic semantic point cloud is input into the point cloud dense super-resolution module. This module upsamples and densifies the completed semantic point cloud to obtain the final output point cloud. The point cloud dense super-resolution module expands the sparse point cloud into a dense point cloud for use by other tasks in the application, thereby improving the spatial density and uniformity of the points in the output point cloud and generating a high-quality, high-precision dense semantic point cloud to meet the high requirements of downstream tasks such as autonomous driving and precise robotic operations for 3D scene representation.

[0090] In summary, this solution, through the collaborative work of the four modules mentioned above, achieves end-to-end completion and reconstruction from sparse, partial input point clouds to dense, complete, and semantically rich output point clouds. By introducing a prior knowledge fusion mechanism based on a pre-trained language model during point cloud completion, the solution leverages the model's deep understanding and reasoning of general knowledge about 3D objects and scenes. This allows for accurate inference and completion of reasonable geometric structures and semantic information for occluded or unscanned areas based on sparse input point clouds. This addresses the technical problems of insufficient point cloud completion accuracy and low reliability of completion results due to a lack of complete prior knowledge of objects in related technologies, thereby improving the accuracy of scene completion.

[0091] Furthermore, this scheme integrates point cloud encoding, prior knowledge injection, feature fusion, and dense generation into a coherent reasoning process through a unified pre-trained language model framework. This avoids the feature inconsistency and error accumulation problems caused by multi-stage independent optimization in traditional methods. While ensuring the accuracy of 3D geometric structure reconstruction, it assigns precise semantic category labels to each point, thereby improving the system's completion accuracy while ensuring the fine-grainedness and reliability of semantic segmentation. This enables efficient and high-precision 3D semantic scene reconstruction in complex and severely occluded scenarios.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0093] Figure 4 A flowchart illustrating the point cloud feature generation process provided in this application embodiment; Figure 5 This is a schematic diagram illustrating the principle of the point cloud feature generation process provided in an embodiment of this application. Specifically, this application provides a point cloud feature generation process executed by an electronic device based on a pre-trained language model. Figure 4 , Figure 5 The point cloud feature generation process is described in detail below:

[0094] S301: Perform voxel encoding on the input point cloud to obtain the encoded voxel features corresponding to the input point cloud.

[0095] S3011: Based on the 3D coordinates of points in the input point cloud and the voxel mesh, obtain the initial voxel matrix corresponding to the input point cloud. The voxel mesh includes multiple voxel elements.

[0096] Specifically, S3011 includes the following steps:

[0097] First, based on the 3D coordinates of points in the input point cloud, the point cloud coordinate information is obtained. Specifically, the input to the point cloud encoding module is a point cloud of size [n, d] (where n represents the number of "points" in the point cloud; d represents the dimension used to describe the data of that point, typically d=6, i.e., the 3D coordinates and RGB color values ​​of the point in space, which are then decomposed and denoted as pc). sp and PC op This refers to the point cloud coordinate information and the point cloud optical information.

[0098] Secondly, the point cloud coordinate information is normalized to obtain normalized point cloud coordinate information; specifically, based on the formula... The normalized spatial coordinate range (a 3-element array, denoted as rc) is calculated and then based on the formula... The normalized point cloud coordinate information is calculated.

[0099] Furthermore, a voxel grid of a first preset size is constructed, which includes multiple voxel units. Finally, based on the normalized point cloud coordinate information, the nearest point cloud of the voxel unit is found in the input point cloud. If the distance between the nearest point cloud and the voxel unit is less than a preset value, the optical information of the nearest point cloud is assigned to the voxel unit to obtain the initial voxel matrix corresponding to the input point cloud.

[0100] Specifically, the lengths l, h, and w of the voxel features in the three dimensions are determined according to the preset scaling ratio port, using the formula l, h, w = rc × port, and a zero matrix v of size [3, l, h, w] is constructed. init Then, traverse the three-dimensional array in the matrix one step according to the last three dimensions, and assign values ​​to the three-dimensional array according to the following formula to obtain the initial voxel matrix:

[0101]

[0102] in,

[0103] in, The above formula indicates that for each 3D array in the voxel matrix (when traversing length, width, and height), if the coordinates of that position can find a sufficiently close point in the point cloud sequence, the optical information (usually RGB 3D data) of that point is assigned to it. In this example, it is assumed that the optical information is 3D RGB. If the optical information is other forms of data, such as HSV color information or high-dimensional spectral information, it is also applicable (v needs to be set to a different value). init The first dimension (dimension 3) can be initialized to the corresponding dimension, and the processed matrix v init This is called the initial voxel matrix.

[0104] S3012: Use a three-dimensional convolutional neural network to encode the initial voxel matrix to obtain the encoded voxel features.

[0105] The three-dimensional convolutional neural network is a stack of several convolutional operation layers, each of which includes a convolutional kernel with a size of [cc]. in ,cc out [3,3,3,l,h,w] learnable parameters of convolutional layers and Leaky-ReLU nonlinear layers, where, cc in This indicates the dimension of the input features of the convolutional layer, cc out This represents the output dimension. Specifically, cc represents the output dimension of the first convolutional layer. in The value is 3, and the output of the last convolutional layer is c. vo This is called the coding voxel feature, and the size of the coding voxel feature is [c vo [,l,h,w], denoted as feat vo .

[0106] S302: Perform point cloud sequence encoding on the input point cloud to obtain the intermediate sequence features corresponding to the input point cloud.

[0107] Specifically, S302 includes the following steps S3021-S3022:

[0108] S3021: Obtain the point cloud optical information of the input point cloud based on the color of the points in the input point cloud.

[0109] For details on how to obtain point cloud optical information, please refer to S3011.

[0110] S3022: Encode point cloud optical information based on multilayer perceptron to obtain intermediate sequence features.

[0111] Specifically, a multilayer perceptron is used to encode the point cloud optical information, and the encoding formula is as follows: The input to the multilayer perceptron is point cloud optical information (pc). op W op1 This indicates a size of [3, c opmid The learnable weight matrix, b op1 This indicates a size of [1, c opmid The learnable bias vector of W; op2 This indicates a size of [c] opmid The learnable weight matrix of [,-3], b op2 This indicates a size of [1, c tem The learnable bias vector of [-3], c opmid c represents a preset intermediate dimension. tem Corresponding to the dimension of the encoded sequence features, the output of this step is the intermediate sequence features, whose dimension is [n, c].tem -3], denoted as feat tem0 .

[0112] S303: Perform sequence cross-coding on the encoded voxel features and intermediate sequence features to obtain the encoded sequence features corresponding to the input point cloud.

[0113] It is understandable that spatial voxel features, which are based on convolutional network encoding, can obtain more global information, but intermediate sequence features are only based on disordered optical information encoding and are difficult to encode effective spatial information. Therefore, in this application, they are interacted with encoded voxel features with spatial information to obtain encoded sequence features.

[0114] Specifically, S303 includes the following steps S3031-S3035:

[0115] S3031: Perform a linear transformation on the encoded voxel features to obtain the transformed encoded voxel features.

[0116] Specifically, the size is [c vo The encoded voxel features of [l,h,w] are input into the perceptual layer, based on the formula Perform a linear transformation on the encoded voxel features to obtain the transformed encoded voxel features, denoted as feat. vom W tvo For size [c vo ,c tem The learnable weight matrix is ​​[-3].

[0117] S3032: Based on the transformed coded voxel features and the position coding matrix, voxel features carrying absolute position information are obtained.

[0118] S3033: Construct a positional relevance mask, which is used to indicate point-voxel interaction information.

[0119] First, based on the formula:

[0120] , construct size [c tem -3,l,h,w] Position encoding matrix pos vom Among them, feat vomc= feat vom +pos vom .

[0121] Next, construct a zero matrix mask. itm The value is [n, l, h, w], and it is assigned a value according to the following formula:

[0122]

[0123] in,

[0124] in, This indicates the preset interaction distance threshold, pc spn This represents the normalized point cloud coordinate information obtained from the above steps. Next, the feat... vomc Reconstructed to size [c tem The form of -3,l,h,w] is denoted as feat vomp ;mask itm Reconstructed into a form of size [n, l, h, w], denoted as the position-related mask. itm .

[0125] S3034: Perform attention calculations on the position-related mask and the coding voxel features to obtain the intermediate features of the coding sequence.

[0126] Specifically, the intermediate features of the encoded sequence are obtained based on the following formula:

[0127]

[0128] Among them, W qit W kit W vit All are of size [c tem -3,c tem The learnable transition weight matrix is ​​[n, c], and the formula output is a matrix of size [n, c]. tem -3] intermediate features of the encoded sequence feature temm .

[0129] S3035: Merge the intermediate features of the encoded sequence with the point cloud coordinate information of the input point cloud to obtain the encoded sequence features corresponding to the input point cloud.

[0130] Specifically, the intermediate features of the encoded sequence are featured temm With point cloud coordinate information pc of size [n,3] sp Merging them yields a result of size [n, c] tem ] coding sequence features feat tem .

[0131] In this embodiment, voxel encoding and point cloud sequence encoding are performed in parallel on the input point cloud to generate two complementary feature representations in terms of data structure and information dimension: encoded voxel features (regularized global features with strong spatial inductive bias) and intermediate sequence features (features that retain point-level resolution and original optical information). This dual-path encoding mechanism provides both structured spatial context and fine-grained point-level attributes for subsequent processing, overcoming the information limitations of a single representation method.

[0132] Secondly, this mechanism establishes geometric distance-based connections between points and voxel units by introducing position-related masks, and dynamically and selectively injects voxel features rich in global spatial information into each point sequence feature using masked cross-attention computation. This process effectively solves the key problem of missing spatial information in pure point sequence features, enabling each point to not only carry its own optical properties but also integrate structured contextual information from its local three-dimensional neighborhood, thereby significantly enhancing the discriminative power and robustness of the features.

[0133] Finally, the generated encoded sequence features integrate the original coordinate information of the points, enhanced optical / semantic information, and local spatial structure information extracted from voxel features. This unified feature representation, possessing both precise geometric location and rich contextual semantics, provides a powerful and reliable feature foundation for downstream prior knowledge fusion, geometric completion, and semantic annotation tasks, and is an important guarantee for improving the accuracy of the entire point cloud semantic scene completion system.

[0134] In some embodiments, point cloud features include encoded voxel features. Figure 6 A flowchart illustrating the prior feature generation process provided in this application embodiment; Figure 7 This application provides a schematic diagram illustrating the principle of a priori feature generation process. Specifically, this application provides a priori feature generation process executed by an electronic device based on a pre-trained language model. For example... Figure 6 , Figure 7 The prior feature generation process is described in detail below:

[0135] S401: Divide the encoded voxel features into blocks according to the second preset size to obtain block voxel features.

[0136] For example, for a size of [c vo The coding voxel features of [l,h,w] feature vo The process involves segmenting the data into blocks (the purpose of which is dimensionality reduction, as inputting such a large number of four-dimensional features into a large model would result in an excessively large computational load). Specifically, S401 includes the following steps:

[0137] Figure 8 The schematic diagram of the block division process provided in the embodiments of this application is shown below. Figure 8 As shown, firstly, the encoded voxel features are... vo Divide into sizes [c] according to the length of gr. vo The characteristic matrix of [×gr×gr×gr,L,H,M], where gr represents the length of each block. Then, use the equation To process the features of each block, where W vow It is of size [c vo×gr×gr×gr,c vob The learnable weight matrix of size [c] is obtained, thus yielding a learnable weight matrix of size [c] vob Fragmented voxel features of [L,H,M] vob .

[0138] S402: Construct the block location embedding matrix, and obtain the encoded dimensionality-reduced voxel features based on the block voxel features and the block location embedding matrix.

[0139] In some embodiments, a size of [c] can be constructed. vob The block position embedding matrix pos of [L,H,M] vob Specifically, S402 includes the following steps S4021-S4023:

[0140] S4021: Construct a zero matrix of the third preset size.

[0141] Specifically, construct a band with [c vob A matrix of all zeros in the range L, H, M.

[0142] S4022: Assign values ​​to the all-zero matrix to obtain the block position embedding matrix.

[0143] Specifically, the zero matrix is ​​assigned values ​​using the following formula:

[0144] .

[0145] in, .

[0146] Where k is L, H, or W. L m H m W The integers must be distinct positive integers; 10000, 20000, and 50000 are recommended.

[0147] S4023: Merge the block voxel features and block position embedding matrix, and perform dimensional reconstruction and linear transformation on the merged matrix to obtain the encoded dimensionality-reduced voxel features.

[0148] Specifically, pos vob with feat vob Merge, refactor, and use a [c] vob The learnable weight matrix of [c] yields encoded dimensionality-reduced voxel features. volThe size is [c, L×H×M] (the last three dimensions are flattened, c is the transformed feature dimension, which corresponds to the common dimensions of popular pre-trained large language model encoders, such as 1024, 768, etc., depending on the large model used. Pre-trained large models can be T5-XXL, etc.).

[0149] S403: Based on the encoder, extract the prior information of the encoded dimensionality-reduced voxel features to obtain the dimensionality-reduced prior features.

[0150] Specifically, the encoded dimensionality reduction voxel features feature vol The prior information is extracted from the encoder of the frozen pre-trained language model to obtain dimensionality-reduced prior features of size [c, L×H×M]. (It should be understood that the pre-trained language model does not change the feature dimensions).

[0151] S404: Combine the block voxel features with the transformed dimensionality-reduced prior features to obtain the first prior intermediate feature, and perform a linear transformation on the first prior intermediate feature to obtain the second prior intermediate feature.

[0152] It should be noted that although the dimensionality reduction prior features encompass prior knowledge, due to the operation in the first step, the dimensionality reduction prior features are not features at the original scale. For semantic understanding tasks at the "point" level, the granularity is too coarse, so it is necessary to reconstruct them to the original length, width and height dimensions.

[0153] Specifically, first use a size of [c,c vob The learnable weight matrix W vow-r1 Convert it to size [c vob The matrix [L,H,M] is used, and the output block voxel features from the first step are combined. vob The summation yields the first prior intermediate feature.

[0154] S405: Reconstruct the prior intermediate second feature to a preset size to obtain the prior feature corresponding to the input point cloud.

[0155] Specifically, use a size of [c vob c vo The learnable weight matrix W of [×gr×gr×gr] vow-r2 Convert it to size [c vo The prior intermediate second feature of [×gr×gr×gr,L,H,M] is then reconstructed into [c vob The size of [l,h,m] is denoted as the prior feature feature. vp .

[0156] In some embodiments, point cloud features include encoded sequence features; panoramic semantic point clouds include sparse point cloud category information and sparse point cloud coordinate information.

[0157] In this embodiment, the high-dimensional, dense encoded voxel features are transformed into low-dimensional, sequential feature representations through two core operations: "block-based dimensionality reduction" and "positional encoding injection." On the one hand, this significantly reduces the computational burden and memory consumption of the input pre-trained language model, making it feasible to process 3D visual features using a large-scale language model. On the other hand, by introducing a positional embedding matrix specifically designed for 3D block-based dimensionality reduction, key spatial structural information is preserved while reducing dimensionality, providing necessary spatial references for the language model to understand the layout and relationships of 3D scenes.

[0158] Secondly, the embodiments of this application design a learnable feature reconstruction path (via W) vow-r1 and W vow-r2 The equal-weighted matrix resamples and fuses the dimensionality-reduced features rich in prior knowledge into the original scaled voxel features. This ensures that general 3D knowledge extracted from the language model (such as the complete shape of an object and the relationships between parts) can be injected back into the specific, fine-grained visual feature space with high fidelity and high spatial correspondence, laying a precise and operable feature foundation for prior knowledge to guide subsequent geometric completion and semantic annotation.

[0159] Furthermore, the final generated prior features are the sum of the original segmented voxel features and the reconstructed prior features. This fusion method means that the prior knowledge provided by the pre-trained language model supplements, enhances, and corrects biases, rather than completely replacing the visual features learned from the original data. This fully utilizes the commonsense reasoning ability of the language model while maintaining the model's sensitivity to the current specific input data, enhancing the system's robustness when handling novel objects or non-standard scenarios, and avoiding distortion or incorrect generalization caused by over-reliance on priors.

[0160] Figure 9 A flowchart illustrating the panoramic semantic point cloud generation process provided in this application embodiment; Figure 10 This application provides a schematic diagram illustrating the principle of a panoramic semantic point cloud generation process. Specifically, this application provides a panoramic semantic point cloud generation process executed by an electronic device based on a pre-trained language model. Figure 9 , Figure 10 As shown, the process of generating panoramic semantic point clouds is described in detail below:

[0161] S501: Construct a point cloud voxel heterogeneous map corresponding to the input point cloud based on the encoded sequence features, prior features, and the positional correlation mask corresponding to the input point cloud.

[0162] The point cloud voxel heterogeneous graph includes multiple nodes and heterogeneous graph edges between nodes. The nodes include voxel nodes and point cloud nodes.

[0163] In some embodiments, a point cloud voxel heterogeneous map is constructed. Among them, Nod v Represents a voxel node; Nod p Represents point cloud nodes; Edg vp An edge representing the connection between two types of nodes is called a heterogeneous graph edge.

[0164] Specifically, S501 includes the following steps S5011-S5015:

[0165] S5011: Convert the encoded sequence features and prior features into a preset dimension.

[0166] Figure 11 This application provides a schematic diagram illustrating the principle of constructing point cloud voxel heterogeneous maps in its embodiments; as follows: Figure 11 As shown in Figure (a), first use a size of [c] tem The learnable transition weight matrix W of [c] g-t1 Encode sequence features feature tem Convert to a size of [n, c]; use a size of [c] vo The learnable transition weight matrix W of [c] g-v1 Prior features vp Convert to a size of [c, l, h, w] to unify the dimension c of the two features.

[0167] S5012: Concatenate the first all-zero matrix and the transformed encoded sequence features to obtain point cloud nodes.

[0168] like Figure 11 As shown in Figure (b), a zero matrix of size [r, c] (i.e., the first zero matrix) is constructed and concatenated with the transformed encoded sequence features to obtain a matrix of size [n+r, c], which represents the point cloud nodes. Here, r represents the preset number of newly added points in the point cloud.

[0169] S5013: Assign values ​​to the second all-zero matrix based on the positional correlation mask, and merge and reconstruct the dimensions of the assigned matrix with the all-one matrix to obtain the heterogeneous graph edges.

[0170] like Figure 11 As shown in Figure (c), construct an all-zero matrix Edg of size [n,l,h,w]. vp1 (i.e., the second all-zero matrix), and assign a value to each element within it. The assignment formula is as follows: .

[0171] in, Then construct a pure one matrix Edg of size [r,l,h,w]. vp2After merging the two and reconstructing their dimensions, we obtain a heterogeneous graph edge Edg of size [n+r, l×h×w]. vp .

[0172] S5014: Perform matrix operations and dimension reconstruction on the transformed prior features to obtain voxel nodes.

[0173] Specifically, the transformed prior feature dimensions are reconstructed into voxel nodes of size [c, l×h×m]. v .

[0174] S5015: Obtain a point cloud voxel heterogeneous graph based on point cloud nodes, heterogeneous graph edges, and voxel nodes.

[0175] S502: Perform operations on the point cloud voxel heterogeneous map to obtain the point cloud semantic features.

[0176] Specifically, S502 includes the following steps S5021-S5023:

[0177] S5021: Determine the mapping weights of voxel nodes to point cloud nodes.

[0178] First, construct four learnable attention matrices W. u W i W v W p All sizes are [c, c].

[0179] Secondly, calculate the voxel node Nod. v Each node nod in v For point cloud node Nod p Each node nod in p The mapping weights are given by the formula: .

[0180] S5022: Normalize the mapping weights to obtain the normalized mapping weights.

[0181] Specifically, based on the formula Perform mapping weight normalization. Wherein, This represents the normalized mapping weights. This represents the exponentiation operator.

[0182] S5023: Based on the heterogeneous graph edges of voxel nodes and point cloud nodes, update the nodes in the point cloud voxel heterogeneous graph to obtain point cloud semantic features.

[0183] Specifically, the edge matrix Edg between two spatial nodes vp Integrate and update node features.

[0184] evp Corresponding to the edge Edg of the heterogeneous graph vp The value of the element in the corresponding row and column.

[0185] This method allows prior features in voxel form to be fused into point cloud features. The output of this step is a point cloud node of size [n+r,c] after the update, denoted as the point cloud semantic feature feat. pcsem .

[0186] S503: Perform semantic decoding on the semantic features of the point cloud to obtain sparse point cloud category information.

[0187] Specifically, the semantic features of point clouds are featured pcsem The input is fed into the semantic decoder (multilayer perceptron), and the formula is: .

[0188] Among them, W decs1 It is a learnable weight matrix of size [c, c], b decs1 W is a learnable bias vector of size [c, 1]. decs2 This is a learnable weight matrix of size [c, nc], where nc is the number of categories to be detected in the specific application of this application. Next, the maximum value in each column is retrieved. If the maximum value is greater than a preset probability threshold thp, the index category of that maximum value is output, such as "person" or "vehicle," and denoted as the sparse point cloud category information pcs. sem .

[0189] S504: Decode the position of the semantic features of the point cloud to obtain the position information.

[0190] Specifically, the semantic features of point clouds are featured pcsem The latter part is input into the position decoder (multilayer perceptron), based on the formula. Location information is obtained.

[0191] Among them, W dect1 It is a learnable weight matrix of size [c, c], b dect1 W is a learnable bias vector of size [c, 1]. dect2 It is a learnable weight matrix of size [c,3], where 3 represents the three-dimensional coordinates of the point in space.

[0192] S505: Concatenate the location information with the point cloud coordinate information of the input point cloud to obtain sparse point cloud coordinate information.

[0193] Specifically, this location information is compared with the point cloud coordinate information (pcs). sp By stitching together, we obtain the sparse point cloud coordinate information pcs sp .

[0194] In some embodiments, the panoramic semantic point cloud includes sparse point cloud category information and sparse point cloud coordinate information.

[0195] In this embodiment, a point cloud-voxel heterogeneous graph is constructed. This graph treats point cloud nodes (representing discrete points and their features) and voxel nodes (representing regularized spatial blocks and their prior knowledge) as two types of heterogeneous nodes, and precisely defines the connection relationship (heterogeneous graph edges) between them through a positional correlation mask based on spatial distance. This places fine-grained information at the point level and globally structured prior knowledge at the voxel level within a unified computational framework capable of message passing, providing a data structure foundation for subsequent feature fusion.

[0196] Furthermore, based on the construction of the heterogeneous graph, this embodiment designs a dedicated graph attention operation. This mechanism allows each point cloud node to adaptively aggregate information from voxel nodes based on its connection strength with all voxel nodes (determined by the edge weights of the heterogeneous graph and the learnable attention weights). This process accurately and discriminatively distributes the voxel features, enhanced by the language model and rich in prior knowledge of the complete shape of objects, to each point that needs to be completed (including original points and newly added points), thereby achieving effective transfer and fusion of knowledge from the "block" level to the "point" level.

[0197] Simultaneously, this embodiment integrates the point cloud semantic features into both the location decoder and the semantic decoder, achieving collaborative parallel decoding. This design ensures that the newly completed points not only have reasonable three-dimensional coordinates (pcs) sp ), and was also given accurate semantic category labels (pcs) sem The output sparse point cloud coordinate information and sparse point cloud category information together constitute a "panoramic semantic point cloud", which is a high-quality intermediate 3D scene representation that is more spatially complete and has semantic interpretation for each point.

[0198] Figure 12 This application provides a flowchart illustrating the output point cloud generation process in its embodiments; Figure 13 This application provides a schematic diagram illustrating the principle of the output point cloud generation process. Specifically, this application provides an output point cloud generation process executed by an electronic device based on a pre-trained language model. Figure 12 , Figure 13 The output point cloud generation process is described in detail below:

[0199] S601: Concatenate the sparse point cloud category information and sparse point cloud coordinate information to obtain a sparse point cloud.

[0200] Specifically, splicing sparse point cloud coordinate information (pcs) sp Sparse point cloud category information pcs semLet be denoted as sparse point cloud pcs. There are pcs[:,:3]. = pcs sp And pcs[:,:3]=pcs sem .

[0201] S602: Copy the sparse point cloud a preset number of times to obtain the initial dense point cloud.

[0202] Specifically, the sparse point cloud pcs is copied t times (i.e., the preset number of copies) to obtain the initial dense point cloud pcd0. The size of t can be set, usually between 5 and 10. The size of pcd0 is [(n+r)×t,3+nc].

[0203] S603: Encode the initial dense point cloud based on the point cloud coding network to obtain dense point cloud features.

[0204] Specifically, the initial dense point cloud pcd0 is input into a common point cloud encoding network (such as PointNet, PointNet++, etc.) for encoding, resulting in a value of [(n+r)×t,c pn The characteristics of ]. Among them, c pn To select the specified dimension for the encoding network, this feature is used with a size of [c]. pn The weight transformation matrix of [(n+r)×t,c] is transformed into a dense point cloud feature of size [(n+r)×t,c].

[0205] S604: Perform semantic decoding on all points in the dense point cloud features to obtain dense point cloud category information.

[0206] Specifically, a semantic decoder is used to decode all points in the dense point cloud features to obtain dense point cloud category information.

[0207] S605: Decode the position of all points in the dense point cloud feature to obtain the coordinate information of the dense point cloud.

[0208] Specifically, a location decoder is used to decode all points in the dense point cloud features to obtain the coordinate information of the dense point cloud.

[0209] S606: Merge the dense point cloud category information, dense point cloud coordinate information, and sparse point cloud information to obtain the output point cloud.

[0210] Specifically, the point cloud information [(n+r)×t,3+nc] (i.e., dense point cloud category information and dense point cloud coordinate information) obtained in steps S605 and S606 above is merged with the sparse point cloud information pcs to obtain dense point cloud information [(n+r)×(t+1),3+nc], which is the output point cloud.

[0211] In this embodiment, firstly, the number of points is increased at low cost through a copying operation (S602), providing a basic point set for densification. Then, a mature point cloud encoding network (such as PointNet++) is used to encode the initial dense point cloud (S603). This network can effectively aggregate the local and global context of points, learn the distribution patterns of the point set, and provide strong feature support for subsequent decoding to generate a more reasonable and smoother point distribution. This method avoids complex and unstable direct upsampling of the original sparse point cloud, ensuring the efficiency and stability of the generation process.

[0212] Secondly, in this embodiment, the pre-trained semantic decoder and position decoder (S60, S605) in the point cloud decoding module are reused to decode the encoded dense point cloud features. On the one hand, this ensures the consistency of semantic prediction between the newly added dense points and the original sparse points; on the other hand, the position decoder generates coordinates for the newly added points based on the learned feature distribution, making the final point cloud more geometrically uniform and more closely conforming to the real geometry of the object surface. This collaborative decoding strategy effectively ensures semantic coherence and geometric fidelity while improving point cloud density.

[0213] Figure 14 The flowchart of the model training process provided in the embodiments of this application is shown below. Figure 14 As shown, embodiments of this application provide a model training process method, which is executed by an electronic device. The model training process is described in detail below:

[0214] S701: Acquire training data.

[0215] In some embodiments, the training data may include pairs of 3D point cloud data. Each pair of data includes:

[0216] Input point cloud: A sparse, incomplete point cloud generated by simulated lidar scanning or manual sampling.

[0217] Target point cloud: The corresponding dense and complete point cloud, used as the ground truth for geometric completion.

[0218] Semantic labels: Category information (such as "vehicle", "tree", "road") labeled for each point in the point cloud, serving as ground truth for semantic prediction.

[0219] S702: Based on the training data, train the initial language prediction model to obtain the preset language prediction model.

[0220] First, the model is trained to learn shape completion using input point cloud and target point cloud pairs. The loss function measures the geometric difference between the predicted point cloud and the true complete point cloud.

[0221] Finally, semantic labels are added, and the model's geometric completion and semantic prediction capabilities are trained simultaneously. A pre-trained language model is also introduced to align the point cloud features with the semantic space of the language model, enabling the model to learn to utilize general knowledge from the language model. The total loss function is a weighted sum of the geometric loss, semantic classification loss, and feature alignment loss.

[0222] In this embodiment, paired training data of "sparse-dense-semantic" is constructed to provide the model with clear and complete learning objectives. Sparse point clouds simulate real sensor input, dense point clouds define ideal geometric output, and semantic labels provide standard answers for scene understanding. This data construction method enables the model to simultaneously optimize geometric reconstruction accuracy and semantic understanding accuracy during the learning process, avoiding feature bias or task conflicts that may occur with traditional single-task training, and ensuring the comprehensiveness of the final model performance from the data source.

[0223] Secondly, by training the model, the reasonableness of the model's completion of sparse and occluded point clouds can be significantly improved, enabling it to "imagine" the reasonable shape of the invisible parts based on common sense, and greatly enhancing the model's ability to generalize to unknown objects or novel scenes.

[0224] Figure 15 This is a schematic diagram of the point cloud processing device provided in an embodiment of this application. Figure 15 As shown, embodiments of this application also provide a point cloud processing apparatus, including:

[0225] Module 1501 is used to acquire the input point cloud;

[0226] Processing module 1502 is used to input the input point cloud into a pre-trained language model to perform semantic completion on the input point cloud based on the pre-trained language model, and obtain the output point cloud corresponding to the input point cloud. The point cloud density of the output point cloud is higher than that of the input point cloud. The pre-trained language model is used to perform the following operations:

[0227] Encode the input point cloud to obtain the point cloud features corresponding to the input point cloud;

[0228] Based on prior knowledge, feature completion is performed on the point cloud features to obtain the prior features corresponding to the input point cloud.

[0229] Decode the point cloud features and prior features to obtain the panoramic semantic point cloud corresponding to the input point cloud;

[0230] The panoramic semantic point cloud is extended to obtain the output point cloud corresponding to the input point cloud.

[0231] In some embodiments, the processing module 1502 is specifically used for:

[0232] Voxel encoding is performed on the input point cloud to obtain the encoded voxel features corresponding to the input point cloud;

[0233] The input point cloud is encoded into a point cloud sequence to obtain the intermediate sequence features corresponding to the input point cloud.

[0234] Sequence cross-coding is performed on the encoded voxel features and intermediate sequence features to obtain the encoded sequence features corresponding to the input point cloud.

[0235] In some embodiments, the processing module 1502 is specifically used for:

[0236] Based on the 3D coordinates of the points in the input point cloud and the voxel grid, the initial voxel matrix corresponding to the input point cloud is obtained. The voxel grid includes multiple voxel units.

[0237] The initial voxel matrix is ​​encoded using a three-dimensional convolutional neural network to obtain the encoded voxel features.

[0238] In some embodiments, the processing module 1502 is specifically used to: obtain point cloud coordinate information of the input point cloud based on the three-dimensional coordinates of points in the input point cloud;

[0239] Normalize the point cloud coordinate information to obtain normalized point cloud coordinate information;

[0240] Construct a voxel grid of a first preset size, wherein the voxel grid includes multiple voxel units;

[0241] Based on the normalized point cloud coordinate information, the nearest point cloud of the voxel unit is found in the input point cloud. If the distance between the nearest point cloud and the voxel unit is less than a preset value, the optical information of the nearest point cloud is assigned to the voxel unit to obtain the initial voxel matrix corresponding to the input point cloud.

[0242] In some embodiments, the processing module 1502 is specifically used for:

[0243] Based on the color of the points in the input point cloud, obtain the point cloud optical information of the input point cloud;

[0244] The optical information of point clouds is encoded using a multilayer perceptron to obtain intermediate sequence features.

[0245] In some embodiments, the processing module 1502 is specifically used for:

[0246] A linear transformation is performed on the encoded voxel features to obtain the transformed encoded voxel features;

[0247] Based on the transformed coded voxel features and the position coding matrix, voxel features carrying absolute position information are obtained;

[0248] Construct a positional relevance mask, which is used to indicate point-voxel interaction information;

[0249] Attention calculations are performed on the position-related mask and the coded voxel features to obtain the intermediate features of the coded sequence;

[0250] The intermediate features of the encoded sequence are merged with the point cloud coordinate information of the input point cloud to obtain the encoded sequence features corresponding to the input point cloud.

[0251] In some embodiments, the point cloud features include encoded voxel features; the processing module 1502 is specifically used for:

[0252] The encoded voxel features are divided into blocks according to the second preset size to obtain block voxel features;

[0253] Construct a block location embedding matrix, and obtain the encoded dimensionality-reduced voxel features based on the block voxel features and the block location embedding matrix;

[0254] Prior information of the encoded dimensionality-reduced voxel features is extracted based on the encoder to obtain the dimensionality-reduced prior features;

[0255] The block voxel features are merged with the transformed dimensionality-reduced prior features to obtain the first prior intermediate feature, and the first prior intermediate feature is linearly transformed to obtain the second prior intermediate feature.

[0256] The prior intermediate second feature is reconstructed to a preset size to obtain the prior feature corresponding to the input point cloud.

[0257] In some embodiments, the processing module 1502 is specifically used for:

[0258] Construct a zero matrix of the third preset size;

[0259] Assign values ​​to the all-zero matrix to obtain the block position embedding matrix;

[0260] The block voxel features and block position embedding matrices are merged, and the merged matrix is ​​reconstructed in dimension and linearly transformed to obtain the encoded dimensionality-reduced voxel features.

[0261] In some embodiments, the point cloud features include encoded sequence features; the panoramic semantic point cloud includes sparse point cloud category information and sparse point cloud coordinate information; the processing module 1502 is specifically used to: construct a point cloud voxel heterogeneous graph corresponding to the input point cloud based on the encoded sequence features, prior features and the positional correlation mask corresponding to the input point cloud, the point cloud voxel heterogeneous graph including multiple nodes and heterogeneous graph edges between nodes, the nodes including voxel nodes and point cloud nodes;

[0262] Perform operations on the point cloud voxel heterogeneous map to obtain the point cloud semantic features;

[0263] Semantic decoding is performed on the semantic features of the point cloud to obtain sparse point cloud category information;

[0264] Position information is obtained by decoding the semantic features of the point cloud.

[0265] The location information is concatenated with the point cloud coordinate information of the input point cloud to obtain sparse point cloud coordinate information.

[0266] In some embodiments, the processing module 1502 is specifically used for:

[0267] Convert encoded sequence features and prior features into a preset dimension;

[0268] The first all-zero matrix and the transformed encoded sequence features are concatenated to obtain point cloud nodes;

[0269] The second all-zero matrix is ​​assigned a value based on the position-related mask, and the assigned matrix is ​​merged and the dimension is reconstructed with the all-one matrix to obtain the heterogeneous graph edges;

[0270] The transformed prior features are subjected to matrix operations and dimension reconstruction to obtain voxel nodes;

[0271] Based on point cloud nodes, heterogeneous graph edges, and voxel nodes, a point cloud voxel heterogeneous graph is obtained.

[0272] In some embodiments, the processing module 1502 is specifically used for:

[0273] Determine the mapping weights of voxel nodes to point cloud nodes;

[0274] The mapping weights are normalized to obtain the normalized mapping weights.

[0275] Based on the heterogeneous graph edges of voxel nodes and point cloud nodes, the nodes in the point cloud voxel heterogeneous graph are updated to obtain the point cloud semantic features.

[0276] In some embodiments, the processing module 1502 is specifically used for:

[0277] The sparse point cloud category information and sparse point cloud coordinate information are concatenated to obtain sparse point cloud information;

[0278] The sparse point cloud is copied a preset number of times to obtain the initial dense point cloud;

[0279] The initial dense point cloud is encoded using a point cloud coding network to obtain dense point cloud features;

[0280] Semantic decoding is performed on all points in the dense point cloud features to obtain the dense point cloud category information;

[0281] The location of all points in the dense point cloud feature is decoded to obtain the coordinate information of the dense point cloud;

[0282] The dense point cloud category information, dense point cloud coordinate information, and sparse point cloud information are merged to obtain the output point cloud.

[0283] It should be noted that the description of the features in the embodiment corresponding to the point cloud processing device can be found in the relevant description of the embodiment corresponding to the point cloud processing method described above, and will not be repeated here.

[0284] Figure 16 A schematic diagram of the structure of the electronic device provided in this application. Figure 16 As shown, the electronic device 160 provided in this embodiment includes at least one processor 1601 and a memory 1602. Optionally, the electronic device 160 also includes a communication component 1603. The processor 1601, memory 1602, and communication component 1603 are connected via a bus.

[0285] In the specific implementation process, at least one processor 1601 executes computer execution instructions stored in memory 1602, causing at least one processor 1601 to execute the above-described point cloud processing method embodiment.

[0286] The specific implementation process of processor 1601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0287] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0288] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0289] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0290] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described point cloud processing method embodiments at runtime.

[0291] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0292] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described point cloud processing and positioning method embodiments.

[0293] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described point cloud processing method embodiments.

[0294] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0295] The point cloud processing method and electronic device provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A point cloud processing method, characterized in that, include: Obtain the input point cloud; The input point cloud is input into a pre-trained language model to perform semantic completion on the input point cloud based on the pre-trained language model, and the output point cloud corresponding to the input point cloud is obtained. The point cloud density of the output point cloud is higher than that of the input point cloud. The pre-trained language model is used to perform the following operations: The input point cloud is encoded to obtain the point cloud features corresponding to the input point cloud; Based on prior knowledge, feature completion is performed on the point cloud features to obtain the prior features corresponding to the input point cloud; Decode the point cloud features and the prior features to obtain the panoramic semantic point cloud corresponding to the input point cloud; The panoramic semantic point cloud is expanded to obtain the output point cloud corresponding to the input point cloud; The step of encoding the input point cloud to obtain the point cloud features corresponding to the input point cloud includes: Voxel encoding is performed on the input point cloud to obtain the encoded voxel features corresponding to the input point cloud; The input point cloud is encoded into a point cloud sequence to obtain the intermediate sequence features corresponding to the input point cloud. The encoded voxel features and the intermediate sequence features are subjected to sequence cross-coding to obtain the encoded sequence features corresponding to the input point cloud; The point cloud features include encoded voxel features; The step of performing feature completion on the point cloud features based on prior knowledge to obtain the prior features corresponding to the input point cloud includes: The encoded voxel features are divided into blocks according to the second preset size to obtain block voxel features; Construct a block location embedding matrix, and obtain encoded dimensionality-reduced voxel features based on the block voxel features and the block location embedding matrix; Based on the prior information of the encoded dimensionality-reduced voxel features extracted by the encoder, the dimensionality-reduced prior features are obtained; The block voxel features are merged with the transformed dimensionality-reduced prior features to obtain the first prior intermediate feature, and the first prior intermediate feature is linearly transformed to obtain the second prior intermediate feature. The prior intermediate second feature is reconstructed to a preset size to obtain the prior feature corresponding to the input point cloud; The point cloud features include encoded sequence features; the panoramic semantic point cloud includes sparse point cloud category information and sparse point cloud coordinate information. Decoding the point cloud features and the prior features to obtain the panoramic semantic point cloud corresponding to the input point cloud includes: Based on the encoded sequence features, the prior features, and the positional correlation mask corresponding to the input point cloud, a point cloud voxel heterogeneous graph corresponding to the input point cloud is constructed. The point cloud voxel heterogeneous graph includes multiple nodes and heterogeneous graph edges between the nodes. The nodes include voxel nodes and point cloud nodes. The point cloud semantic features are obtained by performing calculations on the point cloud voxel heterogeneous map. Semantic decoding is performed on the semantic features of the point cloud to obtain the category information of the sparse point cloud; The semantic features of the point cloud are decoded to obtain location information; The location information is concatenated with the point cloud coordinate information of the input point cloud to obtain the sparse point cloud coordinate information.

2. The method according to claim 1, characterized in that, The step of performing voxel encoding on the input point cloud to obtain the encoded voxel features corresponding to the input point cloud includes: Based on the three-dimensional coordinates of the points in the input point cloud and the voxel grid, the initial voxel matrix corresponding to the input point cloud is obtained, and the voxel grid includes multiple voxel units. The initial voxel matrix is ​​encoded using a three-dimensional convolutional neural network to obtain the encoded voxel features.

3. The method according to claim 2, characterized in that, The process of obtaining the initial voxel matrix corresponding to the input point cloud based on the 3D coordinates of the midpoints of the input point cloud and the voxel mesh includes: Based on the three-dimensional coordinates of the points in the input point cloud, the point cloud coordinate information of the input point cloud is obtained; Normalize the point cloud coordinate information to obtain normalized point cloud coordinate information; Construct the voxel grid of the first preset size, wherein the voxel grid includes multiple voxel units; Based on the normalized point cloud coordinate information, the nearest point cloud of the voxel unit is found in the input point cloud. If the distance between the nearest point cloud and the voxel unit is less than a preset value, the optical information of the nearest point cloud is assigned to the voxel unit to obtain the initial voxel matrix corresponding to the input point cloud.

4. The method according to claim 1, characterized in that, The step of performing sequence cross-coding on the encoded voxel features and the intermediate sequence features to obtain the encoded sequence features corresponding to the input point cloud includes: A linear transformation is performed on the encoded voxel features to obtain the transformed encoded voxel features; Based on the transformed encoded voxel features and the position encoding matrix, voxel features carrying absolute position information are obtained. Construct a positional relevance mask, which is used to indicate point-voxel interaction information; Attention calculations are performed on the position-related mask and the encoded voxel features to obtain intermediate features of the encoded sequence; The intermediate features of the encoded sequence are merged with the point cloud coordinate information of the input point cloud to obtain the encoded sequence features corresponding to the input point cloud.

5. The method according to claim 1, characterized in that, The step of constructing a point cloud voxel heterogeneous map corresponding to the input point cloud based on the encoded sequence features, the prior features, and the positional correlation mask corresponding to the input point cloud includes: The encoded sequence features and the prior features are converted into a preset dimension; The first all-zero matrix and the transformed encoded sequence features are concatenated to obtain the point cloud nodes; The second all-zero matrix is ​​assigned a value based on the positional correlation mask, and the assigned matrix is ​​merged and its dimensions reconstructed with the all-one matrix to obtain the heterogeneous graph edges; The transformed prior features are subjected to matrix operations and dimension reconstruction to obtain the voxel nodes; The point cloud voxel heterogeneous graph is obtained based on the point cloud nodes, the heterogeneous graph edges, and the voxel nodes.

6. The method according to any one of claims 1-4, characterized in that, The panoramic semantic point cloud includes sparse point cloud category information and sparse point cloud coordinate information; The step of expanding the panoramic semantic point cloud to obtain the output point cloud corresponding to the input point cloud includes: The sparse point cloud category information and the sparse point cloud coordinate information are concatenated to obtain sparse point cloud information; The sparse point cloud is copied a predetermined number of times to obtain an initial dense point cloud; The initial dense point cloud is encoded using a point cloud coding network to obtain dense point cloud features; Semantic decoding is performed on all points in the dense point cloud features to obtain dense point cloud category information; The location of all points in the dense point cloud feature is decoded to obtain the dense point cloud coordinate information; The dense point cloud category information, the dense point cloud coordinate information, and the sparse point cloud information are merged to obtain the output point cloud.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the point cloud processing method as described in any one of claims 1 to 6.

Images