A three-dimensional point cloud representation segmentation method and system

Abstract

The embodiment of the application provides a three-dimensional point cloud sign representation segmentation method and system, and relates to the technical field of three-dimensional point cloud processing. After obtaining scene detection data and sign representation text, the method can construct an attribute view through pseudo-color coding, and obtain multi-view two-dimensional semantic features through multi-view off-screen rendering of the attribute view. Then, through depth consistency verification and feature fusion, enhanced visual features are generated according to the two-dimensional semantic features and the three-dimensional point cloud. Then, sparse queries are generated according to the sign representation text and the enhanced visual features, so that language prompts are injected into the sparse queries in a residual manner according to the text query attention entropy, and a segmentation result is obtained. The method introduces a cross-modal interaction mechanism of two-dimensional geometric prior and language guidance, realizes depth consistency verification, text weighted sparse query generation and reliable perception prompt decoding, and improves the processing performance of the city road three-dimensional point cloud sign representation segmentation.

Description

Technical Field

[0001] This application relates to the field of 3D point cloud processing technology, and in particular to a 3D point cloud index expression segmentation method and system. Background Technology

[0002] 3D indexing and segmentation is an image processing technique that locates and segments corresponding 3D targets in a 3D point cloud scene based on the natural language description input by the user. 3D indexing and segmentation is one of the important tasks of 3D visual language understanding, and can serve scenarios such as autonomous driving, high-precision map production, urban road inspection, road asset management, and mobile mapping, enabling the system to extract target regions from 3D point clouds of urban roads according to linguistic expressions.

[0003] Compared to fixed-class detection or semantic segmentation, 3D indexing expression segmentation requires the model to identify target categories and understand attributes, orientations, quantities, relative positions, and negation relationships in natural language. Therefore, it is more suitable for interactive road asset retrieval, understanding complex traffic scenes, and target localization on open roads. Consequently, 3D indexing expression segmentation requires multi-view 2D image augmentation. This can be achieved by using real camera images or general multi-view images to augment 3D point cloud images from multiple perspectives, helping the model understand orientational relationships.

[0004] However, the aforementioned multi-view 2D image enhancement methods are easily affected by incomplete viewpoints, sensor calibration errors, insufficient image coverage, and occlusion projection errors. They are difficult to use stably in road LiDAR data lacking real camera images, thus reducing the performance of 3D point cloud indexing and segmentation for urban roads. Summary of the Invention

[0005] In view of this, embodiments of this application provide a three-dimensional point cloud indexing representation segmentation method and system to solve the problem of low performance in three-dimensional point cloud indexing representation segmentation for urban roads.

[0006] According to a first aspect of this application, a three-dimensional point cloud index expression segmentation method is provided, the method comprising: Acquire scene detection data and instruction expression text. The scene detection data includes 3D point cloud and radar attribute information collected for urban road scenes; the instruction expression text is natural language description text. Based on the scene detection data, an attribute view is constructed through pseudo-color encoding, and multi-view off-screen rendering is performed on the attribute view to obtain a multi-view two-dimensional rendered image and two-dimensional semantic features; the attribute view includes a radial attribute view and a structural attribute view. Enhanced visual features are generated based on the two-dimensional semantic features and the three-dimensional point cloud through deep consistency verification and feature fusion. A sparse query is generated based on the indicated text and the enhanced visual features. The sparse query is obtained by text-weighted semantic farthest point sampling and multi-granularity filtering based on the semantic relevance scores of the indicated text and the enhanced visual features. Based on the text query attention entropy, language cues are injected into the sparse query in a residual manner to obtain the segmentation result, which includes the target segmentation mask and the target existence probability.

[0007] In some embodiments, an attribute view is constructed based on the scene detection data using pseudo-color encoding, including: The three-dimensional point cloud and radar attribute information are extracted from the scene detection data. The radar attribute information includes radiation attribute information and structural attribute information. The radiation attribute information includes at least one of intensity, reflectivity, and amplitude. The structural attribute information includes at least one of height, echo count, and deviation. The radiation attribute information is encoded into the color channel of the radiation attribute view, and the structural attribute information is encoded into the color channel of the structural attribute view.

[0008] In some embodiments, multi-view off-screen rendering is performed on the attribute view to obtain multi-view 2D rendered images and 2D semantic features, including: Layout parameters are obtained based on the scene detection data, including point cloud bounding boxes and road scene centers; Set the virtual camera layout according to the layout parameters; Based on the virtual camera layout, the attribute view is rendered off-screen from multiple perspectives to obtain a multi-view 2D rendered image; the attribute view contains a 3D point cloud with pseudo-color encoding. Two-dimensional semantic features of the multi-view two-dimensional rendered image are extracted using a two-dimensional visual encoder.

[0009] In some embodiments, enhanced visual features are generated based on the two-dimensional semantic features and the three-dimensional point cloud through deep consistency verification and feature fusion, including: The three-dimensional point cloud is projected onto a two-dimensional image plane from a virtual perspective to obtain the projected pixel coordinates and projection depth; The projection pixel index of the homogeneous 3D point cloud is determined based on the projection pixel coordinates; Construct a projection depth buffer based on the projection pixel index and the projection depth; Perform a depth consistency check according to the projection depth buffer to determine the visible projection points; The visible projection points are sampled to correspond to the two-dimensional semantic features from multiple viewpoints, and feature fusion is performed based on the two-dimensional semantic features from multiple viewpoints to generate enhanced visual features.

[0010] In some embodiments, feature fusion is performed based on the two-dimensional semantic features from multiple perspectives to generate enhanced visual features, including: The two-dimensional semantic features from multiple perspectives are aggregated into point-by-point multi-perspective semantic features; Obtain the superpoint assignment relationship and the three-dimensional visual features of the three-dimensional point cloud, wherein the three-dimensional visual features are structural relationship features extracted based on the three-dimensional point cloud; Generate super-point level multi-view semantic features based on the super-point allocation relationship and the point-by-point multi-view semantic features, and map the super-point level multi-view semantic features to an embedding space consistent with the three-dimensional visual features; The enhanced visual features are obtained by fusing the super-point-level multi-view semantic features mapped to the embedding space with the three-dimensional visual features.

[0011] In some embodiments, generating a sparse query based on the indicated text and the enhanced visual features includes: The indicated text is input into a text encoder to generate text features of the indicated text, which include word-level text features and sentence-level text embeddings. Calculate the semantic relevance score of each superpoint based on the enhanced visual features and the text features; The semantic relevance score is injected with an exponential weight into the spatial distance metric to construct a semantically weighted distance. Multi-granularity filtering is performed based on the semantically weighted distance to obtain sparse queries.

[0012] In some embodiments, multi-granularity filtering is performed based on the semantically weighted distance to obtain sparse queries, including: The relevance is calculated based on the semantically weighted distance, and the relevance includes word-level relevance and sentence-level relevance; Generate a seed query based on the indicated text; The seed query is augmented with local geometry to obtain candidate queries; Based on the combined word-level relevance and sentence-level relevance, a sparse query is selected from the candidate queries.

[0013] In some embodiments, language cues are injected into the sparse query in a residual manner based on the text query attention entropy to obtain a segmentation result, including: Calculate the text attention distribution corresponding to each query in the sparse query; The text query attention entropy is calculated based on the text attention distribution, and the text query attention entropy is a normalized information entropy used to measure the concentration and reliability of language prompts; A gating scalar is generated based on the text query attention entropy; According to the gating scalar, the language prompts are injected into the query features corresponding to the sparse query after residual mapping.

[0014] In some embodiments, language cues are injected into the sparse query in a residual manner based on the text query attention entropy to obtain a segmentation result, including: The query features with injected language hints are input into the segmentation model, and the segmentation model outputs the target segmentation mask and the target existence probability. Obtain the preset probability threshold; When the probability of the target's existence is higher than or equal to the preset probability threshold, the segmentation result is generated based on the target segmentation mask and the target's existence probability. When the probability of the existence of all candidate targets is lower than the preset probability threshold, an empty target segmentation result is generated.

[0015] According to a second aspect of this application, a three-dimensional point cloud indexing and segmentation system is provided, the system comprising: The data acquisition module is used to acquire scene detection data and index expression text. The scene detection data includes 3D point cloud and radar attribute information collected for urban road scenes; the index expression text is natural language description text. The encoding and rendering module is used to construct an attribute view based on the scene detection data through pseudo-color encoding, and to obtain a multi-view two-dimensional rendered image and two-dimensional semantic features by performing multi-view off-screen rendering on the attribute view; the attribute view includes a radiation attribute view and a structural attribute view. The semantic fusion module is used to generate enhanced visual features based on the two-dimensional semantic features and the three-dimensional point cloud through deep consistency verification and feature fusion. The query generation module is used to generate a sparse query based on the indicated text and the enhanced visual features. The sparse query is obtained by text-weighted semantic farthest point sampling and multi-granularity filtering based on the semantic relevance score between the indicated text and the enhanced visual features. The segmentation output module is used to inject language cues into the sparse query in a residual manner based on the text query attention entropy to obtain the segmentation result, which includes the target segmentation mask and the target existence probability.

[0016] By employing the above technical solutions, this application provides a method and system for 3D point cloud indexing expression segmentation. After acquiring scene detection data and indexing expression text, the method constructs an attribute view through pseudo-color encoding and obtains multi-view 2D semantic features by performing multi-view off-screen rendering on the attribute view. Then, through depth consistency verification and feature fusion, enhanced visual features are generated based on the 2D semantic features and the 3D point cloud. Next, a sparse query is generated according to the indexing expression text and the enhanced visual features. Based on the text query attention entropy, language prompts are injected into the sparse query in a residual manner to obtain the segmentation result. This method, by introducing a cross-modal interaction mechanism of 2D geometric prior and language guidance, achieves depth consistency verification, text-weighted sparse query generation, and reliability-aware prompt decoding, thereby improving the processing performance of 3D point cloud indexing expression segmentation for urban roads.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the three-dimensional point cloud index expression segmentation method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the segmentation process provided in an embodiment of this application; Figure 3 A schematic diagram of the overall network framework of the segmentation method provided in the embodiments of this application; Figure 4 This is a schematic diagram of multi-view semantic embedding and depth-consistent two-dimensional and three-dimensional fusion provided in the embodiments of this application; Figure 5 This is a schematic diagram of semantic adaptive query generation and multi-granularity filtering provided in the embodiments of this application; Figure 6 This is a schematic diagram of the result filtering process provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of a three-dimensional point cloud pointer representation segmentation system provided in an embodiment of this application. Detailed Implementation

[0019] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0020] 3D indexing and segmentation is an image processing technique that locates and segments corresponding 3D targets in a 3D point cloud scene based on the natural language description input by the user. 3D indexing and segmentation is one of the important tasks of 3D visual language understanding, and can serve scenarios such as autonomous driving, high-precision map production, urban road inspection, road asset management, and mobile mapping, thereby enabling the extraction of target regions from 3D point clouds of urban roads according to linguistic expressions.

[0021] For example, the natural language description text input by the user can be expressions such as "tree on the right side of the intersection," "pole near the edge of the lane," "vehicle in front of the building," or "power line crossing the road." Target areas can then be extracted from the 3D point cloud of urban roads. After acquiring the user-input natural language text, the scene point cloud image acquired by the LiDAR point cloud acquisition device can be combined with the natural language text input segmentation model. This allows the segmentation model to perform image recognition by integrating the textual features of the natural language text and the visual features of the scene point cloud image, thereby extracting target areas such as trees, poles, vehicles, and power lines from the 3D point cloud of urban roads.

[0022] Three-dimensional indexing segmentation requires the segmentation model to be able to identify target categories and understand attributes, orientations, quantities, relative positions, and negation relationships in natural language, in order to adapt to tasks such as human-computer interactive road asset retrieval, understanding complex traffic scenarios, and open road target localization.

[0023] When applying 3D indexing representation segmentation to urban road scenes, the segmentation models primarily target indoor scenes. In indoor scenes, objects are smaller in scale, have more concentrated categories, regular spatial relationships, and relatively uniform point cloud density. However, urban road point clouds are collected by vehicle-mounted, airborne, or mobile LiDAR devices, which have a large coverage area, a high proportion of background point clouds, and point density that varies significantly with distance, scanning angle, and occlusion.

[0024] For example, large background areas such as roads, building facades, and low vegetation may occupy the majority of point clouds, while targets such as trees, vehicles, pedestrians, poles, traffic facilities, and power lines appear as sparse, elongated, discrete, or partially occluded objects.

[0025] It is evident that natural language descriptions may correspond to single-target, multi-target, or zero-target scenarios, requiring segmentation models to avoid missing small targets while also avoiding missegmenting locally similar but not target-specific background regions. Therefore, 3D indexing segmentation necessitates multi-view 2D image enhancement. This can be achieved by using real camera images or general multi-view images to enhance 3D point cloud images from multiple perspectives, helping the model understand orientational relationships.

[0026] However, the aforementioned multi-view 2D image enhancement methods are easily affected by incomplete viewpoints, sensor calibration errors, insufficient image coverage, and occlusion projection errors. They are difficult to use stably in road LiDAR data lacking real camera images, thus reducing the performance of 3D point cloud indexing and segmentation for urban roads.

[0027] Therefore, in some embodiments, query generation can also be based on geometrically farthest point sampling. This involves initially selecting a randomly chosen point as a seed point, then performing iterative sampling. During each iteration, the next farthest point is selected from the unselected points, and the minimum distance is updated based on the newly selected point. After iterative sampling is complete, the coordinates of the selected points can be used as a query point set for subsequent local feature aggregation, attention calculation, or grouping operations.

[0028] However, query generation methods based on geometric farthest point sampling mainly pursue uniform spatial coverage, which easily wastes query capacity in large background areas such as roads, buildings and vegetation, and cannot actively focus on sparse target areas pointed to by text descriptions.

[0029] Furthermore, to improve the accuracy of 3D point cloud index expression segmentation, language prompts can be injected into the query features generated from natural language text. Specifically, in some embodiments, the input natural language description can be encoded to obtain two types of text features: sentence-level embeddings and word-level feature sequences. Then, based on pure geometric farthest point sampling or semantically weighted farthest point sampling, a set of sparse query point locations is selected from the point cloud or image feature map. The sentence-level embeddings or word-level features are then concatenated with the visual features of each query point to obtain enhanced query features, which are then used to perform image segmentation.

[0030] Because the prompt decoder directly concatenates or injects language prompts into query features during prompt decoding, when the attention distribution between the query and the text is relatively scattered, low-quality language prompts will be propagated to subsequent decoding layers, causing semantic noise pollution and reducing the performance of urban road 3D point cloud index expression segmentation.

[0031] To address the issue of low performance in the representation and segmentation of 3D point clouds representing urban roads, this application provides a 3D point cloud representation and segmentation method in some embodiments. This method improves the segmentation accuracy, query targeting, and output reliability in zero-target scenarios by employing radar attribute pseudo-color rendering, depth-consistent 2D and 3D semantic fusion, text-guided query generation, and reliability-aware prompt decoding in large-scale, significantly occluded, and complex background urban road point clouds.

[0032] The method can be applied to a 3D point cloud representation segmentation system, or to an electronic device that establishes a communication connection with the segmentation system and has data processing capabilities. The electronic device includes, but is not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control computers. For ease of description, the segmentation system is used as the execution subject of the method in this embodiment. It should be understood that the method can also be applied to other types of execution subjects, which are not shown one by one in this embodiment. Figure 1 As shown, the method includes: S101. Obtain scene detection data and instruction expression text.

[0033] When performing 3D point cloud indexing and expression segmentation, relevant data used in the 3D point cloud indexing and expression segmentation process can be acquired first, namely, scene detection data and indexing expression text. The scene detection data includes 3D point cloud data and radar attribute information collected for urban road scenes.

[0034] 3D point clouds can be acquired using vehicle-mounted, airborne, backpack-mounted, or fixed LiDAR (Light Detection and Range) devices. For urban road scenes, LiDAR devices can be deployed in the urban road environment and, in response to scene detection commands, perform LiDAR scanning to obtain 3D point clouds of the urban road scene.

[0035] Radar attribute information can be obtained during the acquisition of 3D point clouds by recording relevant information during the acquisition process, or by detection by sensors configured on the radar equipment. Radar attribute information can include 3D coordinates and at least one of the following lidar attributes: intensity, reflectivity, amplitude, height, echo count, and deviation.

[0036] In some embodiments, radar attribute information can be divided into radiation attribute information and structural attribute information according to its intended use. The radiation attribute information includes at least one of intensity, reflectivity, and amplitude; the structural attribute information includes at least one of height, echo frequency, and deviation.

[0037] The referential text is the natural language description text input by the user, used to represent the referential target of referential segmentation. For example, the referential text can be natural language text such as "tree on the right side of the intersection", "pole near the edge of the lane", "vehicle in front of the building", "power line crossing the road".

[0038] S102. Based on scene detection data, construct an attribute view through pseudo-color encoding, and obtain multi-view two-dimensional rendered images and two-dimensional semantic features by performing multi-view off-screen rendering on the attribute view.

[0039] After acquiring scene detection data, pseudo-color encoding and multi-view off-screen rendering can be performed on the data. By performing pseudo-color encoding on the acquired 3D point cloud of urban roads and its LiDAR attributes, an attribute view can be constructed. This attribute view includes a radiation attribute view and a structural attribute view.

[0040] like Figure 2 , Figure 3 As shown, in some embodiments, when constructing an attribute view based on scene detection data using pseudo-color encoding, three-dimensional point cloud and radar attribute information can be extracted from the scene detection data first. The radar attribute information includes radiation attribute information and structural attribute information. The radiation attribute information includes at least one of intensity, reflectivity, and amplitude; the structural attribute information includes at least one of height, echo count, and deviation. Then, the radiation attribute information is encoded into the color channels of the radiation attribute view, and the structural attribute information is encoded into the color channels of the structural attribute view, respectively.

[0041] For example, by performing preprocessing on urban road point cloud input data to obtain 3D point clouds of urban roads collected by vehicle-mounted, airborne, backpack, or fixed LiDAR devices, and receiving natural language descriptions for referring to targets, LiDAR attribute pseudo-color encoding can be performed. Intensity, reflectivity, and amplitude are encoded into the red, green, and blue channels of the radiation attribute view, respectively, and height, echo count, and deviation are encoded into the red, green, and blue channels of the structural attribute view, respectively, to construct the radiation attribute view and the structural attribute view. The radiation attribute view consists of intensity, reflectivity, and amplitude, while the structural attribute view consists of height, echo count, and deviation.

[0042] After constructing the attribute view through pseudo-color encoding, multi-view off-screen rendering can be performed based on the virtual camera layout to obtain multi-view 2D rendered images and their 2D semantic features. The virtual camera layout can be set according to scene detection data, including virtual camera parameters such as the bounding box based on the city road point cloud, the scene center, and preset viewing directions.

[0043] In some embodiments, when performing multi-view off-screen rendering on the attribute view, layout parameters can first be obtained based on scene detection data. These layout parameters include point cloud bounding boxes and the center of the road scene. Then, a virtual camera layout is set according to the layout parameters, and multi-view off-screen rendering is performed on the attribute view based on the virtual camera layout to obtain a multi-view 2D rendered image. The attribute view contains a pseudo-color encoded 3D point cloud, and then a 2D visual encoder is used to extract the 2D semantic features of the multi-view 2D rendered image.

[0044] For example, multi-view off-screen rendering can be performed on pseudo-color encoded point clouds, and multi-view 2D semantic features can be extracted using a 2D visual encoder. When performing virtual multi-view off-screen rendering, virtual camera parameters can be determined based on the bounding box of the city road point cloud, the scene center, and preset viewing directions. For instance, one bird's-eye view, two 45-degree oblique views, and two 15-degree low-angle views can be set, and 10 pseudo-color rendered images can be generated by combining two types of pseudo-color material views. When extracting multi-view 2D semantic features, the multi-view 2D rendered images can be input into a 2D visual encoder to extract the multi-view 2D semantic features.

[0045] S103. Enhanced visual features are generated based on two-dimensional semantic features and three-dimensional point clouds through deep consistency verification and feature fusion.

[0046] After obtaining multi-view 2D rendered images and 2D semantic features, the multi-view 2D semantic features can be fused with 3D point cloud superpoint features based on depth consistency verification to obtain enhanced visual features. A superpoint is a set of points with similar geometric features obtained after super-segmenting the original point cloud; it can be used to reduce computational complexity, preserve geometric structure, and construct relationship graphs. The 3D point cloud superpoint features describe the geometric structure and attribute information of each superpoint. They can be aggregated from the original point information, such as coordinates and color within the superpoints, to form a more representative feature vector. Correspondingly, the 3D point cloud superpoint features are semantic features formed by aggregating 2D semantic features from multiple perspectives and then assigning them to superpoints based on their distribution relationships.

[0047] like Figure 4 As shown, to perform depth consistency verification, in some embodiments, when generating enhanced visual features based on 2D semantic features and 3D point clouds through depth consistency verification and feature fusion, the 3D point cloud can be projected onto a 2D image plane of a virtual viewpoint to obtain projected pixel coordinates and projection depth. Then, the projected pixel index of the source 3D point cloud is determined based on the projected pixel coordinates, and a projection depth buffer is constructed based on the projected pixel index and projection depth. Finally, depth consistency verification is performed according to the projection depth buffer to determine visible projection points.

[0048] For example, by projecting a 3D point onto a 2D viewpoint, the 3D point can be projected onto the 2D image plane of each virtual viewpoint, yielding the projected pixel coordinates and projection depth. Therefore, for any 3D point... The two-dimensional projection coordinates and projection depth of the k-th virtual camera are calculated based on its intrinsic and extrinsic parameters. The projection formula is as follows:

[0049]

[0050] in, and Representing a three-dimensional point The pixel coordinates in the k-th viewpoint image; and These represent the focal lengths of the k-th virtual camera in the horizontal and vertical directions, respectively. and Indicates the coordinates of the principal point; 、 、 This represents the coordinate components of a 3D point in the camera coordinate system after transformation by the k-th viewpoint extrinsic parameter.

[0051] Then, a depth consistency check is performed. This involves constructing a depth buffer based on the projected pixel indices and projection depths of the source 3D point cloud, and determining whether the projected coordinates are within the image range, whether the projection depth is positive, and whether the difference between the projection depth and the corresponding depth in the depth buffer is less than a preset depth threshold. Based on the depth consistency check results, visible projection points can be determined. Then, 2D semantic features corresponding to the visible projection points are sampled from multiple viewpoints, and feature fusion is performed based on these 2D semantic features to generate enhanced visual features.

[0052] During feature fusion, two-dimensional semantic features from multiple perspectives can first be aggregated into point-by-point multi-view semantic features, and superpoint assignment relationships and three-dimensional visual features of the three-dimensional point cloud can be obtained. The three-dimensional visual features are structural relationship features extracted from the three-dimensional point cloud. Then, superpoint-level multi-view semantic features are generated based on the superpoint assignment relationships and point-by-point multi-view semantic features, and these superpoint-level multi-view semantic features are mapped to an embedding space consistent with the three-dimensional visual features. Finally, the superpoint-level multi-view semantic features mapped to the embedding space are fused with the three-dimensional visual features to obtain enhanced visual features.

[0053] For example, for a visible projection point, two-dimensional semantic feature sampling and aggregation can be performed by sampling the two-dimensional semantic features of the corresponding viewpoint. That is, for any scan point P, its coordinates are (X, Y, Z), and the camera parameters for the frontal view include intrinsic parameter K and extrinsic parameter [R|t]. Then, using the camera parameters, the 3D point P can be projected onto the 2D image of the frontal view to obtain pixel coordinates (u, v). Then, on the 2D semantic feature map of the frontal view, bilinear interpolation sampling is performed based on the (u, v) position to obtain a C-dimensional feature vector. This yields the two-dimensional semantic features of the frontal view, denoted as f_P_front. Similarly, for a scan point P, the same operation is performed to sample from the side and top views, obtaining the two-dimensional semantic features of the corresponding side and top views, namely f_P_side and f_P_top.

[0054] The multi-view sampling features are then aggregated into point-by-point multi-view semantic features, followed by super-point fusion. This involves aggregating the point-by-point multi-view semantic features to a super-point level based on super-point allocation relationships, and then fusing them with 3D visual features to obtain enhanced visual features. For the three different viewpoints of the two-dimensional semantic feature vectors of the scanning point P, aggregation strategies such as vector merging, averaging, or weighted summation can be used to form point-by-point multi-view semantic features, i.e., F_point = [f_P_front, f_P_side, f_P_top]. Point-by-point multi-view semantic features can integrate the appearance and semantic information of an object from various angles.

[0055] Then, superpoints are used to package information, forming superpoint-level multi-view semantic features. That is, for the multi-view semantic feature F_point of each scan point, multiple superpoints, such as S1, can be divided according to the pre-calculated superpoint partition. For a superpoint S1, it can contain 100 points. By performing operations such as averaging or max pooling on the multi-view features F_point of each of these 100 points, a multi-view semantic feature vector of each superpoint is generated, denoted as F_superpoint.

[0056] Next, feature fusion is performed, using independent 3D backbone networks such as PointNet++ and KPConv to extract 3D visual features from the original 3D point cloud. These 3D visual features encode information such as the geometric structure and location of the point cloud. Similarly, these 3D visual features are also superpoint level, denoted as G_superpoint, with a dimension of C2. Then, a multilayer perceptron (MLP) is used to map (linearly transform) F_superpoint to the same embedding space as G_superpoint, resulting in F'_superpoint, whose dimension becomes [C2]. Finally, the mapped 2D features are combined with the original 3D features through element-wise addition or by concatenating features before using an MLP, yielding the final enhanced visual features.

[0057] S104. Generate sparse queries based on the indicated text and enhanced visual features.

[0058] After generating enhanced visual features through feature fusion, sparse queries can be generated based on the indexed text and the enhanced visual features. The sparse queries are obtained by text-weighted semantic farthest point sampling and multi-granularity filtering based on the semantic relevance scores of the indexed text and the enhanced visual features.

[0059] To generate sparse queries, it is necessary to calculate semantic relevance scores based on natural language descriptions and enhanced visual features, and generate sparse queries through text-weighted semantic farthest point sampling and multi-granularity filtering at the word and sentence levels.

[0060] like Figure 5As shown, in some embodiments, when generating sparse queries based on the indicated expression text and enhanced visual features, the indicated expression text can first be input into a text encoder to generate text features of the indicated expression text. These text features include word-level text features and sentence-level text embeddings. Then, semantic relevance scores for each superpoint are calculated based on the enhanced visual features and text features. These semantic relevance scores are then injected with exponential weights into a spatial distance metric to construct a semantically weighted distance. Finally, multi-granularity filtering is performed based on the semantically weighted distance to obtain the sparse query.

[0061] For example, the segmentation system can perform text encoding on natural language descriptions. By inputting the natural language descriptions into a text encoder, text feature extraction is performed to obtain word-level text features and sentence-level text embeddings. Semantic relevance estimation is then performed, calculating the semantic relevance score between each superpoint and the natural language description based on enhanced visual features and text features. The semantic relevance score can be obtained by mapping the superpoints and text to the same semantic alignment space and then calculating similarity. For a superpoint S and a text description Q, the enhanced visual features f_v of superpoint S can be input into a lightweight MLP's visual projector head to map the 3D visual features to a semantic space aligned with the text features, obtaining an aligned visual feature vector V. Simultaneously, a pre-trained text encoder is used to encode the sentence of the text description Q into a fixed-dimensional vector, obtaining a text feature vector T. Then, the similarity between the visual feature vector V and the text feature vector T is calculated, such as through cosine similarity, dot product operations, or a learnable similarity head, to calculate the semantic relevance score.

[0062] Then, text-weighted semantic farthest point sampling is performed. That is, to avoid blindly allocating queries in a large road background using ordinary farthest point sampling, the semantic relevance score can be injected into the spatial distance metric with exponential weights to construct a semantically weighted distance, i.e.:

[0063] in, The semantically weighted distance between the i-th and j-th superpoints; and The coordinates of the superpoint; Hyperparameters for adjusting semantic guidance strength; This represents the semantic relevance score between candidate superpoint j and the natural language description.

[0064] Based on the calculated semantic weighted distance, multi-granularity filtering can be performed to obtain sparse queries. Specifically, relevance is calculated based on the semantic weighted distance, including word-level and sentence-level relevance. Seed queries are then generated based on the indicated text, and local geometric enhancement is applied to the seed queries to obtain candidate queries. Finally, by combining word-level and sentence-level relevance, sparse queries are selected from the candidate queries.

[0065] For example, semantic relevance scores can be injected into the distance metric of farthest point sampling, text-weighted semantic farthest point sampling can be performed to obtain seed queries, and then multi-granularity filtering can be performed to perform local geometric enhancement on the seed queries. Finally, word-level relevance and sentence-level relevance can be combined to select the top K sparse queries from the candidate queries. In this case, the number of candidate queries is 256, the number of neighborhoods is 8, and K is 128.

[0066] By injecting semantic relevance scores into farthest point sampling (FPS) and transforming it into text-weighted distance sampling, we can calculate the geometric center coordinates {P_1, P_2, ..., P_N} of N superpoints. Combining this with the semantic score {R_1, R_2, ..., R_N} of each superpoint, a weighted distance is defined, assigning lower effective distances to superpoints with high semantic scores, making them more likely to be selected. Then, sampling is performed, selecting the superpoint S_i with the highest global semantic score as the first point. The weighted distances from all remaining points to S_i are calculated. For example, S_j also has a high semantic score and a moderate geometric distance, but by calculating the weighted distance, it can be selected as the second point. This sampling operation is repeated until M points are selected as seed queries.

[0067] Then, multi-granularity filtering is used to remove false positives from the seed query that may contain semantically high scores but are actually irrelevant. Therefore, verification and filtering can be performed at two granularities: sentence-level relevance and word-level relevance. Sentence-level relevance measures the relevance of the superpoint to the entire sentence; while word-level relevance is obtained by extracting multiple keywords from the sentence and calculating the relevance score of each superpoint to other keywords.

[0068] By setting filtering conditions, such as retaining a superpoint only if it has a high sentence-level score and matches at least one core word, the filtering conditions are applied to each of the M seed queries to determine whether the superpoints corresponding to the seed queries are retained or removed. Through multi-granular filtering, a certain number of superpoints can be retained from the M seed queries to more accurately match all the details of the text description.

[0069] Then, local geometric enhancement is performed on the retained superpoints to supplement the surrounding structural information for the multiple retained seed queries, forming a query vector. During local geometric enhancement, multiple superpoints with the closest geometric distance to the retained seed query superpoint in 3D space are identified as neighboring superpoints. Then, the enhanced visual features of these neighboring superpoints are aggregated using feature aggregation methods such as average pooling, max pooling, and attention mechanisms. A locally geometrically enhanced query feature, i.e., a candidate query Q_enhanced, is generated for each seed query.

[0070] After obtaining the enhanced candidate queries, a comprehensive score is calculated by weighting and summing the sentence-level and word-level scores. The candidate queries are then sorted according to the comprehensive score to obtain a candidate query sequence. Finally, the top K queries are selected from the candidate query sequence as sparse queries for output.

[0071] S105. Based on the text query attention entropy, inject language hints into the sparse query in a residual manner to obtain the segmentation result.

[0072] After obtaining the sparse query, language cues can be injected into the sparse query in a residual manner based on the text query attention entropy to obtain the segmentation result. The segmentation result includes the target segmentation mask and the target presence probability.

[0073] To obtain the segmentation result, the segmentation system can generate a reliability gating scalar based on the "text-query" attention entropy, inject language hints into the sparse query in a residual manner, and then use the segmentation model built into the segmentation system to output the target segmentation mask and the target existence probability.

[0074] In some embodiments, when injecting language cues into sparse queries in a residual manner based on text query attention entropy to obtain segmentation results, the text attention distribution corresponding to each query in the sparse query can be calculated first, and then the text query attention entropy can be calculated based on the text attention distribution. The text query attention entropy is a normalized information entropy used to measure the concentration and reliability of the language cues. Then, a gating scalar is generated based on the text query attention entropy, and the language cues are injected into the query features corresponding to the sparse query after residual mapping according to the gating scalar.

[0075] During decoding, normalized information entropy can be calculated based on the attention distribution between text features and sparse queries. A reliability gating scalar is then generated based on the normalized information entropy, increasing the weight of language cues when the attention distribution is concentrated and decreasing the weight when the attention distribution is dispersed. The language cues are then multiplied by the reliability gating scalar after residual mapping and added to the corresponding query features.

[0076] For example, after generating sparse queries, text-query attention computation can be performed, that is, calculating the text attention distribution corresponding to each query during the decoding process. Then, attention entropy reliability estimation is performed, calculating the normalized information entropy on the text attention distribution to measure whether the language prompts are focused and reliable. The formula for calculating the normalized information entropy is as follows:

[0077] in, This represents the attention distribution entropy value for the j-th query on the i-th word. This represents the total number of positions in the attention distribution. The attention weight for the j-th query at the i-th word; The number of valid lexical units; For example, the numerical stability constant, =10 -8 .

[0078] Then, residual-gated cue injection is performed based on the normalized information entropy. When the attention distribution is concentrated, the normalized information entropy H... i The lower the value, the higher the reliability of the corresponding prompts, and the larger the gating scalar; when the attention distribution is dispersed, the normalized information entropy H... i The higher the value, the more noisy the corresponding prompt becomes, and the smaller the gating scalar is reduced. The language prompt is then injected into the query features after residual mapping, according to the gating scalar.

[0079] After obtaining the query features with injected language prompts, these features can be input into a segmentation model for segmentation operations to obtain the segmentation result. For example, the segmentation model could be a visual segmentation model such as Mask2Former, QueryInst, or Point Cloud based on Transformer. After inputting the query features, the segmentation model can then proceed to a mask prediction branch and an existence probability branch. The mask prediction branch maps the query features to pixel-level or point-level segmentation outputs. In this branch, each query feature can be matched with features at all visual locations, calculating point-by-point similarity. Alternatively, dynamic convolution or masking mechanisms can be used. The query features are input into a Multilayer Perceptron (MLP), generating dynamic convolution kernel parameters and convolving the visual feature map to directly output the corresponding mask logits. Then, a Sigmoid function is applied to obtain the final mask probability.

[0080] The existence probability branch is used to determine whether each query truly corresponds to a valid target. In the existence probability branch, a lightweight MLP can be used to map the query features to a scalar and output the existence probability through the Sigmoid function.

[0081] By applying the technical solutions of the above embodiments, the 3D point cloud indexing and representation segmentation method described in the above embodiments can improve the performance of 3D point cloud indexing and representation segmentation for urban roads by introducing a cross-modal interaction mechanism guided by 2D geometric priors and language. Thanks to depth consistency verification, text-weighted sparse query generation, and reliability-aware prompt decoding mechanism, the method can accurately capture sparse targets such as trees, vehicles, pedestrians, poles, and power lines as indicated by natural language descriptions in road scenes with large scale, strong occlusion, and a high proportion of background point clouds, thereby ensuring the accuracy and completeness of the segmentation results. For example, experimental results on the UrbanRefer dataset show that the method can achieve a mean Intersection over Union (mIoU) ratio of 47.9%, which is better than the 45.1% mIoU ratio of the Image-enhanced Prompt Decoding Network (IPDN).

[0082] The method also addresses issues such as 2D projection occlusion, background query redundancy, and language prompt noise propagation in complex road scenarios. The introduced deep consistency check reduces semantic mismatches caused by occlusion points and incorrect projection points, while text-weighted semantic sampling guides queries to prioritize target-related regions. Simultaneously, the reliability-aware prompt decoder suppresses low-quality language prompts when attention distribution is scattered, improving Acc@0.5 by 4.3 percentage points in multi-target scenarios.

[0083] Therefore, the method not only performs excellently in terms of objective indicators, but also possesses significant engineering application value. This method can be widely applied to outdoor road scenarios such as autonomous driving, high-precision map production, urban road inspection, road asset management, and mobile mapping, providing high-precision and robust technical support for human-computer interactive 3D target segmentation in complex urban road environments.

[0084] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a three-dimensional point cloud finger representation segmentation method. The difference between this method and the above embodiments is that result filtering can be performed when generating the segmentation results, such as... Figure 6 As shown, the method includes: S201. Input the query features with injected language prompts into the segmentation model so that the segmentation model can output the target segmentation mask and the probability of the target's existence. S202, Obtain the preset probability threshold; S203. When the probability of the target's existence is higher than or equal to a preset probability threshold, generate a segmentation result based on the target segmentation mask and the probability of the target's existence. S204. When the probability of the existence of all candidate targets is lower than the preset probability threshold, generate the empty target segmentation result.

[0085] To obtain the segmentation result, language cues are injected into the sparse query using a residual method based on the text query attention entropy. First, the query features with injected language cues are input into the segmentation model, which then outputs a target segmentation mask and the target presence probability. A preset probability threshold is then obtained, and the target presence probability output by the segmentation model is compared with this threshold. If the target presence probability is higher than or equal to the preset threshold, a normal segmentation result is output, i.e., a segmentation result is generated based on the target segmentation mask and the target presence probability. However, if the target presence probability of all candidate targets is lower than the preset probability threshold, an empty target segmentation result is generated.

[0086] For example, when performing segmentation output, query features with injected language hints can be input into the segmentation model. This allows the segmentation model to map the query features to a scalar using a lightweight MLP, and then output the probability Pe of the existence of candidate targets via a sigmoid function. For a preset probability threshold Pt, the probability Pe of the existence of a target can be compared with the preset threshold. When the probability of the existence of all candidate targets is lower than the preset threshold, an empty target result is output. This is achieved when a corresponding target does not exist in the natural language description.

[0087] By applying the technical solutions of the above embodiments, the 3D point cloud indexing and expression segmentation method described in the above embodiments can filter results based on a preset probability threshold when generating segmentation results, thereby selecting candidate targets with a higher probability of existence as segmentation results. When the probability of existence of all candidate targets is lower than the preset threshold, an empty target result is output, realizing the output of an empty target result when there is no corresponding target in the natural language description. The method can improve the effectiveness of segmentation results, increase the adaptability of segmentation results, and enhance the processing performance of 3D point cloud indexing and expression segmentation for urban roads.

[0088] In some embodiments, as a specific implementation of the three-dimensional point cloud pointer expression segmentation method described in the above embodiments, some embodiments of this application also provide a three-dimensional point cloud pointer expression segmentation system, such as... Figure 7 As shown, the system includes: The data acquisition module is used to acquire scene detection data and index expression text. The scene detection data includes 3D point cloud and radar attribute information collected for urban road scenes; the index expression text is natural language description text. The encoding and rendering module is used to construct an attribute view based on the scene detection data through pseudo-color encoding, and to obtain a multi-view two-dimensional rendered image and two-dimensional semantic features by performing multi-view off-screen rendering on the attribute view; the attribute view includes a radiation attribute view and a structural attribute view. The semantic fusion module is used to generate enhanced visual features based on the two-dimensional semantic features and the three-dimensional point cloud through deep consistency verification and feature fusion. The query generation module is used to generate a sparse query based on the indicated text and the enhanced visual features. The sparse query is obtained by text-weighted semantic farthest point sampling and multi-granularity filtering based on the semantic relevance score between the indicated text and the enhanced visual features. The segmentation output module is used to inject language cues into the sparse query in a residual manner based on the text query attention entropy to obtain the segmentation result, which includes the target segmentation mask and the target existence probability.

[0089] By applying the technical solutions of the above embodiments, the 3D point cloud indexing expression segmentation system provided in the above embodiments can, after the data acquisition module acquires scene detection data and indexing expression text, the encoding and rendering module performs pseudo-color encoding to construct an attribute view, and obtain multi-view 2D semantic features by performing multi-view off-screen rendering on the attribute view. The semantic fusion module then generates enhanced visual features based on the 2D semantic features and 3D point cloud through depth consistency verification and feature fusion. Then, the query generation module generates sparse queries according to the indexing expression text and enhanced visual features, so that the segmentation output module can inject language prompts into the sparse queries in a residual manner according to the text query attention entropy to obtain the segmentation result. The system improves the processing performance of 3D point cloud indexing expression segmentation for urban roads by introducing a cross-modal interaction mechanism of 2D geometric prior and language guidance to realize depth consistency verification, text-weighted sparse query generation, and reliability-aware prompt decoding.

[0090] It should be noted that other corresponding descriptions of the functional units involved in the three-dimensional point cloud indexing expression segmentation system provided in the embodiments of this application can be found in the corresponding descriptions in the three-dimensional point cloud indexing expression segmentation method provided in the above embodiments, and will not be repeated here.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A three-dimensional point cloud indexing and representation segmentation method, characterized in that, The method includes: Acquire scene detection data and instruction expression text. The scene detection data includes 3D point cloud and radar attribute information collected for urban road scenes; the instruction expression text is natural language description text. Based on the scene detection data, an attribute view is constructed through pseudo-color encoding, and multi-view off-screen rendering is performed on the attribute view to obtain a multi-view two-dimensional rendered image and two-dimensional semantic features; the attribute view includes a radial attribute view and a structural attribute view. Enhanced visual features are generated based on the two-dimensional semantic features and the three-dimensional point cloud through deep consistency verification and feature fusion. A sparse query is generated based on the indicated text and the enhanced visual features. The sparse query is obtained by text-weighted semantic farthest point sampling and multi-granularity filtering based on the semantic relevance scores of the indicated text and the enhanced visual features. Based on the text query attention entropy, language cues are injected into the sparse query in a residual manner to obtain the segmentation result, which includes the target segmentation mask and the target existence probability.

2. The three-dimensional point cloud representation segmentation method according to claim 1, characterized in that, Based on the scene detection data, an attribute view is constructed using pseudo-color encoding, including: The three-dimensional point cloud and radar attribute information are extracted from the scene detection data. The radar attribute information includes radiation attribute information and structural attribute information. The radiation attribute information includes at least one of intensity, reflectivity, and amplitude. The structural attribute information includes at least one of height, echo count, and deviation. The radiation attribute information is encoded into the color channel of the radiation attribute view, and the structural attribute information is encoded into the color channel of the structural attribute view.

3. The three-dimensional point cloud representation segmentation method according to claim 1, characterized in that, By performing multi-view off-screen rendering on the attribute view, multi-view 2D rendered images and 2D semantic features are obtained, including: Layout parameters are obtained based on the scene detection data, including point cloud bounding boxes and road scene centers; Set the virtual camera layout according to the layout parameters; Based on the virtual camera layout, the attribute view is rendered off-screen from multiple perspectives to obtain a multi-view 2D rendered image; the attribute view contains a 3D point cloud with pseudo-color encoding. Two-dimensional semantic features of the multi-view two-dimensional rendered image are extracted using a two-dimensional visual encoder.

4. The three-dimensional point cloud representation segmentation method according to claim 1, characterized in that, Enhanced visual features are generated based on the two-dimensional semantic features and the three-dimensional point cloud through deep consistency verification and feature fusion, including: The three-dimensional point cloud is projected onto a two-dimensional image plane from a virtual perspective to obtain the projected pixel coordinates and projection depth; The projection pixel index of the homogeneous 3D point cloud is determined based on the projection pixel coordinates; Construct a projection depth buffer based on the projection pixel index and the projection depth; Perform a depth consistency check according to the projection depth buffer to determine the visible projection points; The visible projection points are sampled to correspond to the two-dimensional semantic features from multiple viewpoints, and feature fusion is performed based on the two-dimensional semantic features from multiple viewpoints to generate enhanced visual features.

5. The three-dimensional point cloud representation segmentation method according to claim 4, characterized in that, Feature fusion is performed based on the two-dimensional semantic features from multiple perspectives to generate enhanced visual features, including: The two-dimensional semantic features from multiple perspectives are aggregated into point-by-point multi-perspective semantic features; Obtain the superpoint assignment relationship and the three-dimensional visual features of the three-dimensional point cloud, wherein the three-dimensional visual features are structural relationship features extracted based on the three-dimensional point cloud; Generate super-point level multi-view semantic features based on the super-point allocation relationship and the point-by-point multi-view semantic features, and map the super-point level multi-view semantic features to an embedding space consistent with the three-dimensional visual features; The enhanced visual features are obtained by fusing the super-point-level multi-view semantic features mapped to the embedding space with the three-dimensional visual features.

6. The three-dimensional point cloud indexing and expression segmentation method according to claim 1, characterized in that, Generate a sparse query based on the indicated text and the enhanced visual features, including: The indicated text is input into a text encoder to generate text features of the indicated text, which include word-level text features and sentence-level text embeddings. Calculate the semantic relevance score of each superpoint based on the enhanced visual features and the text features; The semantic relevance score is injected with an exponential weight into the spatial distance metric to construct a semantically weighted distance. Multi-granularity filtering is performed based on the semantically weighted distance to obtain sparse queries.

7. The three-dimensional point cloud representation segmentation method according to claim 6, characterized in that, Multi-granularity filtering is performed based on the semantically weighted distance to obtain sparse queries, including: The relevance is calculated based on the semantically weighted distance, and the relevance includes word-level relevance and sentence-level relevance; Generate a seed query based on the indicated text; The seed query is augmented with local geometry to obtain candidate queries; Based on the combined word-level relevance and sentence-level relevance, a sparse query is selected from the candidate queries.

8. The three-dimensional point cloud representation segmentation method according to claim 1, characterized in that, Based on the text query attention entropy, language cues are injected into the sparse query in a residual manner to obtain segmentation results, including: Calculate the text attention distribution corresponding to each query in the sparse query; The text query attention entropy is calculated based on the text attention distribution, and the text query attention entropy is a normalized information entropy used to measure the concentration and reliability of language prompts; A gating scalar is generated based on the text query attention entropy; According to the gating scalar, the language prompts are injected into the query features corresponding to the sparse query after residual mapping.

9. The three-dimensional point cloud representation segmentation method according to claim 8, characterized in that, Based on the text query attention entropy, language cues are injected into the sparse query in a residual manner to obtain segmentation results, including: The query features with injected language hints are input into the segmentation model, and the segmentation model outputs the target segmentation mask and the target existence probability. Obtain the preset probability threshold; When the probability of the target's existence is higher than or equal to the preset probability threshold, the segmentation result is generated based on the target segmentation mask and the target's existence probability. When the probability of the existence of all candidate targets is lower than the preset probability threshold, an empty target segmentation result is generated.

10. A three-dimensional point cloud indexing and segmentation system, characterized in that, The system includes: The data acquisition module is used to acquire scene detection data and index expression text. The scene detection data includes 3D point cloud and radar attribute information collected for urban road scenes; the index expression text is natural language description text. The encoding and rendering module is used to construct an attribute view based on the scene detection data through pseudo-color encoding, and to obtain a multi-view two-dimensional rendered image and two-dimensional semantic features by performing multi-view off-screen rendering on the attribute view; the attribute view includes a radiation attribute view and a structural attribute view. The semantic fusion module is used to generate enhanced visual features based on the two-dimensional semantic features and the three-dimensional point cloud through deep consistency verification and feature fusion. The query generation module is used to generate a sparse query based on the indicated text and the enhanced visual features. The sparse query is obtained by text-weighted semantic farthest point sampling and multi-granularity filtering based on the semantic relevance score between the indicated text and the enhanced visual features. The segmentation output module is used to inject language cues into the sparse query in a residual manner based on the text query attention entropy to obtain the segmentation result, which includes the target segmentation mask and the target existence probability.

Images