Reinforcement learning driven scalable 3D panorama segmentation method and system based on clustering

Abstract

This invention relates to computer vision and point cloud processing technology, specifically a scalable 3D panoramic segmentation method and system based on reinforcement learning-driven clustering. The method includes the following steps: dividing the input point cloud data into semantic voxels to construct an initial super-point graph; constructing a high-level meta-controller and a low-level controller; modeling panoramic segmentation as a sequential graph clustering problem and iteratively processing the initial super-point graph; evaluating the current super-point subgraph using the high-level meta-controller, extracting global feature embeddings, and outputting the probability of whether to terminate segmentation or continue splitting the current super-point subgraph; if the decision is to terminate segmentation, using the current super-point subgraph as the final instance; if the decision is to continue splitting, traversing the edges in the super-point subgraph to be split using the low-level controller and updating the subgraph structure; and employing a near-end policy optimization algorithm with a hybrid reward function for optimization. This invention reconstructs graph clustering into a multi-step sequential decision-making process, effectively solving the problems of under-segmentation and boundary ambiguity in large-scale point clouds.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and point cloud processing technology, specifically to a scalable 3D panoramic segmentation method and system based on reinforcement learning-driven clustering. Background Technology

[0002] Large-scale 3D panoptic segmentation requires scalable models to handle massive point cloud data, predict fine-grained semantics, and recover all instances with highly diverse spatial scales. Despite some progress in recent years, the vast majority of existing panoptic segmentation methods still rely on heavy backbone networks and single graph optimization algorithms. This greedy clustering or one-step graph optimization approach limits the scalability of the algorithm and often leads to undersegmentation when dealing with objects with fuzzy boundaries.

[0003] Traditional graph clustering methods typically determine the connectivity of all edges at once, making it difficult to adapt to the hierarchical structure of 3D scenes: large objects should first be separated at a coarse level, followed by fine-grained refinement in blurred boundary regions. Furthermore, existing methods struggle to leverage long-term perspectives to optimize decisions, failing to account for the cumulative impact of current edge decisions on the final panoramic quality. Therefore, a novel technical solution is urgently needed that can adaptively handle objects of different scales and address boundary blurring issues in complex scenes. Summary of the Invention

[0004] This invention provides a scalable 3D panoramic segmentation method based on reinforcement learning-driven clustering. It reconstructs graph clustering into a multi-step sequential decision-making process, utilizes a hierarchical reinforcement learning framework, generates high-quality initial superpoints through semantic voxel partitioning, and uses a two-layer surrogate model to iteratively refine the segmentation results, thereby achieving accurate panoramic segmentation of entities in large-scale point cloud scenarios.

[0005] On one hand, embodiments of the present invention provide a scalable 3D panoramic segmentation method based on reinforcement learning-driven clustering, comprising the following steps:

[0006] S1. Divide the input point cloud data into semantic voxels, generate an initial set of superpoints, and construct an initial superpoint graph;

[0007] S2. Construct a hierarchical reinforcement learning agent, which includes a high-level meta-controller and a low-level controller; model 3D panoramic segmentation as a sequential graph clustering problem, and use the hierarchical reinforcement learning agent to iteratively process the initial hypergraph;

[0008] S3. The current super-node subgraph is evaluated using a high-level meta controller. Global feature embeddings are extracted using a graph encoder, and the probability of whether the current super-node subgraph should terminate segmentation or continue splitting is output. If the decision is to terminate segmentation, the current super-node subgraph is taken as the final instance. If the decision is to continue splitting, proceed to step S4.

[0009] S4. Use the low-level controller to traverse the edges in the super-node subgraph that needs to be split, predict the "cut off" or "keep" action based on the node features and semantic probability of the edges, update the subgraph structure based on the action result, and reconstruct the generated object instance.

[0010] S5. During the training of the hierarchical reinforcement learning agent, a proximal policy optimization algorithm is adopted, and a hybrid reward function is used for optimization: the hybrid reward function includes intrinsic reward and extrinsic reward; the intrinsic reward encourages correct edge decisions based on the affinity of the true value; the extrinsic reward is used to evaluate the global improvement and enhancement of the panoptic segmentation quality.

[0011] On the other hand, embodiments of the present invention also provide a scalable 3D panoramic segmentation system based on reinforcement learning-driven clustering, for implementing the above-mentioned scalable 3D panoramic segmentation method; the segmentation system includes:

[0012] The voxel segmentation module is used to divide the input point cloud data into semantic voxels, generate an initial hyperpoint set, and construct an initial hyperpoint graph;

[0013] The hierarchical reinforcement learning agent module includes a high-level meta-controller and a low-level controller. It models 3D panoramic segmentation as a sequential graph clustering problem, iteratively processes the initial super-point graph using the hierarchical reinforcement learning agent. The high-level meta-controller evaluates the current super-point subgraph, extracts global feature embeddings through a graph encoder, and outputs the probability of whether to terminate or continue segmentation of the current super-point subgraph. If the segmentation is terminated, the current super-point subgraph is used as the final instance. If the segmentation continues, the low-level controller traverses the edges in the super-point subgraph to be segmented, predicts "cut" or "keep" actions based on the node features and semantic probabilities of the edges, updates the subgraph structure based on the action results, and reconstructs and generates object instances.

[0014] The optimization module employs a proximal policy optimization algorithm during the training of the hierarchical reinforcement learning agent and uses a hybrid reward function for optimization: the hybrid reward function includes intrinsic rewards and extrinsic rewards; intrinsic rewards encourage correct edge decisions based on the affinity of the true values; extrinsic rewards are used to evaluate the global improvement and enhancement of panoptic segmentation quality.

[0015] Compared with the prior art, the advantages of the present invention include:

[0016] This invention addresses the problems of greediness, lack of long-range planning, and insufficient adaptability to scale differences in existing graph clustering methods when handling complex 3D scenes. It proposes a hierarchical reinforcement learning framework, which for the first time formulates 3D graph clustering as a learnable multi-step strategy, achieving scalable and adaptive instance segmentation. The introduced semantic voxel partitioning module generates robust, semantically aware superpoints, significantly enhancing the performance of subsequent hierarchical reinforcement learning decision-making processes. Through long-range credit assignment, the model not only optimizes immediate edge decisions but also their cumulative impact on the final panoramic quality, achieving finer instance-level segmentation and higher panoramic quality, particularly demonstrating significant advantages in resolving blurred boundaries and multi-scale object segmentation. Attached Figure Description

[0017] Figure 1 This is a flowchart of the scalable 3D panoramic segmentation method in an embodiment of the present invention. Detailed Implementation

[0018] In summary, this invention provides a scalable 3D panoramic segmentation method and system based on reinforcement learning-driven clustering. The method mainly includes the following steps: semantic voxel partitioning, which involves voxelizing the original point cloud and extracting features, merging similar voxels to generate an initial set of superpoints with semantic consistency, and constructing an initial superpoint graph; hierarchical reinforcement learning inference, which uses a two-layer surrogate model to iteratively refine the initial superpoint graph; the two-layer surrogate model includes a high-level meta-controller and a low-level controller; the meta-controller evaluates whether the current subgraph constitutes a complete and semantically consistent object to decide whether to terminate or continue segmentation; if further segmentation is needed, the low-level controller evaluates each edge in the subgraph and performs preservation or truncation operations to progressively segment object instances; the two-layer surrogate model is trained using a proximal policy optimization (PPO) algorithm combined with a hybrid reward mechanism, where the hybrid reward includes an intrinsic reward based on truth affinity and an extrinsic reward for evaluating global panoramic quality improvement. This invention reconstructs graph clustering into a multi-step sequential decision-making process, effectively solving the problems of under-segmentation and boundary ambiguity in large-scale point clouds, and achieving high-precision 3D panoramic segmentation.

[0019] The technical solution of the present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the implementation of the present invention is not limited thereto.

[0020] Example

[0021] This embodiment provides a scalable 3D panoramic segmentation method based on reinforcement learning-driven clustering, such as... Figure 1 As shown, it includes the following steps:

[0022] S1. Divide the input point cloud data into semantic voxels, generate an initial set of superpoints, and construct an initial superpoint graph.

[0023] To initialize a high-quality superpoint graph with strong priors, the original input point cloud data is first over-segmented using a semantic voxel partitioning module, converting the input point cloud data into voxels. Features for each voxel are then extracted using a pre-trained Minkowski UNet network. Highly similar adjacent voxels are merged to generate coherent and robust initial superpoints, resulting in a superpoint set S = {S_1, ..., S_M}. This superpoint set S is used as the nodes of the superpoint graph, and the adjacency relationships between superpoints are used as edges to construct the initial superpoint graph. Initial superpoints are defined as points with the same semantic label and similar spatial coordinates.

[0024] In this step, assume the input point cloud data is P={p_1,...p_i..., p_n}, where each point p_i has coordinates pos_i; given the voxel size v_s and the point-by-point semantic label l_i, the initial superpoint S_k is defined as:

[0025] ;

[0026] Where [·] represents the floor function, and key_k represents a group of points with the same spatial location and the same semantic category.

[0027] This step utilizes a pre-trained Minkowski UNet to extract voxel features, merges similar voxels, and generates a set of superpoints, which serve as nodes in the initial superpoint graph.

[0028] S2. Construct a hierarchical reinforcement learning agent, which includes a high-level meta-controller and a low-level controller; model 3D panoramic segmentation as a sequential graph clustering problem, and use the hierarchical reinforcement learning agent to iteratively process the initial hypergraph.

[0029] This step, given an initial hyperpoint graph, performs multi-step clustering, modeled as a Markov Decision Process (MDP) and solved using a two-layer hierarchical reinforcement learning agent. Specifically, the agent is divided into a high-level policy (meta-controller) and a low-level policy (controller), which will be described in detail in steps S3 and S4.

[0030] S3. Meta controller decision: The high-level meta controller evaluates the current super-node subgraph, extracts global feature embeddings through graph encoder, and outputs the probability of whether the current super-node subgraph terminates segmentation or continues splitting. If it is decided to terminate segmentation, the current super-node subgraph is taken as the final instance. If it is decided to continue splitting, proceed to step S4.

[0031] This step involves high-level policy decision-making. The meta-controller evaluates each hyper-point subgraph and decides whether to terminate (i.e., accept it as the final instance) or continue splitting; the graph encoder generates a global feature embedding, which the policy network uses to output the termination probability. This step naturally aligns with the hierarchical structure of a 3D scene.

[0032] High-level strategy (meta-controller): The meta-controller observes the super-subgraph s, extracts features through the graph encoder F_enc, and calculates the global feature embedding g_s of the super-subgraph.

[0033] ;

[0034] Where GMP(·) represents global average pooling, X_s and E_s represent the node features and edge features of the super-node subgraph s, respectively, and F_enc is the graph encoder.

[0035] This step calculates the probability that the high-level element controller will decide on an action (i.e., terminate the segmentation):

[0036] ;

[0037] Where F_meta is the policy network of the higher-level meta-controller, and σ(·) is the Sigmoid activation function. If the calculated probabilities... If the value exceeds a preset threshold, the corresponding super-node subgraph is considered a complete instance and processing stops; otherwise, it is passed to the lower-level controller.

[0038] S4. Refine the low-level controller. Use the low-level controller to traverse the edges in the super-node subgraph that needs to be split. Predict the "cut off" or "keep" action based on the node characteristics and semantic probability of the edge. Update the subgraph structure based on the action result and reconstruct the generated object instance.

[0039] This step refines the low-level policy. If further refinement is needed (i.e., the higher-level meta-controller decides to continue splitting), the low-level controller examines each edge in the super-node subgraph and predicts the "cut" or "keep" action. Edge states are constructed based on node features and semantic probabilities, and action probabilities are output through the low-level policy network.

[0040] Low-level policy (controller): When refinement is needed, the low-level controller examines each edge e_ij in the super-vertex subgraph. The action decision-making process of the low-level controller includes:

[0041] By combining geometric and semantic features, the edge state s_ij is constructed:

[0042] ;

[0043] Where x_i and x_j are node features, and p_i and p_j are semantic probabilities.

[0044] Calculate the probability of retaining the edge:

[0045] ;

[0046] Where σ(·) represents the Sigmoid activation function, and F_low is the policy network of the lower-level controller.

[0047] S5. During the training of the hierarchical reinforcement learning agent, the Proximal Policy Optimization (PPO) algorithm is adopted, and a hybrid reward function is used for optimization: the hybrid reward function includes intrinsic reward and extrinsic reward; the intrinsic reward encourages correct edge decisions based on the affinity of the true value; the extrinsic reward is used to evaluate the global improvement and enhancement of the panoptic segmentation quality (RQ).

[0048] In this embodiment, a proximal policy optimization algorithm is employed to train the hierarchical reinforcement learning agent, and a hybrid reward mechanism is designed. The intrinsic reward is based on the affinity of truth values, encouraging the correctness of each action; the extrinsic reward is used to evaluate the global improvement in the final segmentation quality.

[0049] Define the extrinsic reward R_e as:

[0050] ;

[0051] Where RQ(·) represents the panoramic segmentation quality, P_final is the final segmentation result after proxy processing, and P_base is the baseline partitioning result before processing.

[0052] The PPO objective function used in this step for:

[0053] ;

[0054] Where E_t is the policy ratio of the empirical expectation r_t(θ) at time step t, and A_t is the advantage estimate at time step t; This represents the ratio of the probability of the current policy choosing action a_t to the probability of the old policy choosing the same action a_t. This represents the shearing parameter used to control the magnitude of policy updates.

[0055] Based on the same inventive concept, this embodiment also provides a scalable 3D panoramic segmentation system based on reinforcement learning-driven clustering, used to implement the above-mentioned scalable 3D panoramic segmentation method. The segmentation system of this embodiment includes:

[0056] The voxel segmentation module is used to divide the input point cloud data into semantic voxels, generate an initial hyperpoint set, and construct an initial hyperpoint graph;

[0057] The hierarchical reinforcement learning agent module includes a high-level meta-controller and a low-level controller. It models 3D panoramic segmentation as a sequential graph clustering problem, iteratively processes the initial super-point graph using the hierarchical reinforcement learning agent. The high-level meta-controller evaluates the current super-point subgraph, extracts global feature embeddings through a graph encoder, and outputs the probability of whether to terminate or continue segmentation of the current super-point subgraph. If the segmentation is terminated, the current super-point subgraph is used as the final instance. If the segmentation continues, the low-level controller traverses the edges in the super-point subgraph to be segmented, predicts "cut" or "keep" actions based on the node features and semantic probabilities of the edges, updates the subgraph structure based on the action results, and reconstructs and generates object instances.

[0058] The optimization module employs a proximal policy optimization algorithm during the training of the hierarchical reinforcement learning agent and uses a hybrid reward function for optimization: the hybrid reward function includes intrinsic rewards and extrinsic rewards; intrinsic rewards encourage correct edge decisions based on the affinity of the true values; extrinsic rewards are used to evaluate the global improvement and enhancement of panoptic segmentation quality.

[0059] The specific implementation process of each of the above modules can be referred to the corresponding steps S1-S5 of the scalable 3D panoramic segmentation method in this embodiment.

[0060] Through the above methods, this invention enables accurate instance segmentation of complex scenes (such as buildings, trees, and vehicles) in large-scale point clouds. This invention demonstrates significant technical advantages in handling large-scale complex 3D scenes by introducing a sequential decision-making mechanism through hierarchical reinforcement learning, which significantly improves the under-segmentation problem. This mechanism allows the model to more accurately identify and perform separation operations when faced with spatially close but semantically different objects, such as closely arranged vehicles or trees adjacent to a wall, thereby greatly reducing the probability of incorrectly merging multiple independent instances into a single object.

[0061] Meanwhile, thanks to the edge-by-edge refinement strategy of the low-level controller, this invention achieves more refined fuzzy boundary segmentation. At object edges, the model combines global contextual information and long-term reward feedback to generate boundaries that better fit the contours of real objects, significantly improving the geometric fidelity of panoramic segmentation. Furthermore, by combining high-quality initialization and iterative optimization with semantic voxel segmentation, this invention enhances its adaptability to multi-scale targets. When faced with extreme scale variations, such as large buildings and small pedestrians appearing in the same scene, it ensures the integrity of large-scale objects while preventing the neglect of small-scale objects. Finally, by introducing extrinsic rewards based on panoramic quality, the model learns to optimize the segmentation results from a global perspective, resulting in an output 3D panoramic map that effectively improves upon existing technologies in terms of instance completeness and category accuracy. This demonstrates the significant effectiveness of this method in solving the robustness problem of panoramic segmentation in complex scenes.

[0062] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. All equivalent changes made in accordance with the methods and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A scalable 3D panoramic segmentation method based on reinforcement learning-driven clustering, characterized in that, Includes the following steps: S1. Divide the input point cloud data into semantic voxels, generate an initial set of superpoints, and construct an initial superpoint graph; S2. Construct a hierarchical reinforcement learning agent, which includes a high-level meta-controller and a low-level controller; model 3D panoramic segmentation as a sequential graph clustering problem, and use the hierarchical reinforcement learning agent to iteratively process the initial hypergraph; S3. Use the high-level meta controller to evaluate the current super-node subgraph, extract global feature embeddings through the graph encoder, and output the probability of whether the current super-node subgraph terminates segmentation or continues to split. If it is decided to terminate the split, the current super-node subgraph is taken as the final instance; if it is decided to continue splitting, proceed to step S4. S4. Use the low-level controller to traverse the edges in the super-node subgraph that needs to be split, predict the "cut" or "keep" action based on the node features and semantic probabilities of the edges, update the subgraph structure based on the action results, and reconstruct the generated object instance. S5. During the training of the hierarchical reinforcement learning agent, a proximal policy optimization algorithm is adopted, and a hybrid reward function is used for optimization: the hybrid reward function includes intrinsic reward and extrinsic reward; the intrinsic reward encourages correct edge decisions based on the affinity of the true value; the extrinsic reward is used to evaluate the global improvement and enhancement of the panoptic segmentation quality.

2. The scalable 3D panoramic segmentation method according to claim 1, characterized in that, Step S1 first performs oversegmentation on the original input point cloud data; the input point cloud data is voxelized, and the features of each voxel are extracted through a pre-trained Minkowski UNet network; adjacent voxels with high similarity are merged to generate a coherent and robust initial superpoint, thus obtaining a superpoint set S. The superpoint set S is used as the nodes of the superpoint graph, and the adjacency relationship between superpoints is used as the edge of the superpoint graph to construct the initial superpoint graph. The initial superpoint is defined as a point with the same semantic label and similar spatial coordinates.

3. The scalable 3D panoramic segmentation method according to claim 2, characterized in that, Let the input point cloud data be P={p_1,...p_i..., p_n}, where each point p_i has coordinates pos_i; given the voxel size v_s and the point-by-point semantic label l_i, the initial superpoint S_k is defined as: ； Where [·] represents the floor function, and key_k represents a group of points with the same spatial location and the same semantic category.

4. The scalable 3D panoramic segmentation method according to claim 1, characterized in that, In step S3, the high-level meta controller observes the super-subgraph, extracts features through the graph encoder, and calculates the global feature embedding g_s of the super-subgraph: ； Where GMP(·) represents global average pooling, X_s represents the node features of the super-subgraph s, E_s represents the edge features of the super-subgraph s, and F_enc is the graph encoder.

5. The scalable 3D panoramic segmentation method according to claim 4, characterized in that, In step S3, the probability of the higher-level element controller deciding on an action is calculated as follows: ； Where F_meta is the policy network of the high-level meta-controller, and σ(·) is the Sigmoid activation function; if the calculated probability If the value exceeds a preset threshold, the corresponding super-subgraph is considered a complete instance, and processing is stopped. Otherwise, pass it to the lower-level controller.

6. The scalable 3D panoramic segmentation method according to claim 1, characterized in that, In step S4, when the higher-level meta-controller decides to continue splitting, the lower-level controller checks each edge e_ij in the super-node subgraph; the action decision process of the lower-level controller includes: By combining geometric and semantic features, the edge state s_ij is constructed: ； Where x_i and x_j are node features, and p_i and p_j are semantic probabilities; Calculate the probability of retaining the edge: ； Where σ(·) represents the Sigmoid activation function, and F_low is the policy network of the lower-level controller.

7. The scalable 3D panoramic segmentation method according to claim 1, characterized in that, Step S5 defines the extrinsic reward R_e as: ； Where RQ(·) represents the panoramic segmentation quality, P_final is the final segmentation result after proxy processing, and P_base is the baseline partitioning result before processing.

8. A scalable 3D panoramic segmentation system based on reinforcement learning-driven clustering, used to implement the scalable 3D panoramic segmentation method according to any one of claims 1-7, characterized in that, The segmentation system includes: The voxel segmentation module is used to divide the input point cloud data into semantic voxels, generate an initial hyperpoint set, and construct an initial hyperpoint graph; The hierarchical reinforcement learning agent module includes a high-level meta-controller and a low-level controller. It models 3D panoramic segmentation as a sequential graph clustering problem, iteratively processes the initial super-point graph using the hierarchical reinforcement learning agent. The high-level meta-controller evaluates the current super-point subgraph, extracts global feature embeddings through a graph encoder, and outputs the probability of whether to terminate or continue segmentation of the current super-point subgraph. If the segmentation is terminated, the current super-point subgraph is used as the final instance. If the segmentation continues, the low-level controller traverses the edges in the super-point subgraph to be segmented, predicts "cut" or "keep" actions based on the node features and semantic probabilities of the edges, updates the subgraph structure based on the action results, and reconstructs and generates object instances. The optimization module employs a proximal policy optimization algorithm during the training of the hierarchical reinforcement learning agent and uses a hybrid reward function for optimization: the hybrid reward function includes intrinsic rewards and extrinsic rewards; intrinsic rewards encourage correct edge decisions based on the affinity of the true values; extrinsic rewards are used to evaluate the global improvement and enhancement of panoptic segmentation quality.

Images