Point cloud scene understanding segmentation method and device based on multi-modal reinforcement reasoning, equipment and storage medium

Abstract

The application provides a point cloud scene understanding segmentation method and device based on multi-modal reinforcement reasoning, equipment and storage medium, relating to the technical field of three-dimensional scene understanding and perception. The method comprises: establishing a point cloud scene understanding segmentation framework, which comprises a point cloud feature extraction module, a large language model, a projection layer and a context perception module; based on a first training strategy, the point cloud scene understanding segmentation framework is supervised and fine-tuned to obtain a supervised fine-tuning framework; based on a second training strategy, the supervised fine-tuning framework is subjected to reinforcement learning to obtain a point cloud scene understanding segmentation framework after reinforcement learning; wherein the second training strategy comprises a reward mechanism based on reinforcement learning, and only optimizes the parameters of the large language model and the context perception module. The method combines the large language model and the reinforcement learning, starts from the end-to-end perspective, explicitly guides the model to make decisions, and thus improves the generalization ability of the model.

Description

Technical Field

[0001] This application relates to the field of 3D scene understanding and perception technology, and in particular to a point cloud scene understanding and segmentation method, apparatus, device and storage medium based on multimodal enhanced reasoning. Background Technology

[0002] Point cloud segmentation, a core challenge in the field of 3D vision, aims to understand and classify each point in a point cloud to obtain its semantic attributes. This technology has driven the rapid development of multiple fields such as robotics, autonomous driving, and virtual reality. With in-depth research, point cloud segmentation has gradually evolved into a series of more refined tasks, each addressing different segmentation needs. These tasks include basic semantic segmentation and instance segmentation, as well as more practically valuable tasks, such as text-based referential segmentation, which segments point clouds using explicit linguistic information; and open-lexical segmentation for complex real-world scenarios, designed to handle dynamic and ever-changing real-world environments.

[0003] Previous approaches have explored leveraging multimodal large language models to enhance reasoning and segmentation capabilities, bridging the gap between multimodal large models and segmentation models through implicit semantic tagging. However, typical methods rely solely on supervised fine-tuning on mixed datasets, which often contain only simple category information or basic factual descriptions. While this paradigm can effectively align multimodal large language models and segmentation models on specific datasets, its generalization ability is significantly insufficient, mainly in the following two aspects: (1) supervised fine-tuning inevitably leads to the model forgetting general capabilities (catastrophic forgetting); (2) the lack of explicit reasoning processes limits the model's effectiveness in complex scenarios. These limitations prompt us to explore how to improve the generalization ability and reasoning performance of segmentation by introducing explicit reasoning mechanisms.

[0004] Furthermore, despite significant progress in point cloud segmentation, existing methods still have limitations in point cloud feature learning. Most methods extract features by sampling only a small number of points, ignoring global information within the complete scene. While this localized strategy improves computational efficiency, it inherently limits segmentation accuracy, resulting in insufficient understanding and generalization ability of the model in complex environments. Summary of the Invention

[0005] This application provides a point cloud scene understanding and segmentation method, apparatus, device, and storage medium based on multimodal reinforced reasoning, aiming to construct a multi-modal reinforced reasoning framework for point cloud scene understanding and segmentation (ReaS3D). This framework designs a global enhancement module to fully model global point cloud information and introduces a reinforcement learning-based decision-making mechanism to improve the system's adaptive reasoning and dynamic decision-making capabilities in complex environments, thereby effectively overcoming the limitations of existing methods.

[0006] Specifically, this application utilizes artificial intelligence algorithms to re-optimize data errors, combining deep learning, statistical learning, and variational set conformance algorithms to overcome systematic biases in current data assimilation processes, particularly in initial field settings and dust aerosol data simulation. By combining the spatial feature extraction capabilities of CNNs with the advantages of SVR in small-sample nonlinear modeling, and introducing a cyclic iterative mechanism to dynamically correct the background error covariance structure, the application enhances the modeling and assimilation capabilities for high-dimensional nonlinear perturbation processes. This will contribute to providing a more advanced, comprehensive, accurate, and convenient method for dust data assimilation.

[0007] Firstly, this application provides a point cloud scene understanding and segmentation method based on multimodal reinforced reasoning, including: A point cloud scene understanding and segmentation framework is established; wherein, the point cloud scene understanding and segmentation framework includes a point cloud feature extraction module, a large language model, a projection layer and a context awareness module, and the point cloud feature extraction module includes a point cloud encoder, a superpoint pooling layer, a global-local interaction awareness layer, a global enhancement module and a mapping layer; Based on the first training strategy, the point cloud scene understanding and segmentation framework is subjected to supervised fine-tuning to obtain a supervised fine-tuning framework; wherein, the first training strategy includes freezing the point cloud encoder, the global-local interaction perception layer and the main part of the large language model, performing lightweight fine-tuning on the large language model, and jointly training the global enhancement module, the mapping layer and the projection layer. Based on the second training strategy, reinforcement learning is performed on the supervised fine-tuning framework to obtain a point cloud scene understanding and segmentation framework after reinforcement learning. Point cloud scene understanding and segmentation is achieved through the point cloud scene understanding and segmentation framework after reinforcement learning. The second training strategy includes a reward mechanism based on reinforcement learning, which optimizes only the parameters of the large language model and the context-aware module.

[0008] In one possible design, the data processing flow for supervised fine-tuning of the point cloud scene understanding and segmentation framework includes: Acquire point cloud data and text features; The point cloud data is processed by the point cloud feature extraction module to obtain the first point cloud feature; The first point cloud feature is concatenated and fused with the text feature to form the first multimodal feature; The first multimodal feature is input into the large language model to learn cross-modal contextual semantic relationships and output a segmentation mask representation; The segmentation mask representation is decoded by the Transformer module in the projection layer and the global-local interactive perception layer to obtain the segmentation features.

[0009] In one possible design, the point cloud data is processed by the point cloud feature extraction module to obtain a first point cloud feature, including: In response to the input point cloud data, multi-scale spatial semantic features are extracted through a point cloud encoder, and then compressed using a superpoint average pooling module to obtain a structured region representation. The computation process is as follows: , (1) In the formula, Representing multi-scale spatial semantic features, This represents a point cloud encoder. Represents point cloud data. For structured region representation, Indicates pooling operation; The structured region representation is input into the global-local interaction perception layer, and the contextual dependencies of the point cloud in the local neighborhood and the global scope are obtained using the following formula. : (2) In the formula, Represents the normalization function. Represents three weight matrices, This represents the transpose function. Indicates feature dimension; A global semantic modeling method is used to perform global semantic modeling on point cloud data to obtain global semantic features. This global enhancement module includes a keypoint convolutional layer, a regularization layer, and an activation function, which integrates the global semantic features with... Preliminary point cloud features were obtained through fusion using a cross-attention mechanism; The initial point cloud features are aligned with the text features through the mapping layer to obtain the first point cloud features.

[0010] In one possible design, the initial point cloud features are calculated using the following formula: , (3) In the formula, Represents global semantic features. Indicates a global enhancement module. It is a learnable parameter variable used to balance global and local scene information.

[0011] In one possible design, the data processing flow for reinforcement learning of the supervised fine-tuning framework includes: Acquire point cloud data, text features, and thought chain features; wherein, the thought chain features are generated during supervised fine-tuning of the point cloud scene understanding and segmentation framework; The point cloud data is processed by the point cloud feature extraction module in the supervised fine-tuning framework to obtain the second point cloud feature, and the second point cloud feature, text feature and thought chain feature are fused to obtain the input sequence. A learnable parameter vector is introduced at the end of the input sequence to obtain a second multimodal feature; wherein, the learnable parameter vector is used to actively extract and aggregate contextual information during the forward propagation process; The second multimodal feature is input into the large language model, and the contextual semantics are extracted from the large language model through the context-aware module and projected onto the segmentation cue space to obtain the mask representation; The mask representation is input into the Transformer module in the global-local interactive perception layer for decoding to obtain the segmentation prediction.

[0012] In one possible design, the reinforcement learning-based reward mechanism includes: generating a set of outputs through sampling using a policy model, calculating the reward value for each output using a reward function, and calculating the advantage function for each output based on the reward value of each output, with the following formula: , , (4) In the formula, Indicates the first i The advantage function of each output It is a constant used to avoid the denominator being 0; This represents the average reward of a given group of outputs, reflecting the average reward level within the group. The standard deviation of a set of output rewards is used to measure the degree of fluctuation in rewards. i This indicates the output sequence number within a given group, and G represents the number of samples. Indicates the first i One output, Indicates the first i A reward function for each output.

[0013] In one possible design, the reward function includes a formatted reward. Location and Comparison Rewards Semantic intersection and comparison reward ; The format reward is calculated using the following formula: (5) The position intersection-union ratio bonus is calculated using the following formula: (6) In the formula, This indicates the predicted location information; Represents the actual coordinate information; express and The intersection and union ratio.

[0014] Semantic Intersection over Union (SIOU) Reward: This reward is used to evaluate the degree of overlap between the segmentation mask generated by the model and the real mask, reflecting the overall segmentation quality. The calculation formula is as follows: (7) In the formula, This represents the predicted point cloud mask information, that is, the model's prediction result for the category (or region) to which each point belongs; Represents the true point cloud mask information; express and The intersection and union ratio.

[0015] Secondly, this application provides a point cloud scene understanding and segmentation device based on multimodal enhanced reasoning, the device comprising: The framework construction module is configured to establish a point cloud scene understanding and segmentation framework; wherein, the point cloud scene understanding and segmentation framework includes a point cloud feature extraction module, a large language model, a projection layer and a context awareness module, and the point cloud feature extraction module includes a point cloud encoder, a superpoint pooling layer, a global-local interaction awareness layer, a global enhancement module and a mapping layer. The supervised fine-tuning module is configured to perform supervised fine-tuning on the point cloud scene understanding and segmentation framework based on the first training strategy to obtain the supervised fine-tuning framework; wherein, the first training strategy includes freezing the point cloud encoder, the global-local interaction perception layer and the main part of the large language model, performing lightweight fine-tuning on the large language model, and jointly training the global enhancement module, the mapping layer and the projection layer. The reinforcement learning module is configured to perform reinforcement learning on the supervised fine-tuning framework based on a second training strategy to obtain a point cloud scene understanding and segmentation framework after reinforcement learning. Point cloud scene understanding and segmentation is achieved through the point cloud scene understanding and segmentation framework after reinforcement learning. The second training strategy includes a reward mechanism based on reinforcement learning, which optimizes only the parameters of the large language model and the context-aware module.

[0016] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the point cloud scene understanding and segmentation method based on multimodal enhanced reasoning as described in the first aspect and various possible designs of the first aspect.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the point cloud scene understanding and segmentation method based on multimodal enhanced reasoning as described in the first aspect and various possible designs of the first aspect.

[0018] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the point cloud scene understanding and segmentation method based on multimodal enhanced reasoning as described in the first aspect and various possible designs of the first aspect.

[0019] The point cloud scene understanding and segmentation method, apparatus, device, and storage medium based on multimodal enhanced reasoning provided in this application have at least the following beneficial effects: 1) This application combines large language models with reinforcement learning, introduces explicit thought chains as prior knowledge, and uses learnable parameter vectors to capture contextual information of large language models, thereby improving the accuracy and generalization ability of point cloud scene understanding and segmentation in an end-to-end framework.

[0020] 2) This application designs a global enhancement module and a global-local interactive perception layer to achieve efficient fusion of global semantic information and local geometric features, thereby improving the representation ability and segmentation effect of point cloud features. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1 A flowchart of a point cloud scene understanding and segmentation method based on multimodal enhanced reasoning provided in this application embodiment; Figure 2 A first-stage supervised fine-tuning model framework diagram provided for embodiments of this application; Figure 3 This is a data processing flowchart for supervised fine-tuning of the point cloud scene understanding and segmentation framework provided in an embodiment of this application; Figure 4 Framework diagram of global-local interaction perception layer and global enhancement module provided in the embodiments of this application; Figure 5 A second-stage reinforcement learning model framework diagram provided for embodiments of this application; Figure 6 This application provides a data processing flowchart for reinforcement learning of a supervised fine-tuning framework. Figure 7 This application provides visualization results on the ScanRefer dataset for embodiments of the present application. Figure 8 This is a structural diagram of a point cloud scene understanding and segmentation device based on multimodal enhanced reasoning, provided in an embodiment of this application.

[0023] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0025] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0027] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0028] With the development of spatial perception technology, 3D point cloud data, due to its high precision and unstructured nature, is widely used in fields such as autonomous driving, robot navigation, and smart cities. However, traditional point cloud processing methods still face many challenges in semantic understanding, semantic segmentation, and instance segmentation, such as insufficient semantic information, difficulties in multimodal information fusion, and a lack of semantic reasoning capabilities. In recent years, the development of technologies such as multimodal learning, large language models, and reinforcement learning has provided new ideas for multimodal data fusion and intelligent decision-making.

[0029] Current research has attempted to fuse multimodal data such as text, images, and point clouds to improve semantic understanding and scene reasoning capabilities. For example, semantic segmentation and instance segmentation under open vocabulary conditions have been achieved through multimodal feature alignment and end-to-end training. However, existing methods still have the following shortcomings in multimodal reasoning, semantic modeling, and intelligent decision-making: 1) Insufficient global point cloud information modeling: Most methods rely only on sampling and segmenting some point cloud features, making it difficult to effectively capture point cloud information in the global scene.

[0030] 2) Lack of intelligent decision-making mechanism: It lacks reasoning and dynamic decision-making methods based on reinforcement learning, making it difficult to achieve adaptive response in complex and dynamic scenarios.

[0031] Based on this, embodiments of this application provide a point cloud scene understanding and segmentation method based on multimodal enhanced reasoning to overcome the above-mentioned shortcomings. For example... Figure 1 As shown, the point cloud scene understanding and segmentation method based on multimodal enhanced reasoning includes the following steps S10-S30.

[0032] S10: Establish a point cloud scene understanding and segmentation framework; the point cloud scene understanding and segmentation framework includes a point cloud feature extraction module, a large language model, a projection layer and a context-aware module. The point cloud feature extraction module includes a point cloud encoder, a super-point pooling layer, a global-local interaction perception layer, a global enhancement module and a mapping layer.

[0033] In this embodiment, the point cloud scene understanding and segmentation framework is used to achieve multimodal point cloud segmentation. During runtime, it comprises three parts: multimodal data preprocessing, large language model inference, and a reinforcement learning-based reward mechanism. Specifically, multimodal data preprocessing includes acquiring input data, which includes point cloud data (such as 3D point clouds) and text information. The point cloud data first undergoes a point cloud encoder to extract low-level geometric and structural features, and then local region aggregation is achieved through a super-pooling layer. Subsequently, a global enhancement module is introduced to model the overall spatial context information, and a global-local interaction perception layer is used to achieve efficient fusion of context information within different spatial ranges. Finally, point cloud features and text features together constitute a cross-modal semantic representation. Large language model inference includes inputting the above multimodal semantic representation into a large language model, utilizing its powerful cross-modal alignment and semantic inference capabilities to achieve high-level semantic modeling and understanding of complex scenes. The reinforcement learning-based reward mechanism includes introducing reinforcement learning during the inference process, using a reward function to guide the dynamic optimization and policy selection of the model, thereby enhancing the system's adaptive decision-making and robustness in complex and variable environments.

[0034] Based on the point cloud scene understanding and segmentation framework established in step S10, the following two stages of the method will be executed: the first stage is the supervised fine-tuning stage, and the second stage is the reinforcement learning stage. The first stage and the second stage correspond to the following steps S20 and S30, respectively.

[0035] S20: Based on the first training strategy, supervised fine-tuning is performed on the point cloud scene understanding and segmentation framework to obtain the supervised fine-tuning framework; wherein, the first training strategy includes freezing the point cloud encoder, the global-local interaction perception layer and the main part of the large language model, performing lightweight fine-tuning on the large language model, and jointly training the global enhancement module, the mapping layer and the projection layer.

[0036] In step S20, point cloud data is first input into the point cloud feature extraction module to obtain spatial representations, which are then concatenated and fused with text features to form multimodal input. Subsequently, these multimodal features are fed into a large language model to learn cross-modal contextual semantic relationships and perform joint modeling. The output features of the large language model are mapped to candidate segmentation mask representations via a projection layer, and then input into the Transformer module in the global-local interaction perception layer for decoding, thereby obtaining the final segmentation features.

[0037] In terms of training strategy, the point cloud encoder, global-local interaction perception layer, and the main body of the large language model are frozen, and only LoRA is used to perform lightweight fine-tuning on the large model. At the same time, the global enhancement module, mapping layer, and projection layer are jointly trained to achieve effective alignment and optimization of cross-modal features.

[0038] It should be noted that LoRA stands for Low-Rank Adaptation. LoRA is an existing method for efficient parameter fine-tuning, but for space limitations, this embodiment will not elaborate on the method in detail here.

[0039] In some embodiments, such as Figure 2 The diagram shown illustrates the framework of the first-stage supervised fine-tuning model provided in this embodiment. The first-stage process mainly involves a point cloud feature extraction module, a large language model, and a projection layer. The point cloud feature extraction module includes: Point cloud encoder: Used to receive input point cloud data and perform preliminary feature recognition.

[0040] Superpooling layer: Connected to the recognizer, used for deeper perception of preliminary features.

[0041] Global-Local Interactive Perception Layer: Connected to the perceptron, it is used to capture and fuse the contextual dependencies of point cloud data in the local neighborhood and the global scope.

[0042] Global Enhancement Module: Receives input point cloud data and interacts with the global-local interactive perception layer to further enhance the global scene modeling capability.

[0043] Mapping layer: Connected to the global-local interactive perception layer, it is used to perform dimensional mapping on the processed point cloud features and output them as three-dimensional point cloud features.

[0044] During supervised fine-tuning, point cloud data flows sequentially through the point cloud encoder, superpooling layer, global-local interactive perception layer, global enhancement module, and mapping layer, ultimately outputting 3D point cloud features. Simultaneously, text input is received. Figure 2 An example of text input is: "I'm standing by the window, deep in thought, with a mirror in front of me. What color is the desk to my left? Please answer with a word or phrase." This text information is concatenated and fused with 3D point cloud features to form a multimodal input. This multimodal input is then fed into a large language model, which leverages its semantic understanding and reasoning capabilities for cross-modal joint modeling. The output features of the large language model are finally mapped to candidate segmentation mask representations via a projection layer, used for subsequent decoding and optimization to obtain the result, i.e., the segmentation features.

[0045] In some embodiments, Figure 3 This is a data processing flowchart for supervised fine-tuning of the point cloud scene understanding and segmentation framework provided in an embodiment of this application. Figure 3 As shown, the data processing flow for supervised fine-tuning of the point cloud scene understanding and segmentation framework includes the following steps S201-S205.

[0046] S201: Acquire point cloud data and text features.

[0047] In this embodiment, the text features are those extracted from the text data provided by the user, and the point cloud data is collected in a manner determined by the scenario in which the method of this application is applied. For example, in an autonomous driving scenario, the point cloud data can be collected by radar point clouds from LiDAR deployed on the vehicle. Of course, the sources of the point cloud data mentioned above are merely exemplary and do not constitute a limitation on this application.

[0048] S202: The point cloud data is processed by the point cloud feature extraction module to obtain the first point cloud feature.

[0049] In some embodiments, Figure 4 This is a framework diagram of the global-local interaction perception layer and the global enhancement module provided in the embodiments of this application. Figure 4 As shown, the point cloud feature extraction module can process the point cloud data to obtain the first point cloud feature in the following way.

[0050] First, point cloud data and text data are used as input. The point cloud data is processed by a point cloud feature extraction module, which mainly consists of a point cloud encoder, a super-pooling layer, a global-local interaction perception layer, a global enhancement module, and a mapping layer. Specifically, the input point cloud data... , N R represents the number of points, R represents the set of real numbers, and 6 is the number of channels. The 6 channels represent the coordinates {x, y, z} and color information {r, g, b} of the point cloud, respectively. In the point cloud feature extraction process, the first step is to... The input is fed into a point cloud encoder, which can use Sparse 3D U-Net as the scene encoder. Voxelization is used to map the sparse point cloud onto a regular voxel grid to extract multi-scale spatial semantic features. Subsequently, the point cloud features are further compressed using a superpoint-based average pooling module to obtain a structured region representation. : , (1) in, This refers to the point cloud encoder. In this embodiment, the point cloud encoder uses Sparse 3D U-Net as the scene encoder. This indicates a pooling operation. Represents point cloud data. For structured region representation, Represents spatial semantic features at multiple scales.

[0051] Will The input is fed into the global-local interaction perception layer. This global-local interaction perception layer, with the Transformer module at its core, is used to capture the contextual dependencies of the point cloud in its local neighborhood and global scope, as shown in the following formula: (2) In the formula, Represents the normalization function. Represents three weight matrices, This represents the transpose function. Indicates the feature dimension; like Figure 4 As shown, the global-local interaction perception layer contains multiple Transformer modules and multiple cross-attention layers. A Transformer module includes at least a self-attention layer, residual & regularization, and a forward propagation layer.

[0052] To further enhance global scene modeling capabilities, this embodiment designs a global enhancement module, such as... Figure 4 As shown, this global enhancement module has multiple layers, each including a keypoint convolutional layer, a regularization layer, and an activation function. The output of this global enhancement module is fused with the local representation in the global-local interaction perception layer through a cross-attention mechanism, thereby achieving effective interaction and joint modeling of local structure and global semantics, and obtaining preliminary point cloud features. The calculation formula is as follows: , (3) In the formula, Represents global semantic features. This represents a global enhancement module consisting of three keypoint convolutional layers, a regularization layer, and an activation function. It is a learnable parameter variable used to balance global and local scene information.

[0053] Finally, the preliminary point cloud features are aligned with the text features through a mapping layer to obtain the first point cloud features. In this embodiment, the mapping layer consists of a linear layer used to align the extracted preliminary point cloud features with the text features.

[0054] S203: The first point cloud feature is concatenated and fused with the text feature to form the first multimodal feature.

[0055] For example, the first point cloud feature and text feature can be concatenated and fused to form the first multimodal feature using a feature dimension concatenation method, that is, the first point cloud feature and text feature are directly concatenated along the feature dimension to obtain the first multimodal feature. Of course, the first point cloud feature and text feature can also be concatenated after projection alignment, for example, the first point cloud feature is projected onto a space with the same dimension as the text feature before concatenation. It should be noted that the above concatenation and fusion methods are only examples and do not constitute a limitation of this application. In other implementations, attention-weighted fusion, cross-modal gating fusion, and other methods can also be used for concatenation and fusion.

[0056] S204: Input the first multimodal feature into the large language model, learn the cross-modal contextual semantic relationship, and output the segmentation mask representation.

[0057] S205: The segmentation mask representation is decoded through the Transformer module in the projection layer and the global-local interactive perception layer to obtain the segmentation features.

[0058] S30: Based on the second training strategy, reinforcement learning is performed on the supervised fine-tuning framework to obtain the point cloud scene understanding and segmentation framework after reinforcement learning. Point cloud scene understanding and segmentation is achieved through the point cloud scene understanding and segmentation framework after reinforcement learning. The second training strategy includes a reward mechanism based on reinforcement learning, which optimizes only the parameters of the large language model and the context-aware module.

[0059] In step S30, the reinforcement learning phase, point cloud data is first input into the point cloud feature extraction module to obtain spatial representations, which are then concatenated and fused with text features and thought chain features. The generation formula for thought chain features is as follows: the model parameters obtained from the first phase of supervised fine-tuning are frozen, generating n thought chain representations. Each thought chain representation contains the text reasoning process and the predicted location information, and is ultimately used as input for multimodal modeling along with other modal features. Subsequently, a learnable parameter vector is introduced at the end of the input sequence to actively extract and aggregate contextual information during forward propagation. Next, a context-aware module is introduced, consisting of a context query and a connector. The former extracts contextual semantics from the large language model, while the latter projects the extracted representations onto the segmentation cue space to obtain effective mask representations. Finally, these mask representations are input into the Transformer module in the global-local interaction perception layer for decoding, thereby achieving end-to-end optimization from cross-modal reasoning to segmentation prediction.

[0060] This embodiment employs a rule-based reinforcement learning mechanism, GRPO, for post-training optimization of large language models. Specifically, given a problem q, GRPO first utilizes a policy model... Sample to generate a set of outputs ,in G This represents the number of samples. Then, the reward function is used. To determine the relative quality of outputs within a group, GRPO calculates the reward value for each output. : , , (4) In the formula, Indicates the first i The advantage function of each output It is a constant used to avoid the denominator being 0; This represents the average reward of a given group of outputs, reflecting the average reward level within the group. The standard deviation of a set of output rewards is used to measure the degree of fluctuation in rewards. i This represents the output sequence number within a certain group, and G represents the number of samples. Indicates the first i One output, Indicates the first i A reward function for each output.

[0061] In this embodiment, the reward function (reward mechanism) consists of the following three parts: format reward. Location and Comparison Rewards Semantic intersection and comparison reward .

[0062] Formatting Reward: Ensures the output of the multimodal large language model conforms to the expected structure and format consistency. It requires that the output inference process be placed within... <thinking> ...< / thinking> The label is internal, and the positioning result is placed in <answer> ...< / answer> The label is internal. If the format is correct, the reward is 1; otherwise, it is 0. The formula is as follows: (5) Location Intersection over Union (CIUC) Reward: During inference, the model may explicitly or implicitly mention the bounding box of a target object. This reward assesses the accuracy of the bounding box localization, and is calculated using the following formula: (6) In the formula, This indicates the predicted location information; Represents the actual coordinate information; express and The intersection and union ratio.

[0063] Semantic Intersection over Union (SIOU) Reward: This reward is used to evaluate the degree of overlap between the segmentation mask generated by the model and the real mask, reflecting the overall segmentation quality. The calculation formula is as follows: (7) In the formula, This represents the predicted point cloud mask information, that is, the model's prediction result for the category (or region) to which each point belongs, such as belonging to "wall", "table", "ground", etc. Represents the true point cloud mask information; express and The intersection and union ratio.

[0064] In some embodiments, Figure 5 This is a diagram illustrating the framework of a second-stage reinforcement learning model provided in an embodiment of this application. Figure 5 As shown, during the second stage of reinforcement learning, point cloud data is input into the point cloud feature extraction module to extract 3D point cloud features. Text features are generated based on the text provided by the user, and thought features are generated based on the thought chain. These thought features are representations of each segment of the thought chain, including the text reasoning process and predicted position information, generated during the supervised fine-tuning process in the first stage. The 3D point cloud features, text features, and thought features are combined to form an input sequence, and a learnable vector (i.e., a learnable parameter vector) is introduced at the end of the input sequence to obtain a multimodal sequence. This multimodal sequence is fed into the large language model. It should be noted that the large language model in the second stage is the model after the supervised fine-tuning in the first stage. The context-aware module extracts mask representations from the large language model and inputs them into the point cloud feature extraction module for decoding to obtain the final point cloud segmentation result.

[0065] In some embodiments, Figure 6 This is a flowchart illustrating the data processing for reinforcement learning of a supervised fine-tuning framework, provided in an embodiment of this application. Figure 6 As shown, the data processing flow for reinforcement learning of the supervised fine-tuning framework includes the following steps S301-S305.

[0066] S301: Acquire point cloud data, text features, and thought chain features; among which, thought chain features are generated during supervised fine-tuning of the point cloud scene understanding and segmentation framework.

[0067] It should be noted that the method of obtaining point cloud data and text features in step S301 is the same as that in step S201, so it will not be described again here.

[0068] S302: The point cloud data is processed by the point cloud feature extraction module in the supervised fine-tuning framework to obtain the second point cloud feature, and the second point cloud feature, text feature and thought chain feature are fused to obtain the input sequence.

[0069] It should be noted that the principle of obtaining the second point cloud feature by processing the point cloud data through the point cloud feature extraction module in the supervised fine-tuning framework is the same as that of extracting the first point cloud feature in step S202 above. The difference lies in the parameters of the sub-modules included in the point cloud feature extraction module. Furthermore, the methods for fusing the second point cloud feature, text features, and thought chain features include, but are not limited to, any one of the following: feature dimension concatenation, attention-weighted fusion, projection-aligned concatenation, and cross-modal gating fusion.

[0070] S303: A learnable parameter vector is introduced at the end of the input sequence to obtain the second multimodal feature; wherein, the learnable parameter vector is used to actively extract and aggregate contextual information during the forward propagation process.

[0071] For example, introducing a learnable parameter vector at the end of the input sequence can be done by directly appending a learnable parameter vector to the end of the input sequence. For instance, if the end of the input sequence is a thought chain feature, then the learnable parameter vector can be directly appended after the thought chain feature.

[0072] S304: Input the second multimodal features into the large language model, extract the contextual semantics from the large language model through the context-aware module, and project them onto the segmentation cue space to obtain the mask representation.

[0073] It should be noted that mask representation, also known as segmentation mask representation, refers to a structured data form that explicitly defines which pixels or points in an image or point cloud scene belong to the target object of interest. For example, a mask representation can be a sequence or matrix with the same number of points as the input point cloud, where each location (corresponding to a point) has a value that typically represents the probability or confidence that the point belongs to the target object.

[0074] S305: The mask representation is input into the Transformer module in the global-local interactive perception layer for decoding to obtain the segmentation prediction.

[0075] In summary, the method proposed in this application can not only fully capture global point cloud information, but also combine the semantic understanding ability of large language models with the dynamic decision-making ability of reinforcement learning to achieve higher accuracy and more robust point cloud scene understanding and segmentation.

[0076] The feasibility and progressiveness of this application will be further illustrated below with specific experimental data.

[0077] This embodiment tests the proposed method (ReaS3D) and the existing method 3D-LLaVA on three datasets: ScanQA, SQA3D, and ScanRefer. These datasets assess the model's ability to locate objects, describe scenes, and perform complex reasoning, respectively. The quality of ScanQA's generated text responses is evaluated using metrics including CiDEr (C), BLEU-4 (B-4), METEOR (M), and Rouge-L (R). Unlike the traditional ScanQA setup, SQA3D is a contextual question-answering dataset with definite answers; therefore, this embodiment uses exact match accuracy (EM) and its improved version EM-R as evaluation metrics. For the referential segmentation task involved in ScanRefer, this embodiment uses the mean intersection-union ratio (mIoU) for evaluation, and the results are shown in Table 1.

[0078] Table 1: Performance Comparison

[0079] It should be noted that 3D-LLaVA refers to the point cloud segmentation model proposed in the paper "Deng J, He T, Jiang L, et al. 3d-llava: Towards generalist 3d lmms with omni superpoint transformer[C] / / Proceedings of the Computer Vision and Pattern Recognition Conference. 2025:3772-3782."

[0080] As shown in Table 1, ReaS3D outperforms 3D-LLaVA on all datasets and almost all metrics. Further... Figure 7 The image shown is a visualization of the ScanRefer dataset provided in this embodiment of the application. Figure 7 In the diagram, (a) shows the visualization result of the Ground Truth, representing the ideal benchmark for segmentation; (b) shows the visualization result of 3D-LLaVA; and (c) shows the visualization result of ReaS3D. Figure 7 As shown in (a), the segmentation mask generated by 3D-LLaVA has obvious shortcomings, such as blurred boundary delineation of the target object and missing areas that are not effectively segmented. Figure 7 As shown in (c), it can be clearly observed that the segmentation mask generated by this application (ReaS3D) has more complete and accurate boundaries, higher consistency with the real mask, and significantly reduces errors and missing areas. Figure 7 The visualization results intuitively demonstrate that, compared to existing technologies, the proposed ReaS3D framework, through its global enhancement module and reinforced inference mechanism, can more effectively capture the global contextual information of the scene, thereby generating higher quality and more robust segmentation results. This visual comparison, along with the quantitative analysis in Table 1 showing the improvement of the mIoU metric of this application (ReaS3D), jointly proves the effectiveness and advancement of the proposed method.

[0081] This application also provides a point cloud scene understanding and segmentation device based on multimodal enhanced reasoning, such as... Figure 8 As shown, the point cloud scene understanding and segmentation device based on multimodal enhanced reasoning includes: The framework construction module 801 is configured to establish a point cloud scene understanding and segmentation framework; wherein, the point cloud scene understanding and segmentation framework includes a point cloud feature extraction module, a large language model, a projection layer and a context awareness module, and the point cloud feature extraction module includes a point cloud encoder, a superpoint pooling layer, a global-local interaction awareness layer, a global enhancement module and a mapping layer. The supervised fine-tuning module 802 is configured to perform supervised fine-tuning on the point cloud scene understanding and segmentation framework based on a first training strategy to obtain a supervised fine-tuning framework; wherein, the first training strategy includes freezing the point cloud encoder, the global-local interaction perception layer and the main part of the large language model, performing lightweight fine-tuning on the large language model, and jointly training the global enhancement module, the mapping layer and the projection layer. The reinforcement learning module 803 is configured to perform reinforcement learning on the supervised fine-tuning framework based on a second training strategy to obtain a point cloud scene understanding and segmentation framework after reinforcement learning, and to realize point cloud scene understanding and segmentation through the point cloud scene understanding and segmentation framework after reinforcement learning; wherein, the second training strategy includes a reward mechanism based on reinforcement learning, and only optimizes the parameters of the large language model and the context awareness module.

[0082] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0083] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0084] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0085] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0086] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the point cloud scene understanding and segmentation method based on multimodal enhanced reasoning described in the above embodiments.

[0087] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the point cloud scene understanding and segmentation method based on multimodal enhanced reasoning in the above embodiments.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0089] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0090] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0091] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0092] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0093] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0094] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0095] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0096] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0097] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A point cloud scene understanding and segmentation method based on multimodal reinforced reasoning, characterized in that, The method includes: A point cloud scene understanding and segmentation framework is established; wherein, the point cloud scene understanding and segmentation framework includes a point cloud feature extraction module, a large language model, a projection layer and a context awareness module, and the point cloud feature extraction module includes a point cloud encoder, a superpoint pooling layer, a global-local interaction awareness layer, a global enhancement module and a mapping layer; Based on the first training strategy, the point cloud scene understanding and segmentation framework is subjected to supervised fine-tuning to obtain a supervised fine-tuning framework; wherein, the first training strategy includes freezing the point cloud encoder, the global-local interaction perception layer and the main part of the large language model, performing lightweight fine-tuning on the large language model, and jointly training the global enhancement module, the mapping layer and the projection layer. Based on the second training strategy, reinforcement learning is performed on the supervised fine-tuning framework to obtain a reinforcement-learned point cloud scene understanding and segmentation framework. Point cloud scene understanding and segmentation is achieved through the reinforcement-learned point cloud scene understanding and segmentation framework. The second training strategy includes a reinforcement learning-based reward mechanism that optimizes only the parameters of the large language model and the context-aware module. The data processing flow for supervised fine-tuning of the point cloud scene understanding and segmentation framework includes: Acquire point cloud data and text features; The point cloud data is processed by the point cloud feature extraction module to obtain the first point cloud feature; The first point cloud feature is concatenated and fused with the text feature to form the first multimodal feature; The first multimodal feature is input into the large language model to learn cross-modal contextual semantic relationships and output a segmentation mask representation; The segmentation mask representation is decoded by the Transformer module in the projection layer and the global-local interactive perception layer to obtain the segmentation features; The point cloud data is processed by the point cloud feature extraction module to obtain the first point cloud feature, including: In response to the input point cloud data, multi-scale spatial semantic features are extracted through a point cloud encoder, and then compressed using a superpoint average pooling module to obtain a structured region representation. The computation process is as follows: , （1） In the formula, Representing multi-scale spatial semantic features, This represents a point cloud encoder. Represents point cloud data. For structured region representation, Indicates pooling operation; The structured region representation is input into the global-local interaction perception layer, and the contextual dependencies of the point cloud in the local neighborhood and the global scope are obtained using the following formula. : （2） In the formula, Represents the normalization function. This represents three weight matrices. This represents the transpose function. Indicates the feature dimension; A global semantic modeling method is used to perform global semantic modeling on point cloud data to obtain global semantic features. This global enhancement module includes a keypoint convolutional layer, a regularization layer, and an activation function, which integrates the global semantic features with... Preliminary point cloud features were obtained through fusion using a cross-attention mechanism; The initial point cloud features are aligned with the text features through the mapping layer to obtain the first point cloud features; The data processing flow for reinforcement learning of the supervised fine-tuning framework includes: Acquire point cloud data, text features, and thought chain features; wherein, the thought chain features are generated during supervised fine-tuning of the point cloud scene understanding and segmentation framework; The point cloud data is processed by the point cloud feature extraction module in the supervised fine-tuning framework to obtain the second point cloud feature, and the second point cloud feature, text feature and thought chain feature are fused to obtain the input sequence. A learnable parameter vector is introduced at the end of the input sequence to obtain a second multimodal feature; wherein, the learnable parameter vector is used to actively extract and aggregate contextual information during the forward propagation process; The second multimodal feature is input into the large language model, and the contextual semantics are extracted from the large language model by the context-aware module and projected onto the segmentation cue space to obtain the mask representation; The mask representation is input into the Transformer module in the global-local interactive perception layer for decoding to obtain the segmentation prediction.

2. The point cloud scene understanding and segmentation method based on multimodal enhanced reasoning according to claim 1, characterized in that, The preliminary point cloud features are calculated using the following formula: , （3） In the formula, Represents global semantic features. Indicates a global enhancement module. It is a learnable parameter variable used to balance global and local scene information.

3. The point cloud scene understanding and segmentation method based on multimodal enhanced reasoning according to any one of claims 1 to 2, characterized in that, The reinforcement learning-based reward mechanism includes: generating a set of outputs through sampling using a policy model, calculating the reward value for each output using a reward function, and calculating the advantage function for each output based on its reward value. The calculation formula is as follows: , , （4） In the formula, Indicates the first i The advantage function of each output It is a constant used to avoid the denominator being 0; This represents the average reward of a given group of outputs, reflecting the average reward level within the group. The standard deviation of a set of output rewards is used to measure the degree of fluctuation in rewards. i This indicates the output sequence number within a specific group. G Indicates the number of samples. Indicates the first i One output, Indicates the first i A reward function for each output.

4. The point cloud scene understanding and segmentation method based on multimodal enhanced reasoning according to claim 3, characterized in that, The reward function includes formatted rewards. Location and Comparison Rewards Semantic intersection and comparison reward ; The format reward is calculated using the following formula: （5） The location intersection-union ratio bonus is calculated using the following formula: （6） In the formula, This indicates the predicted location information; Represents the actual coordinate information; express and The intersection and union ratio; Semantic Intersection over Union (SIOU) Reward: This reward is used to evaluate the degree of overlap between the segmentation mask generated by the model and the real mask, reflecting the overall segmentation quality. The calculation formula is as follows: （7） In the formula, This represents the predicted point cloud mask information, i.e., the model's prediction result for the category or region to which each point belongs; Represents the true point cloud mask information; express and The intersection and union ratio.

5. A point cloud scene understanding and segmentation device based on multimodal reinforced reasoning, used to implement the method as described in any one of claims 1-4, characterized in that, The device includes: The framework construction module is configured to establish a point cloud scene understanding and segmentation framework; wherein, the point cloud scene understanding and segmentation framework includes a point cloud feature extraction module, a large language model, a projection layer and a context awareness module, and the point cloud feature extraction module includes a point cloud encoder, a superpoint pooling layer, a global-local interaction awareness layer, a global enhancement module and a mapping layer. The supervised fine-tuning module is configured to perform supervised fine-tuning on the point cloud scene understanding and segmentation framework based on the first training strategy to obtain the supervised fine-tuning framework; wherein, the first training strategy includes freezing the point cloud encoder, the global-local interaction perception layer and the main part of the large language model, performing lightweight fine-tuning on the large language model, and jointly training the global enhancement module, the mapping layer and the projection layer. The reinforcement learning module is configured to perform reinforcement learning on the supervised fine-tuning framework based on a second training strategy to obtain a point cloud scene understanding and segmentation framework after reinforcement learning. Point cloud scene understanding and segmentation is achieved through the point cloud scene understanding and segmentation framework after reinforcement learning. The second training strategy includes a reward mechanism based on reinforcement learning, which optimizes only the parameters of the large language model and the context-aware module.

6. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes the computer execution instructions stored in the memory to implement the point cloud scene understanding and segmentation method based on multimodal enhanced reasoning as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the point cloud scene understanding and segmentation method based on multimodal enhanced reasoning as described in any one of claims 1-4.

Images