Body robot three-dimensional availability positioning method and system based on multi-modal large language model

Abstract

The invention relates to a multi-modal large language model-based three-dimensional affordability positioning method and system for a body robot, and belongs to the technical field of robot perception and body intelligence. According to the scheme, a multi-modal large model availability positioning framework is provided, the framework effectively combines semantic understanding, multi-modal fusion and 3D space positioning through an end-to-end learning mode, and the features of 3D scene data and natural language instructions are deeply mined and fully utilized. A novel scene-level affordability data representation is constructed, so that robot interaction tasks, such as door opening, drawer pulling and object grabbing, in various complex unstructured environments become more accurate and efficient; according to the method, adaptive sampling and a coarse-to-fine reasoning thought are fused, the calculation efficiency and robustness of understanding of a robot on a large-scale three-dimensional scene are improved by optimizing a data retention strategy and enhancing space consistency, and the limitation of a traditional method under the complex background and the ambiguous instruction in the real world is effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of robot perception and embodied intelligence technology, and relates to a method and system for three-dimensional availability localization of embodied robots based on a multimodal large language model. Background Technology

[0002] With the development of embodied intelligence technology, robots are gradually moving from structured industrial environments to unstructured real-world environments, such as homes and public spaces. In these scenarios, robots need to complete everyday human tasks based on inherently ambiguous natural language instructions, requiring them to understand what actions to perform and where to interact within the scene. A key challenge lies in identifying the interaction locations with objects in the operating environment; therefore, availability becomes a crucial link between perception, decision-making, and execution. Especially in natural language-driven scenarios, robots not only need to identify what the objects are but also to clarify the instructions and connect them to the task, placing even higher demands on the implementation of availability. Traditional robot availability analysis methods often focus on object-level modeling, relying on processing high-quality single-object point clouds under controlled conditions. However, real-world environments are complex and varied, containing diverse objects, complex backgrounds, occlusion, and sensor noise. Traditional object-level methods often fall short when processing these unstructured scene data, struggling to generate accurate and executable interaction locations, thus limiting their application on real-world robotic platforms. Furthermore, human natural language commands are often characterized by ambiguity, combinatorial features, and implicit intentions. Existing technologies and methods face significant challenges in accurately aligning complex linguistic semantics with large-scale 3D spatial representations. Currently, although multimodal large language models have shown significant advantages in jointly modeling visual perception and linguistic semantics, they still face serious challenges in practical applications. On the one hand, real-world robotic platforms are constrained by computational efficiency, making it difficult to support high-cost reasoning on large-scale data. On the other hand, how to effectively integrate language understanding with the precise implementation of enabling features in real-world 3D environments remains an unresolved issue.

[0003] Therefore, given the complex and ever-changing nature of embodied robot manipulation scenarios, it is necessary to develop novel analytical frameworks and technologies. These frameworks should be able to leverage the reasoning capabilities of large language models while possessing high processing efficiency. Through joint modeling of "what to do" and "where to do it," they should achieve a deep understanding and precise anchoring of 3D scene-level availability, thereby comprehensively improving the operational efficiency and safety of the robot system. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method and system for 3D availability localization of embodied robots based on a multimodal large language model. This method and system provide a framework called MLLM-Afford (Multimodal Large Model Availability Localization), which effectively combines semantic understanding, multimodal fusion, and 3D spatial localization through an end-to-end learning approach, deeply mining and fully utilizing the features of 3D scene data and natural language instructions. It constructs a novel scene-level availability data representation, which not only unifies the joint reasoning of "what to do" and "where to do it," but can also be directly used as the execution basis for embodied intelligent robots. This design makes robot interaction tasks in various complex unstructured environments, such as opening doors, pulling drawers, and object grasping, more accurate and efficient. MLLM-Afford integrates adaptive sampling and coarse-to-fine reasoning, improving the computational efficiency and robustness of robots in understanding large-scale 3D scenes by optimizing data retention strategies and enhancing spatial consistency, effectively solving the limitations of traditional methods in complex real-world backgrounds and ambiguous instructions.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for 3D availability localization of embodied robots based on a multimodal large language model is proposed. This method constructs an end-to-end learning framework for availability localization of the robot's 3D scene, achieving 3D availability localization of the robot. The method specifically includes the following steps: S1. Construct an adaptive multimodal representation module that uses the original 3D point cloud, RGB-D observation data and natural language instructions as input. Through an adaptive dynamic farthest point sampling strategy, greedy keyframe selection and point-aligned visual encoding, it transforms multi-view RGB-D observations into compact point-aligned visual labels consistent with 3D geometry. S2. Construct a command-guided multimodal fusion module. By constructing command prompts that include scene context and availability intent, and using gating and cross-attention mechanisms, inject language semantics into the 3D representation to achieve modal selection for task awareness. S3. Construct a spectrum-enhanced coarse-to-fine availability reasoning module. The module first identifies approximate regions that support availability through superpoint clustering, then applies residual spectrum transformation to enhance point features to strengthen spatial consistency, and finally predicts fine availability probabilities through availability decoder. S4. Construct a training optimization module, introduce a motion expert model to parameterize the action into a motion vector, predict the starting point and direction of the action by combining point features and geometric information (such as surface normals), and use asymmetric focal loss function, pairwise ranking loss and point-text contrast loss for joint training. S5. Use the constructed model to perform three-dimensional availability localization of the embodied robot.

[0006] Furthermore, step S1 mainly consists of three parts: point cloud preprocessing and geometric encoding, and greedy keyframe selection and visual encoding. Specific steps include: S11. Point Cloud Preprocessing and Geometric Encoding: For the input image, the Canny operator is used to extract the edge response to obtain the edge map. The image is then divided into blocks and image blocks are calculated. edge strength Image patches with high edge intensity are selected, projected into 3D space, and assigned higher sampling weights for dense sampling. The remaining points are sampled uniformly, thereby achieving adaptive dynamic farthest point sampling (FPS). At the same time, geometric features are extracted and sinusoidal 3D position coding is added. S12. Greedy Keyframe Selection: Defining a Binary Visibility Metric A greedy coverage strategy is adopted to select the key frame set F that maximizes the point coverage, ensuring accurate alignment between 3D points and visual information, while keeping the number of visual markers within a manageable range. S13, Point-aligned visual encoding: Assign the most relevant keyframe to each 3D point based on point visibility, depth consistency, normalized image position and projection block size, and project the point to the corresponding frame using camera intrinsics and pose to encode the visual appearance from a fixed-size image block.

[0007] Furthermore, step S2 specifically includes: S21. Construct a prompt template that includes scene description and action intent, and introduce special scene tokens to inject scene-level multimodal features; S22. Extract the hidden state of special location tokens and the text embedding of instructions from the large language model. S23. Design a gating mechanism to fuse the above representations into a single adjustment vector. This mechanism uses a lightweight multilayer perceptron and a sigmoid function to calculate the gating weights and perform weighted summation and nonlinear transformation on the hidden state and text embedding. S24. Using the generated adjustment vector, the point features are modulated through a cross-attention mechanism to realize the injection of language semantics into the three-dimensional representation space.

[0008] Furthermore, in step S3, a spectral-enhanced coarse-to-fine availability reasoning module is constructed, mainly consisting of coarse-grained localization, spectral feature enhancement, and fine-grained decoding, specifically including: S31. Coarse-grained localization: Based on spatial distance and feature similarity, point clouds are clustered into superpoints, each superpoint is used as the basic unit of coarse-grained localization, and irrelevant background areas are filtered out. S32. Spectral feature enhancement: Construct a k-nearest neighbor graph on the sampled point cloud, and use the low-frequency feature vectors of the graph Laplacian operator to perform residual spectral transformation, propagating the command-aligned response while preserving local boundaries; S33, Fine-grained decoding: A set of learnable query embeddings is introduced, and the instruction adjustment vector is injected as a shared bias; the decoder uses a multi-layer attention mechanism to enable the query embeddings to interact with point features, and finally projects them to the shared space to calculate point-level availability logic values.

[0009] Furthermore, in step S4, a training optimization module is constructed, including an action expert model part and a joint loss optimization part, specifically including: S41, Motion Expert Model Part: The motion is parameterized into a motion vector, and the point with the highest availability score is selected as the motion origin; combined with the features of this point and geometric information such as surface normals, the unit motion direction vector is regressed and predicted. S42. Joint Loss Optimization: Training is performed using a joint loss function, which consists of three parts: an asymmetric focus loss function to address the sparsity and imbalance of option labels; a pairwise ranking loss function to explicitly widen the score gap between high-scoring option points and background points; and a point-text contrast loss function to align instruction embeddings with relevant point features while separating them from background points. The final joint loss function is expressed as follows: in, Indicates asymmetric focal loss. Indicates pairwise ranking loss. This represents the point-to-text contrast loss. and This is the balance coefficient.

[0010] The present invention also provides a three-dimensional availability localization system for embodied robots based on a multimodal large language model.

[0011] The beneficial effects of this invention are as follows: This invention provides a framework called MLLM-Afford (Multimodal Large Model Affordance Localization), which effectively combines 3D point cloud geometric information with visual details through an adaptive multimodal representation module and utilizes an instruction-guided fusion module to deeply mine the semantic intent in natural language instructions. It constructs a unified "semantic-spatial" representation that not only possesses scene-level understanding capabilities but can also directly output executable action parameters. This design enables robots to handle complex instructions containing combined attributes and implicit intents, significantly improving the success rate of interactions in unstructured environments. MLLM-Afford integrates coarse-to-fine reasoning with spectral enhancement techniques, improving the model's operational efficiency and noise resistance on real robotic platforms by reducing computational overhead while maintaining high-resolution details. It effectively addresses the limitations of traditional object-level methods in accurate localization under complex backgrounds and occlusion conditions, improving the robot's adaptability and operational accuracy in real-world 3D environments.

[0012] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0013] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a detailed internal architecture and algorithm flow diagram of the MLLM-Afford framework of this invention; Figure 2 The present invention provides an overall flowchart of a three-dimensional availability localization method for embodied robots based on a multimodal large language model.

[0014] Figure 3 The diagram shows the qualitative comparison results of the availability localization of the present invention on the SceneFun3D dataset, illustrating the visualization effect comparison between the present invention and existing baseline methods under different instruction tasks. Figure 4 The visualization results of the availability localization, category prediction and action estimation generated by this invention demonstrate the model's ability to jointly reason about "what to do" and "where to do it"; Figure 5 The figure shows the results of the stepwise ablation experiment of the key components in this invention, illustrating the impact curves of modules such as dynamic sampling and multimodal fusion on performance indicators. Figure 6This is a schematic diagram of the experimental results of the present invention on a real robot platform, showing successful and unsuccessful cases of the robot performing various interactive tasks (such as opening doors, opening drawers, etc.) in real scenarios. Detailed Implementation

[0015] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0016] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0017] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0018] This invention provides a 3D affordance localization method and system for embodied robots based on a multimodal large language model (MLLM-Afford), which addresses the problem of interactive localization and motion prediction for robots in unstructured real-world environments. Currently, robots are transitioning from industrial to home environments, facing challenges such as complex scenarios, ambiguous instructions, and limited computational resources. Existing object-level modeling methods struggle to accurately locate interaction points, while directly applying large models incurs significant computational overhead. This invention proposes a novel end-to-end framework to address these problems, comprising the following four stages: First, an adaptive multimodal representation module is constructed, utilizing dynamic sampling and keyframe selection strategies to transform massive point cloud data into compact and detail-preserving multimodal features. Second, an instruction-guided multimodal fusion module leverages the semantic understanding capabilities of a large language model to inject natural language instructions into the 3D representation space. Next, in a spectral-enhanced coarse-to-fine availability reasoning module, a hierarchical strategy is employed to progressively narrow the search range, and spectral graph theory is used to enhance spatial consistency, thereby extracting precise availability regions. Finally, in the action expert and training optimization module, specific action execution directions are predicted, and a joint loss function guides the end-to-end optimization of the model. This framework improves the robot's efficiency in understanding complex scenes and the accuracy of action execution. Figure 1 This is a system block diagram of the present invention.

[0019] To better understand this invention, the relevant concepts involved in this invention are explained below: The first concept: 3D scene point cloud. This refers to the three-dimensional geometric representation reconstructed from the robot's RGB-D observation data, represented as... Where N is the number of points and R represents the set of real numbers. In large-scale real-world scenes (such as living rooms and kitchens), point clouds contain the geometry, spatial location, and environmental context information of objects. Scene point clouds are the foundational data for robots to perceive the physical world and are used for subsequent affordance reasoning.

[0020] The second concept: Instruction prompts. To enable large language models to understand specific localization tasks, a specific format of natural language input is constructed. The specific format is " <scene>[Description]. Please find the region in this scene where you can [Action].”。其中, <scene>[[Description]] is a special scene token used to inject visual features; [[Action]] is a high-level semantic description of the scene; [[Action]] specifies the expected type of interaction action (such as "open" or "pull"). This cue design explicitly encodes the scene context and the affordance intent.

[0021] The third concept is adaptive dynamic sampling. This refers to a downsampling strategy for point clouds, designed to resolve the conflict between large-scale scene point clouds and computational budget. Unlike traditional uniform sampling, this strategy assigns sampling weights based on the edge intensity of the image. Regions with high edge intensity (typically corresponding to detailed structures such as object outlines, handles, and switches) are given higher sampling weights, thereby significantly reducing the number of points in flat background areas while preserving details in key interactive areas, thus improving computational efficiency.

[0022] The fourth concept is gating. This refers to a feature fusion technique used to balance the general reasoning capabilities of a large language model with specific task instructions. Through a learnable gating network, a regulation vector *c* is calculated using the sigmoid function to control the weights of the instruction text embedding *t* and the localization intent *g* on the 3D point features. This mechanism allows the model to flexibly combine general knowledge from LLM with specific instruction semantics.

[0023] The fifth concept is spectral feature enhancement. This utilizes spectral analysis methods from graph theory to process point cloud features. By constructing a k-nearest neighbor graph and calculating the low-frequency eigenvector U of its normalized Laplacian matrix, the point cloud features are projected into the spectral domain for transformation. This operation propagates feature responses between neighboring points, enhancing the consistency of local geometry and making the predicted availability region smoother and more continuous, thus addressing the issues of sparse point features and noise.

[0024] The sixth concept is the action vector. It's used to parameterize the robot's actions. For each interaction point in the scene, the action vector is a unit vector. This indicates the direction of the operation. For example, for jointed objects (such as doors and drawers), the action is usually defined as an outward pull along the surface normal, rather than a complex joint axis rotation, to ensure the robot's performance stability in a real environment.

[0025] The seventh concept is superpoints. Based on geometric and feature similarity, superpoints are basic units formed by clustering spatially connected point clouds to reduce the search space for coarse-grained localization.

[0026] like Figure 1 As shown, the system framework of this system mainly includes four modules: adaptive multimodal representation module, instruction-guided multimodal fusion module, spectrum-enhanced coarse-to-fine availability reasoning module, and training optimization module. Figure 2 The diagram shown is a system block diagram of the present invention, illustrating the overall logic from input to output. Wherein: Adaptive Multimodal Representation Module: This module aims to resolve the conflict between large-scale scene point clouds and limited computational budgets. It optimizes data representation by preserving high-frequency details and compressing low-frequency regions, including the following steps: Step 1: Point Cloud Preprocessing and Edge Strength Calculation. The input image I is received, first converted to grayscale, and Gaussian smoothing is applied to reduce image noise. Next, the Canny operator is used to extract the edge response of the image, thus obtaining the edge map E. Specifically, the image is divided into multiple image blocks, and for each image block... Calculate its edge strength Edge strength The calculation formula is: Where (x,y) represents the pixel coordinates within the image patch. This represents the edge response value at that coordinate. Edge strength. The larger the value, the richer the texture or structural details contained in the area corresponding to the image patch, such as the edges of objects, handles, or switches.

[0027] Step 2: Adaptive Dynamic Sampling and Geometric Feature Extraction. Adaptive dynamic farthest point sampling is performed based on the calculated edge intensity. Points with higher edge intensity are selected. Image patches are processed by projecting pixels from these patches into 3D space using depth information, and assigning higher sampling weights to the corresponding 3D points in these regions, resulting in denser sampling during dynamic sampling. For other flat regions with lower edge intensity, sampling is performed using uniform farthest point sampling to maintain coverage of the entire scene. This strategy ensures sufficiently high resolution in key interactive areas with a fixed number of points. After sampling, a sparse 3D U-Net network is used to extract geometric features from the sampled point cloud, yielding a geometric feature vector. In addition, sinusoidal three-dimensional position coding was added to preserve absolute spatial information.

[0028] Step 3: Since availability localization requires precise alignment between 3D points and visual information, and the number of visual tokens needs to be kept controllable, this step adopts a greedy keyframe selection strategy. First, given the sample point set P and candidate video frames f, a binary visibility index is defined. Specifically, if the point If the projection lies within the image boundary of frame f and satisfies the depth consistency constraint, then let... Secondly, a set of the most representative keyframes F is selected by maximizing the total point coverage. The objective function is chosen as follows: A greedy algorithm iteratively adds frames that cover the maximum number of currently uncovered points. Finally, after keyframe selection, each 3D point... Assign the keyframe with the most information The allocation is based on maximizing the score function. .in It is a linearly weighted score that comprehensively considers point visibility, depth consistency, normalized image position, and the size of the projected image patch. Using known camera intrinsics and pose, each point is projected onto its assigned keyframe, and a fixed-size image patch is extracted from that frame to encode its visual appearance, generating point-aligned visual tokens that are naturally consistent with the 3D geometry.

[0029] Instruction-guided multimodal fusion module: This module aims to use semantic information in natural language instructions to guide the model to focus on specific areas in the scene (e.g., focusing on the refrigerator door handle when hearing "open the refrigerator"), including the following steps: Step 1: Extract command features and locate intent. Construct command prompts that include scene context and affordance intents, specifically in the format " <scene>[Description]. Please find the region in this scene where you can [Action]. Input this instruction into the large language model. Extract specific details from the large language model. <ground>The hidden state of the token is denoted as This vector g summarizes the localization intent of the instruction. Simultaneously, the natural language description and availability function labels are encoded into a text embedding, denoted as . .

[0030] Step 2: Gating Fusion and Feature Modulation. To fuse these features in a unified representation space, a gating mechanism is designed. Two lightweight multilayer perceptrons are used, denoted as follows: and and the Sigmoid activation function The hidden state g and the text embedding t are fused into a single adjustment vector c. The specific fusion formula is as follows: .

[0031] in, The vector concatenation operation is represented by . This gating mechanism allows the model to adaptively balance the general reasoning ability from the LLM (represented by g) and the specific instruction text semantics (represented by t). Subsequently, using the generated adjustment vector c, the geometric and visual features of the point cloud are weighted and modulated through a language-conditional cross-attention mechanism to generate an instruction-aware multimodal point feature matrix F.

[0032] The spectral-enhanced coarse-to-fine availability reasoning module addresses the issue that direct point-by-point prediction in large-scale scenes is inefficient and susceptible to noise interference. Therefore, a hierarchical strategy is employed, including the following steps: Step 1: Coarse-grained Availability Localization. First, based on spatial geometric connectivity and feature similarity, the point cloud is clustered into several superpoints. Each superpoint is treated as a basic unit, and its relevance to the instruction is calculated. The model first predicts coarse-grained availability scores at the superpoint level, filtering out a large number of irrelevant background areas (such as walls, floors, and other irrelevant superpoints), thus significantly reducing the search space for subsequent fine-grained processing.

[0033] Step 2: Spectral Feature Enhancement. A k-nearest neighbor graph is constructed on the selected candidate region point cloud. Calculate the normalized Laplacian matrix of the graph and extract its corresponding low-frequency eigenvector matrix, denoted as U. Enhance the point feature matrix X using residual spectral transform; the specific calculation formula is as follows: in, It is a learnable transformation function. This indicates that the features are projected onto the spectral domain. This indicates that the transformed image is projected back into the spatial domain. This step utilizes the smoothing properties of the spectral domain to propagate the command response between adjacent points, enhancing the clarity of local boundaries and spatial consistency, making the features more robust to noise.

[0034] Step 3: Introduce a set of learnable query embeddings Where Q is the number of queries. To utilize the language context, the instruction adjustment vector c is injected into the query as a shared bias, as shown in the formula: The decoder uses a multi-layer attention mechanism, first through... Spatial information is collected from point features F, and then through... The linguistic context is back-injected into the point features. Finally, an availability score for each point i is generated by calculating cosine similarity and aggregating query results. The specific calculation formula is as follows: Where τ is a learnable temperature parameter, and b is a learnable bias term. Let Wq and Wp represent the vector dot product, and Wq and Wp be the projection matrices. This formula ensures that the model can capture multiple implicit availability patterns.

[0035] Training Optimization Module: This module aims not only to find the location but also to guide the robot's actions, and addresses the sample imbalance problem during training through a specific loss function. It includes the following steps: Step 1: Predict the direction of movement. Select the availability score. The highest point is used as the origin of the motion. The motion expert model combines the features of this point with geometric information such as surface normals to regress and predict the unit motion direction vector. In real robot manipulation, for jointed objects (such as doors and drawers), the motion is defined as an outward pull along the surface normal, rather than a precise rotation of the joint axis. During training, using... Loss monitoring action prediction: ,in It is the actual direction vector of the action.

[0036] Step 2: Calculate the joint loss function and train. To train the model, a joint loss function consisting of three parts is constructed. The detailed calculation formula for each part is as follows: (1) Asymmetric Focal Loss. Used to supervise point-level availability scoring, addressing the problem of extremely sparse and imbalanced availability labels (positive samples) in point clouds. For each point i, its predicted probability is... The loss function is defined as: in, It is a modified probability used to suppress simple background negative samples. V is the true label, and V is the set of valid points. The parameters γ+ and γ- control the weights of positive and negative samples, respectively, and δ is the asymmetric truncation threshold.

[0037] Pairwise ranking loss. To enforce the distinction between high-confidence foreground and background points, the following formula is used: ,in It is a high confidence point. is the background point, and m is the preset boundary value. This loss explicitly widens the score gap between the foreground and background, alleviating the flat activation problem.

[0038] Point-to-text contrast loss. This is used to align the instruction text embedding t with the associated point features. Simultaneously, background points are zoomed out, and a loss function in the form of InfoNCE is used: Where P is the set of positive sample points, N is the set of negative sample points, and τ is the temperature coefficient. Indicates the inner product.

[0039] The ultimate training goal is the weighted sum of the above three factors. The gradient of the total loss L with respect to the model parameters is calculated using the backpropagation algorithm, and the network parameters are updated to achieve end-to-end model optimization.

[0040] To verify the effectiveness and generalization ability of the technical solution provided by the present invention, this embodiment conducted extensive evaluation experiments on the benchmark dataset SceneFun3D and a real robot platform, tested the overall performance of the technical solution (MLLM-Afford) of the present invention, and compared it in detail with a number of existing representative methods.

[0041] This embodiment conducts experiments on SceneFun3D, a task-driven 3D power delivery dataset for scene-level 3D power delivery grounding with language instructions. The dataset contains 230 indoor scenes (200 for training and 30 for validation), each providing high-quality 3D point clouds and corresponding power delivery annotations. This embodiment employs detection and execution metrics. For detection, this application reports the mean Intersection over Union (mIoU) and the mean accuracy (AP, AP25, AP50) at different IoU thresholds. For execution, this application reports Top1Dist, measuring the distance between the highest-scoring predicted point and the nearest ground truth point. Succ@r is used to calculate the success rate of the top prediction within the tolerance radius $r$. Hit@K@r is used to evaluate whether at least one of the top K predictions falls within the tolerance range. This application further reports two extended metrics (+T and +TD) based on AP25, which additionally require correct action type and direction of motion. Figure 3 The image shows the qualitative comparison results of the availability localization of the present invention on the SceneFun3D dataset, illustrating the visualization effect comparison between the present invention and existing baseline methods under different instruction tasks.

[0042] This embodiment builds MLLM-Afford on top of LLaVA-Video-7B-Qwen2 and uses LoRA to fine-tune the model for efficient parameter training. The model is trained using the Adam optimizer with a cosine learning rate scheduling scheme for a total of 5 training epochs. Referring to the Scene-Fun3D settings, models with the suffix "-F" indicate that they have undergone task-specific adaptation. Given that LLaVA-3D and Video-3D-LLM were originally designed for general 3D grounding rather than power prediction, this embodiment applies task-specific adaptations to ensure fair comparison under the same point-level power protocols. For LLaVA-3D-F, this embodiment adds a point-to-point scoring head and fine-tunes it using the same downsampled multimodal representation as in this application. For Video-3D-LLM-F, this embodiment replaces object proposals with power component proposals from Mask 3D and fine-tunes the model.

[0043] This embodiment includes comprehensive experiments to evaluate the performance of power grounding under different modeling paradigms. In addition to positioning accuracy, this application also reports model size and inference efficiency to reflect considerations in real-world deployments. This unified evaluation protocol enables comparisons of different methods within a consistent and task-relevant framework.

[0044] Scenario-level powered grounding comparison. To evaluate the scenario-level powered grounding capability of different models in real-world environments, this embodiment compares the model of this application with a representative benchmark model, focusing on the detection performance indicators of scenario-level powered grounding. The experimental results are shown in Table 1.

[0045] Table 1 Table 1 reports the mIoU and AP, AP25, and AP50 metrics used to evaluate the grounding accuracy of the model in a real-world environment. Quantitative results show that MLLM-Afford significantly outperforms the benchmark model across all detection metrics. Specifically, the model in this application achieves an mIoU of 23.28, significantly exceeding OVA-Fields (8.30), LLaVA-3D-F (6.24), and Video-3D-LLM-F (7.78). A similar trend was observed in the detection metrics, with MLLM-Afford reaching 18.72 AP, 36.84 AP25, and 21.43 AP50. These results demonstrate that the model in this application can produce more accurate grounding indications.

[0046] The performance improvement does not stem from a larger model size, but primarily from different modeling strategies. Most of the aforementioned models follow a "classify first, then aggregate" paradigm, that is, first predicting power at the image or instance level, and then projecting or aggregating the predictions into 3D space. This type of design is typically optimized for object-level detection, thus limiting its ability to capture semantic similarity and spatial continuity between power signals in complex scenes. In contrast, MLLM-Afford performs dense power signal grounding directly in the 3D point space. This embodiment unifies semantic understanding, multimodal fusion, and 3D spatial grounding into a single representation space, thereby achieving joint modeling of power signal semantics and continuous spatial distribution. Therefore, MLLM-Afford achieves excellent performance in both mIoU and detection metrics, validating its effectiveness for scene-level power signal grounding.

[0047] This paper compares the predictability of executable capabilities for robotics. In real-world robotic execution scenarios, simply locating semantically relevant regions is insufficient for the robot. The localization must also satisfy the robot's feasibility in the real environment. Quantitative results show that MLLM-Afford achieves substantial advantages across all execution metrics. Specifically, our model achieves 87.86% on Hit@5@r, significantly outperforming OVA-Fields (37.90%), LLaVA-3D-F (33.17%), and Video-3D-LLM-F (44.67%), demonstrating that this model can more reliably predict effective interaction positions within the robot's operating radius. Simultaneously, MLLM-Afford achieves 64.73% on the Succ@r metric, significantly outperforming the benchmark model and validating the reliability of our model. Furthermore, our model reduces the average distance between the optimal predicted point and the actual executable interaction point to 0.38 Top1Dist, demonstrating its more accurate spatial alignment capability compared to the benchmark model.

[0048] The results above indicate that the differences in performance metrics among the models mainly stem from their different emphases on modeling usability goals. Existing models treat scene-level power indication reasoning as a prediction problem of semantic or visual relevance; therefore, their outputs are typically object-level masks or region-level 3D bounding boxes. While such outputs are reasonable, they do not explicitly constrain the consistency between the predicted location and the robot's actual executable space, thus exhibiting certain limitations in performance metrics. In contrast, MLLM-Afford focuses on real-world robot execution. By introducing an adaptive multimodal sampling strategy, the model in this application can efficiently focus on key spatial regions related to language instructions. This design enables the generated power indication grounding to cover feasible regions with strong spatial consistency, thereby generating significant advantages in performance metrics.

[0049] Motion-aware power indication prediction comparison. In addition to scene-level power indication grounding, the robot also needs to identify the interaction type and motion attributes related to power indication. Therefore, this embodiment further analyzes MLLM-Afford in the motion-aware power indication prediction task and compares it with Fun3DU (+motion) and AffordBot. The results are shown in Table 2: Table 2 according to Figure 4 As can be seen, MLLM-Afford outperforms the benchmark model in terms of metrics because it jointly considers interaction type and motion information. Specifically, the model in this application achieves a +T metric of 30.1 and a +TD metric of 22.6. The improvement in +TD is particularly significant, indicating that the model in this application has a stronger ability to identify "content" and "location". This advantage is mainly attributed to the integrated modeling of energy and motion parameters. The benchmark model treats motion parameters as independent additional prediction terms, which can easily introduce inconsistencies in complex scenes; while MLLM-Afford treats motion information as a natural component of energy inference. By guiding the motion expert model to perform joint inference on energy and its corresponding motion through instructions, MLLM-Afford achieves more consistent prediction results.

[0050] To verify the effectiveness of dynamic farthest-point sampling, this embodiment evaluates the impact of dynamic FPS on point-level power grounding. The results are shown in Table 3 and... Figure 5 : Table 3 This indicates that removing dynamic FPS results in a significant performance degradation in both detection and execution metrics. This suggests that uniform sampling typically discards energy-related regions during downsampling and introduces subsequent, unrecoverable errors. In contrast, dynamic FPS preserves critical interaction regions, providing a robust foundation for coarse-to-fine inference and executable energy prediction.

[0051] The examples further investigated the role of the instruction-guided multimodal fusion mechanism. After removing this module, the model exhibited significant performance degradation across multiple metrics. As shown in Table 3, AP25 decreased from 36.84 to 27.46, and Succ@r decreased from 64.73 to 34.91. This result indicates that fixed multimodal fusion introduces irrelevant or conflicting information, especially when different energies depend on different modal data. Instruction-guided fusion enables task-aware modality selection, leading to better semantic alignment and more stable predictions.

[0052] This embodiment analyzes a coarse-to-fine grounding strategy integrating superpoint proposal, spectral enhancement, and a point-level decoder. Removing the coarse stage based on superpoints leads to instability in point-level predictions, indicating that structured region proposals are necessary for reliable grounding. Disabling spectral enhancement further reduces execution quality, demonstrating the importance of spatial consistency between neighboring points. Replacing the point-level decoder with a simple multilayer perceptron also reduces execution accuracy, indicating that coarse luminous cues alone are insufficient for accurate action localization. The aforementioned components are complementary, collectively achieving accurate and executable grounding.

[0053] Finally, this embodiment investigates the effectiveness of ranking and auxiliary losses for supervised energy indication prediction. Due to the extreme sparseness of energy indication points, relying solely on Focal loss is insufficient to learn the relative importance between candidate regions. Removing ranking and auxiliary losses significantly degrades execution performance. These losses guide the model to learn the relative differences and local consistency between candidate regions, thereby encouraging the model to output stable energy indication regions rather than isolated high-response points. This design further enhances the effective integration of MLLM-Afford with end-to-end energy indication modeling and real-world robot execution.

[0054] This embodiment utilizes a hierarchical execution pipeline connecting scene-level power grounding and underlying robot control to deploy MLLM-Afford on a real robot platform, such as... Figure 6 As shown. For real-world robot experiments, this application fine-tunes the model using point cloud data collected from the actual environment. Given an RGB-D-based scene reconstruction, the robot predicts the command-driven enabled region in the global point cloud and selects the target interaction point.

[0055] Subsequently, the mobile chassis navigates to a pre-interactive posture, located approximately 0.4 meters from the target along the estimated surface normal. This design ensures target accessibility and ease of observation. Upon reaching the preset posture, this embodiment utilizes a wrist camera to acquire local depth observation data, refining the target position in a closed-loop manner. This application applies AnyGrasp to the images captured by the wrist camera to estimate the end effector posture to establish grasping.

[0056] This strategy enables reliable execution across a variety of interactive tasks, such as opening doors, pulling drawers, and operating appliance handles. Execution failures primarily occur when the visible area is extremely small or visual features are blurred, making accurate positioning difficult in cluttered environments.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.< / ground> < / scene> < / scene> < / scene>

Claims

1. A three-dimensional availability localization method for embodied robots based on a multimodal large language model, characterized in that, This method constructs an end-to-end learning framework for 3D scene availability localization of the robot, achieving 3D availability localization of the robot. The method specifically includes the following steps: S1. Construct an adaptive multimodal representation module that uses the original 3D point cloud, RGB-D observation data and natural language instructions as input. Through an adaptive dynamic farthest point sampling strategy, greedy keyframe selection and point-aligned visual encoding, it transforms multi-view RGB-D observations into compact point-aligned visual labels consistent with 3D geometry. S2. Construct a command-guided multimodal fusion module. By constructing command prompts that include scene context and availability intent, and using gating and cross-attention mechanisms, inject language semantics into the 3D representation to achieve modal selection for task awareness. S3. Construct a spectrum-enhanced coarse-to-fine availability reasoning module. The module first identifies approximate regions that support availability through superpoint clustering, then applies residual spectrum transformation to enhance point features to strengthen spatial consistency, and finally predicts fine availability probabilities through availability decoder. S4. Construct a training optimization module, introduce a motion expert model to parameterize the action into a motion vector, predict the starting point and direction of the action by combining point features and geometric information, and use asymmetric focal loss function, pairwise ranking loss and point-text contrast loss for joint training. S5. Use the constructed model to perform three-dimensional availability localization of the embodied robot.

2. The method for three-dimensional availability localization of an embodied robot based on a multimodal large language model according to claim 1, characterized in that, Step S1 mainly consists of three parts: point cloud preprocessing and geometric encoding, and greedy keyframe selection and visual encoding. The specific steps include: S11. Point Cloud Preprocessing and Geometric Encoding: For the input image, the Canny operator is used to extract the edge response to obtain the edge map. The image is then divided into blocks and image blocks are calculated. edge strength Image patches with high edge intensity are selected, projected into 3D space, and assigned higher sampling weights for dense sampling. The remaining points are sampled uniformly, thereby achieving adaptive dynamic farthest point sampling. At the same time, geometric features are extracted and sinusoidal 3D position coding is added. S12. Greedy Keyframe Selection: Defining a Binary Visibility Metric A greedy coverage strategy is adopted to select the key frame set F that maximizes the point coverage, ensuring accurate alignment between 3D points and visual information, while keeping the number of visual markers within a manageable range. S13, Point-aligned visual encoding: Assign the most relevant keyframe to each 3D point based on point visibility, depth consistency, normalized image position and projection block size, and project the point to the corresponding frame using camera intrinsics and pose to encode the visual appearance from a fixed-size image block.

3. The method for three-dimensional availability localization of an embodied robot based on a multimodal large language model according to claim 2, characterized in that, Step S2 specifically includes: S21. Construct a prompt template that includes scene description and action intent, and introduce special scene tokens to inject scene-level multimodal features; S22. Extract the hidden state of special location tokens and the text embedding of instructions from the large language model. S23. Design a gating mechanism to fuse the above representations into a single adjustment vector. This mechanism uses a lightweight multilayer perceptron and a sigmoid function to calculate the gating weights and perform weighted summation and nonlinear transformation on the hidden state and text embedding. S24. Using the generated adjustment vector, the point features are modulated through a cross-attention mechanism to realize the injection of language semantics into the three-dimensional representation space.

4. The method for three-dimensional availability localization of an embodied robot based on a multimodal large language model according to claim 3, characterized in that, In step S3, a coarse-to-fine availability reasoning module with spectral enhancement is constructed, mainly consisting of coarse-grained localization, spectral feature enhancement, and fine-grained decoding, specifically including: S31. Coarse-grained localization: Based on spatial distance and feature similarity, point clouds are clustered into superpoints, each superpoint is used as the basic unit of coarse-grained localization, and irrelevant background areas are filtered out. S32. Spectral feature enhancement: Construct a k-nearest neighbor graph on the sampled point cloud, and use the low-frequency feature vectors of the graph Laplacian operator to perform residual spectral transformation, propagating the command-aligned response while preserving local boundaries; S33, Fine-grained decoding: A set of learnable query embeddings is introduced, and the instruction adjustment vector is injected as a shared bias; the decoder uses a multi-layer attention mechanism to enable the query embeddings to interact with point features, and finally projects them to the shared space to calculate point-level availability logic values.

5. The method for three-dimensional availability localization of an embodied robot based on a multimodal large language model according to claim 4, characterized in that, In step S4, a training optimization module is constructed, including an action expert model part and a joint loss optimization part, specifically including: S41, Motion Expert Model Part: The motion is parameterized into a motion vector, and the point with the highest availability score is selected as the motion origin; combined with the features of this point and geometric information such as surface normals, the unit motion direction vector is regressed and predicted. S42. Joint Loss Optimization: Training is performed using a joint loss function, which consists of three parts: an asymmetric focus loss function to address the sparsity and imbalance of option labels; a pairwise ranking loss function to explicitly widen the score gap between high-scoring option points and background points; and a point-text contrast loss function to align instruction embeddings with relevant point features while separating them from background points. The final joint loss function is expressed as follows: in, Indicates asymmetric focal loss. Indicates pairwise ranking loss. This represents the point-to-text contrast loss. and This is the balance coefficient.

6. A three-dimensional availability localization system for embodied robots based on a multimodal large language model, characterized in that, The system employs the method described in any one of claims 1 to 5.

Images