Multi-modal retrieval zero-shot robot operation method and device based on 3DGS

Abstract

The application discloses a multi-modal retrieval zero-sample robot operation method and device based on 3DGS. First, a multi-source operation knowledge base containing object contact frames, task completion frames and pose parameters is constructed. In the reasoning process, hierarchical multi-modal retrieval is adopted, a three-priority hybrid retrieval strategy is used to find object prototypes related to the task, and the closest reference sample in geometry is selected based on pixel-level similarity and instance matching distance (IMD). Second, a 3D perception pose optimization based on 3D Gaussian splash is introduced to align the pose of the reference object with the target object in 3D space. Finally, the aligned result is projected onto the image plane again and provided as input to the multi-modal large language model (MLLM) to enhance the generation of the final operation pose parameters and improve the operation generalization and accuracy in the unseen object scene.

Description

Technical Field

[0001] This invention relates to the fields of robotics and artificial intelligence, and more specifically, to a pose estimation method for robot operation, particularly a robot operation method and apparatus based on 3D Gaussian sputtering (3DGS) that can guide a robot to interact accurately with unseen objects in a zero-shot scenario. Background Technology

[0002] Robotics manipulation is a core branch of modern robotics, fundamentally requiring robots to interact accurately and reliably with diverse objects in the physical world. In all interactive tasks, whether grasping, pushing, pulling, or more complex operations, accurate prediction of the contact point and the three-dimensional (3D) pose is a prerequisite for success. Without reliable estimation of object pose, robots will frequently fall into a cycle of trial and error, leading to inefficiency and task failure. Therefore, endowing robots with powerful reasoning and generalization capabilities, enabling them to accurately predict low-level motion parameters and adapt to dynamic environments, is a key objective pursued in this field.

[0003] While research into end-to-end learning methods for grasping pose prediction has made some progress, these methods generally face two major challenges. First, their generalization ability is severely limited by the scale and diversity of training data, making it difficult to cover the infinite number of object categories and scenes in the real world. Second, the "black box" nature of deep learning models makes them lack the common-sense reasoning abilities possessed by humans, resulting in vulnerability when faced with novel situations.

[0004] With the rise of Multimodal Large Language Models (MLLMs), they have shown great potential in unifying visual perception, language understanding, and action planning. Some works, such as RT-2 and RoboMamba, utilize large-scale pre-training to translate natural language instructions into executable robot control commands. However, existing research indicates that while MLLMs excel in high-level semantic planning, their understanding of fine geometry and spatial layout remains insufficient, directly limiting their effectiveness in grasping pose prediction tasks requiring sub-centimeter accuracy. For example, even when MLLMs know that opening a drawer requires manipulating a handle, their predicted contact point coordinates often deviate from the object itself and cannot be directly used for robot execution.

[0005] Zero-shot generalization is the cornerstone of building general-purpose agents. It requires robots to manipulate previously unseen objects and adapt to different robot hardware and environments without specific training. To achieve this goal, existing methods either rely on large-scale expert teaching or attempt to transfer learning skills from videos of everyday human activities. For example, RAM methods perform operations by retrieving similar 2D affordances from a database and "lifting" them into 3D space. However, the bottleneck of these methods lies in the extreme fragility of their 2D-to-3D lifting process when the object's perspective or geometry changes, and the fact that MLLM's inference process is detached from the geometric core, making it difficult to guarantee pose accuracy in complex scenes.

[0006] In summary, a significant "semantic-geometric gap" exists in the current field of robotics manipulation: there is a lack of effective connection between high-level task understanding capabilities (provided by MLLM) and low-level precise geometric execution capabilities (required for robot manipulation). Existing methods either lack generalization ability or sufficient geometric accuracy, making it difficult to meet the complex and ever-changing operational needs of the real world. Summary of the Invention

[0007] The core objective of this invention is to overcome the aforementioned deficiencies of existing technologies and provide a multimodal retrieval zero-shot robot operation method and apparatus based on 3D Gaussian splashing. This invention aims to effectively bridge the gap between semantic understanding and geometric execution through an innovative method, thereby significantly improving the success rate and accuracy of robots operating on unseen objects in zero-shot scenarios.

[0008] To achieve this objective, the specific technical solution of the present invention is as follows:

[0009] A zero-shot robot operation method for multimodal retrieval based on 3DGS, characterized by the following steps:

[0010] Step 1: Construct a multi-source operation knowledge base, which contains multiple operation cases. Each case includes at least one key action frame, instruction text describing the task, and 6D grasping pose parameters. The key action frame includes an object contact frame and / or a task completion frame.

[0011] Step 2: Receive the natural language target task instructions and the current scene observation image containing the target object issued by the user;

[0012] Step 3: Based on the natural language target task instructions, a three-priority hybrid retrieval strategy is adopted to search the knowledge base. First, sparse retrieval is used for exact matching. If it fails, it switches to dense retrieval based on semantic embedding to obtain the semantically closest candidate sample set.

[0013] Step 4: Calculate the cosine similarity between the current scene observation image and the contact frames of each example in the candidate example set using the visual language model CLIP, and filter out the top-n candidate examples that are most similar in visual appearance to narrow the search range.

[0014] Step 5: Using the Instance Matching Distance (IMD) algorithm, calculate the geometric shape difference between the observed object and each of the top-n candidate examples, and select the best reference example that is geometrically the best match by re-sorting;

[0015] Steps 3, 4, and 5 are combined to form a hierarchical multimodal retrieval.

[0016] Step 6: If the IMD value of the best reference sample and the target object is higher than the preset threshold, then start 3D perception pose optimization; use 3D Gaussian sputtering (3DGS) technology to generate its three-dimensional representation from the RGB-D image of the reference sample, and find the pose that is best aligned with the target object in 3D space by performing small-angle rotation transformation and differentiable re-rendering on the three-dimensional representation, and generate the optimized reference sample.

[0017] Step 7: Combine the natural language target task instructions, the current scene observation image, and the best reference example from Step 5 or the optimized reference example from Step 6 into a structured prompt.

[0018] Step 8: Input the structured prompts into a pre-trained multimodal large language model (MLLM), which infers and generates an accurate final operation pose for performing the target task, including 2D contact points and 3D end effector orientation.

[0019] Furthermore, the hierarchical multimodal retrieval specifically includes:

[0020] Text semantic retrieval: A three-priority hybrid retrieval strategy is adopted. First, sparse retrieval is used to match object names; if no match is found, it switches to dense retrieval based on semantic embedding; if the highest similarity of the dense retrieval is lower than a preset threshold, the retrieval scope is expanded to other data sources, thereby filtering out the candidate sample set with the closest semantics. The calculation formula is as follows:

[0021]

[0022] in This is a semantic embedding vector for natural language instruction text. For example, the instruction text semantic embedding vector is the knowledge base example, and Enc() represents the text encoder; find the semantically closest substitute object in the semantic space; for example, the instruction "take the phone" may retrieve the operation example "take the tablet", because the two have similar features in the semantic space;

[0023] Visual similarity filtering: Calculate the cosine similarity between the observed image and each contact frame in the candidate sample set using a pre-trained visual language model (such as CLIP) to select the top-n most visually similar candidate samples.

[0024] Geometric matching: The instance matching distance (IMD) is used to calculate the geometric difference between the observed object and visually similar candidate examples, and the example with the lowest IMD score is selected as the best reference example.

[0025] Furthermore, the 3D perception pose optimization specifically includes:

[0026] A 3D Gaussian sputtering (3DGS) representation is generated from an RGB-D image of a reference sample using a pre-trained generative model;

[0027] Apply a set of predefined small-angle rotation transformations to the reference grasp pose to generate multiple candidate poses;

[0028] Each candidate pose is reprojected onto a two-dimensional image plane using 3D Gaussian sputtering differentiable rendering to generate the corresponding candidate contact frame.

[0029] The instance matching distance (IMD) between each candidate contact frame and the observed image is recalculated, and the candidate pose with the smallest IMD is selected as the optimized reference pose.

[0030] Furthermore, the multimodal large language model MLLM is fine-tuned using low-rank adaptive (LoRA) technology, and its joint loss function combines masked language modeling (MLM) loss and pose regression loss.

[0031] Furthermore, the pose regression loss is defined as follows:

[0032]

[0033] Where (x, y) are the actual two-dimensional contact point coordinates, and (x^, y^) are the predicted contact point coordinates; d u , d f Let d^ be the upward and forward direction vectors of the actual end effector. u , d^ f λ1 and λ2 are the predicted direction vectors; λ1 and λ2 are the balancing weight coefficients.

[0034] A 3DGS-based multimodal retrieval zero-shot robot operation device includes:

[0035] The knowledge base construction unit is used to build a multi-source operation knowledge base containing operation cases.

[0036] The retrieval unit is used to perform hierarchical multimodal retrieval based on user instructions and observed images to obtain the optimal reference example from the knowledge base;

[0037] The pose optimization unit is used to perform 3D perception pose optimization on the retrieved reference sample using 3DGS technology, so that it is aligned with the target object;

[0038] The generation unit is used to input observation data, user instructions, and optimized reference poses into the multimodal large language model to generate the final robot operation pose.

[0039] Furthermore, the retrieval unit specifically includes:

[0040] Text semantic retrieval is used to execute a three-priority hybrid retrieval strategy;

[0041] Visual similarity filtering is used to filter candidate samples by calculating the similarity of visual features;

[0042] Geometric matching is used to determine the best reference sample by calculating the instance matching distance.

[0043] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the aforementioned 3DGS-based multimodal retrieval zero-shot robot operation method.

[0044] The beneficial effects of this invention compared to the prior art are reflected in:

[0045] 1) Excellent zero-shot generalization ability: Through the retrieval enhancement paradigm, this invention enables robots to effectively utilize external knowledge bases to handle objects and tasks that have never been seen before.

[0046] 2) High geometric accuracy: Innovative 3D perception pose optimization uses 3D Gaussian sputtering technology to solve the inherent inconsistency of viewpoints in 2D retrieval, achieving accurate 3D spatial alignment and providing a solid geometric foundation for the final pose generation.

[0047] 3) Deep integration of semantics and geometry: The method of this invention successfully connects the powerful high-level semantic reasoning capability of MLLM with the low-level geometric parameter execution capability required for robot operation, so that the robot not only "understands" the task, but also "accurately" executes it.

[0048] 4) Inherent interpretability and reliability: Unlike the end-to-end black box model, this invention makes the robot's decision-making process transparent and traceable through the retrieved specific reference examples, which is easy for humans to understand, debug and trust. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the method flow provided in an embodiment of the present invention;

[0050] Figure 2 This is a visual flowchart of the hierarchical multimodal retrieval strategy in an embodiment of the present invention, showing the three-level filtering process from text retrieval to visual filtering to geometric matching;

[0051] Figure 3 This is a schematic diagram of the 3D perception pose optimization workflow in an embodiment of the present invention, illustrating the process of pose alignment using 3D Gaussian sputtering;

[0052] Figure 4 This is a structural block diagram of the robot operating device provided in an embodiment of the present invention;

[0053] Figure 5 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be described more clearly and completely below with reference to the accompanying drawings. It should be noted that the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0055] This invention proposes a zero-shot robot operation method based on 3DGS for multimodal retrieval, named RobMRAG (Robotic Multimodal Retrieval-Augmented Generation). Its technical solution includes the following aspects:

[0056] First, this invention proposes constructing a rich and diverse operational knowledge base. This knowledge base integrates high-quality simulation data, real robot operation data, and extensive internet video data to provide the robot with a vast amount of operational prior knowledge. Each entry in the knowledge base is meticulously designed, containing task instructions, key action frames (such as the moment the robot's hand contacts an object), and associated precise pose parameters. This knowledge base forms the foundation for the robot's "learning by analogy" approach.

[0057] Secondly, this invention designs an ingenious hierarchical multimodal retrieval mechanism. When the robot receives new task instructions and scene images, this mechanism initiates a cascading retrieval process. The first layer performs textual semantic retrieval, quickly filtering the most relevant operation cases from the knowledge base. The second layer performs visual similarity filtering, further selecting examples from semantically relevant cases that are most similar in appearance to objects in the current scene. The third layer performs refined geometric matching, comparing the similarity of three-dimensional shapes of objects by calculating the instance matching distance (IMD), ultimately locking in the best reference example that best matches the current target object geometrically. This hierarchical strategy ensures that the retrieval results are highly relevant to the current task in semantic, visual, and geometric dimensions.

[0058] Furthermore, to address the potential viewpoint differences between the retrieved reference sample and the current target object, this invention introduces a core innovation: a 3D perception pose optimization based on 3D Gaussian Splatting. This module can reconstruct a differentiable 3D representation of the object from a single or a few images of the reference sample. Then, by applying a series of rotational transformations to this 3D model and utilizing its differentiable rendering properties, images viewed from different virtual perspectives can be generated. By comparing these newly generated virtual views with the real observed images (again using IMD), the system can find an optimal pose that perfectly aligns the reference object and the target object in 3D space. This step significantly improves the geometric accuracy of the reference information.

[0059] Finally, this invention packages the carefully selected and optimized reference examples (including their images and 3D-aligned precise pose parameters) along with the original user commands and observed images into an informative prompt, which is then fed into a multimodal large language model (MLLM). By providing high-quality, geometrically calibrated contextual information, the MLLM can more deeply understand the geometric constraints of the task, thereby generating a final operational pose that is far more accurate and reliable than traditional methods, which can be directly used to drive the robot to perform the task.

[0060] Example

[0061] See Figure 1 The RobMRAG method proposed in this embodiment aims to solve the zero-shot robot operation problem. Its core idea is that when the robot faces a new task (e.g., the user command "turn on the microwave"), it does not start planning from scratch. Instead, it uses a complex and sophisticated retrieval system to find the most similar and relevant past case from a large knowledge base, and then "borrows" the successful experience of this case to guide the current operation.

[0062] The entire workflow can be summarized as: Input -> Retrieval -> Optimization -> Generation -> Output.

[0063] Input: User-generated natural language command I and current scene image F captured by the robot's camera. obs And depth map F dep .

[0064] Search: Through hierarchical multimodal retrieval, three rounds of filtering are performed in a multi-source knowledge base to finally find the best reference example e. best .

[0065] Optimization: Through 3D perception pose optimization, 3D Gaussian sputtering technology is used to optimize the e best The pose is fine-tuned to ensure that it is aligned with the pose of the current target object in three-dimensional space.

[0066] Generation: The original input and the optimized reference sample are packaged together to form a rich contextual hint, which is then fed into MLLM.

[0067] Output: MLLM generates a precise 6-DoF maneuver pose, and the robot performs actions based on this pose.

[0068] I. Construction of a Multi-Source Operational Knowledge Base

[0069] The first step in this embodiment is to construct a comprehensive and diverse operational knowledge base K. This knowledge base is the source of the robot's "intelligence".

[0070] Data sources: The knowledge base uses data from multiple sources to ensure its breadth and depth.

[0071] 1. Simulation Data: In advanced physics simulation environments such as SAPIEN, 3D model libraries with detailed part annotations, such as PartNet-Mobility, are used to systematically generate robot manipulation data. For example, using the Franka Panda robotic arm, approximately 20,000 successful "pull" actions were performed on objects from 20 categories. Because this is done in simulation, the complete 6D grasping parameters for each successful operation can be recorded without error, including the 2D pixel coordinates of the contact points. e =(x,y), and the upward direction vector d of the end effector. u =(x u ,y u ,z u ) and forward direction vector d f This data forms the core of a high-quality knowledge base.

[0072] 2. Real-world data: Primarily derived from large video datasets of human-object interactions, such as HOI4D. By applying advanced computer vision algorithms, such as hand keypoint detection, the contact points and movement directions of humans manipulating objects can be extracted from the videos and annotated as 2D arrows on keyframe images. This data provides robots with operational knowledge derived from the real physical world that is more in line with human intuition.

[0073] 3. Internet Data: To further broaden the coverage of the knowledge base, this invention also collected a large number of diverse operational scenarios from the internet (such as tutorial videos on Bilibili, DIY animations, etc.). Although this data usually lacks precise 3D pose information, a semi-automated annotation process can be used to generate 2D grasping sketches or functional area diagrams. This data greatly enriches the robot's ability to cope with long-tailed and rare objects.

[0074] Data Structure: Each entry (i.e., an operation case) in the knowledge base is stored as a standardized tuple: K = {(I e , F ctc , F suc , p e D e )}. Among them, I e It is a natural language instruction describing the task (such as "open the door"), F ctc It is a key image frame that records the moment when the robot / human hand makes correct contact with the object, F suc These are image frames that demonstrate the successful completion of a task, while p e and D e These are the corresponding contact point and direction parameters.

[0075] II. Hierarchical Multimodal Retrieval

[0076] When the system receives a new task, hierarchical multimodal retrieval will be initiated, such as... Figure 2 As shown. This process aims to efficiently and accurately find the unique best reference from a vast knowledge base.

[0077] The first layer: text semantic retrieval. This layer is responsible for understanding the user's "intent".

[0078] It employs a "three-priority hybrid" strategy to ensure both retrieval efficiency and robustness:

[0079] -P1 (Highest Priority): Uses the classic sparse search algorithm BM25 to find cases that exactly match the object name in the user command from the highest quality simulation data source. For example, if the command is "open microwave oven", it will directly look for cases with "microwave oven". This is the fastest and most direct method.

[0080] -P2 (Second Priority): If not found in P1 (e.g., the object in the user instruction is "rice cooker," but it doesn't exist in the simulation library), the system automatically downgrades to P2, employing a deep learning-based intensive retrieval. It uses a powerful pre-trained language encoder (such as BERT) to compare the user instruction I with all instructions I in the knowledge base. e All are converted into high-dimensional mathematical vectors (embeddings), and then the cosine similarity between them is calculated. This method can understand deeper semantics, such as knowing that "rice cooker" and "pressure cooker" are functionally similar.

[0081] -P3 (Third Priority): If the score of the most similar case found in P2 is still below a preset threshold tau den (For example, 0.70) indicates that not even a functionally similar example can be found in a high-quality simulation library. At this point, the system will trigger P3 to expand the search scope to cover a wider range of real-world and internet data sources, performing a cross-source search.

[0082] After this layer is completed, the system will obtain a list C of candidate cases aligned with the current instruction in terms of "task intent". text .

[0083] The second layer: visual similarity filtering. This layer is responsible for matching "appearance".

[0084] Often, even for the same task (such as "opening a door"), different doors can look vastly different. To find the reference that most closely resembles an object in the current scene, this embodiment utilizes an image encoder based on the powerful visual language model CLIP.

[0085] It calculates the observed image F currently seen by the user. obs With C text Contact frame F for each candidate case ctc The visual features between them are calculated using cosine similarity. Then, only the top-n candidates with the highest similarity (e.g., n=10) are retained. Figure 2 As shown in the "Cosine Similarity" step, this step filters out cases that, although they have the same task, look very different.

[0086] The third layer: Geometric matching and reordering. This layer is responsible for matching "3D shapes".

[0087] This is the most delicate step. In order to select a single winner from the final few candidates, this embodiment uses a metric called Instance Matching Distance (IMD).

[0088] IMD measures the geometric differences between objects by comparing their dense feature maps in two images. Simply put, it examines every feature point on the currently observed object to see if a very close corresponding point can be found in the image of the reference object. The more similar the shapes of the two objects, the smaller this distance value.

[0089] like Figure 2 As shown in the "Instance Matching Distance Reranking" step, the system calculates the IMD for each of the top-n candidates selected in the second layer, and selects the one with the lowest IMD score (i.e., the most geometrically similar) as the final best reference instance e. best .

[0090] III. 3D Perception Pose Optimization

[0091] Even after three layers of meticulous searching, the best reference example e is obtained. best There may still be pose deviations between the target object and the target object due to different shooting angles. To solve this "last mile" alignment problem, this embodiment introduces a core innovative module—3D perception pose optimization, the process of which is as follows: Figure 3 As shown.

[0092] 3D Reconstruction: When the system determines e best The IMD value is higher than a preset geometric misalignment threshold tau IMD A time interval (e.g., 135) signifies the need for 3D alignment. The system first invokes a pre-trained, advanced technique—3D Gaussian Splatting (3DGS)—that generates 3D models from a single RGB-D image. It transforms the reference image into a set G consisting of thousands of differentiable, colored 3D Gaussian spheres. This 3DGS representation is essentially a high-quality, mathematically operable "digital twin" of the reference object.

[0093] Virtual Viewpoint Generation: Because the 3DGS representation is differentiable, it can be arbitrarily rotated and translated, and a 2D image viewed from a new perspective is generated in real time through a process called "differentiable rendering." Utilizing this property, a set of predefined small-angle rotation transformations {R} are applied to the grab pose of the reference sample. k} K k=1 (For example, K=6, meaning trying 6 tiny different angles). For each transformation, the system calculates the new 3D contact point and orientation, and renders a new candidate contact frame F. r ctc .

[0094] Optimal pose selection: The system obtains K new reference images rendered from different fine-tuning perspectives. Then, it uses the IMD algorithm again to calculate the ratio of these K new images to the user's currently observed image F. obs The geometric distance between them. Ultimately, the rotation transformation corresponding to the image with the lowest IMD score is considered the best transformation that can perfectly eliminate the viewpoint difference. Its corresponding pose p r D r This becomes the final optimized reference pose K. ref .

[0095] like Figure 3 As shown, the initial reference (Misaligned) is processed by this module to obtain a reference (Aligned) that is completely consistent with the pose of the target object.

[0096] IV. MLLM Pose Generation and Optional Fine-tuning

[0097] Hint: Construction and Reasoning: This is the final step in the method of this embodiment. The system will store all key information—the original user command I, the currently observed image F. obs And the final reference sample K, which is highly relevant and 3D geometrically aligned, obtained through the aforementioned complex process. ref —Integrated into a structured prompt.

[0098] Here's an example of the prompt template: Context: The current robot is {robot type}, and its task is: {action} {object_name}. Input: The first image is your current observation. The second image is a reference image, with the grab point marked by a red dot. This reference image is similar to your observation, with the grab point at (point), the upward direction as {rag_up}, and the forward direction as {rag_forward}. Your responsibility: Based on the task requirements, predict the precise grab point and direction in the observed image. Output format:

[0099] 1. Fetch point: Normalized (x, y).

[0100] 2. Grab direction: Normalized upward [x, y, z] and forward [x, y, z].

[0101] This highly informative prompt is fed into a powerful multimodal large language model (MLLM), such as Qwen2-VL-7B-Instruct. Thanks to concrete and precise geometric references, the MLLM is no longer "blind men and an elephant," but can reason based on solid geometric evidence, thereby generating a highly accurate and directly executable final operation pose p,D.

[0102] Optional LoRA Fine-tuning: To further improve performance, the MLLM in this embodiment can be efficiently fine-tuned using Low-Rank Adaptation (LoRA) technology. The fine-tuning process uses a joint loss function. Among them, L MLM It is the masked language model loss, which forces the model to learn to infer numerical values ​​from context by randomly masking pose parameters in the input text. L pose It is a pose regression loss that directly supervises the accuracy of the output contact point coordinates (using mean square error) and direction vectors (using cosine similarity).

[0103] Experimental verification

[0104] To demonstrate the effectiveness of the RobMRAG method proposed in this invention, a series of comprehensive experiments were conducted. The experimental environment was built based on the SAPIEN physics simulator and the PartNet-Mobility dataset. The test set included 30 categories of household items, divided into two parts: "seen categories" (objects seen during training but not identical instances) and "unseen categories" (completely new object categories), to comprehensively evaluate the model's generalization ability. The main evaluation metric was the success rate (SR), which is the proportion of trials in which the robot successfully moved an object beyond a specific distance threshold.

[0105] Ablation experiments: To verify the contribution of each design component in this invention, ablation studies were conducted, and the experimental results are shown in Table 1. All experiments were conducted under a zero-sample setting.

[0106] Table 1: RobMRAG Component Ablation Study

[0107]

[0108] The results in Table 1 clearly show the following: Baseline performance: When no retrieval enhancements are used (w / oRetrieval), relying solely on the visual understanding capabilities of LM, the model performance is very poor, with a success rate of less than 20%. This proves that in complex geometric manipulation tasks, the capabilities of MLLM alone are far from sufficient. The value of retrieval: Once multimodal retrieval is introduced, even using only simple cosine similarity (CosSim) for visual filtering, performance is greatly improved. This demonstrates that the basic idea of ​​finding reference cases from a knowledge base is effective. The importance of geometric matching: When using geometrically more refined IMD for matching, performance is further improved, especially in unseen categories. This indicates that, in addition to appearance similarity, 3D shape matching is crucial for generalization. The decisive role of 3D alignment: Finally, when the core innovation of this invention—3D perceptual pose optimization (Full MRAG)—is added, the model achieves optimal performance in both seen and unseen categories. This irrefutably proves that solving the perspective deviation problem between the reference and the target through 3D Gaussian sputtering plays a decisive role in generating accurate grasping poses.

[0109] Comparative experiment: The method of this invention (Ours) was compared with the current state-of-the-art baseline methods, including the zero-shot method RAM and the supervised learning method ManipLLM. The results are shown in Table 2.

[0110] Table 2: Performance comparison with the state-of-the-art (SOTA) method (average success rate, %)

[0111]

[0112] Analysis of the data in Table 2 shows that, under the zero-shot setting, the method of this invention (Ours (Zero-shot)) achieved a success rate of 43.53% on unseen categories, which is 7.76 percentage points higher than the then-state-of-the-art zero-shot method RAM. This fully demonstrates the powerful ability of the framework of this invention to generalize to new objects. After Lora fine-tuning (All), the advantages of the method of this invention are even more obvious. On unseen categories, the model of this invention achieved an average success rate of 57.54%, which is 6.54 percentage points higher than the strongest supervised learning baseline ManipLLM. In some specific tasks, such as Oven and Laptop, the success rate of this invention can even reach 86.00% and 96.08%, respectively, far exceeding the baseline.

[0113] The apparatus and device of the present invention

[0114] See Figure 4The present invention also provides a robot manipulation device 400 for implementing the above-described method. This device can be integrated into a robot system and includes: a knowledge base construction unit 401, an input interface 402, a retrieval unit 403, a pose optimization unit 404, a generation unit 405, and an execution control unit 406. Each unit works collaboratively to complete the entire process from receiving instructions to executing operations.

[0115] See Figure 5 The method and apparatus of the present invention can be implemented by a computer device 500. This device includes a processor 501, a memory 502, and an input / output interface 503. The memory stores a computer program implementing the method of the present invention; when the processor executes the program, it can complete all or part of the method steps described in the present invention.

[0116] In summary, this invention proposes a 3DGS-based multimodal retrieval method and apparatus for zero-shot robot operation. By constructing a multi-source knowledge base, designing a hierarchical multimodal retrieval strategy, and innovatively introducing 3DGS-based pose optimization, it successfully combines the powerful semantic understanding capabilities of MLLM with the precise geometric execution capabilities required for robot operation. The effectiveness of this invention has been fully verified through extensive simulation experiments, and it outperforms existing technologies in all aspects. It provides a new and efficient solution to the challenging problem of zero-shot robot operation, and has certain academic research value and commercial application prospects.

[0117] The specific embodiments of the present invention have been described in detail above, but they are merely examples and not intended to limit the invention. Those skilled in the art should recognize that all modifications, equivalent substitutions, and improvements within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A zero-shot robot operation method for multimodal retrieval based on 3DGS, characterized in that, The specific steps include the following: Step 1: Construct a multi-source operation knowledge base, which contains multiple operation cases. Each case includes at least one key action frame, instruction text describing the task, and 6D grasping pose parameters. The key action frame includes an object contact frame and / or a task completion frame. Step 2: Receive the natural language target task instructions and the current scene observation image containing the target object issued by the user; Step 3: Hierarchical multimodal retrieval 3.1: Based on the natural language target task instructions, a three-priority hybrid retrieval strategy is adopted to search the knowledge base. First, sparse retrieval is used for exact matching. If it fails, it switches to dense retrieval based on semantic embedding to obtain the semantically closest candidate sample set. 3.2: Calculate the cosine similarity between the current scene observation image and the contact frames of each example in the candidate example set using the visual language model CLIP, and filter out the top-n candidate examples that are most similar in visual appearance to narrow the search range; 3.3: The Instance Matching Distance (IMD) algorithm is used to calculate the geometric shape difference between the observed object and the top-n candidate samples, and the best reference sample that is geometrically the best match is selected by re-sorting. Step 4: If the IMD value of the best reference sample and the target object is higher than the preset threshold, then start 3D perception pose optimization; use 3D Gaussian sputtering technology to generate its three-dimensional representation from the RGB-D image of the reference sample, and find the pose that is best aligned with the target object in 3D space by performing small-angle rotation transformation and differentiable re-rendering on the three-dimensional representation, and generate the optimized reference sample. Step 5: Combine the natural language target task instructions, the current scene observation image, and the best reference example from Step 3 or the optimized reference example from Step 4 into a structured prompt. Step 6: Input the structured prompts into a pre-trained multimodal large language model (MLLM), which infers and generates an accurate final operation pose for performing the target task, including 2D contact points and 3D end effector orientation.

2. The multimodal retrieval zero-shot robot operation method according to claim 1, characterized in that, The hierarchical multimodal retrieval specifically includes: Text semantic retrieval: A three-priority hybrid retrieval strategy is adopted. First, sparse retrieval is used to match object names; if no match is found, it switches to dense retrieval based on semantic embedding; if the highest similarity of the dense retrieval is lower than a preset threshold, the retrieval scope is expanded to other data sources, thereby filtering out the candidate sample set with the closest semantics. The calculation formula is as follows: in This is a semantic embedding vector for natural language instruction text. For example, the semantic embedding vector of the instruction text in the knowledge base, where Enc() represents the text encoder; find the semantically closest substitute object in the semantic space; Visual similarity filtering: The cosine similarity between the observed image and each contact frame in the candidate sample set is calculated using a pre-trained visual language model, and the top-n most visually similar candidate samples are selected. Geometric matching: The instance matching distance (IMD) is used to calculate the geometric difference between the observed object and visually similar candidate examples, and the example with the lowest IMD score is selected as the best reference example.

3. The multimodal retrieval zero-shot robot operation method according to claim 1, characterized in that, The 3D perception pose optimization specifically includes: A 3D Gaussian sputtering representation is generated from an RGB-D image of a reference sample using a pre-trained generative model; Apply a set of predefined small-angle rotation transformations to the reference grasp pose to generate multiple candidate poses; Each candidate pose is reprojected onto a two-dimensional image plane using 3D Gaussian sputtering differentiable rendering to generate the corresponding candidate contact frame. The instance matching distance (IMD) between each candidate contact frame and the observed image is recalculated, and the candidate pose with the smallest IMD is selected as the optimized reference pose.

4. The multimodal retrieval zero-shot robot operation method according to claim 1, characterized in that, The Multimodal Large Language Model (MLLM) is fine-tuned using low-rank adaptive techniques, and its joint loss function combines masked language modeling loss and pose regression loss.

5. The multimodal retrieval zero-shot robot operation method according to claim 4, characterized in that, The pose regression loss is defined as follows: Where (x,y) are the actual two-dimensional coordinates of the contact point. d represents the predicted contact point coordinates. u , d f For the actual end effector's upward and forward direction vectors, , λ1 and λ2 are the predicted direction vectors; λ1 and λ2 are the balancing weight coefficients.

6. A multimodal retrieval zero-shot robot operation device based on the method of claim 1, characterized in that, include: The knowledge base construction unit is used to build a multi-source operation knowledge base containing operation cases. The retrieval unit is used to perform hierarchical multimodal retrieval based on user instructions and observed images to obtain the optimal reference example from the knowledge base; The pose optimization unit is used to perform 3D perception pose optimization on the retrieved reference sample using 3DGS technology, so that it is aligned with the target object; The generation unit is used to input observation data, user instructions, and optimized reference poses into the multimodal large language model to generate the final robot operation pose.

7. The multimodal retrieval zero-shot robot operation device according to claim 6, characterized in that, The retrieval unit specifically includes: Text semantic retrieval is used to execute a three-priority hybrid retrieval strategy; Visual similarity filtering is used to filter candidate samples by calculating the similarity of visual features; Geometric matching is used to determine the best reference sample by calculating the instance matching distance.

8. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1 to 5.

Images