Robot task planning method based on hierarchical multi-modal retrieval enhanced generation

Abstract

The invention discloses a robot task planning method based on hierarchical multi-modal retrieval enhanced generation, and relates to the field of robots, comprising: constructing a multi-modal knowledge base; receiving a user instruction, and shooting a real-time image of target equipment; performing a two-stage retrieval for obtaining a reference content text; executing two-stage generation, generating a button sequence, and converting the button sequence into a control code; and the robot executes the control code and monitors the execution state of each action. Visual perception, knowledge retrieval and action generation are integrated, and a high-level human intention is converted into an accurate and executable robot action sequence through a layered framework.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more particularly to a robot task planning method based on hierarchical multimodal retrieval enhancement generation. Background Technology

[0002] The rapid development of artificial intelligence and robotics has enabled intelligent robots to handle increasingly complex tasks across various fields. Large-scale models, including large language models (LLMs) and visual language models (VLMs), have demonstrated exceptional capabilities in natural language processing and computer vision. When these models are integrated with robotic systems, they significantly enhance the robots' perception, decision-making, and interaction capabilities. However, the effective deployment of large-scale models requires substantial amounts of labeled data and computational resources. Furthermore, relying solely on model inference can lead to problems such as outdated knowledge and insufficient real-time response. As task complexity increases, enabling robots to perform tasks efficiently in diverse and dynamic environments remains a fundamental scientific challenge, especially when single-modal information or traditional algorithms are insufficient to solve the problem.

[0003] Real-world scenarios often involve complex and specialized tasks that even humans cannot solve solely based on prior knowledge, necessitating real-time information retrieval or consultation with external resources. These tasks typically involve multi-step sequential operations, where high-level intentions must be translated into precise low-level actions. Existing research on robot task planning primarily focuses on tasks that can be intuitively understood and decomposed, such as combining navigation-based methods with simple operations (like "grasping" and "placing"), using symbolic reasoning, or large-scale model-based approaches. However, existing methods face significant limitations when dealing with domain-specific tasks requiring specialized knowledge, precise sequential control, and adaptation to diverse interface constraints. Furthermore, these methods often require substantial retraining when faced with entirely different scenarios or task types and cannot effectively utilize external documentation or real-time information sources, limiting their generalization ability and practical applicability.

[0004] Therefore, those skilled in the art are dedicated to developing a robot task planning method based on hierarchical multimodal retrieval enhancement generation, enabling robot systems to complete various complex multi-step tasks based solely on simple human instructions. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is that existing robot systems are unable to handle complex sequential tasks that require specialized domain knowledge, and are unable to effectively understand ambiguous human instructions and convert them into precise operation sequences for specific devices.

[0006] To achieve the above objectives, this invention provides a robot task planning method based on hierarchical multimodal retrieval enhancement generation, the method comprising the following steps: Step 1: Construct a multimodal knowledge base, which includes reference images of at least one device and corresponding user manuals; Step 2: Receive user instructions and capture real-time images of the target device; Step 3: Perform a two-stage search to obtain the reference content text; Step 4: Perform a two-stage generation process, including: Step 4.1: Generate a button sequence based on the user instructions and the reference content text; Step 4.2: Convert the button sequence into control codes; Step 5: The robot executes the control code and monitors the execution status of each action.

[0007] Furthermore, step 1 also includes: Step 1.1: Obtain a reference image of at least one device and its corresponding user manual; Step 1.2: Divide the user manual into multiple semantically coherent text blocks; Step 1.3: Use a multimodal embedding model to process the reference image and text block, and generate the corresponding reference image embedding vector and text block embedding vector; Step 1.4: The reference image embedding vector and the text block embedding vector are stored in a structured manner to form the multimodal knowledge base.

[0008] Furthermore, step 1.4 also includes: for each device, using its reference image embedding vector as the index of the device, and storing it in association with the set of all text block embedding vectors corresponding to the device, so as to support a hierarchical retrieval process of first retrieving the device and then retrieving the device-related text.

[0009] Furthermore, step 3 also includes: Step 3.1: Perform the first-stage retrieval to identify the target device from the multimodal knowledge base; Step 3.2: Perform the second stage of retrieval, find the specific operation instructions most relevant to the user commands in the user manual of the target device, and obtain the reference content text.

[0010] Furthermore, step 3.1 also includes: Step 3.1.1: Use a multimodal embedding model to convert real-time images into real-time image embedding vectors; Step 3.1.2: Calculate the cosine similarity between the real-time image embedding vector and all reference image embedding vectors in the multimodal knowledge base, and determine the best matching device based on the similarity. Step 3.1.3: After identifying the best matching device, extract the embedding vectors of all text blocks corresponding to the best matching device to form a temporary sub-database.

[0011] Furthermore, step 3.2 also includes: Step 3.2.1: Convert the user's natural language commands into embedding vectors; Step 3.2.2: In the temporary sub-database, calculate the cosine similarity between the embedding vector of the user instruction and the embedding vectors of all text blocks; Step 3.2.3: Based on the similarity score, retrieve the m highest-ranking text blocks to form the reference content text.

[0012] Furthermore, step 4.1 also includes: Step 4.1.1: Use a visual detection algorithm to identify all available buttons and their names on the device panel from the real-time image, forming a button set; Step 4.1.2: Design the first prompt by inputting the user command, the reference content text, and the button set into the generated model LLM; Step 4.1.3: Generate the button sequence based on the first prompt.

[0013] Furthermore, step 4.2 also includes: Step 4.2.1: The system loads the software development kit specification containing all available robot actions, which defines each control function and parameter in detail; Step 4.2.2: Design the second prompt by inputting the button sequence and software development kit specification into the generated model LLM; Step 4.2.3: Generate the LLM model. Based on the second prompt, map each button name in the button sequence to a specific SDK function call, fill in the correct coordinates and control parameters, and generate control code.

[0014] Furthermore, the constraints imposed by the first prompt on the output of the generative model LLM include: Feasibility constraint: In the generated button sequence, each button is a member of the set of buttons detected visually; Efficiency constraint: Encourage the model to generate the shortest effective button sequence; Formatting constraints: The output is a structured list format.

[0015] Furthermore, in step 4.2.3, the generated control code is a sequence of specific function calls that can be directly executed by the robot control system.

[0016] Compared with the prior art, the present invention has at least the following beneficial technical effects: 1. This invention achieves end-to-end conversion from high-level natural language commands to low-level, precise robot physical actions. Experiments on a real robot platform have demonstrated that this method can successfully complete complex equipment operation tasks with a success rate of over 90%. 2. This invention achieves high accuracy in both virtual and physical experiments. Even with significant differences in viewing angle, lighting conditions, and ambiguous commands, the method can still accurately identify the target device and retrieve relevant text, demonstrating strong robustness. 3. It successfully bridges the semantic gap between human intent and robot physical execution. The generated action sequences are not only correct, but also tend to choose simpler and more efficient operation methods by introducing an efficiency penalty mechanism.

[0017] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the present invention; Figure 2 This is a flowchart illustrating the construction of a multimodal knowledge base according to a preferred embodiment of the present invention; Figure 3 This is a two-stage retrieval flowchart of a preferred embodiment of the present invention; Figure 4 This is a schematic diagram of a two-stage retrieval algorithm according to a preferred embodiment of the present invention; Figure 5 This is a two-stage generation flowchart of a preferred embodiment of the present invention; Figure 6 This is a schematic diagram of a two-stage generation algorithm according to a preferred embodiment of the present invention. Detailed Implementation

[0019] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0020] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components is appropriately exaggerated in the drawings.

[0021] Example 1

[0022] This embodiment provides a robot task planning method that employs Hierarchical Multimodal Retrieval-Augmented Generation (HiMRAG) to address the challenges of instruction ambiguity and visual diversity faced by robots when operating complex equipment based on natural language commands.

[0023] This embodiment addresses the problem of generating an optimal button press sequence S = {b1, b2, ..., b}, given a user instruction q in natural language (e.g., "heat the food for two minutes") and an image I of a device control panel captured in real-time by a robot. n To fulfill the user's intent.

[0024] like Figure 1 As shown, it includes the following steps: Step 1: Construct a structured multimodal knowledge base D, including at least one device document D = {(d_img_i, d_txt_i)}, where d_img_i is the reference image of the device (e.g., a product promotional image), and d_txt_i is its corresponding user manual, such as... Figure 2 As shown, it includes: Step 1.1: For each target device (such as different models of microwave ovens), collect its complete official documentation, mainly including high-definition product promotional images and PDF user manuals. Step 1.2: Use advanced document parsing tools to segment the PDF manual into semantically coherent text blocks. During segmentation, preserve the integrity of the operational context, avoiding breaking a complete operation step into different text blocks, and ensuring that each text block contains one or more related operation instructions or function descriptions; Step 1.3: Process the reference image and text block using a multimodal embedding model to generate corresponding reference image embedding vectors and text block embedding vectors. The multimodal embedding model can map images and text to the same high-dimensional vector space (e.g., 1536-dimensional), allowing for semantic similarity comparisons of information from different modalities. This process can be represented as: Image embedding: e_img_i = Embed(d_img_i) Text block embedding: e_txt_(i,j) = Embed(t_(i,j)) Step 1.4: The reference image embedding vector and the text block embedding vector are stored in a structured manner to form a natural tree structure, resulting in a multimodal knowledge base. Specifically, for each device, the embedding vector e_img_i of its reference image is used as the root node of the device's "branch," followed by the embedding vectors e_txt_(i,j) of all manual text blocks for that device as "leaf nodes." This organization allows all information related to the same device to be aggregated together in the vector space, facilitating subsequent hierarchical retrieval.

[0025] Step 2: Receive user instructions. The robot uses its onboard camera to capture real-time images (I) from the device control panel.

[0026] Step 3: Perform a two-stage search, such as... Figure 3 As shown, it specifically includes: Step 3.1: Perform the first-stage retrieval to accurately identify the target device currently facing the robot from the multimodal knowledge base, including: Step 3.1.1: Using the same multimodal embedding model as when building the knowledge base, convert the real-time images into real-time image embedding vectors e_I = Embed(I); Step 3.1.2: Calculate the cosine similarity between the real-time image embedding vector and the embedding vectors of all reference images in the multimodal knowledge base. Based on the similarity, determine the device that best matches the real-time image, i. = argmax_i sim(e_I, e_img_i); Step 3.1.3: Identify the most matching device i Then, the system narrows the search scope to the "branch" of that device, that is, extracts the embedding vectors of all text blocks corresponding to the most matching device {e_txt_(i ,1), ..., e_txt_(i ,n)}, forming a temporary sub-database D_i This process greatly reduces the computational load and interference information in subsequent searches.

[0027] Step 3.2: Perform the second-stage search, find the most relevant operation instructions in the target device's user manual, and obtain the reference content text, specifically including: Step 3.2.1: Convert the user's natural language command q into an embedding vector e_q = Embed(q); Step 3.2.2: In the temporary sub-database Di In the calculation, the cosine similarity e_txt_(i) between the embedding vector e_q of the user instruction and the embedding vectors of all text blocks is calculated. ,j); Step 3.2.3: Based on the similarity score, retrieve the m highest-ranking text blocks to form the reference content text, R = {t_i} ,j | j ∈ top-m(sim(e_q, e_txt_i These retrieved text blocks R will serve as contextual information, providing precise, device-specific knowledge for subsequent action generation.

[0028] like Figure 4 The diagram shown is a schematic of the two-stage retrieval algorithm.

[0029] Step 4: Perform two-stage generation, such as... Figure 5 As shown, it includes: Step 4.1, First Stage: Generate Button Sequence. The goal of this stage is to translate the user's high-level intent into a device-specific, ordered sequence of button actions, including: Step 4.1.1: Use a visual detection algorithm to identify all available buttons and their names on the device panel from the real-time image I, forming a button set B; Step 4.1.2: Design a carefully crafted first prompt, inputting the user command q, reference text R, and button set B into the generative model LLM; the first prompt imposes strict constraints on the output of the generative model: Feasibility constraint: In the generated button sequence, each button must be a member of the visually detected button set B.

[0030] Efficiency constraint: Encourage the model to generate the shortest effective button sequence to improve operational efficiency.

[0031] Format constraints: The output must be in a structured list format (e.g., ["button1", "button2",...]) to facilitate subsequent processing; Step 4.1.3: Generate the LLM model to generate a button sequence S = LLM(prompt_1(q, R,B)) based on the first prompt.

[0032] Step 4.2, the second stage, translates the abstract button sequence S generated in the previous stage into low-level control code that the robot can execute, including: Step 4.2.1: The system loads the Software Development Kit (SDK) specification SDK_spec, which contains all available robot actions. This specification defines each control function and parameter in detail. Step 4.2.2: Design the second prompt by inputting the button sequence S and the software development kit specification into the generated model LLM; Step 4.2.3: Generate Model LLM. Based on the second prompt, map each button name in the button sequence to a specific SDK function call, fill in the correct coordinates and control parameters, and generate control code C = LLM(prompt_2(S,SDK_spec)). The generated code C is a sequence of specific function calls that can be directly executed by the robot control system.

[0033] like Figure 6 The diagram shown is a schematic of the two-stage execution algorithm.

[0034] Step 5, System Integration and Execution: During runtime, all the above modules are integrated into a seamless pipeline. After receiving user commands, the robot executes them sequentially: Capture images and perform preprocessing; Perform a two-stage search to retrieve relevant information from the equipment manual; Perform a two-stage generation process to produce executable robot code; The generated control code is executed sequentially, and the execution status of each action is monitored.

[0035] The method provided in this embodiment enables robots to efficiently and accurately understand and execute complex operational tasks, successfully linking human natural language instructions with precise operations in the physical world.

[0036] Example 2

[0037] The performance of the multimodal embedding model in device recognition tasks was validated, including: 1. Experimental Objective This embodiment aims to comprehensively evaluate the performance of the multimodal embedding model used in the first-stage retrieval (device identification) of this invention. The core objective of the experiment is to verify whether the model can still achieve accurate cross-modal matching when the query image (real-world photograph) and the knowledge base image have vastly different styles.

[0038] 2. Experimental Setup

[0039] Build the dataset, including the knowledge base and query set.

[0040] Knowledge Base: A knowledge base has been built containing 6 different microwave oven models. Each model includes one official high-definition promotional image as a reference image, and additional images from different categories of home appliances have also been collected.

[0041] The query set comprises a virtual query set and a physical query set. The virtual query set contains 17 real-world microwave oven photos collected from e-commerce platforms, with backgrounds and lighting conditions significantly different from official promotional images. The physical query set uses a RealSense camera mounted on a robotic arm to photograph each microwave oven from six different angles (front, top, top left, left, top right, and right), collecting a total of 29 physical images taken under different lighting and angles.

[0042] 3. Verification of basic equipment identification capabilities

[0043] Objective: To verify the model's basic recognition ability when faced with domain gaps between promotional images and real photos.

[0044] Process: Use a virtual query set (17 real photos) to query a knowledge base that contains only 6 microwave oven promotional images.

[0045] Results: For each real photo query, the model accurately ranked the corresponding promotional image first in the search results. This demonstrates that despite differences in shooting style, lighting, and background, these models can still effectively capture the core visual features of the device and complete cross-modal matching.

[0046] 4. Cross-category differentiation capability verification

[0047] Objective: To verify the model's ability to distinguish between target and non-target categories in more complex knowledge bases.

[0048] Process: Use a virtual query set to query the extended knowledge base that has added 25 other home appliance images.

[0049] Results: The model was able to successfully identify microwave ovens while effectively eliminating interference from other types of home appliances.

[0050] In-depth analysis: Although the accuracy is 100%, the distribution of similarity scores of the models reveals the difference in their discrimination ability; the greater the gap between the highest score (correct match) and the second highest score (incorrect match) given by the model, the stronger its discrimination ability.

[0051] 5. Robustness verification from the perspective of a real robot

[0052] Objective: To verify the robustness of the model in real robot operating environments when faced with drastic changes in viewpoint and lighting.

[0053] Process: Use the physical query set (29 photos taken by the robot) to query the basic knowledge base.

[0054] Results: The model was able to successfully retrieve the target microwave oven under different viewing angles and lighting conditions. Experimental results show that viewing angle and lighting have a significant impact on the absolute similarity score: the similarity score is highest when shooting from the front; when shooting from the side or at an angle, the absolute similarity score decreases significantly because key identification features may be partially obscured or perspective distortion may occur.

[0055] Despite the dramatic fluctuations in absolute similarity scores, all models consistently maintain the correct relative ranking, meaning that the correctly matched device reference map always receives the highest similarity score.

[0056] 6. Conclusion

[0057] Through a series of experiments in this embodiment, the following conclusions can be drawn: 1. Effectiveness: The selected multimodal embedding model can effectively overcome the domain gap between real photos and promotional images, and achieve accurate device recognition.

[0058] 2. Robustness: Even when the knowledge base contains multiple interference items, or when the robot's actual shooting perspective and lighting conditions change drastically, the model can still maintain a very high recognition accuracy.

[0059] Example 3

[0060] Perform end-to-end virtual performance evaluation of the HiMRAG method, including: 1. Experimental Objective This embodiment aims to provide a comprehensive virtual evaluation of a complete Retrieval Enhanced Generation (RAG) pipeline. The experiment evaluates not only the performance of the retrieval module but also the quality of the generation module, and analyzes the overall performance of the system when handling tasks of varying complexity and type.

[0061] 2. Experimental Setup

[0062] (1) System Configuration

[0063] Retrieval module: Employs a multimodal embedding model for generating and retrieving embedding vectors.

[0064] Generation module: GPT-4O is used as the generation model to perform a two-stage generation task.

[0065] Knowledge Base: Contains documents (promotional images and manual text blocks) for 5 different microwave oven models.

[0066] (2) Tasks and Datasets

[0067] Task Categories: Several task categories covering the main functions of the microwave oven have been designed.

[0068] Query set: For each task category, multiple natural language instructions were designed for each microwave oven model.

[0069] (3) Ground Truth

[0070] Retrieval Benchmark: For each task, all potentially relevant text blocks in the manual are manually annotated, including the manual content corresponding to all feasible methods; Generation Benchmark: For each task, one or more optimal, intuitive button operation sequences are provided as reference answers.

[0071] 3. Evaluation Indicators

[0072] (1) Retrieval quality indicators: Precision: Measures the proportion of relevant text blocks retrieved out of all retrieved text blocks; Mean Reciprocal Rank (MRR): Measures the inverse average of the ranking of the first relevant text block in the search results, reflecting the speed at which the system finds key information.

[0073] (2) Generate quality indicators: ROUGE-1 / ROUGE-L: Evaluate the overlap between the generated sequence and the reference sequence on a single button (ROUGE-1) and the longest common subsequence (ROUGE-L), measuring the similarity of functional integrity and operational flow; BLEU Score: Evaluates the accuracy of the generated sequence through n-gram matching, with higher requirements for the accuracy of button order.

[0074] 4. Experimental Results and Analysis

[0075] (1) Search performance analysis: The MRR values ​​remained at a high level (averaging over 90%). This indicates that although it may not be able to find all relevant information, the system always prioritizes the most crucial manual sections in the search results, providing strong support for subsequent generation.

[0076] Precision is relatively low and fluctuates significantly. This may be because user commands are typically short, containing insufficient keywords and semantic information to match all relevant text blocks in the benchmark.

[0077] (2) Generation performance analysis: All generated quality metrics (ROUGE-1, ROUGE-L, BLEU) demonstrated high stability and excellent performance (averaging over 80%). This indicates that even with fluctuations in the accuracy of retrieved contextual information, as long as the key information is found (high MRR), the generation module can robustly produce correct and efficient button sequences.

[0078] 5. Conclusion

[0079] (1) Robustness: The HiMRAG pipeline of this invention exhibits strong robustness. Even when faced with incomplete retrieval results, its generation module can still accurately understand the task intent and generate a high-quality execution plan. This is thanks to the powerful reasoning and completion capabilities of LLM, as well as the carefully designed prompting engineering.

[0080] (2) Effectiveness of the hierarchical design: The successful decoupling and collaboration between the retrieval module and the generation module proves the effectiveness of the hierarchical design of this invention. The retrieval module and the generation module together ensure the success of the task.

[0081] (3) Adaptation to the complexity of modern home appliances: The experimental design fully considers the diverse operation paths of modern smart home appliances. Evaluation results show that the present invention can adapt to this complexity and generate one of several feasible optimal solutions.

[0082] Example 4

[0083] Perform end-to-end physical execution verification of the HiMRAG system, including: 1. Experimental Objective This embodiment aims to perform end-to-end verification of the complete workflow of the HiMRAG method of the present invention in a real-world scenario. The purpose of the experiment is to demonstrate that the system can be seamlessly integrated with robot hardware, translating high-level natural language instructions into precise physical operations, thereby verifying its feasibility and reliability in practical applications.

[0084] 2. Experimental Platform and Environment Setup

[0085] (1) Robot hardware

[0086] Operating platform: Flexiv Rizon4 7-DOF robotic arm.

[0087] Vision system: Intel RealSense D435i depth camera.

[0088] (2) Target device: Media PM20M3 microwave oven

[0089] (3) Software system: Deploy the complete HiMRAG software pipeline described in Example 1, including a knowledge base, a retrieval module and a generation module.

[0090] 3. End-to-end execution process demonstration

[0091] Using a representative task as an example, we will demonstrate the complete execution flow of the system.

[0092] Execution steps: (1) Command reception and visual perception: After the system receives the user's voice command, the robot arm moves to the preset observation position and takes a picture of the front of the microwave oven.

[0093] (2) Hierarchical retrieval

[0094] Device identification: The system embeds the captured image and compares it with reference images in the knowledge base to accurately identify the model of the microwave oven currently in operation.

[0095] Task Information Retrieval: The system generates and embeds instruction text, then searches within the identified equipment manuals. Search results may include relevant text blocks such as "How to set cooking time," "How to adjust power levels," and "Preset menus."

[0096] (3) Two-stage generation

[0097] Button sequence generation: LLM comprehensively analyzes user commands and retrieved manual content to generate an optimal button sequence.

[0098] Robot control command generation: The system inputs the generated button sequence and robot SDK description into the LLM to generate a series of specific, executable SDK function calls.

[0099] (4) Physical execution

[0100] The robotic arm moves directly above each target button according to the code instructions and performs the pressing action.

[0101] 4. Conclusion

[0102] The end-to-end physical verification was a complete success, and the following key conclusions were drawn: (1) Technical feasibility: The technical feasibility of the HiMRAG framework proposed in this invention in the real world has been fully demonstrated. It successfully combines the reasoning ability of language models with the physical interaction of the real world.

[0103] (2) System integrity: This invention demonstrates its ability as a complete system, seamlessly integrating multiple complex modules such as multimodal perception, knowledge retrieval, advanced reasoning, and low-level robot control.

[0104] (3) Application value: The success of this embodiment fully demonstrates the huge application potential and commercial value of the invention in the fields of smart home, elderly / disabled assistance, and automated operation of complex equipment.

[0105] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A robot task planning method based on hierarchical multimodal retrieval enhancement generation, characterized in that, The method includes the following steps: Step 1: Construct a multimodal knowledge base, which includes reference images of at least one device and corresponding user manuals; Step 2: Receive user instructions and capture real-time images of the target device; Step 3: Perform a two-stage search to obtain the reference content text; Step 4: Perform a two-stage generation process, including: Step 4.1: Generate a button sequence based on the user instructions and the reference content text; Step 4.2: Convert the button sequence into control codes; Step 5: The robot executes the control code and monitors the execution status of each action.

2. The robot task planning method based on hierarchical multimodal retrieval enhancement generation as described in claim 1, characterized in that, Step 1 further includes: Step 1.1: Obtain a reference image of at least one device and its corresponding user manual; Step 1.2: Divide the user manual into multiple semantically coherent text blocks; Step 1.3: Use a multimodal embedding model to process the reference image and text block, and generate the corresponding reference image embedding vector and text block embedding vector; Step 1.4: The reference image embedding vector and the text block embedding vector are stored in a structured manner to form the multimodal knowledge base.

3. The robot task planning method based on hierarchical multimodal retrieval enhancement generation as described in claim 2, characterized in that, Step 1.4 further includes: for each device, using its reference image embedding vector as the index of the device, and storing it in association with the set of all text block embedding vectors corresponding to the device, so as to support a hierarchical retrieval process of first retrieving the device and then retrieving the device-related text.

4. The robot task planning method based on hierarchical multimodal retrieval enhancement generation as described in claim 1, characterized in that, Step 3 also includes: Step 3.1: Perform the first-stage retrieval to identify the target device from the multimodal knowledge base; Step 3.2: Perform the second stage of retrieval, find the specific operation instructions most relevant to the user commands in the user manual of the target device, and obtain the reference content text.

5. The robot task planning method based on hierarchical multimodal retrieval enhancement generation as described in claim 4, characterized in that, Step 3.1 further includes: Step 3.1.1: Use a multimodal embedding model to convert real-time images into real-time image embedding vectors; Step 3.1.2: Calculate the cosine similarity between the real-time image embedding vector and all reference image embedding vectors in the multimodal knowledge base, and determine the best matching device based on the similarity. Step 3.1.3: After identifying the best matching device, extract the embedding vectors of all text blocks corresponding to the best matching device to form a temporary sub-database.

6. The robot task planning method based on hierarchical multimodal retrieval enhancement generation as described in claim 4, characterized in that, Step 3.2 further includes: Step 3.2.1: Convert the user's natural language commands into embedding vectors; Step 3.2.2: In the temporary sub-database, calculate the cosine similarity between the embedding vector of the user instruction and the embedding vectors of all text blocks; Step 3.2.3: Based on the similarity score, retrieve the m highest-ranking text blocks to form the reference content text.

7. The robot task planning method based on hierarchical multimodal retrieval enhancement generation as described in claim 1, characterized in that, Step 4.1 further includes: Step 4.1.1: Use a visual detection algorithm to identify all available buttons and their names on the device panel from the real-time image, forming a button set; Step 4.1.2: Design the first prompt by inputting the user command, the reference content text, and the button set into the generated model LLM; Step 4.1.3: Generate the button sequence based on the first prompt.

8. The robot task planning method based on hierarchical multimodal retrieval enhancement generation as described in claim 1, characterized in that, Step 4.2 further includes: Step 4.2.1: The system loads the software development kit specification containing all available robot actions, which defines each control function and parameter in detail; Step 4.2.2: Design the second prompt by inputting the button sequence and software development kit specification into the generated model LLM; Step 4.2.3: Generate the LLM model. Based on the second prompt, map each button name in the button sequence to a specific SDK function call, fill in the correct coordinates and control parameters, and generate control code.

9. The robot task planning method based on hierarchical multimodal retrieval enhancement generation as described in claim 7, characterized in that, The first suggestion imposes constraints on the output of the generative model LLM, including: Feasibility constraint: In the generated button sequence, each button is a member of the set of buttons detected visually; Efficiency constraint: Encourage the model to generate the shortest effective button sequence; Formatting constraints: The output is a structured list format.

10. The robot task planning method based on hierarchical multimodal retrieval enhancement generation as described in claim 8, characterized in that, In step 4.2.3, the generated control code is a sequence of specific function calls that can be directly executed by the robot control system.

Images