Family robot action generation method based on Re-Plan principle

Abstract

The invention provides a home robot action generation method based on the Re-Plan principle, and belongs to the technical field of robots. According to the invention, a dynamic gating fusion network is designed to improve the multi-mode sensing efficiency of the home service robot, and a joint space-time security optimization mechanism is introduced to improve the quality of multi-mode feature fusion; on the basis of a HomeAct-1. 2k data set in combination with a curriculum learning strategy, lightweight fine tuning of a Qwen-VL-Chat model is completed, an action generation framework based on the Re-Plan principle is designed on the basis of the model, and the task execution capacity of the home service robot in complex tasks is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to a method for generating motions for home robots based on the Re-Plan principle. Background Technology

[0002] The fine-tuning performance of the Qwen-VL-Chat model based on the HomeAct-1.2k family service task dataset heavily depends on the efficiency of the multimodal feature fusion module. Current mainstream bimodal information encoding architectures typically employ a heterogeneous encoder stacking scheme, such as the encoding architecture based on the Qwen-VL-Chat model: utilizing a 12-layer Transformer network to extract semantic features of natural language instructions. Then, the CLIP-ViT-L / 14 visual encoder extracts visual features from the robot's main field of view image. Two different characteristics The fused features are generated by projecting through a cross-modal projection matrix.

[0003] While traditional coding frameworks demonstrate good performance on basic datasets, they still face three major bottlenecks when dealing with complex task sets like those in home scenarios: (1) The latency problem caused by the traversal processing: the serial processing mechanism of this structure causes the total encoding latency to be equal to the text feature latency. and visual features The sum of encoding delays when processing 2K resolution images ( =63ms) and long text commands (such as When =256, =41ms), total latency The response time is as high as 104ms, far exceeding the real-time response threshold (80ms) of home service robots. (2) The problem of low parameter efficiency caused by projection redundancy: Traditional splicing introduces nearly 78% of invalid parameters to force projection, which causes the number of parameters in the projection matrix to increase sharply. These invalid parameters not only prolong the entire encoding process by several times, but also introduce the risk of overfitting. (3) Dimensional adaptation problem, i.e. manifold misalignment caused by heterogeneous dimensions: text features and visual features are located in different Lie group spaces. Direct splicing will lead to information dilution, causing the model to be unable to complete sufficient fine-tuning, i.e. it does not fully learn the entire content of the family service task dataset.

[0004] Therefore, it is necessary to provide a method for generating motion for home robots based on the Re-Plan principle to solve the above problems. Summary of the Invention

[0005] This invention provides a method for generating motions in home robots based on the Re-Plan principle. It designs a dynamic gating fusion network to improve the multimodal perception efficiency of home service robots and introduces a joint spatiotemporal safety optimization mechanism to enhance the quality of multimodal feature fusion. Based on the HomeAct-1.2k dataset and a course learning strategy, it completes lightweight fine-tuning of the Qwen-VL-Chat model and designs a motion generation framework based on the Re-Plan principle, effectively improving the task execution capability of home service robots in complex tasks. This addresses at least one of the technical problems mentioned in the background.

[0006] In order to solve the above-mentioned technical problems, the present invention is achieved as follows: A method for generating motion for home robots based on the Re-Plan principle includes the following steps: Step S1: Construct a gated fusion network. Input the images and service instructions collected by the home service robot into the gated fusion network. Use CUDA streaming parallel technology to extract the visual features of the images and the text features of the service instructions. Correct the feature space based on the joint spatiotemporal security optimization mechanism. Use data-driven weight allocation to dynamically balance the contribution of the corrected visual features and text features. Fuse the visual features and text features to form fused features. Step S2: Construct a multimodal large language model and use a hybrid fine-tuning scheme that integrates low-rank adaptation and visual adapter to perform lightweight fine-tuning of the multimodal large language model: For the text features of service instructions, LoRA is used to inject a low-rank matrix to achieve targeted enhancement of semantic understanding ability; for the visual features of images, a cross-attention mechanism is used to enhance fine-grained spatial localization ability. Step S3 introduces the course learning strategy into the fine-tuning process of the multimodal large language model. By constructing a three-dimensional progressive course system, the multimodal large language model can follow the human learning path of "single-point learning - combined reinforcement - comprehensive use" to progressively perform hierarchical reasoning on the fusion features of the input to obtain atomic action sequences, thereby completing the action generation of the home service robot.

[0007] As a preferred improvement, the dynamic gating fusion network uses a gating coefficient matrix. To achieve nonlinear modulation between modes, its mathematical model is expressed as: In the formula, Represents the gate weight matrix; This represents the Sigmoid activation function; Representing text features; Indicates visual features; Indicates fusion characteristics; , Represents the projection matrix; This represents the Hadamard product.

[0008] As a preferred improvement, step S1, "correcting the feature space based on a joint spatiotemporal security optimization mechanism," specifically includes the following process: Step S11: The feature space is mapped to the physical space through bilinear mapping to achieve cross-modal alignment of visual and textual features. The mapping process is represented as follows: In the formula, This represents the aligned text and visual features; The first feature representing the text One component; The first characteristic representing visual features One component; This represents the Hadamard product operation; Represents the weight matrix; Indicates dimension; Step S12: A multi-dimensional perception model integrating scene semantic entropy and motion state was designed, using obstacle distribution entropy from the visual features of the family task dataset. Task criticality score in service instruction text features To dynamically adjust the safe distance. Determine the safety boundaries, and the adjustment process is represented as follows: In the formula, Represents the obstacle distribution entropy in visual features; This indicates the maximum moving speed of the home service robot; This represents the task criticality score within the text features; Step S13: Design a reverse feature correction mechanism. When a violation of safety boundaries is detected in the home service robot, the feature space is corrected through gradient backpropagation. The reverse feature correction mechanism is expressed as follows: In the formula, Represents the safety loss function; Represent the feature space; Indicates a time step; Indicates the maximum time step; Represents the collision loss function; Indicates time step The pose of the home service robot at that time; This represents the aligned feature space.

[0009] As a preferred improvement, the hybrid fine-tuning scheme is implemented through a hybrid fine-tuning model, which includes four key components: a frozen Qwen-7B language model base layer, a frozen OpenCLIP visual encoding backbone network, a configurable visual adapter, and a LoRA injection module. The collaborative working mechanism among these components is as follows: The weights of the 24-layer Transformer structure of the frozen Qwen-7B language model base layer remain frozen, fully preserving its performance in general language understanding, logical reasoning, and task decomposition; the parameters of the 24-layer visual Transformer of the frozen OpenCLIP visual encoding backbone network remain frozen, maintaining the output as a 1024-dimensional visual feature vector. This weight solidification strategy preserves the geometric characteristics of the CLIP feature space while avoiding visual representation drift; the configurable visual adapter introduces a cross-attention mechanism, enhancing the spatial localization capability of the Qwen-VL-Chat model while maintaining computational efficiency; the LoRA injection module implements LoRA low-rank adaptation in the Q, K, and V projection layers of the Qwen-VL-Chat model's attention module, and its parameter update mechanism is mathematically expressed as follows: In the formula, This indicates the parameter update of the Qwen-VL-Chat model; The dimension is Small matrix; The dimension is Small matrix; This represents the input / output dimensions of the Qwen-VL-Chat model; Represents the rank dimension. r =8.

[0010] As a preferred improvement, a dual feedback mechanism is introduced into the course learning strategy: escalation is triggered by task success rate, and degrade is triggered by loss detection. Its mathematical expression is: In the formula, Indicates the level of difficulty of the course The overall score below; , This represents the weighting coefficient, with values ​​ranging from 1 to 2. , ; Represents cross-entropy loss; This represents the maximum value of the cross-entropy loss; Indicates the task success rate; Indicates a task sample; Indicates the maximum number of tasks.

[0011] As a preferred improvement, in the task reinforcement stage of the course learning strategy combination, a "scaffolding" teaching strategy was adopted to construct a hierarchical cognitive system. First, a task decomposition tree was generated, and the correlation between its sub-tasks was determined. The calculation formula is: In the formula, Subtasks labeled by experts The correlation function; For task nodes penetration, For task nodes out degree; Based on this difficulty quantification model, a task difficulty evaluation function is established: In the formula, the parameter values ​​are determined through grid search. , , The instruction entropy value The following formula is used to calculate: .

[0012] As a preferred improvement, the generated action sequence is represented as: In the formula, Indicates the gradient coefficient of the safety constraint; Indicates a safety envelope; , They represent , Actions at any given moment; Indicates predicted noise; express The state at any given moment; This represents the gradient of the action value function with respect to the action.

[0013] As a preferred improvement, the generated action sequence is subjected to a two-layer verification mechanism that integrates spatiotemporal coupling analysis and symbolic logic verification to determine its executability.

[0014] The beneficial effects of this invention are as follows: (1) The present invention designs a dynamic gating fusion network for multimodal information fusion, which significantly improves the efficiency of multimodal perception of home service robots, and designs a joint spatiotemporal safety optimization mechanism to improve the quality of multimodal feature fusion; (2) Based on the HomeAct-1.2k dataset, this invention completed the lightweight fine-tuning of the Qwen-VL-Chat model using a course learning strategy, and designed an action generation framework based on the Re-Plan principle based on the model, which effectively improved the task execution capability of the home service robot in complex tasks. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 A flowchart illustrating the home robot motion generation method based on the Re-Plan principle provided by this invention; Figure 2 This describes the task execution framework for home service robots in a home service scenario. Figure 3 This diagram illustrates the architecture of a dynamic gating fusion network. Figure 4 This represents the LoRA schematic diagram; Figure 5 This is a schematic diagram of a three-dimensional progressive curriculum system. Detailed Implementation

[0016] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please refer to Figure 1-Figure 5 This embodiment provides a method for generating motion for a home robot based on the Re-Plan principle, including the following steps: Step S1: Construct a gated fusion network. Input the images and service instructions collected by the home service robot into the gated fusion network. Use CUDA streaming parallel technology to extract the visual features of the images and the text features of the service instructions. Correct the feature space based on the joint spatiotemporal security optimization mechanism. Use data-driven weight allocation to dynamically balance the contribution of the corrected visual features and text features. Fuse the visual features and text features to form a fused feature.

[0018] The dynamic gated fusion network comprises a parallel feature extraction layer, a dynamic gate generator, and a fusion module. The parallel feature extraction layer employs CUDA streaming parallelism to asynchronously encode text and visual features, reducing latency by 43% compared to traditional serial architectures, thus overcoming the latency bottleneck of traditional serial architectures. The dynamic gate generator achieves adaptive mapping of the feature space based on a hierarchical attention mechanism and dynamically balances the contribution of different modalities through learnable parameters. The fusion module utilizes a projection matrix... , This drives weight allocation and enables dynamic calibration of cross-modal features, and, combined with channel compression technology, reduces the parameter size by 78%.

[0019] The dynamic gating fusion network uses a gating coefficient matrix. To achieve nonlinear modulation between modes, its mathematical model is expressed as: In the formula, Represents the gate weight matrix; This represents the Sigmoid activation function; Representing text features; Indicates visual features; Indicates fusion characteristics; , Represents the projection matrix; This represents the Hadamard product.

[0020] To quantitatively evaluate the performance of the dynamic gating fusion network, this invention uses real-time performance, computational efficiency, and parameter efficiency as evaluation indicators. The invention is compared with traditional methods. A test environment is built based on the ISO13482:2014 standard. The multimodal input information tested is a home scene image with a resolution of 1280×720 and a user natural language service instruction with a text length L equal to 256. The benchmark test results compared with mainstream multimodal fusion methods are shown in Table 1.

[0021] Table 1 Performance test results of various multimodal fusion methods As shown in Table 1, the dynamic gating fusion network provided by this invention significantly outperforms traditional multimodal fusion methods in key indicators such as latency control and parameter efficiency, verifying the technical advantages of the parallel coding architecture and dynamic gating mechanism. By introducing a dynamic weight allocation mechanism into the dynamic gating fusion network, the performance of home service robots performing multimodal tasks is significantly improved while maintaining the lightweight nature of the cross-modal feature fusion model, effectively solving problems such as encoding latency and dimensionality mismatch.

[0022] The home environment is characterized by a compact layout (item density of 2.3 / m²), and home service tasks have a very strong temporal dependency (such as the operation chain of "open, retrieve, close"). Traditional simple feature alignment cannot meet the actual needs of home service robots.

[0023] To ensure that the motion generation of home service robots meets the requirements of temporal continuity (e.g., "acceleration of joint movements"), reasonable task timing (e.g., "sequence of multi-stage operations"), and spatial geometric constraints (e.g., "obstacle avoidance by the end effector"), this invention designs a joint spatiotemporal safety optimization mechanism (STS-JOINT). Its core lies in establishing a closed-loop feedback mechanism for feature alignment and safety verification. The system achieves collaborative optimization through three stages: dynamic feature projection, safety boundary prediction, and reverse correction.

[0024] In the dynamic feature projection stage, the feature space is mapped to the physical space through bilinear mapping, achieving cross-modal alignment of visual and textual features. The mapping process is represented as follows: In the formula, This represents the aligned text and visual features; The first feature representing the text One component; The first characteristic representing visual features One component; This represents the Hadamard product operation; Represents the weight matrix; Indicates dimension.

[0025] In the safety boundary prediction stage, this application designs a multi-dimensional perception model that integrates scene semantic entropy and motion state. This is achieved by using obstacle distribution entropy from visual features in a family task dataset. Task criticality score in instruction text features To dynamically adjust the safe distance. The adjustment process is represented as follows: In the formula, Represents the obstacle distribution entropy in visual features; This indicates the maximum moving speed of the home service robot; This represents the task criticality score in the text features.

[0026] In the reverse correction stage, this application designs a reverse feature correction mechanism. When a violation of safety boundaries is detected in the home service robot, the feature space is corrected through gradient backpropagation. The reverse feature correction mechanism is expressed as follows: In the formula, Represents the safety loss function; Represent the feature space; Indicates a time step; Indicates the maximum time step; Represents the collision loss function; Indicates time step The pose of the home service robot at that time; This represents the aligned feature space.

[0027] The execution principle of the joint spatiotemporal safety optimization mechanism is as follows: at time step At that time, the robot's pose If a collision is determined to be close to or violate a safety boundary (e.g., too close to an obstacle), the collision loss is calculated. The first term in the formula tells the robot which direction to move in physical space (away from obstacles) to reduce collision risk. This first term represents the mapping from feature space to physical space, converting the robot's motion in physical space into adjustments in the aligned feature space. The third term represents the mapping from the original feature space to the aligned feature space, converting adjustments in the aligned feature space into adjustments in the original feature space. The core idea of ​​this mechanism is to use gradient information to trace back to the root cause of unsafe behavior (feature space). ), and by correcting the feature space This will help guide future behavior to be safer from the "source".

[0028] To comprehensively evaluate the safety optimization mechanism of joint time and space, this invention sets up four types of task scenarios: item handling, cleaning service, user assistance, and hazard disposal. Among them, 30% of the scenarios contain dynamic obstacles, and 20% of the scenarios have visual occlusion. The performance of the safety optimization mechanism of joint time and space is comprehensively evaluated using positioning error, safety violation rate, and trajectory smoothness as evaluation indicators. The evaluation results are shown in Table 2.

[0029] Table 2 Test Results of the Joint Spatiotemporal Security Optimization Mechanism As shown in Table 2, the joint spatiotemporal safety optimization mechanism significantly improves the quality of multimodal feature fusion. In terms of positioning accuracy, the average error is reduced to 5.1 cm. Regarding safety, it improves by fusing obstacle distribution entropy. With mission criticality The dynamic safety distance calculation reduces the safety violation rate to 3.2% of the baseline method. At the same time, the trajectory smoothness is improved to 0.91 through the reverse correction mechanism, which effectively solves the problem of abrupt joint motion changes in home service robots in traditional methods, significantly improves the task execution capability of home service robots, and provides reliable support for subsequent motion sequence generation.

[0030] Step S2: Construct a multimodal large language model and use a hybrid fine-tuning scheme that integrates low-rank adaptation and visual adapter to perform lightweight fine-tuning of the multimodal large language model: For the text features of service instructions, a low-rank matrix with r=8 is injected through LoRA to achieve targeted enhancement of semantic understanding ability; for the visual features of images, a cross-attention mechanism is used to enhance fine-grained spatial localization ability.

[0031] In the field of home service robots, fine-tuning of multimodal large models faces two major challenges. First, it requires real-time processing of data streams from different modalities (visual images and user natural language commands). This invention effectively addresses this issue through the aforementioned research on cross-modal feature fusion. Second, there is the training efficiency problem caused by limited computing resources. While traditional full-scale fine-tuning schemes have a high upper limit (action generation success rate), their massive memory usage and parameter scale far exceed the processing capabilities of ordinary computing units. Furthermore, the dynamic safety constraints of service robots in home scenarios require models with very accurate cross-modal alignment capabilities. The modal decoupling characteristics of adapters (which reduce multimodal alignment accuracy) and the long-sequence fluctuation defects of prefix-tuning (significantly reducing trajectory smoothness) in mainstream methods are difficult to meet practical requirements.

[0032] By comparing the mainstream fine-tuning methods in existing technologies, it was found that the LoRA method benefits from the directional optimization of the Q, K, and V projection layers, which can control the parameter size within a reasonable range (within 0.1%) through low-rank matrix factorization, while maintaining the structural integrity of the attention mechanism. However, the LoRA method alone has limitations in its adaptability to vision-language multimodal collaboration, resulting in a high response delay for trajectory correction under dynamic safety constraints.

[0033] The hybrid fine-tuning scheme provided by this invention combines the parametric efficiency of LoRA with the dynamic resolution support of the visual adapter, making up for the lack of adaptability of LoRA in complex home scenarios. It effectively balances the advantages and disadvantages of the above methods, and significantly reduces the computational resource requirements of the fine-tuning process while ensuring that the multimodal large language model has high generalization ability.

[0034] The hybrid fine-tuning scheme is implemented through a hybrid fine-tuning model, which includes four key components: a frozen Qwen-7B language model base layer, a frozen OpenCLIP visual encoding backbone network, a configurable visual adapter, and a LoRA injection module. The collaborative working mechanism among these components is as follows: the weights of the 24-layer Transformer structure of the frozen Qwen-7B language model base layer remain frozen, fully preserving its performance in general language understanding, logical reasoning, and task decomposition; the parameters of the 24-layer visual Transformer of the frozen OpenCLIP visual encoding backbone network remain frozen, maintaining the output as a 1024-dimensional visual feature vector. This weight solidification strategy preserves the geometric characteristics of the CLIP feature space while avoiding visual representation drift; the configurable visual adapter introduces a cross-attention mechanism, enhancing the spatial localization capability of the Qwen-VL-Chat model while maintaining computational efficiency; the LoRA injection module implements low-rank LoRA adaptation in the Q, K, and V projection layers of the Qwen-VL-Chat model's attention module, and its parameter update mechanism is mathematically expressed as follows: In the formula, This indicates the parameter update of the Qwen-VL-Chat model; The dimension is Small matrix; The dimension is Small matrix; This represents the input / output dimensions of the Qwen-VL-Chat model; Represents the rank dimension. r =8, adjust the model parameters by 0.07% (approximately 18.6M), and ensure the orthogonality of the injected parameters with the frozen model through gradient masking.

[0035] This application introduces a LoRA structure in the fully connected layer of the visual adapter to build a cross-modal parameter sharing mechanism, which can effectively improve the alignment accuracy of multimodal features.

[0036] Finally, regarding training optimization, this study employs a two-stage fine-tuning strategy. In the pre-alignment stage, contrastive learning loss is used for cross-modal representation alignment to improve image-text accuracy. In the fine-tuning stage, a reinforcement learning strategy is combined to set the reward function to dynamically balance the success rate and safety factor of task execution. This invention also reduces the GPU memory usage (24.1GB) of the model training by using a hybrid precision training and gradient accumulation (step size = 4) technique.

[0037] The hybrid fine-tuning scheme provided by this invention achieves collaborative optimization of multimodal information, and while maintaining the general capabilities of the Qwen-VL-Chat model, it provides targeted enhancements for the specific needs of home-based service tasks. The performance comparison results of the hybrid fine-tuning scheme of this application with other existing schemes are shown in Table 3.

[0038] Table 3 Comparison Results of Fine-tuning Schemes As can be seen from Table 3, compared with the traditional fine-tuning method, there are significant improvements in the number of fine-tuning parameters, computational resource consumption, and the success rate of service robot actions.

[0039] Step S3 introduces the course learning strategy into the fine-tuning process of the multimodal large language model. By constructing a three-dimensional progressive course system, the multimodal large language model can follow the human learning path of "single-point learning - combined reinforcement - comprehensive use" to progressively perform hierarchical reasoning on the fusion features of the input to obtain atomic action sequences, thereby completing the action generation of the home service robot.

[0040] Household service tasks are essentially a hierarchical cognitive process for robots (e.g., "tidying the bedroom" requires breaking down into multiple stages such as organizing items, wiping tables, and arranging furniture). However, the actual home environment is a dynamic environment with interfering factors such as changes in lighting and movement obstructing the view. Therefore, household service robots rely heavily on the environmental adaptability of multimodal large language models. Currently, while traditional end-to-end training models can achieve coarse-grained alignment between semantic understanding and action generation, they face several challenges: First, the sample distribution of long-tail tasks in actual home scenarios varies (high-complexity tasks account for only 5% of the training data), and direct mixed training may cause the model to get stuck in local optima. Second, the coupling between the aforementioned cross-modal feature alignment and safety constraints requires the model to establish a closed-loop capability of "perception-decision-execution". Third, ISO standards have strict requirements for the action coherence of household service robots, making it difficult to directly transfer experience from simple tasks to complex tasks.

[0041] To address the aforementioned issues, this application introduces curriculum learning strategies from the human cognitive domain into the fine-tuning process of the Qwen-VL-Chat model. By constructing a three-dimensional progressive curriculum system, the Qwen-VL-Chat model can progressively overcome complex operational challenges in home service tasks, mimicking the human learning path of "single-point learning - combined reinforcement - comprehensive application." This three-dimensional progressive curriculum system allows the Qwen-VL-Chat model to imitate the human learning process, starting with easy samples and then progressing to more complex samples and knowledge. Its core is highly compatible with the task execution of home service robots, as complex tasks in home scenarios are often a collection of simple tasks. Introducing curriculum learning strategies can significantly improve the Qwen-VL-Chat model's generalization ability in home service tasks.

[0042] The design process of the course learning strategy follows the principle of complexity hierarchy. Based on the cognitive characteristics of family service tasks, this application constructs a system that includes the length of the action sequence (…). ), environmental interference level ( ) and natural language instruction entropy ( The three-dimensional evaluation matrix of ( ). Taking the simple task "rotating a door handle" as an example, its instruction entropy value It equals 0.8 bits. However, in environmental interference levels... In complex tasks such as "opening a door to get something," the instruction entropy value increases due to the temporal dependencies between different actions. This will be increased to 2.1 bits. This quantitative grading system provides a quantitative basis for curriculum design. Simultaneously, this paper introduces a dual feedback mechanism: upgrading is triggered by task success rate, and downgrading is triggered by loss detection. Its mathematical expression is: In the formula, Indicates the level of difficulty of the course The overall score below; , This represents the weighting coefficient, with values ​​ranging from 1 to 2. , ; Represents cross-entropy loss; This represents the maximum value of the cross-entropy loss; Indicates the task success rate; Indicates a task sample; Indicates the maximum number of tasks.

[0043] A three-stage progressive training framework is adopted, which is linked with the LoRA parameter configuration in Section 4.2.1 to form a strategy, as shown in Table 4.5.

[0044] Table 4 Three-stage progressive training framework In the combined task reinforcement stage, this application employs a "scaffolding" teaching strategy to construct a hierarchical cognitive system. First, a task decomposition tree is generated, and the sub-tasks are related in degree. The calculation formula is: In the formula, Subtasks labeled by experts The correlation function; For task nodes In-degree (number of predecessor dependencies) For task nodes Out-degree (subsequent derivatives). Based on this difficulty quantification model, a task difficulty evaluation function is established: The parameter values ​​were determined through a grid search. , , The instruction entropy value The following formula is used to calculate: To verify the actual performance of fine-tuning the Qwen-VL-Chat model after introducing a course learning strategy, this invention conducted ablation experiments using a small number of samples (200 image instruction pairs) from the HomeAct-1.2k dataset, comparing and analyzing key performance indicators under different configurations. The test scenarios included four typical household tasks (item handling, cleaning services, user assistance, and hazard handling), with 30% of the scenarios containing dynamic obstacles and 20% exhibiting visual occlusion. Key performance indicators under different configurations are shown in Table 5 below.

[0045] Table 5. Course Learning Strategies Ablation Experiment Based on the results in Table 5, the course learning strategy significantly improves the performance of the Qwen-VL-Chat model. Crucially, it achieves a substantial improvement over the fixed-rank approach in terms of generalization ability for complex tasks, validating the key role of the task decomposition tree and the correlation between subtasks, thus enabling the service robot to exhibit better adaptability when performing complex household tasks.

[0046] Home service robots, by integrating a cross-modal feature fusion module with a finely tuned Qwen-VL-Chat model, construct an efficient and reliable multimodal information processing channel. This enables them to quickly acquire scene information (from user natural language commands and the main field of view image) and complete end-to-end mapping to a structured action primitive library (207 atomic actions specific to home services). However, if a home service robot only statically executes the action sequence mapped by a single command, the reliability of home service tasks cannot be guaranteed. Its limitations are twofold: firstly, it lacks real-time environmental state perception and dynamic decision-making mechanisms, especially in complex multi-step task scenarios (such as object handling in scenarios with dynamic obstacles). Faced with sudden occlusion, home service robots often cannot respond as flexibly as humans. Secondly, the linear execution mode of the action sequence has an adaptation deviation from real-world home scenarios. When executing complex tasks with more than five steps, such as "desk tidying," the accumulated errors due to the displacement of tableware lead to a sharp increase in the failure rate of the service task. Clearly, traditional planning systems lack the ability to coordinate action execution and state verification, failing to meet the task requirements of service robots in home scenarios.

[0047] To address the above issues, this application designs an action generation framework based on the Re-Plan principle. This framework constructs a closed-loop action execution chain of "visual & textual multimodal perception - dynamic task planning - physical execution verification - environmental state update - multimodal information perception", realizing dynamic collaborative optimization of "perception-decision-execution-verification" for home service robots.

[0048] Based on this framework, the home service robot first acquires the user's natural language commands and the home scene image in its main field of vision through multimodal perception. After feature fusion is completed by a dynamic gating fusion network, the Qwen-VL-Chat model performs hierarchical reasoning to obtain the corresponding atomic action sequence. Based on this atomic action sequence, unlike the static execution of traditional planning frameworks, the home service robot refreshes the main field of vision image after the first atomic action is executed. It then repeats the above process of feature fusion, action reasoning, and physical execution until the task ends. The home service robot can map the atomic actions to the actual home scene through a motion controller and interact with the task target. Its core process includes the following key steps: First, multimodal perception and feature fusion are performed. The home service robot acquires home scene information (1280×720 resolution image) from its main field of view using an RGB-D camera, and recognizes the text content in the verbal commands of family members through a speech receiving module. The semantic features of these two modalities are then obtained through a Dynamically Gated Fusion Network (DGFN). and visual features Ultimately, dynamic fusion features are formed. .

[0049] Next comes semantic understanding and task analysis. The Qwen-VL-Chat model, after fine-tuning with HomeAct-1.2k and reinforcement through course learning, already possesses mature multimodal understanding capabilities, based on the aforementioned fusion features. The atomic action sequences corresponding to user service instructions are parsed out. In particular, the introduction of the temporal dependency detection module gives the action sequences inferred by the Qwen-VL-Chat model the characteristics of spatiotemporal and physical constraints (e.g., opening the door must precede retrieving the item). In this process, this invention employs the MLDT multi-level decomposition method, combined with a course learning strategy to dynamically adjust the granularity of task difficulty. A diffusion strategy is introduced to generate candidate action sequences. In the formula, Indicates the gradient coefficient of the safety constraint; Indicates a safety envelope; , They represent , Actions at any given moment; Indicates predicted noise; express The state at any given moment; This represents the gradient of the action value function with respect to the action.

[0050] The first term in the diffusion strategy formula uses a noise predictor. From noisy actions The first step removes noise progressively, generating more accurate motion estimates; the second step... Introducing reward signals from reinforcement learning, optimizing actions in real time, and through... and Mandatory actions satisfy safety constraints.

[0051] Finally, there is the state verification and physical execution layer. The action sequence output by the Qwen-VL-Chat model is subjected to a two-layer verification mechanism that integrates spatiotemporal coupling analysis and symbolic logic verification to determine its executability.

[0052] After verification, the atomic action sequences all originated from the 207 predefined home service-specific atomic actions in HomeAct-1.2k. Within this action sequence, only... It is in a pending execution state, and the home service robot is executing correctly. After that, it will not automatically redirect to Instead of executing commands directly, the robot re-acquires the current home scene image through the main field-of-view camera, thus completing a closed loop for motion generation. Combining historical natural language commands with the latest visual observations, the Qwen-VL-Chat model serves as the inference engine, generating a new library of atomic actions and executing them within the new sequence. This process continues until the task is completely completed.

[0053] To evaluate the performance of the Re-Plan-based action generation framework in practical applications, this invention conducted experimental verification using a small sample (200 image instruction pairs) from the HomeAct-1.2k dataset. The test scenarios included four typical household tasks (item handling, cleaning services, user assistance, and hazard handling), with 30% of the scenarios containing dynamic obstacles and 20% exhibiting visual occlusion. This invention used a controlled variable approach to compare the performance with traditional task planning frameworks in terms of task success rate, safety violation rate, and response latency. The experimental results are shown in Table 6.

[0054] Table 6 compares the performance of traditional planning methods The experimental results show that the action generation framework based on the Re-Plan principle in this study effectively solves the problem of lack of real-time perception and dynamic matching of environmental state in traditional methods, and effectively improves the task execution capability of home service robots in complex tasks.

[0055] This invention focuses on the multimodal perception, dynamic decision-making, and task execution of home service robots. First, a dynamic gating fusion network for multimodal information fusion is designed, significantly improving the efficiency of multimodal perception in home service robots. A joint spatiotemporal safety optimization mechanism is also designed to enhance the quality of multimodal feature fusion. Second, based on the HomeAct-1.2k dataset, this invention employs a course learning strategy to perform lightweight fine-tuning of the Qwen-VL-Chat model. Based on this model, an action generation framework based on the Re-Plan principle is designed, effectively improving the task execution capability of home service robots in complex tasks.

[0056] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for generating motion for a home robot based on the Re-Plan principle, characterized in that, Includes the following steps: Step S1: Construct a gated fusion network. Input the images and service instructions collected by the home service robot into the gated fusion network. Use CUDA streaming parallel technology to extract the visual features of the images and the text features of the service instructions. Correct the feature space based on the joint spatiotemporal security optimization mechanism. Use data-driven weight allocation to dynamically balance the contribution of the corrected visual features and text features. Fuse the visual features and text features to form fused features. Step S2: Construct a multimodal large language model and use a hybrid fine-tuning scheme that integrates low-rank adaptation and visual adapter to perform lightweight fine-tuning of the multimodal large language model: For the text features of service instructions, LoRA is used to inject a low-rank matrix to achieve targeted enhancement of semantic understanding ability; for the visual features of images, a cross-attention mechanism is used to enhance fine-grained spatial localization ability. Step S3 introduces the course learning strategy into the fine-tuning process of the multimodal large language model. By constructing a three-dimensional progressive course system, the multimodal large language model can follow the human learning path of "single-point learning - combined reinforcement - comprehensive use" to progressively perform hierarchical reasoning on the fusion features of the input to obtain atomic action sequences, thereby completing the action generation of the home service robot.

2. The method for generating motion for a home robot based on the Re-Plan principle according to claim 1, characterized in that, The dynamic gating fusion network uses a gating coefficient matrix. To achieve nonlinear modulation between modes, its mathematical model is expressed as: In the formula, Represents the gate weight matrix; This represents the Sigmoid activation function; Representing text features; Indicates visual features; Indicates fusion characteristics; , Represents the projection matrix; This represents the Hadamard product.

3. The method for generating motion for a home robot based on the Re-Plan principle according to claim 2, characterized in that, Step S1, "correcting the feature space based on the joint spatiotemporal security optimization mechanism," specifically includes the following process: Step S11: The feature space is mapped to the physical space through bilinear mapping to achieve cross-modal alignment of visual and textual features. The mapping process is represented as follows: Where, This represents the aligned text features and visual features; The first feature representing the text One component; The first characteristic representing visual features One component; This represents the Hadamard product operation; Represents the weight matrix; Indicates dimension; Step S12: A multi-dimensional perception model integrating scene semantic entropy and motion state was designed, using obstacle distribution entropy from the visual features of the family task dataset. Task criticality score in service instruction text features To dynamically adjust the safe distance. Determine the safety boundaries, and the adjustment process is represented as follows: Where, Represents the obstacle distribution entropy in visual features; This indicates the maximum moving speed of the home service robot; This represents the task criticality score within the text features; Step S13: Design a reverse feature correction mechanism. When a violation of safety boundaries is detected in the home service robot, the feature space is corrected through gradient backpropagation. The reverse feature correction mechanism is expressed as follows: Where, Represents the safety loss function; Represent the feature space; Indicates a time step; Indicates the maximum time step; Represents the collision loss function; Indicates time step The pose of the home service robot at that time; This represents the aligned feature space.

4. The method for generating motion for a home robot based on the Re-Plan principle according to claim 1, characterized in that, The hybrid fine-tuning scheme is implemented through a hybrid fine-tuning model, which includes four key components: a frozen Qwen-7B language model base layer, a frozen OpenCLIP visual encoding backbone network, a configurable visual adapter, and a LoRA injection module. The collaborative working mechanism among these components is as follows: The weights of the 24-layer Transformer structure of the frozen Qwen-7B language model base layer remain frozen, fully preserving its performance in general language understanding, logical reasoning, and task decomposition; the parameters of the 24-layer visual Transformer of the frozen OpenCLIP visual encoding backbone network remain frozen, maintaining the output as a 1024-dimensional visual feature vector. This weight solidification strategy preserves the geometric characteristics of the CLIP feature space while avoiding visual representation drift; the configurable visual adapter introduces a cross-attention mechanism, enhancing the spatial localization capability of the Qwen-VL-Chat model while maintaining computational efficiency; the LoRA injection module implements low-rank LoRA adaptation in the Q, K, and V projection layers of the Qwen-VL-Chat model's attention module. The mathematical expression of its parameter update mechanism is as follows: Where, This indicates the parameter update of the Qwen-VL-Chat model; The dimension is Small matrix; The dimension is Small matrix; This represents the input / output dimensions of the Qwen-VL-Chat model; Represents the rank dimension. r =8.

5. The method for generating motion for a home robot based on the Re-Plan principle according to claim 1, characterized in that, The course learning strategy incorporates a dual feedback mechanism: escalation is triggered by task success rate, and degrade is triggered by loss detection. Its mathematical expression is as follows: Where, Indicates the level of difficulty of the course The overall score below; , This represents the weighting coefficient, with values ​​ranging from 1 to 2. , ; Represents cross-entropy loss; This represents the maximum value of the cross-entropy loss; Indicates the task success rate; Indicates a task sample; Indicates the maximum number of tasks.

6. The method for generating motion for a home robot based on the Re-Plan principle according to claim 5, characterized in that, In the task reinforcement phase of the course learning strategies, a "scaffolding" teaching strategy was adopted to construct a hierarchical cognitive system. First, a task decomposition tree was generated, and the relevance of its sub-tasks was determined. The calculation formula is: In the formula, Subtasks labeled by experts The correlation function; For task nodes penetration, For task nodes out degree; Based on this difficulty quantification model, a task difficulty evaluation function is established: In the formula, the parameter values ​​are determined through grid search. , , The instruction entropy value The following formula is used to calculate: 。 7. The method for generating motion for a home robot based on the Re-Plan principle according to claim 1, characterized in that, The generated action sequence is represented as: Where, Indicates the gradient coefficient of the safety constraint; Indicates a safety envelope; , They represent , Actions at any given moment; Indicates predicted noise; express The state at any given moment; This represents the gradient of the action value function with respect to the action.

8. The method for generating motion for a home robot based on the Re-Plan principle according to claim 1, characterized in that, The generated action sequence is evaluated for executability through a two-layer verification mechanism that integrates spatiotemporal coupling analysis and symbolic logic verification.

Images