Task instruction hierarchical reconstruction-based embodiment video generation model fine-tuning method

Abstract

A method for fine-tuning a multimodal embodied video generation model based on hierarchical reconstruction of task instructions includes steps such as multi-source heterogeneous navigation data acquisition and unified preprocessing, data augmentation based on random sub-trajectory sampling, sub-instruction reconstruction based on a multimodal large model, hierarchical cue word template construction, LoRA fine-tuning training of the video generation model, and instruction construction and video generation during the inference stage. In the inference stage, a hierarchical instruction reconstruction pipeline and structured cue word templates are constructed to solve problems such as insufficient multimodal alignment, low data utilization, and difficulty in heterogeneous data fusion when fine-tuning the video generation model in visual language navigation scenarios.

Description

Technical Field

[0001] This application relates to the field of embodied intelligence technology, and more specifically, to a method for fine-tuning a multimodal embodied video generation model based on hierarchical reconstruction of task instructions. Background Technology

[0002] In the field of embodied intelligence, vision-and-language navigation (VLN) is a core task, requiring agents to perform navigation actions in unknown environments based on natural language instructions. In recent years, with the rapid development of large-scale visual language models and video generation models, researchers have begun to explore the introduction of video generation capabilities into embodied navigation systems to achieve the prediction and planning of future visual trajectories, forming a "think first, then do" dual-system architecture.

[0003] In existing technologies, methods such as NavigateDiff and AstarNav attempt to generate future navigation video frames using diffusion models to provide visual targets or trajectory references for downstream decision-making systems. However, these methods suffer from the following major drawbacks in practical applications: (1) Insufficient multimodal alignment between vision and instructions. Existing methods typically use the original navigation instructions directly as textual conditional inputs to the video generation model for fine-tuning training. However, the original VLN instructions describe the complete navigation process from the starting point to the end point. When the video clips in the training data only cover part of the trajectory, there is a serious semantic misalignment between the complete instructions and the truncated video clips. This misalignment causes severe visual illusions when the model generates subsequent frames, reducing the spatial consistency and instruction compliance of the generated video.

[0004] (2) Low data utilization efficiency and lack of effective data augmentation strategies. Existing methods process training data in a relatively simple way, usually by preprocessing the data by taking a fixed number of the first N frames. This approach leads to a large amount of long trajectory data being wasted or truncated unreasonably. On the other hand, it makes it easy for the model to implicitly learn pseudo-patterns that stop at a fixed number of frames, thus reducing the model's generalization ability.

[0005] (3) Lack of a unified training framework for heterogeneous data sources. In practical applications, navigation data comes from various sources, including simulation environment rendering data (such as Matterport3D) and real-world video data (such as YouTube navigation videos). Different data sources have significant differences in terms of viewpoint height, resolution, frame rate, and instruction format. Existing methods lack a unified data organization and instruction construction scheme, making it difficult to effectively integrate multi-source heterogeneous data for joint training, thus limiting the model's transferability in real-world environments.

[0006] (4) The instruction prompts have a simple structure and lack hierarchical task context information. Existing methods, when fine-tuning video generation models, only use the original navigation instructions as the sole text prompt input, lacking structured encoding of contextual information such as the current action, task progress, observation domain, and perspective. This simple prompt structure limits the video generation model's ability to understand the navigation task structure, making it difficult to generate video content that accurately corresponds to the current navigation stage.

[0007] For example, Chinese patent document CN120147918A discloses an embodied intelligence data processing method, including: acquiring a task reference video of an embodied intelligent agent, the task reference video containing behavioral information of the task execution object and state information of task-related objects; performing multimodal feature extraction on the task reference video to obtain visual semantic features of the task reference video and body posture features of the task execution object; fusing the visual semantic features and the body posture features to obtain a first fused feature; performing task recognition on the task reference video and behavioral intent recognition on the task execution object based on the first fused feature to obtain a recognition result; and determining valid video segments of the task reference video based on the recognition result, the valid video segments being used for verification and / or reference of operations during the embodied intelligent agent's task execution. However, this technical solution is clearly insufficient in terms of hierarchical reconstruction and unified training.

[0008] For example, Chinese patent document CN121531204A discloses a video generation method based on embodied masking. This method includes determining an embodied mask sequence based on a target video; encoding the embodied mask sequence and the initial frame image of the target video respectively to obtain embodied mask features and image encoding features; inputting the embodied mask features into a control network module to obtain injected features, and inputting the image encoding features into the backbone network of a video generation model to obtain backbone layer features; fusing the injected features and the backbone layer features to obtain fused features, and generating a predicted video based on the fused features. This technical solution also suffers from problems such as insufficient multimodal alignment, low data utilization, and difficulty in fusing heterogeneous data. Summary of the Invention

[0009] In view of the technical problems existing in the prior art, the present invention aims to provide a method for fine-tuning a multimodal embodied video generation model based on hierarchical reconstruction of task instructions. By constructing a hierarchical instruction reconstruction pipeline and structured prompt word templates, the method solves the problems of insufficient multimodal alignment, low data utilization, and difficulty in heterogeneous data fusion when fine-tuning video generation models in visual language navigation scenarios.

[0010] Specifically, the following technical solution is adopted to solve this technical problem.

[0011] According to one aspect of the present invention, a method for fine-tuning a multimodal embodied video generation model based on hierarchical reconstruction of task instructions is provided, comprising the following steps: Step (1): Multi-source heterogeneous navigation data acquisition and unified preprocessing: Acquire navigation trajectory data; perform unified preprocessing on the acquired multi-source data; Step (2): Data augmentation based on random sub-trajectory sampling: For each navigation trajectory, adaptive processing is performed according to the preset training frame number parameter N: Step (3): Sub-instruction reconstruction based on multimodal large model: For the sub-trajectory segments that need to be reconstructed in step (2b), hierarchical reconstruction of instructions is carried out using the open-source multimodal visual language large model; Step (4): Construction of hierarchical prompt word template: Based on the output results of steps (2) and (3), construct hierarchical structured prompt words for each training sample; Step (5): LoRA fine-tuning training of the video generation model: Based on the structured training data constructed above, the pre-trained text-image to video (TI2V) diffusion generation model is fine-tuned using low-rank adaptive (LoRA). Step (6): Command construction and video generation in the inference phase: In the inference phase, given the complete command of the navigation task and the first frame image of the current observation, inference prompts are constructed according to the hierarchical prompt template consistent with that during training.

[0012] According to the above method, the unified preprocessing in step (1) specifically includes: normalizing the resolution of image data from different sources, for example, adjusting the simulation data to a resolution of 640×480, and cropping the real video data to align with the size and field of view of the simulation data; performing frame count statistics and metadata registration for each navigation trajectory, constructing a unified metadata file, and recording the scene identifier, agent parameter configuration (including observation height and pitch angle, such as 60cm_30deg), image storage path, trajectory number, original navigation instructions and other field information for each trajectory.

[0013] According to the method described, the adaptive processing in step (2) includes: (2a) if the actual number of frames of a certain trajectory is less than or equal to N, then all frame data of the trajectory are retained, no reconstruction is performed, the sub-trajectory window is directly recorded as [0, frames-1], the sub-instruction is consistent with the original instruction, and the task progress is marked as 0%-100%.

[0014] (2b) If the actual number of frames of a certain trajectory is greater than N, then a continuous sub-trajectory segment with a length of exactly N frames is randomly sampled from the entire frame sequence of the trajectory, and the starting frame index isart and the ending frame index iend of the sub-trajectory segment are recorded.

[0015] According to the method described above, the specific reconstruction process in step (3) is as follows: (3a) Auxiliary frame sampling: Several frames are uniformly extracted from the sub-trajectory segments of length N frames as auxiliary reference frames, which together with the start frame and the end frame form a visual reference set to provide spatial-temporal visual context for multimodal large models.

[0016] (3b) Multimodal Alignment Inference: The original complete navigation command, the starting frame index isart, the ending frame index iend, the total number of trajectory frames, and the aforementioned visual reference frames are input into the multimodal visual language large model. The model is guided to perform the following tasks through carefully designed prompt word templates: Based primarily on visual information and supplemented by frame indexes, determine the semantic interval corresponding to the sub-trajectory segment in the original complete command; Based on visual anchoring and command semantics, extract and rewrite the sub-command that precisely corresponds to the sub-trajectory segment from the original command, using only the vocabulary and structure already present in the original command and without introducing new actions, landmarks, or viewpoint descriptions; Dynamically evaluate the progress interval of the sub-trajectory segment in the complete navigation task and output it in the form of a percentage integer interval.

[0017] (3c) Structured output: The reconstruction results of the multimodal large model are output in JSON format, including frame range, instruction span, sub-instruction, action mode and progress interval.

[0018] According to the method described, the prompt words in step (4) include the following hierarchical fields: (4a) Task field: Records the original complete navigation instructions, providing a global task context for the video generation model.

[0019] (4b) Current Action field: Records the reconstructed sub-instruction, accurately describing the navigation action corresponding to the current sub-track segment, and achieving precise alignment between text conditions and video content.

[0020] (4c) Progress field: Records the progress percentage range of the current sub-trajectory in the complete task, providing the model with temporal positioning information.

[0021] (4d) Domain field: indicates the source domain type of the current data, such as Virtual (simulation environment) or Real (real world), to help the model distinguish and adapt to the features of different visual domains.

[0022] (4e) View field: Label the observation view type of the current data, such as Robot Ground Level (robot ground view, height about 0.5m) or Human-Eye-Level (human eye level view, height about 1.25m), so that the model can learn and distinguish visual features under different view.

[0023] According to the method, the training in step (5) specifically includes: (5a) Basic model selection: Wan2.2-TI2V-5B is used as the basic video generation model. This model is based on the DiT (Diffusion Transformer) architecture and supports the generation of subsequent video frame sequences with text prompts and the first frame image as conditional inputs.

[0024] (5b) LoRA fine-tuning strategy: LoRA adapters are injected into the Cross-Attention Layer and Temporal Attention Block / Time Embedding of the DiT architecture. The model parameters are finely tuned efficiently by using low-rank matrix factorization. While maintaining the original generative ability of the model, visual-language alignment knowledge from the navigation domain is injected.

[0025] (5c) Multi-stage training strategy: A phased training approach is adopted. First, the model is fine-tuned on simulation environment data (such as VLN-CE R2R data) to learn the visual generation pattern of navigation scene. Then, the model is fine-tuned on real-world data (such as YouTube navigation video data). The second stage adopts a cosine annealing learning rate scheduling strategy with warm-up and loads the LoRA weights of the first stage as initialization to achieve a gradual transfer from the simulation domain to the real domain.

[0026] (5d) Training frame number setting: During training, use N frames (e.g., 49 frames) as the target video length. During inference, the number of generated frames can be flexibly adjusted (e.g., 17 frames). The number of training frames should be greater than or equal to the number of inference frames to ensure the generation quality.

[0027] According to the method described, in step (6), the Current Action field in the prompt is set to a general description (such as "Continue navigation based on the current view.") during inference, the Task field retains the complete navigation instructions, and the Progress field is dynamically set according to the current navigation progress. By inputting the constructed prompt and the first frame image into the fine-tuned video generation model, a future visual trajectory prediction video that is semantically aligned with the current navigation stage can be generated, providing visual target guidance information for the downstream decision-making system.

[0028] According to another aspect of the present invention, this application also provides an electronic device, characterized in that it includes: a processor, and a memory connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the fine-tuning method described in any of the preceding claims.

[0029] According to another aspect of the present invention, this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described fine-tuning method.

[0030] According to another aspect of the present invention, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described fine-tuning method.

[0031] Compared with the prior art, the present invention has the following advantages: (1) Significantly improves the accuracy of visual-instruction multimodal alignment. By introducing a sub-instruction reconstruction mechanism based on a large multimodal visual language model (step 3), the complete navigation instruction is decomposed into sub-instructions that precisely correspond to the sub-trajectory segments, fundamentally solving the problem of semantic misalignment between complete instructions and truncated video segments in the prior art. Experimental results show that on the R2R Val Unseen dataset, the method of the present invention achieves a PSNR of 12.13, an SSIM of 0.51, and a CLIPScore of 26.61 under the condition of 49 frames generated. Compared with the original Wan2.2-TI2V-5B baseline model without instruction reconstruction (PSNR 10.41, SSIM 0.37, CLIPScore 25.35), all of these are significantly improved. The FID decreases from 106.15 to 63.73, and the FVD decreases from 2159.05 to 354.42, demonstrating the significant effect of the present invention in reducing visual illusions and improving generation quality.

[0032] (2) Effectively improve data utilization efficiency and model generalization ability. By using the random sub-trajectory sampling strategy (step 2b), multiple sub-segments in the long trajectory data are fully utilized for training, avoiding data waste and pseudo-pattern learning caused by fixed truncation. At the same time, random sampling itself has the effect of data augmentation, enabling the model to maintain stable generation quality when facing navigation tasks with different starting positions and different lengths.

[0033] (3) Achieving unified training of multi-source heterogeneous navigation data. By introducing Domain and View fields into hierarchical prompts (steps 4d and 4e), this invention constructs a unified training framework that is compatible with both simulation data and real-world data. Data from different sources and from different perspectives are distinguished by structured labels, and in conjunction with a multi-stage training strategy (step 5c), effective domain transfer from the simulation environment to the real scene is achieved, significantly enhancing the model's adaptability in actual deployment scenarios.

[0034] (4) Provide rich hierarchical task context information. Unlike existing methods that only use a single text instruction as a prompt word, the hierarchical prompt word template constructed in this invention (step 4) contains multi-dimensional information such as global task, current action, task progress, data domain and perspective, providing sufficient structured context for the video generation model, enabling the model to more accurately understand and follow the phased goals of the navigation task and generate video content that is highly consistent with the current navigation phase. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the overall process of a method for fine-tuning a multimodal embodied video generation model based on hierarchical reconstruction of task instructions, according to a specific embodiment of the present invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Those skilled in the art will understand that this description is exemplary and the present invention is not limited to the specific embodiments described herein.

[0037] Figure 1 The overall flowchart of the multimodal embodied video generation model fine-tuning method based on hierarchical reconstruction of task instructions in a specific implementation is shown, as follows: Figure 1 As shown, the method includes the following steps: Step (1): Acquisition and unified preprocessing of multi-source heterogeneous navigation data First, navigation trajectory data was collected from various publicly available sources, including but not limited to: (a) VLN datasets based on simulation platforms, such as R2R navigation trajectory data based on Matterport3D scenes in the VLN-CE dataset. This type of data includes RGB image sequences rendered step by step by the simulator according to discrete actions and corresponding natural language navigation instructions; (b) navigation datasets based on real-world videos, such as real indoor navigation video frames and corresponding navigation instruction annotations extracted from NaVILA's YouTube navigation video dataset.

[0038] The collected multi-source data undergoes unified preprocessing, specifically including: resolution normalization of image data from different sources, such as adjusting simulation data to a uniform 640×480 resolution, and center-cropping real video data to align with the size and field of view of the simulation data; frame count and metadata registration for each navigation trajectory, constructing a unified metadata file that records field information such as scene identifier, agent parameter configuration (including observation height and pitch angle, such as 60cm_30deg), image storage path, trajectory number, and original navigation instructions for each trajectory.

[0039] Step 2: Data Augmentation Based on Random Sub-Trajectory Sampling For each navigation trajectory, adaptive processing is performed based on the preset training frame count parameter N (e.g., N=49): (2a) If the actual number of frames of a certain trajectory is less than or equal to N, then all frame data of the trajectory are retained without reconstruction. The sub-trajectory window is directly recorded as [0, frames-1]. The sub-instruction is consistent with the original instruction, and the task progress is marked as 0%-100%.

[0040] (2b) If the actual number of frames for a certain trajectory is greater than N, then a continuous sub-trajectory segment with a length of exactly N frames is randomly sampled from the entire frame sequence of that trajectory, and the starting frame index istar and the ending frame index iend of the sub-trajectory segment are recorded. This random sampling strategy avoids the traditional approach of fixedly truncating the first N frames, effectively preventing the model from learning a pseudo-pattern that terminates the motion at a fixed frame position, while maximizing the utilization efficiency of long trajectory data and achieving data-level enhancement.

[0041] Step 3: Sub-instruction reconstruction based on a multimodal large model For the sub-trajectory segments that need to be reconstructed in step (2b), this invention utilizes an open-source multimodal visual language large model (e.g., Qwen2.5-VL-32B-Instruct) for instruction-level hierarchical reconstruction. The specific process is as follows: (3a) Auxiliary frame sampling: Several frames are uniformly extracted from the sub-trajectory segments of length N frames as auxiliary reference frames, which together with the start frame and the end frame form a visual reference set to provide spatial-temporal visual context for multimodal large models.

[0042] (3b) Multimodal Alignment Inference: The original complete navigation command, the starting frame index isart, the ending frame index iend, the total number of trajectory frames, and the aforementioned visual reference frames are input into the multimodal visual language large model. The model is guided to perform the following tasks through carefully designed prompt word templates: Based primarily on visual information and supplemented by frame indexes, determine the semantic interval corresponding to the sub-trajectory segment in the original complete command; Based on visual anchoring and command semantics, extract and rewrite the sub-command that precisely corresponds to the sub-trajectory segment from the original command, using only the vocabulary and structure already present in the original command and without introducing new actions, landmarks, or viewpoint descriptions; Dynamically evaluate the progress interval of the sub-trajectory segment in the complete navigation task and output it in the form of a percentage integer interval.

[0043] (3c) Structured output: The reconstruction results of the multimodal large model are output in JSON format, including frame range, instruction span, sub-instruction, action mode and progress interval.

[0044] Step 4: Building a Hierarchical Hint Template Based on the output of steps 2 and 3, this invention constructs hierarchical structured prompts for each training sample. These prompts contain the following hierarchical fields: (4a) Task field: Records the original complete navigation instructions, providing a global task context for the video generation model.

[0045] (4b) Current Action field: Records the reconstructed sub-instruction, accurately describing the navigation action corresponding to the current sub-track segment, and achieving precise alignment between text conditions and video content.

[0046] (4c) Progress field: Records the progress percentage range of the current sub-trajectory in the complete task, providing the model with temporal positioning information.

[0047] (4d) Domain field: indicates the source domain type of the current data, such as Virtual (simulation environment) or Real (real world), to help the model distinguish and adapt to the features of different visual domains.

[0048] (4e) View field: Label the observation view type of the current data, such as Robot Ground Level (robot ground view, height about 0.5m) or Human-Eye-Level (human eye level view, height about 1.25m), so that the model can learn and distinguish visual features under different view.

[0049] Here is an example of a complete training prompt word: Task: Walk directly between the refrigerator and the island, then continue to the fireplace and stop in front of the TV. Current Action: Walk between the refrigerator and the island. Progress: 0% - 25%. Domain: Virtual. View: Robot Ground Level. Step 5: LoRA fine-tuning training of the video generation model Based on the structured training data constructed above, the pre-trained text-image-to-video (TI2V) diffusion generation model is fine-tuned using low-rank adaptive (LoRA) training. Specifically: (5a) Basic model selection: Wan2.2-TI2V-5B is used as the basic video generation model. This model is based on the DiT (Diffusion Transformer) architecture and supports the generation of subsequent video frame sequences with text prompts and the first frame image as conditional inputs.

[0050] (5b) LoRA fine-tuning strategy: LoRA adapters are injected into the Cross-Attention Layer and Temporal Attention Block / Time Embedding of the DiT architecture. The model parameters are finely tuned efficiently by using low-rank matrix factorization. While maintaining the original generative ability of the model, visual-language alignment knowledge from the navigation domain is injected.

[0051] (5c) Multi-stage training strategy: A phased training approach is adopted. First, the model is fine-tuned on simulation environment data (such as VLN-CE R2R data) to learn the visual generation pattern of navigation scene. Then, the model is fine-tuned on real-world data (such as YouTube navigation video data). The second stage adopts a cosine annealing learning rate scheduling strategy with warm-up and loads the LoRA weights of the first stage as initialization to achieve a gradual transfer from the simulation domain to the real domain.

[0052] (5d) Training frame number setting: During training, use N frames (e.g., 49 frames) as the target video length. During inference, the number of generated frames can be flexibly adjusted (e.g., 17 frames). The number of training frames should be greater than or equal to the number of inference frames to ensure the generation quality.

[0053] Step 6: Instruction Construction and Video Generation in the Inference Phase During the inference phase, given the complete instructions for the navigation task and the first observed frame image, inference prompts are constructed according to a hierarchical prompt template consistent with that used during training. Specifically, the Current Action field is set to a general description during inference (e.g., "Continue navigation based on the current view."), the Task field retains the complete navigation instructions, and the Progress field is dynamically set based on the current navigation progress. By inputting the constructed prompts and the first frame image into a fine-tuned video generation model, a future visual trajectory prediction video semantically aligned with the current navigation stage can be generated, providing visual target guidance information for the downstream decision-making system.

[0054] In another embodiment of the invention, an electronic device is also provided, which can be provided as a server or computer, including one or more processors and memory resources represented by memory for storing instructions executable by the processor, such as application programs. The application programs stored in the memory may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor is configured to execute the instructions to perform the above-described fine-tuning method. The electronic device may also include a power supply component, a wired or wireless network interface, and an input / output (I / O) interface, etc. The electronic device can operate on an operating system stored in the memory, such as Windows Server™, Mac OSX™, Unix™, Linux™, FreeBSD™, or similar. This application also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the above-described fine-tuning method. This application also provides a computer program product including a computer program that, when executed by a processor, implements the above-described fine-tuning method. The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. The scope of protection of this invention is defined by the appended claims.

Claims

1. A method for fine-tuning a multimodal embodied video generation model based on hierarchical reconstruction of task instructions, comprising the following steps: Step (1): Multi-source heterogeneous navigation data acquisition and unified preprocessing: Acquire navigation trajectory data; The collected multi-source data is preprocessed in a unified manner; Step (2): Data augmentation based on random sub-trajectory sampling: For each navigation trajectory, adaptive processing is performed according to the preset training frame number parameter N: Step (3): Sub-instruction reconstruction based on multimodal large model: For the sub-trajectory segments that need to be reconstructed in step (2b), hierarchical reconstruction of instructions is carried out using the open-source multimodal visual language large model; Step (4): Construction of hierarchical prompt word template: Based on the output results of steps (2) and (3), construct hierarchical structured prompt words for each training sample; Step (5): LoRA fine-tuning training of the video generation model: Based on the structured training data constructed above, the pre-trained text-image to video (TI2V) diffusion generation model is fine-tuned using low-rank adaptive (LoRA). Step (6): Command construction and video generation in the inference phase: In the inference phase, given the complete command of the navigation task and the first frame image of the current observation, inference prompts are constructed according to the hierarchical prompt template consistent with that during training.

2. The method of claim 1, wherein, The unified preprocessing described in step (1) includes: normalizing the resolution of image data from different sources, performing frame count and metadata registration for each navigation trajectory, constructing a unified metadata file, and recording the scene identifier, agent parameter configuration, image storage path, trajectory number, and original navigation instructions for each trajectory.

3. The method of claim 1, wherein, The adaptive processing described in step (2) includes: (2a) If the actual number of frames of a certain trajectory is less than or equal to N, then all frame data of the trajectory are retained without reconstruction. The sub-trajectory window is directly recorded as [0, frames-1]. The sub-instruction is consistent with the original instruction, and the task progress is marked as 0%-100%. (2b) If the actual number of frames of a certain trajectory is greater than N, then a continuous sub-trajectory segment with a length of exactly N frames is randomly sampled from the entire frame sequence of the trajectory, and the starting frame index isart and the ending frame index iend of the sub-trajectory segment are recorded.

4. The method of claim 1, wherein, The reconstruction process described in step (3) is as follows: (3a) Auxiliary frame sampling: Several frames of images are uniformly extracted from the sub-trajectory segments of length N frames as auxiliary reference frames, which together with the start frame and the end frame form a visual reference set to provide spatial-temporal visual context for multimodal large models. (3b) Multimodal alignment reasoning: Input the original complete navigation command, the start frame index isart, the end frame index iend, the total number of trajectory frames, and the above visual reference frames into the multimodal visual language large model. Guide the model to perform the following tasks through prompt word templates: Based on visual information as the main basis and frame index as an auxiliary clue, determine the semantic interval corresponding to the sub-trajectory segment in the original complete command; Based on visual anchoring and command semantics, extract and rewrite the sub-command that precisely corresponds to the sub-trajectory segment from the original command. When rewriting, only use the vocabulary and structure already in the original command and do not introduce new actions, landmarks, or viewpoint descriptions; Dynamically evaluate the progress interval of the sub-trajectory segment in the complete navigation task and output it in the form of percentage integer intervals. (3c) Structured output: The reconstruction results of the multimodal large model are output in JSON format, including frame range, instruction span, sub-instruction, action mode and progress interval.

5. The method of claim 1, wherein, The prompt words in step (4) include the following hierarchical fields: (4a) Task field: Records the original complete navigation instructions, providing a global task context for the video generation model; (4b) Current Action field: Records the reconstructed sub-instruction, describing the navigation action corresponding to the current sub-track segment, and achieving precise alignment between text conditions and video content; (4c) Progress field: Records the progress percentage range of the current sub-trajectory in the complete task, providing the model with temporal positioning information; (4d) Domain field: indicates the source domain type of the current data, helping the model to distinguish and adapt to the features of different visual domains; (4e) View field: Marks the observation view type of the current data, enabling the model to learn and distinguish visual features under different viewpoints.

6. The method of claim 1, wherein, The training in step (5) includes: (5a) Basic model selection: Wan2.2-TI2V-5B is used as the basic video generation model, which supports the generation of subsequent video frame sequences with text prompts and the first frame image as conditional inputs; (5b) LoRA fine-tuning strategy: LoRA adapters are injected into the cross-attention layer and temporal modeling module of the DiT architecture, and the model parameters are finely tuned efficiently by low-rank matrix factorization. While maintaining the original generative ability of the model, visual-language alignment knowledge in the navigation field is injected. (5c) Multi-stage training strategy: adopt a phased training method. First, fine-tune the model on the simulation environment data to enable the model to learn the visual generation mode of the navigation scene. Then, fine-tune the model on the real world data. The second stage adopts a cosine annealing learning rate scheduling strategy with preheating and loads the LoRA weights of the first stage as initialization to achieve a gradual transfer from the simulation domain to the real domain. (5d) Training frame number setting: During training, N frames (N is a positive integer) are used as the target video length. During inference, the number of generated frames can be flexibly adjusted. The number of training frames is greater than or equal to the number of inference frames to ensure the generation quality.

7. The method of claim 1, wherein, In step (6), the CurrentAction field is set to a general description during inference, the Task field maintains complete navigation instructions, and the Progress field is dynamically set according to the current navigation progress. By inputting the constructed prompts and the first frame image into the fine-tuned video generation model, a future visual trajectory prediction video that is semantically aligned with the current navigation stage can be generated, providing visual target guidance information for the downstream decision-making system.

8. An electronic device, comprising: include: A processor, and a memory connected to the processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the fine-tuning method according to any one of claims 1-7.

9. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement any one of the fine-tuning methods of claims 1-7.

10. A computer program product comprising a computer program that, when executed by a processor, implements a scheme of fine-tuning method as described in any one of claims 1-7.

Images