Method for automatic generation of cross-ontology visual navigation dataset based on multi-model coupling

Abstract

The application discloses a kind of cross ontology visual navigation dataset automatic generation method based on multi-model coupling.The method obtains first person original long video and carries out pre-processing and slicing by automation agent;Then depth map, camera pose, camera internal parameter and depth confidence are recovered using visual geometry base model, and camera trajectory is formed;Long context multi-modal model is then used to generate structured report containing shot continuity, camera motion, scene semantics and dynamic entity;And the report, key frame and target ontology physical constraint rule are input into logical inference model, and video slice suitable for target ontology is screened;Finally, according to dynamic entity information, time sequence dynamic mask is generated;RGB image, geometric information, dynamic mask and ontology label are aligned according to uniform timestamp, and packaged as training data.The method is used to generate visual navigation dataset for different robot ontology from internet video.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligence and end-to-end visual navigation technology, and in particular relates to an automated generation method for cross-ontology visual navigation datasets based on multi-model coupling. Background Technology

[0002] End-to-end visual navigation establishes a mapping between visual observations and motion control outputs through neural network models. With the development of visual language action models and fundamental robot models, cross-ontology learning is beginning to map heterogeneous data, such as wheeled robots, quadruped robots, drones, and human first-person video, to a shared representation space to extract transferable navigation knowledge. However, different entities differ in motion patterns, pose ranges, velocities, and sensor configurations, requiring structured, temporally continuous training data that matches the physical constraints of the target entity.

[0003] To reduce the cost of real-world robot data acquisition and expand scene coverage, existing methods are beginning to utilize first-person tours, house visits, and urban walking videos from internet platforms to construct navigation training data. Typical methods include: constructing task signals based on object detection results; estimating camera pose changes and generating action labels using visual odometry or simultaneous localization and mapping methods; and combining language models to generate navigation text instructions, thereby forming image, text, and action-paired data.

[0004] Existing technologies include pipeline frameworks for automatically generating navigation supervision signals based on internet video, such as those proposed by CityWalker, LeLaN, and NaVILA. These methods typically involve the following steps: capturing first-person walking video from the internet; estimating camera ego motion using existing visual odometry models (such as DROID-SLAM or TartanVO); processing video frames using a visual language model to generate natural language descriptions corresponding to the trajectory; filtering out unsuitable segments based on thresholds such as motion speed and VO tracking loss rate; and outputting image-text-action pairs.

[0005] However, the aforementioned existing technologies still have the following problems:

[0006] First, traditional visual odometry suffers from insufficient geometric estimation stability in online videos. Internet videos often involve motion blur, drastic lighting changes, and unknown camera intrinsic parameters, making them prone to failure in low-texture areas or fast-moving scenes. Furthermore, they struggle to output pixel-level depth or pose reliability information, impacting data quality control.

[0007] Second, filtering mechanisms based on single frames or short sequences struggle to identify complex long-term shot logic. Online videos commonly employ shot language such as soft transitions, rapid panning, and montage editing. Existing methods rely solely on the similarity of features between adjacent frames or short-term motion continuity indicators, lacking a holistic understanding of the long-term context and easily preserving visually approximate but physically discontinuous segments.

[0008] Third, the filtering criteria lack the ability to conditionally respond to the target ontology. Existing technologies mostly adopt uniform heuristic filtering rules, which make it difficult to perform differentiated screening based on the kinematic and dynamic constraints of the target ontology. For example, when the Z-axis displacement, pitch angle, or roll angle changes in a video clip exceed the allowable range of a wheeled robot, the clip is not suitable as training data for the wheeled robot, but may meet the constraints of other ontologs with corresponding terrain-crossing capabilities.

[0009] Fourth, it lacks fine-grained spatiotemporal isolation capabilities for dynamic interference. Existing solutions typically treat the entire video frame as a static scene, failing to perform pixel-level separation of dynamic targets such as pedestrians and vehicles. In the first-person view, when a large area of ​​dynamic objects enters, the motion extraction module is prone to misinterpreting external motion as camera motion, generating significant false label noise.

[0010] In summary, existing technologies lack an automated dataset generation method capable of sequentially performing geometric restoration, long-term temporal continuity review, target ontology adaptation, dynamic region identification, and multimodal alignment encapsulation on internet videos. Summary of the Invention

[0011] The technical problem to be solved by this invention is that: first-person videos on the Internet lack geometric information to support navigation training; there are clippings or temporal breaks in the videos; unified filtering rules are difficult to adapt to different robot bodies; dynamic entities can easily interfere with camera motion and scene structure representation; and the data of each modality lacks automatic annotation and encapsulation under a unified time reference.

[0012] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The first aspect of this invention: an automated method for generating cross-ontology visual navigation datasets based on multi-model coupling, comprising: S1. Obtain a first-person video file from the video source as the original long video through an automated agent; preprocess the original long video to generate a set of RGB image sequence slices; S2. Perform joint geometric restoration on the RGB image sequence using a visual geometric basic model, output the depth map, camera pose, camera intrinsic parameters and depth confidence map of each frame, and construct camera trajectory information in chronological order; S3. Input the RGB image sequence slices and the camera trajectory information into a long context multimodal model to generate a structured JSON object containing lens continuity, camera motion, motion distance, scene semantics and dynamic entity information; S4. Based on the input target ontology category, input the structured JSON, the corresponding physical constraint rules, and the key frames sampled from the video slice into the logical reasoning model, perform a qualification judgment on the video slice, and output the judgment result and reason. S5. Parse the dynamic entity information in the structured JSON, use the dynamic entity name as an open vocabulary prompt input to the video segmentation model, generate frame-by-frame masks and fuse them into a dynamic mask sequence; S6. Align RGB images, dynamic masks, dynamic entity tracking labels, depth maps, camera poses, depth confidence maps, and camera intrinsics according to a unified timestamp and slice index, and encapsulate them with target ontology labels and quality review information into training data units to form a cross-ontology visual navigation dataset.

[0013] Specifically, the automated agent in step S1 is implemented as follows: based on the input ontology category, motion mode and scene type, the video platform's search API is called to obtain a candidate video list sorted by relevance. Then, first-person videos are selected based on keywords in the video title, description and tags. The original video file is downloaded and stored as the original long video, provided that the resolution is not less than 1080p, the duration is greater than 30 seconds and the frame rate is not less than 30fps.

[0014] Specifically, the preprocessing in step S1 includes: detecting subtitles and watermarks using optical character recognition, detecting scene boundaries using inter-frame image gradients, and performing resampling and slicing using video processing tools; filtering out segments containing watermarks or black borders exceeding a preset ratio, continuous static duration exceeding a preset duration, solid color transition frames, advertising inserts, or non-navigation scenes.

[0015] Further, step S2 specifically includes: The RGB image sequences are input into the visual geometry fundamental model in batches, and the visual geometry fundamental model jointly outputs the following tensor data based on shared visual features: An absolute depth map sequence is used to characterize the geometric depth of each pixel in each frame relative to the camera optical center; The camera pose sequence is used to characterize the rotation and translation of the camera in a unified coordinate system for each frame, and is composed of the camera trajectory information according to the timestamp; The camera intrinsic parameter matrix sequence is used to characterize the focal length and principal point parameters of the camera for each frame; A sequence of depth confidence maps is used to quantify the confidence level of the predicted depth for each pixel; The low-confidence region consists of pixels whose depth confidence is lower than a preset confidence threshold, which is determined based on the statistical results of the verification samples or the quantiles of the confidence distribution.

[0016] Further, step S3 specifically includes: The RGB image sequence slices and camera trajectory information are input into a long context multimodal model, wherein the camera trajectory information is converted into a pose text sequence with timestamps; the long context multimodal model is constrained by preset system prompts to output a structured JSON object; The structured JSON object includes at least the following fields: The shot description field, shot_description, is used to record shot continuity and editing or transition information; The camera motion field camera_kinematics is used to record changes in the camera's translation, pitch, roll, and yaw. The motion distance field, motion_distance, is used to record the displacement and relative velocity level of the segment obtained based on the visual content and camera trajectory. The scene semantic field scene_semantics is used to record scene type, spatial structure, and lighting conditions; The dynamic_agents field is used to record dynamic entities that move independently in the video.

[0017] Further, step S4 specifically includes:

[0018] The logical reasoning model determines the suitability of candidate video slices as training data for the target ontology based on the physical constraint rules. When the structured JSON object records montage editing, fast jump cuts, or shot breaks, or when the motion mode corresponding to the video slice exceeds the motion constraint range of the target ontology, it is determined to be an unqualified segment. The remaining segments are processed in the subsequent dynamic region.

[0019] The logical reasoning model is a multimodal model with the ability to jointly reason using visual and textual information; The physical constraint rules include at least one or more of the following: allowed translation mode, allowed rotation mode, pitch angle tolerance range, roll angle tolerance range, yaw angle or yaw rate range, Z-axis displacement threshold, speed range, and acceptable viewing angle change mode; the viewing angle change mode includes the change mode of horizontal rotation viewing angle and / or vertical pitch viewing angle.

[0020] Further, step S5 specifically includes: Parse the dynamic entity field dynamic_agents and extract the dynamic entity name as an open vocabulary suggestion; Identify the corresponding dynamic entities in the video start frame or key frame and assign target identifiers; The target identifier is kept consistent across frames through a video memory tracking mechanism, and bounding boxes and pixel-level masks are output frame by frame. The dynamic mask sequence is obtained by performing a pixel-by-pixel logical OR operation on the mask of each dynamic entity.

[0021] A second aspect of the present invention: an automated generation apparatus for cross-ontology visual navigation datasets, the apparatus comprising: The data screening and slicing module obtains first-person video files from the video source as the original long video through an automated agent; it preprocesses the original long video to generate a set of RGB image sequence slices. The physical geometry restoration module uses a visual geometry basic model to perform joint geometric restoration on the RGB image sequence, outputting the depth map, camera pose, camera intrinsic parameters and depth confidence map of each frame, and organizing the camera pose of each frame into camera trajectory information in chronological order; The temporal semantic deconstruction module inputs the RGB image sequence slices and the camera trajectory information into a long context multimodal model to generate a structured JSON object containing lens continuity, camera motion, motion distance, scene semantics, and dynamic entity information. The ontology logic reasoning module, based on the input target ontology category, inputs structured JSON, corresponding physical constraint rules, and key frames sampled from the video slice into the logic reasoning model, performs a qualification judgment on the video slice, and outputs the judgment result and reason; The dynamic mask extraction module parses the dynamic entity information in the structured JSON, uses the dynamic entity names as open vocabulary prompts to input the video segmentation model, generates frame-by-frame masks, and merges them into a dynamic mask sequence. The multimodal alignment and encapsulation module aligns RGB images, dynamic masks, dynamic entity tracking labels, depth maps, camera poses, depth confidence maps, and camera intrinsics according to a unified timestamp and slice index, and encapsulates them with target ontology labels and quality review information into training data units to form a cross-ontology visual navigation dataset.

[0022] A third aspect of the present invention: an electronic device comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors enable the one or more processors to implement the method for automatically generating cross-ontology visual navigation datasets.

[0023] A fourth aspect of the present invention: a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the aforementioned method for automatically generating cross-ontology visual navigation datasets.

[0024] Compared with the prior art, the present invention has the following technical effects: 1) Use the physical constraint rules of the target ontology as input to the logical reasoning model so that the same video slice can obtain corresponding retention or removal results for different ontologs; 2) The visual geometry model jointly outputs depth, camera pose, camera intrinsics, and depth confidence, providing geometric information and credibility basis for data screening and downstream training; 3) Input video slices and camera trajectories into a long-context multimodal model to identify clips, transitions, and temporally discontinuous segments; 4) Generate temporally consistent pixel-level dynamic masks based on the dynamic entity information in the structured report to identify dynamic regions that may affect geometric recovery and motion supervision; 5) Align RGB images, geometric information, dynamic masks, and metadata according to a unified timestamp and slice index to reduce secondary alignment processing before downstream training; 6) By automating the acquisition, slicing, filtering, labeling, and packaging processes, the manual screening of each segment and manual labeling of each frame is reduced. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method for automatically generating cross-ontology visual navigation datasets based on multi-model coupling, as described in this invention. Figure 2 This is a schematic diagram of the main content composition of the prompt words in the ChatGPT large model system of this invention. Detailed Implementation

[0026] The technical solution of the present invention will be described below with reference to the accompanying drawings and specific embodiments. These embodiments are used to explain the present invention and do not constitute a limitation on the scope of protection.

[0027] The cross-ontology visual navigation dataset automated generation device of the present invention includes a data initial screening and slicing module, a physical geometry recovery module, a temporal semantic deconstruction module, an ontology logic reasoning module, a dynamic mask extraction module, and a multimodal alignment and encapsulation module. Each module transmits intermediate results according to video slice identifiers and timestamps.

[0028] Data screening and slicing module: Downloads first-person video files from video sources through an automated agent and saves them as raw long videos; performs preprocessing, resampling and slicing on the raw long videos, and outputs a set of RGB image sequences with uniform frame numbers and slice identifiers; The physical geometry restoration module utilizes a visual foundation model with unified ray-depth representation capabilities (such as DepthAnything V3 (DA3)), taking the original RGB sequence as input and jointly outputting depth, pose, intrinsic parameters, and confidence information to alleviate geometric distortion and scale blur issues; the temporal semantic deconstruction module inputs video slices and timestamped camera trajectory information into a long-context multimodal model, and outputs a structured JSON object containing shot continuity, camera motion, scene semantics, and dynamic entity information; Ontology logic reasoning module: Input structured JSON objects, target ontology physical constraint rules and video keyframes into the logic reasoning model, and output a Boolean judgment result on whether the video slice fits the target ontology and the reason; Dynamic mask extraction module: Parses the dynamic entity names in the structured JSON object and uses them as open vocabulary prompts to input the video segmentation model, and uses the video memory tracking mechanism to generate a temporal dynamic mask with target identifiers; Multimodal alignment and encapsulation module: Aligns RGB images, depth maps, camera poses, camera intrinsics, depth confidence maps, dynamic masks, object ontology labels, and quality review information based on slice identifiers and timestamps, and encapsulates them into training data units.

[0029] like Figure 1 As shown, the original video is acquired and cropped to form a cross-ontology RGB image sequence; the visual geometry model processes the RGB image sequence to output a depth map and camera pose; the long context multimodal model combines the RGB image sequence and camera pose to extract metadata such as scene description, camera changes, and pose changes; the logical reasoning model filters qualified video frames based on image frames and the metadata; the video segmentation model generates dynamic masks and dynamic object tracking results for the qualified video frames. The above results are finally aligned and encapsulated according to a unified timestamp and slice index.

[0030] Step 1: First-person video acquisition and preliminary spatiotemporal slicing processing

[0031] First, the system performs the search through an automated agent. This automated agent downloads first-person or near-first-person video files from internet video sources based on the input, and records the downloaded files as the original long video. Search criteria can include modes of movement such as walking, cycling, pushing a cart, flying, or traversing narrow passages, as well as scene types such as indoor, street, or unstructured terrain. The resolution of the original long video can be no less than 1080p, and the frame rate can be between 30fps and 60fps.

[0032] The system performs preprocessing on the original long video. Specifically, it uses optical character recognition to detect subtitles and watermarks, uses inter-frame image gradients to detect scene boundaries, and performs decoding, resampling, and slicing using FFmpeg. Based on preset area ratios, static durations, and scene category thresholds, it filters out watermarks or black borders that are too high, long static periods, solid color transitions, advertising inserts, or non-navigation scene segments.

[0033] After preprocessing, the system slices the continuous video according to a preset sampling frame rate and the number of slice frames. For example, the video is resampled to 30fps, and each slice is 4 seconds long, so that each slice contains 120 frames of RGB images, generating a set of RGB image sequence slices of uniform length. ; in, This indicates the total number of frames corresponding to the slice length. Through this step, the original long-duration unstructured video is transformed into a standardized set of candidate segments with a uniform time scale, which is convenient for subsequent batch processing modules.

[0034] Step 2: Jointly extract depth, pose, intrinsic parameters, and confidence scores based on the visual geometric model.

[0035] After initial slicing, the system recovers the physical geometric information required for navigation training for each RGB image sequence. This embodiment uses a visual geometry model, such as Depth Anything V3 (DA3) and its accompanying post-processing module, to perform joint geometric recovery on the input sequence so that the depth, pose, intrinsic parameters, and confidence outputs have the same frame index and spatial size reference.

[0036] Specifically, the RGB image sequence generated in step one is input into the visual geometry foundation model in batches. The model jointly outputs the following tensor data based on shared visual features: Absolute depth map sequence Used to characterize the geometric depth relationship of each pixel relative to the camera optical center in each frame; Camera pose sequence It is used to characterize the rotation and translation of the camera in a unified coordinate system for each frame, and to form the camera trajectory information according to the timestamp; Camera intrinsic parameter matrix sequence , used to characterize the focal length and principal point parameters of the camera corresponding to each frame;

[0037] Depth confidence plot This is used to quantify the confidence level of the predicted depth for each pixel, so that subsequent steps can identify low-confidence regions.

[0038] In one embodiment, a preset confidence threshold is set as follows: For the depth confidence map of frame t... , will satisfy The pixel coordinates (u,v) form a low-confidence region; The depth error statistics or confidence distribution quantiles of the validation samples are used to determine the parameters. Frames or slices with a low-confidence region ratio exceeding a preset proportion can be removed, or low-confidence regions can be masked or have their loss weights reduced during training.

[0039] Through this step, the system obtains the depth map, camera pose, camera intrinsic parameters, and depth confidence map corresponding to each frame of the RGB image sequence, and organizes the camera pose into camera trajectory information in chronological order. Each output result is cached using slice identifiers and timestamps as indices for subsequent temporal semantic analysis, ontology adaptation review, and data encapsulation.

[0040] Step 3: Generate structured JSON of video scenes and shots using a long-context multimodal model.

[0041] After obtaining the physical geometry information, the system performs semantic and temporal logic analysis on the video slices to extract shot continuity, camera motion, scene semantics, motion distance, and dynamic entity information. This analysis is performed by a multimodal model with long context processing capabilities, jointly processing the complete video slices and camera trajectories.

[0042] like Figure 2 As shown, the system inputs RGB image sequence slices and camera trajectory information generated in step two into a long-context multimodal model. The camera trajectory information can be converted into a pose text sequence with timestamps. System prompts are mainly used for visual continuity, motion rationality, kinematic features, and scene analysis to identify shot discontinuities, determine whether the motion is smooth and conforms to physical laws, and extract motion modes and scene attributes. Based on this, the model forms a judgment result based on temporal continuity, logical consistency, robot learning applicability, dynamic interference level, and navigation value, and returns a structured JSON object for subsequent ontology adaptation review and dynamic entity parsing.

[0043] The structured JSON object includes at least the following fields: The shot description field, shot_description, is used to record shot continuity as well as editing, montage, fade-in / fade-out, or jump cut information; The camera motion field camera_kinematics is used to record changes in the camera's translation, pitch, roll, and yaw. The motion distance field, motion_distance, is used to record the displacement and relative velocity level of the segment obtained based on the visual content and camera trajectory. The scene semantic field scene_semantics is used to record scene type, spatial structure, and lighting conditions; The dynamic_agents field is used to record dynamic entities that move independently in the video, such as pedestrians, vehicles, or animals.

[0044] Through this step, video slices are converted into structured temporal semantic reports that can be invoked for subsequent logical reasoning and dynamic mask generation.

[0045] Step 4: Perform conditional screening based on the target ontology's physical constraint rules.

[0046] Different target entities have different motion modes, attitude ranges, velocity ranges, and operational domains. Therefore, the same video slice needs to be conditionally reviewed in conjunction with the target entity's physical constraint rules. Using the Qwen3-VL multimodal model or other multimodal models with joint vision-language reasoning capabilities, video clips are first filtered out for non-terrestrial scenes, static or abnormal viewpoints, those centered on local objects, those with obvious editing, and those with poor visual quality. Then, the moving entities from the video source are classified based on features such as camera motion speed, stability, movement path, and road environment; the figure uses cars and pedestrians as examples. The filtering and classification results serve as input for subsequent target entity adaptation review. After completing the above filtering and classification, the system specifies the target entity category, such as a wheeled robot, quadruped robot, or micro-drone, and inputs the structured JSON object generated in step three, the target entity's physical constraint rules, and keyframes uniformly sampled from the video slice into the logical reasoning model. The physical constraint rules can be represented by text or structured parameters, and include at least one or more of the following: allowed translation, allowed rotation, pitch angle tolerance range, roll angle tolerance range, yaw angle or yaw rate range, Z-axis displacement threshold, velocity range, and viewpoint change mode. Yaw represents a change in horizontal turning, and is not used as a uniform exclusion condition, but is determined based on the allowable yaw angle increment or yaw rate range of the target body; viewpoint change modes include horizontal rotation viewpoint and / or vertical pitch viewpoint. The system calculates the Z-axis displacement within adjacent frames or a preset time window based on the camera trajectory. When its absolute value exceeds the preset Z-axis displacement threshold, the corresponding motion is recorded as exceeding the limit; the threshold is determined based on the target body's ground clearance, obstacle clearance height, and time window length. The logical reasoning model outputs the pass / fail judgment result and reason accordingly.

[0047] This step outputs a Boolean judgment result and the corresponding reason. Only video slices that are judged as qualified will proceed to subsequent dynamic region processing. The same video slice can obtain corresponding judgment results when different target ontology physical constraint rules are input.

[0048] Step 5: Perform zero-shot dynamic target tracking and mask generation using an open vocabulary segmentation model.

[0049] For qualified segments that pass the review in step four, the system identifies and labels dynamic entities within them to reduce the impact of objects such as pedestrians and vehicles on geometric reconstruction and motion supervision. This embodiment employs an open-vocabulary video segmentation model, such as SAM3, to perform zero-shot tracking and pixel-level mask generation on dynamic entities in the video.

[0050] The system parses the `dynamic_agents` field from the structured JSON object, extracts the entity names (e.g., person, bicycle, or moving car), and uses these as open-vocabulary prompts to input into the video segmentation model. The model identifies dynamic entities matching the prompts in the video's start frame or keyframes and assigns them target identifiers. It then utilizes a video memory tracking mechanism to maintain consistency of target identifiers throughout the entire slice sequence, outputting bounding boxes and pixel-level masks frame by frame.

[0051] For each video slice, the system performs a pixel-by-pixel logical OR operation on the pixel-level mask of each dynamic entity to obtain a unified dynamic mask sequence: ; The system simultaneously retains the target identifiers and tracking labels of dynamic entities as training aids or metadata. This step identifies dynamic regions without requiring manual frame-by-frame annotation, thereby reducing the interference of dynamic entities on geometric reconstruction and motion supervision.

[0052] Step Six: Multi-source Feature Fusion and Standard Encapsulation of End-to-End Navigation Dataset

[0053] After completing the aforementioned geometric restoration, semantic analysis, logical filtering, and dynamic region identification, the system aligns the outputs of each modality at the same timestamp. The data elements for each frame include an RGB image, dynamic mask, dynamic entity tracking labels, depth map, camera pose, depth confidence map, and camera intrinsics; video slice-level metadata includes scene semantics, target ontology labels, quality assessment results, and a list of dynamic entities.

[0054] After completing multimodal alignment, the system serializes and encapsulates the above information into training data units. These data units can be used for imitation learning, reinforcement learning, visual language action modeling, or multi-task pre-training of cross-ontology end-to-end navigation models.

[0055] Example 1: Multi-model Cascaded System Architecture and Computing Hardware Deployment

[0056] This embodiment provides a multi-model cascaded system architecture. Each module is associated with data via slice IDs and timestamps. The data interface between modules uses a combination of "slice-level JSON metadata + tensor file path or memory reference," where the slice ID serves as the primary key throughout all processing stages, ensuring the traceability and alignment of the output results from each module. Each module can perform serial, parallel, or asynchronous scheduling based on computing resource availability. The system can be deployed on a computing cluster with parallel computing capabilities. In one implementation, a single node can be configured with multiple high-memory GPUs, large-capacity memory, and high-speed storage to meet the inference requirements of the visual geometric reconstruction model and the video-level segmentation model in high-resolution scenes.

[0057] The system comprises the following six processing levels according to the data flow sequence: Layer 1: Data Initial Screening and Slicing Module. This module is responsible for video acquisition, resampling, spatiotemporally standardized slicing, and basic quality screening. Specifically, it uses video processing tools to uniformly resample the video and filters out unqualified segments based on preset conditions such as watermark area ratio, continuous still duration, and scene category. This layer outputs a slice record (SliceRecord), which includes at least the slice number, original video number, start and end times, frame rate, number of frames, image size, frame sequence storage address, and quality marker. The frame sequence storage address points to color image frames or video segments saved in chronological order, and the quality marker records the screening results such as watermark, still segments, and scene category. The second layer is the physical geometry reconstruction module. This module receives the slice record and color image sequence output from the first layer, uses the visual geometry model to reconstruct the depth information, camera motion trajectory, and camera intrinsic parameters of the video clips, and outputs a geometry record. This record contains at least the slice number, timestamp sequence, depth data storage address or reference, camera pose sequence, camera intrinsic parameter sequence, confidence data, and geometry reconstruction status. Specifically, the camera pose sequence records the relative pose changes of the camera in a unified reference coordinate system according to the timestamps, and the camera intrinsic parameter sequence records the camera parameters corresponding to each frame. The third layer: Temporal semantic deconstruction module. This module receives the color image sequence and camera trajectory information from the geometric record output by the second layer, organizing them as input to a long-context multimodal model. The timestamped pose sequence is described in the form of "timestamp, translation, rotation, motion velocity, yaw angle, pitch angle, and roll angle," and is input to the model along with the RGB image sequence. This module outputs a semantic report, which includes at least a shot description, camera motion features, motion distance, scene semantics, dynamic targets, temporal continuity score, and removal markers, used to characterize the temporal continuity and scene semantic information of the video slices. The fourth layer: Ontology logic reasoning module. This module maintains the target robot ontology constraint parameter library (RobotProfile), which describes the target robot's translation and rotation methods, speed range, pitch and roll angle tolerance range, yaw angle or yaw rate range, vertical displacement threshold, ground clearance, and applicable operating environment. This module receives the semantic review report, ontology constraint parameters, and keyframes output from the third layer, determines whether the video clip conforms to the motion characteristics and physical constraints of the target ontology, and outputs the ontology adaptation result (ReviewResult). This result includes at least the slice number, target ontology identifier, whether it passed the judgment, the judgment basis, risk level, and constraint evidence. The judgment basis explains the specific reasons for retaining or removing the clip for subsequent data filtering and quality traceability. Layer 5: Dynamic Mask Extraction Module. For video segments that have passed the pre-examination review, this module calls an open-vocabulary video segmentation model to generate temporally consistent pixel-level masks for dynamic interference targets such as pedestrians, vehicles, and animals. This module receives the slice numbers that have passed the review, the color image sequence, and the ontology adaptation results, and outputs a dynamic mask record (MaskRecord). This record contains at least the mask data storage address or reference, target tracking number, target category, timestamp sequence, and mask generation status, used to describe the position, category, and cross-frame tracking relationship of the dynamic interference target in each frame; Layer 6: Multimodal Alignment and Encapsulation Module. This module receives the data output from the previous layers, aligns the RGB images, dynamic masks, dynamic entity tracking labels, depth maps, camera poses, depth confidence maps, and camera intrinsics according to a unified timestamp and slice index, and encapsulates the aligned multimodal data together with the target ontology labels and quality review information into training data units.

[0058] Through the above six processing layers, the first to fourth layers complete the screening and semantic understanding of video data, while the fifth and sixth layers complete the dynamic region recognition and multimodal data encapsulation, realizing the automatic conversion from Internet video to cross-ontology visual navigation training data units.

[0059] Example 2: End-to-end data processing flow and multimodal tensor transfer

[0060] The end-to-end data flow process of this invention will be described below using a specific scenario.

[0061] This embodiment acquires and saves a 12-second, 1920×1080 resolution, 30fps first-person perspective video of a campus walk from an internet video source, using this video as the original long video. The system slices the original long video into 4-second time windows, with each slice containing 120 frames of RGB images. Taking the first segment as an example, it is represented as follows: ; in, Indicates time step The corresponding RGB image frame.

[0062] During the slicing stage, frame difference detection or adjacent frame abrupt change analysis can be run in parallel to identify potential hard transitions, frame interpolation anomalies, sudden changes in the picture, and generate corresponding metadata tags for subsequent semantic review.

[0063] Subsequently, the sequence is input into the Depth Anything V3 (DA3) model or its associated geometric reconstruction process; in one implementation, the model first outputs a downsampled depth feature map or a low-resolution depth estimate, then upsamples it to the input image resolution through interpolation, convolutional decoding, or other post-processing methods, and aligns it with the input RGB frames by timestamp to obtain the following types of tensors: (1) Absolute depth map sequence ; Where D represents the depth map sequence after post-processing upsampling and alignment with the input frame; if the spatial size of the depth feature map generated internally by the model is h×w, it is mapped to H×W=1080×1920 through post-processing such as interpolation or convolutional decoding. The above 120×1080×1920 represents the three-dimensional tensor dimension of 120 frames, with a height of 1080 and a width of 1920.

[0064] Used to describe the physical distance corresponding to each pixel in each frame.

[0065] (2) Camera pose sequence ; Used to represent the 6-DoF pose change of the camera in a unified reference coordinate system for each frame.

[0066] (3) Dynamic intrinsic parameter matrix sequence ; It is designed to address the common issues of unknown focal length and zoom in internet videos, and to support subsequent geometric backprojection and multi-view alignment.

[0067] (4) Depth confidence map sequence ; Each pixel value represents the reliability of the depth estimate at that location. Areas such as the sky, glass, water surfaces, and motion-blurred edges typically have lower confidence levels.

[0068] After completing the geometric tensor extraction, the system derives the relative displacement and rotation changes of adjacent frames based on the camera pose sequence, organizes them into camera trajectory information according to timestamps, and converts them into auxiliary text prompts; the auxiliary text prompts and the current RGB image sequence slices are input into a long contextual multimodal model for temporal semantic review.

[0069] After passing temporal semantic review and ontology adaptation adjudication, the video segmentation model further generates dynamic entity masks: ; Here, Mdyn represents a binary dynamic mask sequence aligned frame-by-frame with the RGB input frames; if the video segmentation model first generates low-resolution mask logits or a probability map, then the tensor is formed by upsampling to the input resolution through post-processing. In the tensor dimensions above, 120 represents the number of frames, and 1080 and 1920 represent the height and width of the input image, respectively.

[0070] Finally, the system aligns information such as RGB, depth, pose, intrinsic parameters, confidence map, dynamic mask, and target body label according to a unified timestamp and encapsulates them into training data units.

[0071] Example 3: Temporal Continuity Review Mechanism Based on Long Context Multimodal Model

[0072] Internet videos naturally contain traces of post-processing, such as editing, jump cuts, dissolves, and fade-in / fade-out effects. If used directly for navigation training without review, the model will learn state transitions that do not conform to the laws of real physics, such as entering a bedroom directly from the living room without passing through a door frame, or "teleporting" between corridors with very similar scenes.

[0073] Traditional filtering methods typically rely on optical flow continuity or SSIM similarity within a short time window, but these methods often fail when faced with soft transitions, jump cuts between similar scenes, and cinematic narrative structures.

[0074] To perform long-context cross-modal review, the system first processes the camera pose sequence, extracting the relative displacement and angle changes between adjacent frames, and generating auxiliary text descriptions, such as recording the forward distance and pitch, roll, and yaw angle changes in each time interval. Subsequently, the auxiliary text, along with slices of the current RGB image sequence, is input into the long-context multimodal model.

[0075] By using a predefined prompt template, the model is forced to output structured JSON instead of free text. The JSON may contain at least the following fields: 1) Shot description: Whether the shot is continuous, and whether there is any editing or transition; 2) camera_kinematics: Camera motion modes; 3) motion_distance: Whether the displacement logic is interpretable; 4) scene_semantics: Changes in scene semantics; 5) dynamic_agents: A list of dynamic entities.

[0076] For example, in a house tour video, the first two seconds show the character walking from the living room to the bedroom door, and the next two seconds show the character directly inside the bedroom. Traditional visual methods might simply interpret this as a normal movement after a slight interruption, but the long context model in this invention can determine, based on global semantic understanding, that the video lacks the crucial physical transition of "opening the door—crossing the door frame—entering the bedroom," thus indicating a break in spatial connectivity. The system can then reject this segment from the subsequent dataset.

[0077] This mechanism uses global semantic changes in video slices and camera trajectory changes to jointly determine physical continuity, in order to filter out temporal breaks caused by editing or transitions.

[0078] Example 4: Conditional Logic Decision-Making for Different Robot Bodies

[0079] This embodiment performs conditional adjudication based on the physical constraints of the target ontology to determine whether a video slice is suitable as a training sample for a specified target ontology.

[0080] In one implementation, the system constructs a target ontology description file or parameter configuration pool, the contents of which may include: 1) Permissible linear velocity range; 2) Permissible translation methods, rotation methods, turning radii, and angular velocity ranges; 3) Permissible pitch angle range, roll angle range, and yaw angle increment or yaw rate range; 4) Z-axis displacement threshold, ground clearance, and obstacle clearance height; 5) Acceptable horizontal rotation and vertical pitch angle changes, as well as applicable terrain types and operating range.

[0081] Subsequently, the system inputs the target ontology configuration, the structured JSON object generated in the preceding steps, and the video keyframes into the logical reasoning model, which then performs a qualification judgment by combining scene semantics, camera motion, and target ontology constraints.

[0082] ①Example A (Target body is an indoor wheeled robot): For wheeled robots that mainly operate on flat indoor floors, the linear velocity range, yaw rate range, pitch angle and roll angle tolerance range can be set according to the platform's rated speed, turning ability and camera mounting attitude, and the Z-axis displacement threshold can be set according to the ground clearance and the height of obstacles that can be passed.

[0083] After reading the structured JSON, if the camera motion field indicates that the Z-axis displacement in adjacent frames or a preset time window exceeds the Z-axis displacement threshold, and the terrain type recorded in the scene semantic field exceeds the operating domain of the wheeled robot, then an unqualified conclusion will be output.

[0084] Judgment command output: The model may output, for example: {"is_qualified": false, "reason": "Motion constraints and operating domain do not match"}.

[0085] This means that the video slice will not be included in the training data stream for indoor wheeled robots.

[0086] ②Example B (The target body is a quadruped robot): For quadruped robots, the corresponding speed range, attitude change range, Z-axis displacement threshold, and terrain types that can be traversed can be set according to the platform's rated parameters.

[0087] For the same video slice, if the changes in Z-axis displacement, pitch angle, roll angle, and yaw angle in the camera motion field are all within the allowable range of the quadruped robot configuration, and the terrain type recorded in the scene semantic field belongs to its operating domain, then the logical reasoning model outputs a qualified conclusion.

[0088] Judgment command output: The model can output, for example: {"is_qualified": true, "reason": "Motion characteristics match the quadruped robot constraints"}.

[0089] Therefore, this segment can be included in a subset of data for quadrupedal navigation tasks.

[0090] Example 5: Dynamic Entity Isolation and Data Encapsulation Based on Video-Level Memory Tracking

[0091] Video slices that have undergone geometric review and ontological adjudication may still contain dynamic entities such as pets, vehicles, or crowds moving across the frame. The independent motion of these dynamic entities may affect camera motion estimation, scene structure reconstruction, and motion supervision extraction.

[0092] To this end, the system extracts the dynamic entity field `dynamic_agents` from the preceding structured JSON object, converts the dynamic entity names into open-vocabulary cues, and inputs them into the video segmentation model. The video segmentation model first locates the corresponding targets in keyframes and assigns target identifiers, then utilizes video memory and identity consistency tracking mechanisms to output a temporally consistent mask tensor throughout the sequence. ; In subsequent training, the dynamic mask can be processed in one or more of the following ways: 1) Directly mask the corresponding pixels; 2) Set the characteristic response of this region to zero; 3) Block the gradient in this region during backpropagation; 4) Further adjust the local loss weights in conjunction with the confidence plot.

[0093] In the final data encapsulation stage, the system strictly aligns and packages the following information according to a unified timestamp: 1) RGB image sequence; 2) Depth map sequence; 3) Pose matrix sequence; 4) Camera intrinsic parameter sequence; 5) Depth confidence map sequence; 6) Dynamic entity mask sequence; 7) Corresponding target robot body tag; 8) Necessary quality review and metadata fields.

[0094] The data encapsulation format can adopt HDF5, WebDataset, or other formats suitable for large-scale parallel training to meet the high-throughput loading requirements of offline reinforcement learning, supervised imitation learning, and VLA large model training scenarios.

[0095] After the above processing, the internet video slices are converted into visual navigation training samples containing RGB images, geometric tensors, dynamic masks, target ontology labels, and quality review information.

[0096] Through the above steps, the system generates multimodal training data with unified slice indexes and timestamps from first-person videos on the Internet, and forms corresponding cross-ontology visual navigation datasets based on the physical constraint rules of the target ontology.

[0097] To visually demonstrate the advantages of this invention over other traditional technologies, Table 1 lists comparative data for the core dimensions:

[0098] Table 1 .

[0099] In summary, compared to existing data construction methods that rely on long-term remote acquisition by real robots, manual screening, and manual annotation, this invention directly uses massive amounts of unstructured video data from the internet as the raw data source. Through the collaborative efforts of multiple models, it completes geometric reconstruction, temporal review, ontology filtering, dynamic cleaning, and standard encapsulation, significantly reducing the degree of human intervention. The system does not require redesigning numerous rules or performing frame-by-frame annotation for every new scene, ontology, or video source, exhibiting a high level of automation and scalability.

[0100] With the continuous growth of available video sources and the continuous enhancement of the capabilities of the basic model, this invention can also continuously expand the data scale, update the scene coverage, and adapt to new ontology categories, thereby forming a sustainable data production infrastructure. This capability is of great significance for training next-generation embodied intelligence basic models and cross-ontology navigation large models.

[0101] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0102] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0103] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the above-described method for automatically generating cross-ontology visual navigation datasets based on multi-model coupling.

[0104] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the aforementioned method for automatically generating cross-ontology visual navigation datasets based on multi-model coupling. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart memory card, SD card, flash memory card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0105] Furthermore, the specific models mentioned in the embodiments (such as Depth Anything V3 (DA3), Gemini 2.5 Pro, Qwen3-VL, SAM3, etc.) are merely examples. The logical reasoning model can be Qwen3-VL, or other multimodal models with joint visual-language reasoning capabilities, such as GPT-4V, LLaVA, etc.; those skilled in the art can choose according to actual needs. Other existing models with the same functionality or their upgraded versions, as long as they follow the data processing flow and cascading logic of this invention, can be considered to fall within the protection scope of this invention. The protection focus of this invention is on the combination relationship of steps such as spatiotemporal slicing, geometric recovery, conditional logic adjudication, dynamic mask generation, and multimodal standard encapsulation. Equivalent workflow replacements or variations in step combinations used to solve the above technical problems should also be included within the protection scope.

[0106] This invention is not limited to the specific ontology such as robots, animals, or vehicles exemplified in the text. Any other ontology scenario that can be formally described in terms of its kinematics, dynamics, or operational domain constraints, and that adopts the technical approach of "target ontology constraint pre-injection + data condition adjudication," falls within the scope of protection of this invention.

Claims

1. A method for automatically generating cross-ontology visual navigation datasets based on multi-model coupling, characterized in that, include: S1. Obtain first-person video files from the video source as the original long video through an automated agent; The original long video is preprocessed to generate a set of RGB image sequence slices; S2. Perform joint geometric reconstruction on the RGB image sequence using a visual geometric model, outputting the depth map, camera pose, camera intrinsic parameters, and depth confidence map for each frame, and constructing camera trajectory information in chronological order; specifically including: The RGB image sequences are input into the visual geometry fundamental model in batches, and the visual geometry fundamental model jointly outputs the following tensor data based on shared visual features: An absolute depth map sequence is used to characterize the geometric depth of each pixel in each frame relative to the camera optical center; The camera pose sequence is used to characterize the rotation and translation of the camera in a unified coordinate system for each frame, and is composed of the camera trajectory information according to the timestamp; The camera intrinsic parameter matrix sequence is used to characterize the focal length and principal point parameters of the camera for each frame; A sequence of depth confidence maps is used to quantify the confidence level of the predicted depth for each pixel; The low-confidence region consists of pixels whose depth confidence is lower than a preset confidence threshold, which is determined based on the statistical results of the verification samples or the quantiles of the confidence distribution. S3. Input the RGB image sequence slices and the camera trajectory information into a long context multimodal model to generate a structured JSON object containing lens continuity, camera motion, motion distance, scene semantics and dynamic entity information; S4. Based on the input target ontology category, input the structured JSON, corresponding physical constraint rules, and sampled keyframes from the video slice into the logical reasoning model to determine the pass / fail status of the video slice and output the determination result and reasoning; specifically including: The logical reasoning model determines the suitability of candidate video slices as training data for the target ontology based on the physical constraint rules. When the structured JSON object records montage editing, fast jump cuts, or shot breaks, or when the motion mode corresponding to the video slice exceeds the motion constraint range of the target ontology, it is determined to be an unqualified segment. The remaining segments are processed in the subsequent dynamic region. The logical reasoning model is a multimodal model with the ability to jointly reason using visual and textual information; The physical constraint rules include at least one or more of the following: allowed translation methods, allowed rotation methods, pitch angle tolerance range, roll angle tolerance range, yaw angle or yaw rate range, Z-axis displacement threshold, speed range, and acceptable viewing angle change methods; the viewing angle change methods include changes in horizontal rotation viewing angle and / or vertical pitch viewing angle. S5. Parse the dynamic entity information in the structured JSON, use the dynamic entity name as an open vocabulary prompt input to the video segmentation model, generate frame-by-frame masks and fuse them into a dynamic mask sequence; S6. Align RGB images, dynamic masks, dynamic entity tracking labels, depth maps, camera poses, depth confidence maps, and camera intrinsics according to a unified timestamp and slice index, and encapsulate them with target ontology labels and quality review information into training data units to form a cross-ontology visual navigation dataset.

2. The method for automatically generating cross-ontology visual navigation datasets according to claim 1, characterized in that, The automated agent in step S1 is implemented as follows: based on the input ontology category, motion mode and scene type, the video platform's search API is called to obtain a candidate video list sorted by relevance. Then, first-person videos are selected based on keywords in the video title, description and tags. The original video file is downloaded and stored as the original long video, provided that the resolution is not less than 1080p, the duration is more than 30 seconds and the frame rate is not less than 30fps.

3. The method for automatically generating cross-ontology visual navigation datasets according to claim 1, characterized in that, The preprocessing in step S1 includes: detecting subtitles and watermarks using optical character recognition, detecting scene boundaries using inter-frame image gradients, and performing resampling and slicing using video processing tools; filtering out segments containing watermarks or black borders exceeding a preset ratio, continuous static duration exceeding a preset duration, solid color transition frames, advertising inserts, or non-navigation scenes.

4. The method for automatically generating cross-ontology visual navigation datasets according to claim 1, characterized in that, Step S3 specifically involves: The RGB image sequence slices and camera trajectory information are input into a long context multimodal model, wherein the camera trajectory information is converted into a pose text sequence with timestamps; the long context multimodal model is constrained by preset system prompts to output a structured JSON object; The structured JSON object includes at least the following fields: The shot description field, shot_description, is used to record shot continuity and editing or transition information; The camera motion field camera_kinematics is used to record changes in the camera's translation, pitch, roll, and yaw. The motion distance field, motion_distance, is used to record the displacement and relative velocity level of the segment obtained based on the visual content and camera trajectory. The scene semantic field scene_semantics is used to record scene type, spatial structure, and lighting conditions; The dynamic_agents field is used to record dynamic entities that move independently in the video.

5. The method for automatically generating cross-ontology visual navigation datasets according to claim 4, characterized in that, Step S5 specifically involves: Parse the dynamic entity field dynamic_agents and extract the dynamic entity name as an open vocabulary suggestion; Identify the corresponding dynamic entities in the video start frame or key frame and assign target identifiers; The target identifier is kept consistent across frames through a video memory tracking mechanism, and bounding boxes and pixel-level masks are output frame by frame. The dynamic mask sequence is obtained by performing a pixel-by-pixel logical OR operation on the mask of each dynamic entity.

6. An automated generation device for cross-ontology visual navigation datasets, characterized in that, The device includes: The data screening and slicing module obtains first-person video files from the video source as the original long video through an automated agent; it preprocesses the original long video to generate a set of RGB image sequence slices. The physical geometry restoration module uses a visual geometry foundation model to perform joint geometric restoration on the RGB image sequence, outputting depth maps, camera poses, camera intrinsic parameters, and depth confidence maps for each frame. The camera poses of each frame are organized into camera trajectory information in chronological order. Specifically, this includes: inputting the RGB image sequence into the visual geometry foundation model in batches, and the visual geometry foundation model jointly outputting the following tensor data based on shared visual features: An absolute depth map sequence is used to characterize the geometric depth of each pixel in each frame relative to the camera optical center; The camera pose sequence is used to characterize the rotation and translation of the camera in a unified coordinate system for each frame, and is composed of the camera trajectory information according to the timestamp; The camera intrinsic parameter matrix sequence is used to characterize the focal length and principal point parameters of the camera for each frame; A sequence of depth confidence maps is used to quantify the confidence level of the predicted depth for each pixel; The low-confidence region consists of pixels whose depth confidence is lower than a preset confidence threshold, which is determined based on the statistical results of the verification samples or the quantiles of the confidence distribution. The temporal semantic deconstruction module inputs the RGB image sequence slices and the camera trajectory information into a long context multimodal dataset. The model generates a structured JSON object containing information on shot continuity, camera motion, motion distance, scene semantics, and dynamic entities. The ontology logic reasoning module, based on the input target ontology category, inputs structured JSON, corresponding physical constraint rules, and keyframes sampled from video slices into the logic reasoning model. It then performs a qualification assessment on the video slices and outputs the assessment results and reasons. Specifically, this includes: The logical reasoning model determines the suitability of candidate video slices as training data for the target ontology based on the physical constraint rules. When the structured JSON object records montage editing, fast jump cuts, or shot breaks, or when the motion mode corresponding to the video slice exceeds the motion constraint range of the target ontology, it is determined to be an unqualified segment. The remaining segments are processed in the subsequent dynamic region. The logical reasoning model is a multimodal model with the ability to jointly reason using visual and textual information; The physical constraint rules include at least one or more of the following: allowed translation methods, allowed rotation methods, pitch angle tolerance range, roll angle tolerance range, yaw angle or yaw rate range, Z-axis displacement threshold, speed range, and acceptable viewing angle change methods; the viewing angle change methods include changes in horizontal rotation viewing angle and / or vertical pitch viewing angle. The dynamic mask extraction module parses the dynamic entity information in the structured JSON, uses the dynamic entity names as open vocabulary prompts to input the video segmentation model, generates frame-by-frame masks, and merges them into a dynamic mask sequence. The multimodal alignment and encapsulation module aligns RGB images, dynamic masks, dynamic entity tracking labels, depth maps, camera poses, depth confidence maps, and camera intrinsics according to a unified timestamp and slice index, and encapsulates them with target ontology labels and quality review information into training data units to form a cross-ontology visual navigation dataset.

7. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the cross-ontology visual navigation dataset automatic generation method according to any one of claims 1-5.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, they implement the cross-ontology visual navigation dataset automatic generation method according to any one of claims 1-5.

Images