Multi-frame image space-time perception enhancement method, system and device

Abstract

The invention discloses a multi-frame image space-time perception enhancement method, system and device. The method solves the problem of poor consistency of spatial orientation and time sequence reasoning when a multi-modal large language model processes multiple frames of images. The method comprises the following steps: acquiring an image sequence and a text instruction; estimating camera poses of the image sequence to form a pose sequence; constructing a unified reference coordinate system and calculating a relative motion abstract amount; generating a pose track prompt graph based on the pose sequence and the abstract quantity, wherein the pose track prompt graph comprises a track top view panel for displaying a horizontal projection track and an observation direction of a camera center point, and an intra-frame orientation HUD panel which is superposed on a key frame and displays a course angle and a reference direction; jointly inputting the prompt graph and the text instruction into a multi-modal large language model; and receiving and outputting an answer or action instruction generated by the model. According to the method, the cross-frame space-time perception and consistency reasoning capability of the multi-modal large language model is enhanced.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, artificial intelligence and multimodal understanding, and specifically to a method, system and electronic device for enhancing spatial and temporal consistency reasoning of multimodal large language models (MLLM) in multi-frame image or video input scenarios. Background Technology

[0002] Multimodal Large Language Models (MLLMs) have demonstrated powerful capabilities in tasks such as single-image question answering and document understanding. However, when the input becomes a collection of multi-frame video or multi-view image sets, the models perform poorly in terms of spatially and temporally consistent reasoning. The main problems include: 1. Unstable orientation reference frame: When the camera viewpoint moves or turns, the model is prone to confusion regarding relative spatial relationships such as "left / right," "front / back," and "near / far away," especially in problems requiring viewpoint shifts or perspective-taking, where the error rate is high. 2. Difficulty in maintaining consistency across frame relationships: For temporal reasoning problems involving the order of events, location change trajectories, whether a region has been traversed, or whether the same location has been returned to, the model is prone to contradictory or incorrect interpretations. 3. Conflict between processing cost and input information: Increasing the number of input frames helps provide more comprehensive contextual information, but it also significantly increases the number of image tokens the model needs to process, leading to increased computational overhead and potentially exceeding the model's processing length limit.

[0003] To improve the multi-frame inference capability of MLLM, existing technologies typically employ the following approaches: introducing additional 3D sensors (such as depth cameras and LiDAR) to acquire accurate geometric information; specifically training the MLLM for multi-frame tasks or modifying the model architecture; or converting multi-frame information into text descriptions before processing them by a large, pure text model. These methods each suffer from problems such as high deployment costs, large training data requirements, high technical complexity, and error accumulation due to multi-level processing.

[0004] Therefore, there is an urgent need in this field for a visual cueing technique that can effectively enhance the spatial-temporal perception and understanding of multi-frame inputs of multimodal models without changing the general MLLM structure and without relying on task-specific training as much as possible. Summary of the Invention

[0005] The present invention aims to address the deficiencies in the prior art. The main technical problem to be solved is: how to effectively improve the accuracy and reliability of the multimodal large language model in making consistent inferences on the spatial relationships (such as left and right, front and back, orientation) and temporal order (such as sequence, path) of multi-frame image or video inputs without modifying the structure of the multimodal large language model itself and without relying on a large amount of additional training.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A spatial-temporal awareness enhancement method for multi-frame images includes the following steps: S1. Obtaining an image sequence (I1, I2, ..., I... n ) and text instructions ( The image sequence is a collection of multiple frames or multiple viewpoints from the same scene; S2. Based on the image sequence, estimate the camera pose of each frame ( ), constituting a camera pose sequence, the camera pose ( ) includes rotation matrix ( Translation vector S3. Based on the camera pose sequence, construct a unified reference coordinate system, and calculate the relative motion summary of each frame based on the reference coordinate system. The relative motion summary includes the heading angle ( ) and displacement strength ( S4. Based on the camera pose sequence and the relative motion summary, generate a pose trajectory cue map (S), the pose trajectory cue map (S) including: a trajectory top view panel (Straj), used to visualize the projection trajectory of the camera center point on the horizontal plane and the viewing direction of each frame image; and an intra-frame orientation head-up display system (SHUD) panel, used to overlay graphic symbols on at least one keyframe image of the image sequence, the graphic symbols including at least the heading angle representing the current frame ( S5. The compass pointer and fixed mark representing the reference direction (Ref); S6. The pose trajectory prompt diagram (S) and the text command ( S6. Receive and output the input from the MLLM based on the pose trajectory cue map (S) and the text instructions (S) The generated answer or action instruction (A).

[0007] In some embodiments, at least one of the following technical means is also included: Step S2 includes: processing the image sequence using a Visual Simultaneous Localization and Mapping (SLAM) algorithm or a Structure for Motion Reconstruction (SfM) algorithm to estimate the camera pose. ).

[0008] In step S2, sensing data from an inertial measurement unit (IMU) is also fused, the IMU data being used to assist in stable attitude estimation or to provide scale information.

[0009] In step S3, a unified reference coordinate system is constructed according to the following priority rules: if the Manhattan world axis is detected, the Manhattan world axis is used as the horizontal reference; otherwise, if the gravity direction is available, the gravity alignment coordinate system of the first frame image is used as the world coordinate system, and the projection direction of the camera forward vector of the first frame onto the horizontal plane is defined as the reference direction (Ref); otherwise, the camera coordinate system of the first frame image is used as the world coordinate system, and the camera forward vector of the first frame is defined as the reference direction (Ref).

[0010] Step S4 generates the trajectory top view panel (Straj), including: selecting the camera center point from the camera pose sequence ( Projecting the data onto a horizontal plane yields two-dimensional trajectory points; adaptively scaling and translating these two-dimensional trajectory points to map them onto a canvas of a predetermined resolution; connecting the mapped points to draw a trajectory line; and drawing graphic elements representing the viewing direction of each frame of the image at specific locations on the trajectory line.

[0011] The graphic element is an arrow or a view cone, and the length of the arrow or the opening angle of the view cone is related to the displacement intensity. This is related to either the camera's intrinsic parameters or the camera's internal parameters.

[0012] Step S4 generates the intra-frame orientation HUD panel (SHUD), which includes: superimposing the HUD panel on the corner area of ​​the keyframe image; the display area of ​​the HUD panel does not exceed 10% of the total area of ​​the keyframe image in which it is located; and the graphic symbols displayed in the HUD panel include at least one of the following: a steering symbol representing a change in heading relative to the previous frame; a displacement symbol representing the displacement intensity relative to the previous frame; and fixedly displayed global left (L) and right (R) markers representing the reference direction (Ref).

[0013] The heading angle ( The compass pointer is discretized into multiple sectors, with the pointer pointing towards the center of its sector; and / or the magnitudes of the steering and displacement symbols are discretized into a small number of increments for display.

[0014] The pose trajectory hint map (S) generated in step S4 also includes a legend, which is used to interpret the semantics of the graphic symbols in the trajectory top view panel (Straj) and the intra-frame orientation HUD panel (SHUD).

[0015] In step S2, the tracking status flag for each frame of image is also output. ); when the tracking status flag ( When tracking is lost, step S4 performs at least one of the following processes: draws the predicted trajectory segment using dashed lines in the trajectory top view panel (Straj); suppresses the display of displacement symbols in the intra-frame orientation HUD panel (SHUD); and stops updating the pose trajectory hint map (S) if the tracking loss continues to exceed a preset threshold.

[0016] The present invention also adopts the following technical solutions: An electronic device includes: a processor; a memory for storing a computer program; wherein, when the computer program is executed by the processor, it implements the method as described in any one of the above.

[0017] The present invention also adopts the following technical solutions: A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the method as described in any one of the foregoing.

[0018] The present invention also adopts the following technical solutions: A multi-frame space-time awareness enhancement system based on camera pose trajectory cue maps includes: an input module for performing step S1 to acquire an image sequence (I1, I2, ..., I...). n ) and text instructions ( ); camera pose estimation module, used to estimate the camera pose of each frame of the image based on the image sequence, based on step S2. The system comprises: a camera pose sequence; a reference system construction and summary calculation module, used to execute step S3, to construct a unified reference coordinate system based on the camera pose sequence and calculate the relative motion summary of each frame; and a pose trajectory cue map generation module, used to execute step S4, to generate a pose trajectory cue map (S) based on the camera pose sequence and the relative motion summary, wherein the cue map includes a trajectory top view panel (S). traj ) and intra-frame orientation HUD panel (S HUD The image-text joint reasoning module is used to execute steps S5 and S6 to combine the pose trajectory prompt image (S) with the text instruction (S). The inputs are shared to the multimodal large language model (MLLM) and the output answers or action instructions (A) are obtained.

[0019] Furthermore, The camera pose estimation module is implemented using visual SLAM or SfM algorithms, and optionally integrates an IMU data fusion unit.

[0020] Furthermore, The pose trajectory prompt image generation module also includes a tracking state processing unit, which is used to adaptively modify the generated prompt image when camera pose tracking is lost.

[0021] The present invention also adopts the following technical solutions: A robot includes: a camera for capturing a sequence of images of an environment; a system as described in any one of the preceding claims; and an actuator; wherein motion commands (A) output by the system are used to control the movement of the actuator.

[0022] The present invention also adopts the following technical solutions: An autonomous vehicle includes: an environmental perception sensor for acquiring image sequences of the vehicle's surrounding environment; a system as described in any one of the above; and a vehicle control system; wherein the action command (A) output by the system is used to assist or control the driving behavior of the vehicle.

[0023] Compared with existing technologies, the multi-frame space-time awareness enhancement method, system, and device based on camera pose trajectory cue maps provided by this invention have the following beneficial effects: 1. Significantly Improved Spatial-Temporal Reasoning Consistency: The core benefit of this invention lies in providing a unified, stable, and visualized spatial-temporal reference framework for multimodal large language models by generating and utilizing pose trajectory cue maps. These cue maps explicitly encode camera motion geometry information (including motion trajectories and orientation in each frame) that is originally implicit in multiple frames of images and difficult for the model to extract directly and stably, into easily understandable visual symbols (such as trajectory lines, compass pointers, and turning symbols). This allows MLLM to align and interpret these symbolic geometric constraints with the semantic content (such as objects and scenes) in the images within a unified coordinate system during reasoning. This effectively reduces confusion in left / right, front / back, and other orientation judgments caused by changes in viewpoint, as well as inconsistencies in the interpretation of event sequence and motion paths across frames. Ultimately, this significantly improves the accuracy and reliability of the model's consistent reasoning in both spatial and temporal dimensions.

[0024] 2. Low invasiveness and high compatibility: As a preprocessing enhancement method, this invention does not require changing the internal structure or parameters of existing general MLLMs, nor does it require targeted model retraining. It has good versatility and compatibility and is easy to integrate into existing multimodal application pipelines.

[0025] 3. Efficient information compression and low occlusion: By compressing the motion information of multiple frames into a compact cue map (containing a global trajectory overview and a local intra-frame HUD) and presenting it in a low-occlusion manner (small HUD area, located in a corner), key spatial-temporal information is effectively conveyed while reducing occlusion of the main content of the image and controlling the number of image tokens input to the MLLM, which helps to balance performance and computational overhead.

[0026] 4. Robustness and Flexibility: The technical solution incorporates a priority strategy for reference frame construction, a tracking loss handling mechanism, and a discretized display of summary data. This allows it to adapt to different scenarios (with or without IMU, Manhattan world, etc.) and potential anomalies during processing (such as short-term tracking loss), improving the method's practicality and robustness. Furthermore, it supports multiple pose estimation sources (visual SLAM / SfM, IMU, ground truth, etc.) and adjustable cue graph styles, making it highly flexible in application.

[0027] 5. Enhance zero-shot understanding: By including legends, visual symbols in cue diagrams are self-described, which helps MLLM to correctly understand the semantics of these symbols in zero-shot or few-shot settings, reducing the risk of out-of-distribution generalization. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a multi-frame spatial-temporal awareness enhancement method based on camera pose trajectory cue maps provided in an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram showing the detailed design of an intra-frame orientation HUD panel provided in an embodiment of the present invention.

[0030] Figure 3 This is a schematic diagram of a pose trajectory prompting image jigsaw puzzle structure provided in an embodiment of the present invention.

[0031] Figure 4 This is a schematic diagram of the closed-loop structure of online navigation and embodied intelligence application provided in an embodiment of the present invention.

[0032] Figure 5 This is a schematic diagram of the structure of a multi-frame spatial-temporal awareness enhancement system based on a camera pose trajectory cue map provided in an embodiment of the present invention. Detailed Implementation

[0033] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0034] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be used for both fixing and circuit / signal connectivity.

[0035] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0036] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0037] The basic concept of the embodiments of the present invention is as follows: This invention proposes a "Pose-Trace Visual Prompting" method: Instead of overlaying a large number of target markers in the main image area, "camera motion and orientation summary" is used as the main prompt carrier; The relative pose trajectory of the camera is estimated from multiple frames of images (optionally fused with IMU (Inertial Measurement Unit) / odometry, etc.) to construct a globally or locally consistent reference coordinate system; Generate a low-occlusion, high-information-density pose trajectory prompt sheet, which consists of two core parts: (1) Trajectory top view panel: Displays the camera's motion trajectory in space, the viewing direction (frustum / arrow) for each frame, and the frame sequence index; (2) Intra-frame orientation HUD (Head-Up Display, here referring to "the orientation information layer superimposed on the image") panel: a compact "compass / orientation / turn" prompt (e.g., reference direction arrow, heading angle, rotation and displacement level relative to the previous frame) is superimposed in the corner of each frame.

[0038] Note: The “Reference Direction / Initial Direction (Ref)” in the HUD described in this article is a direction reference determined by the reference frame construction module from the pose sequence (such as the projection of the first frame forward onto the horizontal plane, or the Manhattan axis direction). It is not equivalent to any absolute geographic orientation by default. The Ref can only be further aligned to an absolute orientation when the input contains an absolute orientation source such as an electronic compass / magnetometer or map alignment.

[0039] By explicitly encoding "viewpoint changes" into readable visual cues, the Multimodal Large Language Model (MLLM) can integrate information from each frame into a unified directional reference when answering questions about spatial relationships and temporal reasoning, thereby reducing horizontal flipping, front-back confusion, and inconsistencies in path narrative.

[0040] System structure: A space-time awareness enhancement system based on pose-trajectory cues may include: (a) Input module: Acquires image sequences or multi-view image sets. and text issues / instructions Optionally, sensor information such as IMU, wheel velocities, ARKit (Apple's augmented reality development platform) / SLAM pose can also be acquired. .

[0041] (ii) Camera pose estimation module: based on (and optional) Estimate the camera pose for each frame. ,in , That is, the three-dimensional rigid body transformation group, which includes rotation and translation.

[0042] (III) Reference Frame Construction Module: This module determines a consistent reference frame and outputs an availability flag, used to calculate the summary values ​​for each frame, such as heading angle, relative displacement, and critical steering. The reference frame can be selected with the following priority: When Manhattan World is available: the main axis is detected as a horizontal reference, and the vertical axis is determined by combining the direction of gravity; Otherwise, if the gravity direction is available (IMU or visual-inertial estimation): perform gravity alignment with the first frame and define the forward projection of the camera onto the horizontal plane in the first frame as the "reference direction / initial direction (Ref)"; Degeneracy selection: The camera coordinate system of the first frame is used as the local world coordinate system, and the "reference direction / initial direction (Ref)" is the forward projection of the first frame; at this time, the heading angle is the relative heading relative to the first frame.

[0043] (iv) Pose Trajectory Prompt Image Generation Module: Generates a trajectory top view panel based on the pose sequence, and generates an orientation HUD overlay prompt for each frame, which is finally stitched together to form a prompt image (which can be a single image or a small number of multi-image images). .

[0044] (v) Graphic and Text Joint Reasoning Module: Input the prompt image and text question together into the general MLLM. Output the answer or action .

[0045] To facilitate understanding of the above module relationships, the system structure is as follows: Figure 5 As shown.

[0046] Example 1: like Figure 1 As shown, this embodiment provides a spatial-temporal perception enhancement method for multi-frame images, which includes the following steps (S1~S5).

[0047] Step S1: Estimate the camera pose sequence.

[0048] For image sequences In one example implementation, the publicly available ORB-SLAM3 algorithm is used to estimate camera pose: the input is an image sequence and a camera intrinsic parameter matrix. (Optional distortion parameters are used for distortion correction); ORB feature points and descriptors are extracted for each frame, and matching is performed based on Hamming distance, combined with ratio test to filter out ambiguous matches; RANSAC is used for two-look geometric verification to eliminate outliers, and the intrinsic parameters are adjusted accordingly. Under constraints, the camera pose for each frame is obtained by minimizing PnP and reprojection error (with optional local BA optimization). :

[0049] in Indicates the first Camera pose of a frame image. for Rotation matrix, for Translation vector, This represents a three-dimensional rotation matrix group. Optional IMU fusion: gravity direction estimated by accelerometers, attitude priors provided by gyroscope integration, and joint optimization with vision to constrain pitch and roll, thereby stabilizing heading and attitude estimation.

[0050] In challenging scenarios such as texture loss, drastic lighting changes, interference from dynamic objects, pure rotation / small parallax, and blurring caused by rapid motion, visual tracking may experience loss or short-term drift. To prevent the generated trajectory cue map from transmitting incorrect spatial relationship information to the MLLM, the pose estimation module outputs the tracking status for each frame. (or an equivalent availability flag), and handle exceptions according to the following rules: Loss determination (if any condition is met) ): Effective tracking cannot be completed for several consecutive frames (e.g., consecutive frames). frame); The number of valid in-points is lower than the lower limit (e.g.) ) or the proportion of interior points is lower than the lower limit (e.g. ); Pose calculation residuals consistently exceed limits (e.g., average reprojection error greater than...). (Pixels and continue for several frames).

[0051] Loss handling (when) (Execution time) Pause to update pose sequence And trigger relocation; If relocation is successful within the limited number of frames (e.g.) (frame), continue to output subsequent poses, and mark the lost segment as a missing segment; If relocation fails, reinitialize with the current frame (using this frame as the new local origin reference frame to continue estimation) and display the trajectory segmented in the prompt graph.

[0052] IMU short-term prediction (only when an IMU exists and) ): Attitude prediction based on IMU integration within a short time interval to maintain orientation continuity (e.g., duration not exceeding...) or no more than frame); If visual tracking is not restored after this time, stop outputting predictions and process according to the "reinitialization" procedure.

[0053] Step S2: Construct a unified frame of reference and a summary of relative motion.

[0054] The construction of a frame of reference can be carried out in the following order of priority (from highest to lowest): The Manhattan axis is available: use the Manhattan world axis as a horizontal reference and combine it with the direction of gravity to determine the vertical axis; Gravity direction available: The first frame gravity alignment coordinate system is used as the world coordinate system, and the projection of the first frame camera forward onto the horizontal plane is defined as the "reference direction / initial direction (Ref)"; Degeneracy selection: The camera coordinate system of the first frame is used as the local world coordinate system, and the "reference direction / initial direction (Ref)" is the forward projection of the first frame; the heading angle only represents the relative heading with respect to the first frame.

[0055] To reduce the impact of cumulative drift on trajectory cues from large loops / long sequences Figure 1 To address the impact of consistency, the following measures can be taken: With loop closure detection / revisit constraints and sufficient computing power: perform global optimization on the pose sequence (e.g., pose graph optimization or global BA, fused with loop closure constraints) to reduce trajectory non-closure and heading drift; Lack of reliable loop constraints: The hint diagram mainly presents "relative path and relative turn" in a consistent manner, without requiring strict closure.

[0056] After selecting a reference frame, the relative motion is calculated with the first frame as the local origin:

[0057] and from Extract: Translation Scale (When a metric scale is available, it corresponds to the actual distance; under purely monocular conditions, it is used as relative displacement intensity and normalized / discretized). Rotation Scale ; Heading angle: obtained by projecting the camera's forward vector onto the horizontal plane. and take When only in the "first frame degradation" situation, the coordinates of the first frame are used. - The plane is approximated as horizontal, and the relative heading relative to the first frame is calculated.

[0058] The above-mentioned summary data is used to generate "turn / move forward" information in the cue map, helping MLLM attribute changes in viewpoint. The scale source is determined and used for cue map display according to the following rules: Metric-scale input: When the input is from a stereo camera, RGB-D camera, or fused with sensor information such as IMU / wheel speedometer, the metric scale is restored. It has real physical units; Pure monocular input (scale uncertain): As a relative displacement intensity, it is normalized according to statistics (median displacement) within the sequence (or within the online sliding window), and the normalized displacement intensity is discretized into a small number of levels for displacement prompts, trajectory arrow length and scale display in the HUD, thereby ensuring that the meaning of the prompts is consistent within the same sequence.

[0059] Step S3: Generate the trajectory top view panel.

[0060] Assume the resolution of the trajectory panel is In one example implementation, the camera center point is first... Projecting onto a horizontal top-down plane yields two-dimensional trajectory points. When the direction of gravity has been obtained from the IMU, define the horizontal plane with gravity as the normal and... Project onto that plane; if there is no IMU, then use the coordinates of the first frame. - The plane is taken as the top view plane and taken .

[0061] To ensure that trajectories of different scales are readable and displayable within the panel, Perform adaptive scaling and translation mapping to pixel coordinates :make For each dimension's minimum / maximum value, , , Take the scaling factor

[0062] in White space factor (e.g.) ), To prevent division by zero of small constants; then press

[0063] Obtain the coordinates within the panel. Then connect them with a polyline. Draw the trajectory and mark the frame sequence at keyframe points. (For example, every) (Frames or labels at points of significant rotation). To enhance the intuitiveness of scale, a scale bar can be drawn in the legend: the scale bar length is... The value is labeled as "one step (median displacement)". When using a recoverable metric scale implementation such as binocular / RGB-D / visual-inertial, this scale corresponds to the actual distance; when using a pure monocular implementation, this scale represents the normalized relative displacement unit, used to maintain display consistency within the same sequence.

[0064] To represent the viewing direction for each frame in the top-down view, the forward unit vector of the camera coordinate system is taken. (For example ), mapped to world coordinates Then project it onto the top view plane and normalize it to obtain the two-dimensional orientation. At point Draw a directional arrow at the location, the length of which is related to the pose data, and take...

[0065] in Indicates will Cut off to interval , Upper and lower limits (e.g.) and ), the arrow endpoint is Alternatively, the arrow can be replaced with "view cone": the half-angle of the view cone can be approximated by intrinsic parameters. (in Image width, From ), and draw two lines with The included angle is ray.

[0066] The purpose of this panel is to express "relative path and orientation changes" in a readable way, without requiring precise geometric reconstruction.

[0067] Step S4: Generate intra-frame orientation HUD.

[0068] For each frame of image A compact HUD is obtained by overlaying it without obscuring the main semantic region. In one example implementation, the HUD is placed in one of the four corners of the image (preferably the top left corner), with an outer margin from the image boundary. (For example The side length of the HUD circumscribed square is taken as... And use a semi-transparent base (e.g., opacity) "Low occlusion" is defined as follows: the HUD occupies only about [percentage missing] of the image area. It is located in the corner edge area and does not cover the central area of ​​the image.

[0069] The graphical elements of the HUD can be drawn directly from the pose summary as follows: (1) Compass / Orientation: Draw a compass with a radius of The circular ring serves as the compass base map; a fixed marker (such as Ref or an upward arrow) is drawn on the HUD using the "Reference Direction / Initial Direction (Ref)" output by the reference system construction module; then the heading angle is... Mapped to the compass pointer direction, draw the pointer at the center of the circle, with a pointer length of [value missing]. .

[0070] The “reference direction / initial direction (Ref)” is provided by the reference frame construction module. By default, the “projection of the first frame camera forward onto the horizontal plane” (or the Manhattan axis direction) can be taken as the Ref, and an absolute orientation is not required.

[0071] (2) Steering and displacement summary: The heading change and translation intensity relative to the previous frame (or relative to the previous keyframe) are discretized into a small number of gears to improve readability and expressed with compact symbols: steering is represented by "left / right curved arrow + number of gears", and displacement is represented by "1~3 short bars / arrows".

[0072] (3) Global left and right prompts: The global “L / R” (left / right relative to the reference direction) is marked in a fixed style in the HUD, which is fixed with the HUD coordinates and does not change with the camera rotation, thereby reducing semantic drift.

[0073] To avoid ambiguity and facilitate implementation, the symbols in the HUD are generated according to the following simple rules (the example thresholds can be adjusted according to the task or replaced with quantiles): Ref label: The Ref label (or up arrow) above the HUD indicates "reference direction / initial direction (Ref)".

[0074] Compass pointer: drawn with a heading relative to Ref, defined ,in The angle corresponding to the Ref direction (e.g., taking the first frame). ), This indicates that the angle is normalized to... Interval. The angle the pointer points to. .

[0075] Turn signal (left / right + gear): Use determination: Left turn indicated. The system indicates a right turn; otherwise, it indicates going straight / no obvious turn. The minimum angle threshold for determining whether a significant turn has occurred. The amplitude is calculated as follows: Divided into three categories: small, medium, and large (e.g.) (Two thresholds).

[0076] Displacement symbol (gear): Use (See step 2) represents displacement intensity; normalization under monocular conditions. Then, based on the threshold, they are divided into three levels: small, medium, and large (e.g., 0.5, 1.5).

[0077] L / R (optional): The HUD displays L / R, which means "left / right relative to the Ref" and remains unchanged within the same sequence, not changing as the lens rotates.

[0078] To clearly illustrate the composition and encoding method of a HUD, the diagram is as follows: Figure 2 As shown.

[0079] Step S5: Create a prompt image by splicing the images and input MLLM.

[0080] Specifically as follows: The trajectory top view panel is stitched together with several keyframes with HUD to form a single tooltip mosaic. .

[0081] When pose estimation is detected to be unavailable or tracking loss causes trajectory discontinuity, the prompt graph generation module outputs according to the following rules: If the loss is short-term and an IMU is present: the trajectory panel draws the short-term prediction segment with a dashed line; the HUD suppresses the displacement setting and only retains the orientation information; If the loss continues beyond the preset window: the track panel stops updating and a blank background is used as a placeholder, causing the input to degenerate into a baseline form with "no track hints". An example of a preset window is: continuous loss exceeding... Frames, or the proportion of frames effectively tracked within the used sequence / sliding window is less than .

[0082] Puzzle layout rules: The track panel is positioned at the top (or left). Keyframe thumbnails are placed at the bottom (or right) and arranged in chronological order (e.g., from left to right, from top to bottom).

[0083] Labeling rule: Retain the frame number in the corner of each keyframe thumbnail. And retain the reference system label (e.g., "reference direction / initial direction datum (Ref)").

[0084] Labeling rule: Retain the frame number in the corner of each keyframe thumbnail. And retain the reference system label (e.g., "reference direction / initial direction datum (Ref)").

[0085] Legend rules: Draw a compact legend on the trajectory panel or the edge of the puzzle, using minimal text to self-describe the semantics of the symbols (e.g., Ref = reference direction; compass pointer = orientation relative to Ref; curved arrow = turning relative to the previous frame (left / right, the number of bars represents the magnitude); short bar = displacement level; dashed line = short-term prediction / uncertain segment). The purpose of the legend is to improve the readability of zero-shot data and reduce symbol ambiguity; if individual models are still unstable with respect to symbols, 1-2 lines of restatement of the legend can be added to the text side, or a "few-shot" question and answer can be provided for calibration.

[0086] Input method: With text issues Enter the general MLLM together .

[0087] Model processing: MLLM usually first... The image tokens are converted into a sequence by a visual encoder, and then combined with text tokens for joint attention reasoning to output the answer. .

[0088] Optionally, minimalist prompts can be added to the text side (e.g., "Please combine the trajectory top view with each frame of the HUD to determine the left-right, front-back, and sequential order under a unified reference frame") to form a visual-text dual-channel guidance.

[0089] The above-mentioned "Track Top View Panel + Keyframe HUD Thumbnail" tooltip puzzle structure is shown in the image. Figure 3 .

[0090] Implementation format The technical solution of the present invention can be implemented in the following forms: (1) Method: A pose-trajectory cue generation and multi-frame inference enhancement method executed on a computing device.

[0091] (2) Device: A device comprising an input module, a pose estimation module, a prompting graph generation module and an inference module.

[0092] (3) Electronic device: includes processor, memory and program, to execute the above method.

[0093] (4) Computer-readable storage medium: storing a program for performing the above methods.

[0094] To verify the technical effects of the present invention, the following experiments were conducted: Compared to multi-frame input methods without prompts, this invention achieves at least the following technical effects (the magnitude of the effect varies with the model and data). For ease of understanding, a set of comparative experimental data is provided below to illustrate the potential improvements of this invention under the same MLLM and the same image input budget; this data does not constitute a limitation on the scope of protection of this invention, and the actual effect will vary depending on the dataset, pose estimation quality, keyframe selection strategy, cue map layout, and the MLLM used.

[0095] Experimental example: MLLM Model and Version: The open-source multimodal model LLaVA-1.5 7B was used as the inference model. The following parameters were fixed and publicly disclosed in the experimental records: visual encoder input resolution, max_new_tokens, decoding strategy (greedy), and random seed.

[0096] Dataset and Official Split: To conform to the research conventions in this field, the evaluation uses the official evaluation split of the dataset, and an "evaluation subset" is constructed on top of it according to defined rules (only filtering / sampling, without changing the labels): Multi-frame video question answering (Accuracy): NExT-QA (NextQA) official test split.

[0097] Online closed-loop navigation (Success Rate): Room-to-Room (R2R, Matterport3D) official val_unseen split.

[0098] Evaluation subset construction rules: The basic unit of the sample is uniformly "video / round + keyframe index set + question / instruction + standard answer / success judgment".

[0099] NextQA-Dir (fixed selection of 200 questions): In the NextQA test, samples containing explicit locative words in the question text are selected. The selection rule is to use regular expressions to match left|right|front|behind. If the selection results are more than 200, the first 200 are selected in lexicographical order of video_id + question_id; if there are fewer than 200, all are selected and the actual number is noted in the experimental record (in the table, the actual number is used). count).

[0100] NextQA-Temp (fixed selection of 150 questions): The NextQA test filters samples containing explicit temporal terms. The filtering rules are: first, then, after, before, next. If the number of results is greater than 150, the first 150 are selected in lexicographical order of video_id + question_id; if it is less than 150, all results are selected, and the actual number is noted (in the table, the actual number is used). count).

[0101] R2R-Nav (fixed 100 rounds): The first 100 rounds in R2R val_unseen are selected as evaluation rounds in lexicographical order of path_id.

[0102] Keyframes and input budget: Fixed for each sample Keyframes are extracted, and deterministic, equally spaced frame extraction is used to avoid ambiguity caused by threshold selection. The video / round is first resampled to 5 fps, and its length is truncated to the first 200 frames (if less than 200 frames, the full length is used); let the total number of frames after truncation be... If the frame is a keyframe, then the keyframe index is...

[0103] Pose estimation is performed on the resampled sequence; the trajectory panel and HUD are only superimposed on the above 4 keyframes.

[0104] In-camera participation in pose estimation: NextQA (pose estimation): First, use COLMAP (pinhole model, self-calibration) to estimate camera intrinsics on the resampled frames of each video segment. Then The pose of each frame is obtained by inputting the resampled frame sequence into ORB-SLAM3 (monocular mode, loop closure detection enabled). The failure rejection rule is fixed as follows: samples with a valid tracking frame ratio of less than 90% are not included in the statistics, and the rejection ratio is reported in the experimental record.

[0105] R2R (Take-to-Pose Truth): R2R provides the camera pose and orientation at each step in the simulation environment. This experiment directly uses the pose provided by the environment as the true value. (Visual odometry is no longer used) to avoid SLAM instability caused by discrete viewpoints and to better conform to the research conventions of embodied navigation evaluation.

[0106] Comparison method definition: Baseline (no prompt): To ensure consistency in image input budget, the baseline uses the same jigsaw puzzle canvas size and the same keyframe thumbnail position as this invention; the trajectory panel area is occupied by a blank background, and the keyframes are not overlaid with the HUD.

[0107] This invention (pose trajectory prompting diagram + HUD): Generate a "trajectory top view panel + keyframe HUD (including Legend)" according to steps 3 to 5, and then stitch them together into a single prompting diagram before inputting it.

[0108] Eliminate / Add Contrast: To eliminate accidental improvements caused by non-critical factors such as "only due to layout changes / additional text", the following contrast is added under the same canvas and the same keyframe position (without changing the MLLM structure and parameters): Trajectory-only: Only the trajectory top view panel (including frame order and orientation arrows / view cones) is used; keyframes are not overlaid with the HUD.

[0109] HUD-only: The keyframe overlays the HUD (including Ref and gear), but the track panel area is a blank background.

[0110] Legend-only (symbol and legend comparison): Only the Legend text / small legend is placed on the baseline puzzle, the trajectory panel is blank, and the HUD is not overlaid on the keyframes. This is used to test whether "only adding explanatory text" will bring false improvement.

[0111] w / o Legend: Same as this invention but with Legend removed, used to measure the contribution of self-describing legends to zero-shot understanding.

[0112] Cue Map Understandability Verification: To directly verify that "MLLM can interpret trajectory top-down views and HUD symbols," a probe task that can answer questions solely based on cue map symbols can be constructed without introducing the original keyframe semantic content. The probes are then tested on the selected MLLM with zero-sample responses. Probe-Heading: Given a cue map (or HUD / track panel only), ask "the first..." "Which side is the frame facing relative to the Ref, on the left / right / front / back?" or "The first..." Is the frame a left turn / right turn / straight line relative to the previous frame? What are the amplitude levels?

[0113] Probe-Order / Trace (Trace Interpretation): Asks questions such as "Does the frame sequence move monotonically along the trajectory? Has the camera returned to the vicinity of the starting point? Is there a clear backtracking?", which can be determined solely by the trajectory and index.

[0114] Judgment method: This probe can automatically generate a standard answer (calculated directly from the pose summary), avoiding subjective manual annotation.

[0115] Expected outcome: If the model can indeed interpret the symbols, it should perform significantly better than random guessing on the probes mentioned above (e.g., significantly better than 50% for binary classification).

[0116] Statistical significance: For multiple-choice questions in NextQA, paired tests (e.g., McNemar's test) or bootstrap methods can be used for reporting. Confidence intervals; for the R2R success rate, a bootstrap approach can be used to calculate confidence intervals at the round level. This approach is used to support the claim that "the hint graph provides a statistically significant improvement," rather than random fluctuations.

[0117] Inference and Statistical Approach: Deterministic decoding is employed: temperature is set to 0 (greedy), and the maximum output length is fixed. For NextQA multiple-choice questions, the model output is mapped to one of {A, B, C, D, E}; for R2R navigation, the action sequence is output until STOP or a condition is met. .

[0118] Evaluation metrics definition: Accuracy (%): The percentage of NextQA multiple-choice questions where the statistical model output matches the standard options.

[0119] Success Rate (%): For R2R rounds, the official success criterion is used (reaching the target distance threshold and within the target range). (Termination within the period), the percentage of successful rounds out of the total number of rounds.

[0120] Experimental results: The table below presents the comparative results obtained by statistical analysis of the "Experimental Examples," and clarifies the calculation method for the figures in the table: for the NextQA subset, the accuracy is obtained by averaging the correct answers for each question; for the R2R subset, the success rate is obtained by averaging the success rates for each round. This experiment uses deterministic decoding, therefore, the standard deviation (std) does not need to be reported.

[0121] (A) Main Experiment:

[0122] The main experiment demonstrates the effectiveness of this method on multiple tasks.

[0123] (B) Ablation experiment:

[0124] Ablation experiments ruled out the possibility that non-critical factors could lead to significant improvements in this method: Legend-only is close to the baseline, eliminating the false boost caused by "only adding text descriptions / layout changes"; The improvements between Trajectory-only and HUD-only indicate that a single sub-cue can already be partially utilized by the model; The difference between w / o Legend and containing Legend is used to demonstrate the enhanced readability of zero-sample data.

[0125] (C) Prompt-only Probe: On a probe that can automatically determine right or wrong by simply inputting a symbol from a prompt image, the model can correctly interpret the symbol, significantly outperforming random ones.

[0126]

[0127] The probe results above demonstrate that the cue graph is not "only readable by humans," but rather has a similar form to visual language such as diagrams, illustrations, and arrow annotations commonly used in MLLM training, thus possessing the potential for zero-shot or few-shot understanding. At the same time, Legend's self-descriptive design further reduces the risk of generalization caused by out-of-distribution symbols.

[0128] Application examples: Case 1 (Location): Question: "When the camera pans from frame 1 to frame 3, is the cabinet on the left or right side of the sofa relative to it?" Common baseline error: confusing "left" and "right" (switching them as the camera turns).

[0129] After using this invention: the model can first read the orientation of the third frame relative to the Ref and the turning direction relative to the previous frame in the HUD, and then map the relative position of "cabinet-sofa" in the third frame back to the Ref coordinates, thus giving a consistent answer.

[0130] Case 2 (Timing / Path): Question: "Does the person in the video first pass through the corridor and then enter the kitchen, or vice versa? Is there any turning back?"

[0131] Common baseline error: difficulty in determining the chronological order.

[0132] After using this invention: the model can determine the order of events based on the frame order of the trajectory panel along the path, and use the trajectory shape (e.g., the appearance of a clear U-shape / reversal segment) to answer "whether to turn back".

[0133] As can be seen from the above results, by explicitly providing a "motion and orientation summary in a unified reference frame", this invention can improve the model's consistent understanding of spatial orientation and cross-frame narrative without increasing the number of keyframes, and bring a higher success rate in closed-loop decision-making tasks.

[0134] (1) Reduce orientation confusion: By explicitly providing a summary of the camera's heading and trajectory, errors such as "left and right flip, front and back reversal, and head / turn not being correctly understood" are reduced.

[0135] (2) Enhance cross-frame narrative consistency: The trajectory panel provides a unified temporal-spatial skeleton, making it easier for the model to maintain a coherent explanation of issues such as "sequence, path traversal, and whether it returns to the same area".

[0136] (3) Low occlusion and low information overload: The prompts are mainly presented by HUD and trajectory overview, without the need to overlay a large number of markers in the main area of ​​the screen, thus occluding less semantic content.

[0137] (4) Lower deployment dependency: It can run with only RGB images (pose estimated by visual odometry); stability can be further improved with IMU / ARKit / robot odometry.

[0138] (5) Save MLLM input budget: By compressing multi-frame motion information into one / a small number of mosaics, the image token consumption is reduced while preserving key motion relationships.

[0139] Example 2

[0140] An example of spatial relationship determination: Indoor multi-view question and answer (spatial relationship): Data and Input: Using the same indoor scene as the subject, data was collected. Zhang's perspective image (The image was taken by moving the phone around the scene while holding it), and the image resolution was uniformly scaled to 512 pixels on the shorter side.

[0141] Pose estimation method and parameters: COLMAP was used for... Incremental SfM reconstruction is performed to obtain the relative pose of the camera.

[0142] Feature: SIFT (COLMAP default); Matching: Uses COLMAP's default global matching (exhaustive matching) and performs geometric verification (RANSAC) to remove outliers; Optimization: Perform global / local BA to obtain camera pose. Compared with (self-calibrated) internal reference From this, we obtain It is directly used for the construction of the reference system and the generation of the prompt diagram in steps 2 to 5.

[0143] HUD heading angle quantization (8 discrete directions): for each frame / viewpoint, from the rotation matrix The heading angle is obtained by taking the camera's forward vector and projecting it onto the horizontal plane. (Same as step 2). The heading relative to the reference direction... Discretized into 8 directional sectors (each sector) ):

[0144] in This indicates that the angle is normalized to... The HUD compass pointer points to the center corner of the sector. and numbered in the HUD (Or marked with Ref, Ref+45°, Ref+90°,...).

[0145] Prompt image stitching parameters: Fixed selection Zhang key perspectives (take the 1st / 3rd / 6th / 8th shots in the shooting order) overlaid with HUD; track panel resolution taken as... Key perspective thumbnails for each image Follow step 5 to stitch them together into a single prompt image. .

[0146] Detailed Q&A: Enter the question text: "When walking from the entrance towards the sofa area, is the coffee table to the left or right of the sofa?" MLLM outputs the answer: "The coffee table is to the left of the sofa." Example 3: An example of temporal and path reasoning, for long-video comprehension (event sequence and path): Data and Input: Select a video segment from the test split of the publicly available video question-answering dataset NextQA, resample it at 5 fps and truncate it into the first 200 frames to obtain the sequence. ( ).

[0147] Keyframes and Input Budget: Fixed Zhang keyframes, indexed according to the equal-interval rule described in the embodiment. These four frames were then used for HUD overlay and puzzle input.

[0148] Pose estimation method and parameters: For the resampled frame sequence of this video segment, the intrinsic parameters are first estimated using COLMAP (self-calibration). Then, ORB-SLAM3 (monocular, loop closure detection enabled) is used to estimate the pose of each frame. If the proportion of effective tracking frames is less than 90%, the sample will be removed from the statistics.

[0149] HUD heading angle quantization (8 discrete directions): calculated according to the same 8-sector rule as in Example 1. In each keyframe, the HUD displays the discrete heading and the steering gear (small / medium / large) relative to the previous keyframe.

[0150] Hint: Image stitching parameters: Track panel Keyframe thumbnails ; spliced ​​into a single sheet Then input MLLM along with the question text.

[0151] Detailed Q&A: Enter the question text: "Did this person in the video enter the kitchen first and then the bedroom, or vice versa?" MLLM outputs the answer: "Enter the kitchen first, then the bedroom." Example 4 Implementation examples for online decision-making, online navigation / embodied intelligence (stepwise observation): Environment and Round Definition: Embody navigation baseline R2R (Matterport3D) is adopted, and one round is taken from the official val_unseensplit by path_id as an example; each observation is an RGB image. Camera pose provided by the environment .

[0152] Pose source and reference system: Directly using the true pose provided by the environment. Construct a frame of reference and a HUD (without running visual odometry) to conform to common practices in navigation evaluation.

[0153] Online window and keyframe strategy: Setting the sliding window length Steps, only keep the most recent ones Step pose is used in the trajectory panel; it is taken at equal intervals from within the window. Keyframes are used for HUD and puzzle input (e.g., retrieving keyframes). ).

[0154] HUD heading angle quantization (8 discrete directions): For each heading angle step Discretized into 8 sectors And displayed on the compass needle; relative steering amount is determined by The settings are divided into three levels: small, medium, and large, used to indicate the "steering range".

[0155] Action space and maximum number of steps: The action space is fixed at {TURN_LEFT, TURN_RIGHT, FORWARD, STOP}, and the maximum number of steps per round is... .

[0156] Detailed navigation: Enter the instruction text: "Walk from the starting point to the bedroom and stop." Example of an action sequence output by MLLM: TURN_RIGHT → FORWARD → FORWARD → TURN_LEFT → FORWARD → STOP.

[0157] In this online navigation scenario, the generation and updating strategy of the prompt map can be flexibly adjusted. The update frequency of the prompt map can be triggered by key turning events. An event-triggered update mechanism can be adopted. For example, the trajectory panel and HUD panel can only be updated after the agent performs key turning actions such as TURN_LEFT or TURN_RIGHT, thereby efficiently capturing changes in the topology of the path and avoiding unnecessary recalculation on straight sections.

[0158] This cues are used to stabilize orientation references and improve temporal consistency. The introduction of this cues is crucial for sequence decision-making tasks that require stable orientation references and improved temporal consistency. It enables MLLM to understand its relative position to the environment at each step based on a stable world coordinate system (not just the local viewpoint of the current frame), thus making more consistent and reliable navigation decisions. This effectively solves the problems of 'losing direction after turning' or 'misjudging the same location due to different viewpoints'.

[0159] The closed-loop structure of "observation—pose update—cue graph generation—MLLM decision—environment execution" in online scenarios is as follows: Figure 4 As shown.

[0160] In optional implementations, the modules or steps in the above embodiments of the present invention can be replaced and expanded in various ways according to actual needs.

[0161] One implementation method: pose source replacement, 1) Visual odometry / SLAM; 2) ARKit / ARCore; 3) Robot odometry + IMU fusion; 4) Multi-camera pose mapping.

[0162] As another implementation method: replacing the prompt image style. 1) The top-down trajectory diagram is replaced with a side-view height change diagram; 2) The view cone is replaced with a heading arrow; 3) The heading angle is encoded with discrete sectors / color bands.

[0163] One implementation method: keyframe selection strategy, 1) Based on rotation threshold With displacement threshold Event triggering: Preferred adaptive threshold , (Statistics on the entire sequence or sliding window, among which) (representing the 70th percentile); when or Time selection as keyframe; alternative empirical values , ;2) Maximize trajectory coverage;3) Trigger based on keywords in the question text (e.g., increase the density of turning frames when "turn / look back / left" appears).

[0164] As one implementation method: lightweight integration with depth / layout, Optionally, monocular depth or ground plane estimation can be introduced to draw a rough outline of passable areas / obstacles on the trajectory panel to further indicate spatial relationships such as "passing through / around / occluding before and after", but precise 3D reconstruction is not required.

[0165] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spatial-temporal awareness enhancement method for multi-frame images, characterized in that, Includes the following steps: S1. Obtain an image sequence and text instructions, wherein the image sequence is a collection of multiple frames or multiple perspective images from the same scene; S2. Based on the image sequence, estimate the camera pose of each frame to form a camera pose sequence, wherein the camera pose includes a rotation matrix and a translation vector; S3. Based on the camera pose sequence, a unified reference coordinate system is constructed, and the relative motion summary of each frame image is calculated based on the reference coordinate system. The relative motion summary includes the heading angle and displacement intensity. S4. Based on the camera pose sequence and the relative motion summary, generate a pose trajectory cue map, the pose trajectory cue map including: The trajectory top-view panel is used to visualize the projected trajectory of the camera's center point on the horizontal plane and the viewing direction of each frame; and The HUD panel of the intra-frame orientation head-up display system is used to overlay graphic symbols on at least one keyframe image of the image sequence, wherein the graphic symbols include at least a compass pointer representing the heading angle of the current frame and a fixed mark representing the reference direction. S5. Input the pose trajectory cue map and the text command together into a multimodal large language model (MLLM); and S6. Receive and output the answer or action command generated by the MLLM based on the pose trajectory prompt map and the text command.

2. The method according to claim 1, characterized in that, Step S2 includes: The image sequence is processed using either Visual Simultaneous Localization and Mapping (SLAM) or Structure for Motion Restoration (SfM) algorithm to estimate the camera pose.

3. The method according to claim 2, characterized in that, In step S2, sensing data from an inertial measurement unit (IMU) is also fused, the IMU data being used to assist in stable attitude estimation or to provide scale information.

4. The method according to claim 1, characterized in that, In step S3, a unified reference coordinate system is constructed according to the following priority rules: If the Manhattan World main axis is detected, then the Manhattan World main axis is used as the horizontal reference. Otherwise, if the gravity direction is available, the gravity-aligned coordinate system of the first frame image is used as the world coordinate system, and the projection direction of the camera forward vector of the first frame onto the horizontal plane is defined as the reference direction. Otherwise, the camera coordinate system of the first frame image is used as the world coordinate system, and the forward vector of the first frame camera is defined as the reference direction.

5. The method according to claim 1, characterized in that, Step S4 generates the trajectory top view panel, including: Projecting the camera center point in the camera pose sequence onto a horizontal plane yields two-dimensional trajectory points; The two-dimensional trajectory points are adaptively scaled and translated to map them onto a canvas of a predetermined resolution; Connect the mapped points to draw the trajectory line; and At specific locations along the trajectory line, graphic elements representing the viewing direction of each frame of the image are drawn.

6. The method according to claim 5, characterized in that, The graphic element is an arrow or a view cone, and the length of the arrow or the opening angle of the view cone is associated with the displacement intensity or camera intrinsic parameters.

7. The method according to claim 1, characterized in that, Step S4 generates the intra-frame orientation HUD panel, including: The HUD panel is overlaid on a corner area of ​​the keyframe image; The display area of ​​the HUD panel does not exceed 10% of the total area of ​​the keyframe image in which it is located; and The graphic symbols displayed in the HUD panel also include at least one of the following: A steering symbol representing a change in heading relative to the previous frame; The sign representing the displacement intensity relative to the previous frame; A fixed display of global left and right markers relative to the reference direction.

8. The method according to claim 7, characterized in that, The heading angle is discretized into multiple sectors, and the compass pointer points to the center direction of its respective sector; and / or The magnitudes of the steering and displacement symbols are discretized into a small number of gears for display.

9. The method according to claim 1, characterized in that, The pose trajectory hint map generated in step S4 also includes a legend, which is used to interpret the semantics of the graphic symbols in the trajectory top view panel and the intra-frame orientation HUD panel.

10. The method according to claim 1, characterized in that, In step S2, the tracking status flag for each frame of image is also output; When the tracking status flag indicates that tracking has been lost, step S4 performs at least one of the following processes: Use dashed lines to draw the predicted trajectory segments in the trajectory top view panel; Suppress the display of displacement symbols in the intra-frame orientation HUD panel; If the tracking loss continues to exceed a preset threshold, the pose trajectory hint map will stop being updated.

Images