VLN inference self-correction method and device based on future visual imagination verification

Abstract

The application discloses a VLN reasoning self-correction method and device based on future visual imagination verification, comprising: acquiring a current VLN model state and an instruction, parsing the instruction into a sub-target predicate that can be verified by a future perspective, and generating a plurality of candidate forward route hypotheses; determining a future perspective anchor point for each candidate route, generating a future visual imagination view through a generative visual imagination model, extracting future semantic structure information and constructing a future perspective evidence package; performing consistency verification on the evidence package to obtain a visual language history consistency score, and then calculating a route error risk, rejecting a low credibility route to generate a self-correction decision; outputting a navigation action according to the verification result, and writing the verification result, the rejection reason and the actual observation into a navigation memory for optimizing the future visual imagination model. The method can predict the future state of the candidate route before the action is executed, discover the wrong direction and the high-risk path in advance, and improve the reliability of the VLN model.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligence technology, specifically relating to a VLN inference self-correction method and device based on future visual imagination verification. Background Technology

[0002] With the development of embodied intelligence, visual language navigation (VLN) technology is gradually being applied to scenarios such as warehousing and logistics, indoor delivery, service robots, industrial inspection, and hospital guidance. In these scenarios, natural language commands often contain multiple landmarks, sequential relationships, conditional actions, and negative constraints.

[0003] Existing VLN methods typically output actions directly based on current visual observation data and historical trajectories, or generate multiple candidate routes during the testing phase and select a route through model scoring, searching, or reordering. However, many failures in visual language navigation do not stem from obvious errors in the current action itself, but rather from the agent discovering, after moving several steps in a certain direction, that it has entered the wrong room, wrong passage, dead end, or repetitive area. In other words, current local observations are insufficient to directly determine the long-term consequences of candidate routes.

[0004] While some future-view prediction or path-imagine techniques exist for data augmentation or auxiliary representation learning during the training phase, they are typically not used directly as candidate route verification mechanisms during the inference phase. Furthermore, they do not perform joint consistency verification with natural language sub-objectives, current observation transition relationships, and historical trajectory memories. Therefore, existing technologies still struggle to identify candidate routes that "seem reasonable but will likely lead to the wrong path" before action execution.

[0005] Therefore, there is an urgent need for a visual-language navigation method that can generate future visual evidence for candidate routes during testing and perform route rejection and self-correction based on the consistency between the future perspective and language target, current observation and historical trajectory. Summary of the Invention

[0006] The purpose of this application is to provide a VLN inference self-correction method and apparatus based on future visual imagination verification, in order to solve the technical problems in related technologies where visual language navigation models usually only output navigation actions based on current visual observations and historical navigation information, making it difficult to predict and verify the future visual state of candidate routes before the action is executed. This results in the inability to identify erroneous navigation directions in a timely manner, difficulty in assessing the credibility of generative future perspectives, and the easy occurrence of route deviation, repeated exploration, and high-cost erroneous execution in the case of missing visual information, ambiguous instructions, or changes in the scene.

[0007] According to a first aspect of the embodiments of this application, a VLN inference self-correction method based on future visual imagination verification is provided, comprising: The system acquires current visual observation data, historical navigation trajectories, and natural language navigation instructions. It parses the natural language navigation instructions into multiple sub-target predicates that can be verified from a future perspective and generates multiple candidate forward route hypotheses based on the current navigation state. For each candidate prospective route hypothesis, a generative visual imagination model is constructed, and a future view anchor point corresponding to it is determined to generate a future visual imagination view. Semantic structure information is extracted from the view, and the reliability of future view generation is calculated to obtain a future view evidence package. The future perspective evidence package is verified for consistency with the sub-target predicate, current visual observation data and historical navigation trajectory to obtain a visual language historical consistency score. Based on the visual language historical consistency score, the reliability and irreversible cost of candidate routes are generated, and the error risk of the candidate prospective route assumption is calculated. Based on the error risk, candidate prospective route hypotheses are divided into a set of candidates to be verified and a set of candidates to be rejected, and rejection reasons and self-correction decisions are generated for the rejected candidate prospective route hypotheses. When the candidate set is empty, a self-correction action is output based on the self-correction decision. When the candidate set is not empty, a target route is selected from it and the corresponding action is output. Finally, by executing the self-correction action or the corresponding action, new action data is obtained and written into the navigation memory as training sub-data of the generative visual imagination model.

[0008] According to a second aspect of the application embodiments, a VLN inference self-correction device based on future visual imagination verification is provided, comprising: The candidate route generation module is used to acquire current visual observation data, historical navigation trajectories and natural language navigation instructions, parse the natural language navigation instructions into multiple sub-target predicates that can be verified by future perspectives, and generate multiple candidate forward route hypotheses based on the current navigation state. The future perspective evidence construction module is used to construct a generative visual imagination model for each candidate prospective route hypothesis, determine the corresponding future perspective anchor point to generate a future visual imagination view, extract semantic structure information from the view, calculate the reliability of future perspective generation, and obtain a future perspective evidence package. The consistency verification module is used to verify the consistency of the future perspective evidence package with the sub-target predicate, the current visual observation data and the historical navigation trajectory, respectively, and obtain a visual language historical consistency score. The error risk calculation module is used to generate reliability and irreversible cost of candidate routes based on the visual language historical consistency score, and to calculate the error risk of candidate prospective route assumptions. The candidate segmentation and self-correction module is used to segment candidate prospective route hypotheses into a set of candidates to be verified and a set of candidates to be rejected based on the error risk, and to generate rejection reasons and self-correction decisions for the rejected candidate prospective route hypotheses. The navigation decision output module outputs a self-correction action based on the self-correction decision when the candidate set is empty, and selects a target route and outputs the corresponding action when the candidate set is not empty. Finally, it obtains new action data by executing the self-correction action or the corresponding action and writes it into the navigation memory as training sub-data for the generative visual imagination model.

[0009] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0010] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the method described in the first aspect.

[0011] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects: (1) It can detect long-range navigation errors in advance. This application predicts the future visual results of candidate routes before the action is executed and verifies whether they meet the language target, thereby reducing the situation of "finding the wrong direction after taking a few steps".

[0012] (2) It can improve the reliability of candidate route verification. This application does not only determine whether the future view contains the target object, but also jointly verifies the consistency of language target, visual transfer consistency and historical memory consistency, thereby reducing the misjudgment caused by single image-text matching.

[0013] (3) It can achieve interpretable route rejection. This application generates rejection reasons for rejected candidate routes, such as inconsistent language goals, unreasonable visual transfer, conflict of historical memory, insufficient generation reliability or excessive irreversible cost, so as to make the navigation self-correction process interpretable.

[0014] (4) It can balance accuracy and safety. This application incorporates irreversible costs into the error risk calculation, and can preferentially reject candidate routes that enter dead ends, error areas, repetitive areas or areas that are difficult to backtrack.

[0015] (5) It can form a continuous improvement closed loop. This application uses rejection reasons and actual observation results as sub-data to train or update the generative visual imagination model, so that the model can reduce similar erroneous imaginations in subsequent navigation tasks and improve the prediction quality of future perspectives. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a VLN inference self-correction method based on future visual imagination verification, according to an exemplary embodiment.

[0017] Figure 2 This is a block diagram illustrating a VLN inference self-correction device based on future visual imagination verification according to an exemplary embodiment.

[0018] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0022] Figure 1This is a flowchart illustrating a VLN inference self-correction method based on future visual imagination verification, according to an exemplary embodiment. In this example, a warehouse logistics packaging scenario is used as an example. The logistics navigation agent is equipped with an RGB-D camera, an odometer, and a local topology mapping module to complete package packaging in the warehouse according to natural language instructions. Of course, it is not limited to this; it can also be an intelligent robot or intelligent vehicle equipped with a VLN model, which can be processed and implemented by this invention.

[0023] refer to Figure 1 The present invention provides a VLN inference self-correction method based on future visual imagination verification, which may include the following steps: S1: Acquire current visual observation data, historical navigation trajectory, and natural language navigation instructions; parse the natural language navigation instructions into multiple sub-target predicates that can be verified from a future perspective; and generate multiple candidate forward route hypotheses based on the current navigation state; this step includes the following sub-steps: S11: Obtain current visual observation data, historical navigation trajectory, and natural language navigation instructions to obtain a representation of the current navigation status; Specifically, the navigation agent acquires the current visual observation data at the current time t. Historical navigation trajectory And natural language navigation instructions I.

[0024] Among them, current visual observation data This includes current image, depth information, object detection results, landmark recognition results, and traversable area recognition results; historical navigation trajectory. This includes visited locations, actions performed, observed landmarks, obstacle-avoided areas, and backed-up nodes.

[0025] The navigation agent is based on historical navigation trajectories Constructing local topological memory The current visual observation data, current location, historical trajectory nodes, and command execution progress are aligned to the same navigation time point to obtain the current navigation state representation:

[0026] in, This indicates the current navigation status; Represents the state alignment function; This represents local topological memory.

[0027] S12: Based on the current navigation state representation, perform semantic parsing on the natural language navigation instructions to obtain a set of sub-target predicates that can be verified from a future perspective; Specifically, the navigation agent performs semantic parsing on the natural language navigation command I, extracting target objects, visual landmarks, spatial relationships, action constraints, sequence constraints, and negative constraints, and representing them as multiple sub-target predicates. :

[0028] in, This represents the predicate of the k-th sub-target; Indicates the target object; Indicates a visual landmark; Indicate spatial relationships; Indicates action constraints; Indicates order constraints; This indicates a negative constraint.

[0029] Specifically, the navigation agent will use predicates for each sub-target. Compared with historical navigation trajectory and current visual observation data Perform matching to determine whether the sub-target predicate has been completed or validated. The predicate completion status is indicated by... for:

[0030] in, Indicates the completion status of the k-th sub-goal predicate; when When, it indicates that the predicate of that sub-target has been completed; when When this occurs, it indicates that the predicate for that sub-target has not yet been completed; This function represents the predicate completion status verification function.

[0031] This yields the current set of predicates for the sub-targets to be verified. :

[0032] in, This represents a set of sub-goals that are not yet complete at the current moment and require further verification from a future perspective.

[0033] S13: Based on the current navigation state representation and the sub-target predicate set, generate multiple candidate action or candidate route nodes, and perform look-ahead expansion on each of the multiple candidate action or candidate route nodes to obtain multiple candidate look-ahead route hypotheses.

[0034] Specifically, the navigation agent is based on the current navigation state. and the current set of predicates for the sub-targets to be verified This generates multiple candidate action or candidate route nodes, and expands each candidate action or candidate route node into a prospective route segment.

[0035] First, a set of candidate actions is generated based on the navigation backbone model. :

[0036] in, Represents the set of candidate actions; Indicates candidate actions; This represents the action probability function output by the navigation backbone model; This indicates that the top K candidate actions with the highest probability are selected.

[0037] Subsequently, a look-ahead extension is performed on each candidate action to obtain candidate look-ahead path hypotheses. :

[0038] in, This represents the hypothesis of the nth candidate forward route; This represents the nth candidate action; This represents the lookahead expansion function; Indicates the maximum look-forward step size. This represents the l-th action in the assumed route.

[0039] During the expansion process, the navigation agent does not simply extend along the direction with the highest action probability, but prioritizes retaining route segments that make at least one sub-goal predicate to be verified observable in the future view. For example, if the current predicate to be verified is "the B2 sorting station should appear in the future view", then the candidate look-ahead route should be expanded to the location where the sorting station marker can be observed.

[0040] Finally, a set of candidate forward route hypotheses was obtained. :

[0041] Where N represents the number of candidate forward-looking route hypotheses.

[0042] Through the above steps, the navigation agent does not directly output actions based on current observations. Instead, it first determines which sub-goals in the natural language instructions have been completed and which still need to be verified through future perspectives. Then, it generates candidate prospective route hypotheses around the incomplete sub-goals. This avoids the candidate route generation from becoming disconnected from the current task progress, giving subsequent future visual imagination and consistency verification a clear verification object, thereby improving the targeting of self-correction during the inference phase.

[0043] S2: For each candidate prospective route hypothesis, construct a generative visual imagination model, determine its corresponding future viewpoint anchor point to generate a future visual imagination view, extract semantic structure information from the view, and calculate the reliability of the future viewpoint generation to obtain a future viewpoint evidence package; this step includes the following sub-steps: S21: For each candidate forward route hypothesis, predict the future location, future orientation, and future observable range after executing the candidate forward route hypothesis, and obtain the future viewpoint anchor point; Specifically, the navigation agent selects from the set of candidate look-ahead route hypotheses. Take any candidate forward route assumption from the data. And indicate based on the current navigation status and local topological memory It gradually predicts the future location, future orientation, and future observable range after executing the candidate route.

[0044] Furthermore, for the candidate prospective route hypothesis, the navigation agent predicts the future perspective according to the following state transition process. :

[0045] in, Indicate execution The node at the position before the j-th action; This indicates the orientation before the j-th action; This represents the j-th action; Represents local topological memory The state transition function.

[0046] When the candidate forward route hypothesis reaches its final stage, the future perspective anchor point is obtained. :

[0047] in, This represents the future perspective anchor point corresponding to the nth candidate forward-looking route hypothesis; Indicates the future location after executing the route; Indicates the future orientation after executing this route; This indicates the observable range under the future location and orientation.

[0048] This future perspective anchor point is used to constrain the subsequent generative visual imagination model, ensuring that the generated future visual imagination view is consistent with the spatial endpoint and observation direction of the candidate route.

[0049] S22: Based on the future perspective anchor point, construct a generative visual imagination model, and input the current visual observation data, historical navigation trajectory, candidate forward route hypothesis and future perspective anchor point into the generative visual imagination model to generate a future visual imagination view corresponding to the candidate forward route hypothesis.

[0050] Specifically, the navigation agent is based on a future-oriented anchor point. A generative visual imagination model is constructed. This generative visual imagination model takes current visual observation data, historical navigation trajectories, candidate prospective route hypotheses, future viewpoint anchors, and the current set of predicates for sub-targets to be verified as input conditions to generate corresponding future visual imagination views.

[0051] The generative visual imagination model includes a visual encoding unit, a trajectory conditional encoding unit, a viewpoint anchor point encoding unit, a predicate conditional fusion unit, and a future view decoding unit. The visual encoding unit is used to decode current visual observation data. Extract current scene features; trajectory conditional coding unit is used to encode candidate look-ahead route hypotheses. The action sequence; the viewpoint anchor encoding unit is used to encode the future position and future orientation; the predicate conditional fusion unit is used to make the generated result focus on the predicates of the sub-targets that have not yet been completed; the future view decoding unit is used to output the future visual imagination view. .

[0052] Specifically, the visual coding unit is used to extract data from the current visual observation data. Extract current scene features from current visual observation data. This can include the current RGB image, depth map, semantic segmentation results, object detection results, and passable region detection results. The visual encoding unit encodes the above information to obtain the current scene features. :

[0053] in, This represents a visual coding unit.

[0054] The current scene features are used to describe the passageways, landmarks, obstacles, shelves, sorting gate signs, and passable directions that the agent has currently observed.

[0055] Trajectory conditional coding units are used to encode candidate look-ahead route hypotheses. The action sequence, and can be combined with historical navigation trajectory. Characterizing the movement trend of the candidate route relative to historical paths, the trajectory condition encoding unit encodes the candidate look-ahead route assumptions and historical navigation trajectories to obtain trajectory condition features:

[0056] in, This represents the trajectory condition features of the nth candidate look-ahead route hypothesis; This represents the trajectory condition encoding unit.

[0057] This trajectory condition feature is used to enable the model to distinguish different candidate routes, such as the future view after turning right, the future view after continuing forward, and the future view after going around to the left.

[0058] The viewpoint anchor encoding unit is used to encode future viewpoint anchors. The viewpoint anchor encoding unit encodes future viewpoint anchors to obtain viewpoint anchor features:

[0059] in, This represents the viewpoint anchor point feature corresponding to the nth candidate look-ahead route hypothesis; This represents the viewpoint anchor point encoding unit.

[0060] This viewpoint anchor feature is used to constrain the observation position, observation orientation, and field of view of the future visual imagination view, so as to avoid the generation result being inconsistent with the spatial endpoint of the candidate route.

[0061] The predicate conditional fusion unit is used to make the generated result focus on the predicates of the sub-targets that have not yet been completed. The predicate conditional fusion unit fuses the current scene features, trajectory condition features, viewpoint anchor features and the set of predicates of the sub-targets to be verified:

[0062] in, This represents the fusion condition feature corresponding to the nth candidate look-ahead route hypothesis; This represents the predicate condition fusion unit.

[0063] The predicate condition fusion unit weights the current scene features, trajectory condition features, and viewpoint anchor point features based on the target object, visual landmarks, spatial relationships, and negative constraints in the predicate of the sub-target to be verified, so that the future view generation process prioritizes the area related to the current navigation decision.

[0064] Future view decoding unit is used for fusion condition feature-based decoding Output future visual imagination view

[0065]

[0066] in, Indicates the candidate forward-looking route hypothesis The corresponding vision of the future; This represents the future view decoding unit.

[0067] Future Vision Imagination View It can be an RGB view, a semantic view, a panoramic view, or a low-resolution view with semantic labels. Its purpose is to provide subsequent steps with information on future semantic structure and to perform consistency verification, and it is not limited to photorealistic images.

[0068] Further overall Represents a generative visual imagination model; Model parameters representing generative visual imagination models; future visual imagination views It can be represented as:

[0069] in, Indicates the candidate forward-looking route hypothesis The corresponding vision of the future; This represents the current visual observation data; Indicates historical navigation trajectory; Indicates the candidate forward-looking route hypothesis; Indicates a future-oriented anchor point; This represents the set of predicates for the sub-targets currently to be verified.

[0070] Therefore, the future visual imagination view is not an unconditionally generated scene image, but a prediction result constrained by candidate routes, future perspectives, and language targets to be verified.

[0071] S23: Perform semantic analysis on the future visual imagination view to extract target objects, visual landmarks, region types, passable areas and spatial relationships to obtain future semantic structure information; Specifically, the navigation agent's vision of the future. Semantic parsing is performed to identify target objects, visual landmarks, region types, traversable areas, obstacles, and spatial relationships, and the future view in image form is converted into future semantic structure information. :

[0072] in, Indicates the candidate forward-looking route hypothesis The corresponding future semantic structure information; This represents a semantic parsing function.

[0073] S24: Based on the source, geometric continuity, and semantic clarity of the future visual imagination view, calculate the generation reliability, and combine the candidate forward route hypothesis, future perspective anchor point, future visual imagination view, future semantic structure information, and generation reliability into a future perspective evidence package.

[0074] Specifically, the navigation agent does not directly present a future visual vision. Instead of serving as definitive evidence, its generation reliability is further calculated. Generation reliability is used to characterize whether the future view is credible, consistent with the future viewpoint anchor, semantically clear, and meets the input condition constraints.

[0075] Generate reliability It can be represented as:

[0076] in, Represents a vision of the future. The reliability of its generation; Geometric reliability is used to measure the relationship between the future view and the future viewpoint anchor. Does it match? It represents semantic reliability and is used to measure whether targets, landmarks, and area types are clearly identifiable in future views; It represents the conditional consistency reliability, used to measure whether the future view is consistent with the current visual observation data, historical navigation trajectory, candidate look-ahead route assumptions, and predicates of the sub-targets to be verified; This represents the model's uncertainty reliability, used to measure the uncertainty of a generative visual imagination model under current input conditions. The lower the model uncertainty, the better. The higher.

[0077] Subsequently, the navigation agent combines candidate forward route hypotheses, future perspective anchors, future visual imagined views, future semantic structure information, and generation reliability into a future perspective evidence package:

[0078] in, Indicates the candidate forward-looking route hypothesis A future-oriented evidence package.

[0079] Through the above steps, the navigation agent can construct future visual evidence of "what might be seen after traveling along the candidate route" before actually executing the candidate route. Unlike directly scoring candidate routes, this step encapsulates the candidate route, future position, future orientation, language sub-target, and generation reliability into a future perspective evidence package, enabling subsequent steps to distinguish between different situations, thereby providing a calculable and interpretable basis for self-correction during the inference stage.

[0080] S3: Verify the consistency of the future perspective evidence package with the sub-target predicate, current visual observation data, and historical navigation trajectory to obtain a visual language historical consistency score; this step includes the following sub-steps: S31: Match the future semantic structure information with the multiple sub-target predicates to obtain language target consistency; Specifically, the navigation agent will use future semantic structure information With the current set of predicates for the sub-targets to be verified Perform item-by-item matching to determine whether the target object, visual landmark, spatial relationship, and action progress required by the natural language command appear in the future perspective, and at the same time determine whether the negative constraints are violated.

[0081] The set of predicates for the current sub-target to be verified is known to be:

[0082] Each sub-target predicate is represented as:

[0083] For each sub-target predicate Calculate its future semantic structure Satisfaction in :

[0084] in, Indicates the candidate forward-looking route hypothesis The degree of satisfaction of the predicate of the k-th sub-goal from the future perspective; This represents the target object matching function, used to determine whether the target object appears in the future view; This represents a visual landmark matching function used to determine whether a specified landmark appears in a future view; This represents a spatial relationship matching function, used to determine whether a target object or landmark satisfies relationships such as "right side", "left side", "front", and "after passing". This represents the action progress matching function, used to determine whether the action corresponding to the candidate route advances the sub-target towards completion; This represents the sequence constraint matching function, used to determine whether future views conform to the order specified in the instructions; This represents a negative constraint matching function, used to determine whether the future view has not entered an error region, dead end, or duplicate region; to The weights of each matching item.

[0085] Subsequently, candidate forward route hypotheses were calculated. Language goal consistency :

[0086] in, Indicates the candidate forward-looking route hypothesis Consistency of language goals; This represents the importance weight of the k-th sub-target predicate.

[0087] S32: Based on the current visual observation data, the candidate prospective route hypothesis and the future viewpoint anchor point, determine whether the future visual imagined view can be reasonably transferred from the current visual observation data through the candidate prospective route hypothesis, and obtain visual transfer consistency. Specifically, the navigation agent judges the future visual imagination view. Is it possible to obtain the current visual observation data? Along the candidate prospective route hypothesis This was achieved through a reasonable transfer. First, based on current visual observation data... Local topological memory and candidate forward-looking route hypotheses Predict the geometry and set of visible landmarks after executing the candidate route:

[0088] in, This represents the predicted geometric semantic structure obtained based on the current observations and candidate route projections; This represents the geometric projection function.

[0089] Then, the predicted geometric semantic structure will be... With future semantic structures Compare and calculate geometric deviation and semantic mutation deviation. :

[0090] in, Indicates geometric deviation; This indicates the future location predicted based on the candidate routes; Indicates the future position within the future perspective anchor point; This indicates the future orientation predicted based on the candidate routes; Indicates the future orientation within the future perspective anchor point; Indicates positional deviation; Indicates orientation deviation; This indicates a visible semantic structure deviation.

[0091] Further calculate semantic change bias :

[0092] in, This represents a semantic mutation detection function, used to determine whether the semantic changes from the current visual observation data to the future visual imagined view exceed the range of changes allowed by the candidate route.

[0093] Finally, visual transfer consistency was obtained. :

[0094] in, Indicates the candidate forward-looking route hypothesis Consistency of visual transfer; and These represent the weights of geometric deviation and semantic change deviation, respectively.

[0095] S33: Based on the historical navigation trajectory, visited areas and observed landmarks, determine whether the future semantic structure information conflicts with the historical navigation memory, and obtain the consistency of historical memory; Specifically, the navigation agent will use future semantic structure information Compared with historical navigation trajectory The visited areas, observed landmarks, and obstacle-avoided areas are compared to determine the candidate look-ahead route hypothesis. Will this lead to repeated visits, entry into already confirmed error areas, or future observations that contradict historical memory?

[0096] The degree of conflict in historical memory can be expressed as:

[0097] in, Indicates the candidate forward-looking route hypothesis The degree of conflict in historical memory; This indicates the degree of repeated visits, used to determine whether a future view corresponds to an already visited area; It indicates the degree of contradiction in directional relationships and is used to determine whether the future direction of a landmark conflicts with historical records; Indicates the degree of conflict between restricted or incorrect areas; It indicates the degree of conflict in the observed content and is used to determine whether landmarks appearing in future views contradict historically confirmed landmarks. It indicates the degree of conflict of the fallback path, and is used to determine whether the candidate route weakens or destroys the fallback path; to The weights of each conflicting term.

[0098] Calculating historical memory consistency based on the degree of historical memory conflict :

[0099] in, Indicates the candidate forward-looking route hypothesis Consistency of historical memory.

[0100] S34: Based on the consistency of language goals, visual transfer, and historical memory, obtain the visual language historical consistency score of the corresponding candidate prospective route hypothesis.

[0101] Specifically, the navigation agent will ensure consistency of language goals. Consistency of visual transfer Consistency with historical memory Combining candidate forward-looking route hypotheses Consistency verification results:

[0102] in, Indicates the candidate forward-looking route hypothesis Consistency verification results; This indicates consistency with language goals; Indicates visual transfer consistency; This indicates consistency in historical memory.

[0103] To further diagnose error risks, a consistency missing vector is constructed. :

[0104] in Indicates the candidate forward-looking route hypothesis Consistency missing vector; Indicates the degree of inconsistency between language goals; Indicates the degree of inconsistency in visual transfer; This indicates the degree of inconsistency in historical memory.

[0105] Consistency verification results and consistency missing vector Combining the results yields a visual language history consistency score. :

[0106] This represents the hypothesis of the nth candidate look-ahead route. The visual language historical consistency score.

[0107] Through the above steps, the navigation agent can verify the credibility of the imagined future visual view from three aspects: language target, visual transfer, and historical memory, before executing a candidate route. This step avoids making decisions solely based on the confidence level of the navigation backbone model's actions, instead breaking down the future view into computable verification evidence. As a result, the system can identify potential error sorting areas, incorrect channels, duplicate regions, or geometrically unreasonable future views that may result from candidate routes in advance, providing a clear basis for subsequent error risk calculation and self-correction decisions.

[0108] S4: Based on the visual language historical consistency score, generate the reliability and irreversible cost of candidate routes, and calculate the error risk of the candidate prospective route assumptions; this step includes the following sub-steps: S41: Calculate the irreversible cost based on the route length, deviation from the target, repeated visits, backtracking difficulty, collision risk, restricted area risk, and task time delay assumed by the candidate prospective routes; Specifically, the navigation agent makes assumptions for each candidate look-ahead route. Calculate the irreversible cost that may occur once the route is executed. The irreversible cost is used to characterize how much the agent needs to pay to correct the error if the candidate route is misjudged.

[0109] The irreversible cost can be expressed as :

[0110] in, Indicates the candidate forward-looking route hypothesis The irreversible cost; Indicates the cost of the route length; Indicates the degree of deviation from the target direction; Indicates the degree of repeated visits; This indicates the difficulty of rolling back to a safe node after execution; This indicates a risk of collision, congestion, or entering a restricted area; This indicates the delay caused by the timeliness of the logistics and repackaging tasks; to This indicates the weight of each cost item.

[0111] S42: Based on the visual language historical consistency score, generate the reliability and calculate the error risk of the corresponding candidate prospective route assumption in combination with the irreversible cost; Specifically, the navigation agent will use the visual language history consistency score obtained from S3. The generation reliability obtained from S2 And the irreversible costs of S41 To perform joint computation, firstly, computationally inconsistent terms are calculated. :

[0112] in, Indicates the candidate forward-looking route hypothesis The degree of inconsistency.

[0113] Secondly, the generation reliability obtained from S24 Calculate unreliable terms :

[0114] in, This indicates the degree of unreliability in generating future visual imagination views.

[0115] Then, calculate the candidate route error risk. :

[0116] in, Indicates the degree of inconsistency; Indicates the degree of unreliability in generation; This indicates an irreversible cost; This represents the coupling risk term between insufficient consistency and irreversible costs; to This represents the weight of each risk item. The coupled risk item is... When the future perspective of a candidate route is inconsistent and difficult to correct after execution, its error risk should be amplified. In other words, the risk of a low-consistency route can be low if it is only observed in place; however, if a low-consistency route enters a dead end, an error sorting area, or a region that is difficult to backtrack, the risk should be significantly increased.

[0117] S5: Based on the error risk, divide the candidate prospective route hypotheses into a verification candidate set and a rejection candidate set, and generate rejection reasons and self-correction decisions for the rejected candidate prospective route hypotheses; this step includes the following sub-steps: S51: Compare the error risk with a preset risk threshold, add candidate prospective route hypotheses with error risks not exceeding the preset risk threshold to the verified candidate set, and add candidate prospective route hypotheses with error risks exceeding the preset risk threshold to the rejected candidate set. Specifically, the navigation agent will assess the risk of errors in each candidate look-ahead route assumption. With preset risk threshold When comparing, When the risk threshold is not exceeded The corresponding candidate forward route hypothesis is added to the validated candidate set:

[0118] when If the risk exceeds a preset threshold, the corresponding candidate forward-looking route hypothesis will be added to the rejection candidate set.

[0119] in, This represents the candidate set that is validated from a future perspective. This represents the set of rejected candidates.

[0120] S52: For the candidate prospective route hypothesis in the rejection candidate set, determine the rejection reason based on its language goal consistency, visual transfer consistency, historical memory consistency, generation reliability and irreversible cost, and generate a self-correcting decision based on the rejection reason.

[0121] Specifically, for the set of rejected candidates Each candidate forward route hypothesis The navigation agent generates rejection reasons based on the primary sources of error risk:

[0122] in, Indicates the candidate forward-looking route hypothesis Reasons for refusal; This function represents a rejection reason diagnostic function. Indicates the degree of inconsistency; Indicates the degree of unreliability in generation; This indicates an irreversible cost; This represents a vector indicating a lack of consistency.

[0123] Among them, the consistency loss vector can include the degree of inconsistency in language goals, the degree of inconsistency in visual transfer, and the degree of inconsistency in historical memory.

[0124] In this case, the reasons for rejection can be broken down as follows: (1) inconsistent language goals; (2) unreasonable visual transfer; (3) conflict of historical memory; (4) unreliable generation of future view; and (5) excessive cost of irreversible candidate routes.

[0125] Subsequently, a self-correcting decision is generated based on the reasons for rejection. :

[0126] in, This indicates a self-correcting decision; This represents the mapping function from the rejection reason to the self-correcting decision.

[0127] In this case, the mapping relationships include: (1) if the reason for rejection is inconsistent language goals, a rerouting decision is generated; (2) if the reason for rejection is unreasonable visual transfer, a decision to observe in place or perform small-step verification is generated; (3) if the reason for rejection is conflict of historical memory, a decision to retreat to the previous safe node is generated; (4) if the reason for rejection is unreliable future view generation, a supplementary observation decision is generated; (5) if the reason for rejection is that the irreversible cost is too high, a decision to stop or proceed conservatively is generated.

[0128] For example, if the future view of the candidate route "turn right into the passage" displays a zone C sign, it will lead to a lower visual language history consistency score. If the error is low, the reason for rejection could be "inconsistent language goals," and the self-correction decision is to reselect a candidate route; if the future view and historical trajectory show that the route will circle back to the visited area, the reason for rejection is "historical memory conflict," and the self-correction decision is to backtrack or select an unvisited channel.

[0129] Through the steps described above, the navigation agent does not simply select the highest-scoring route based on the consistency score of the future view. Instead, it further combines the reliability generated by the future view with the irreversible cost of the candidate routes to calculate the error risk of the candidate routes. In particular, by coupling the risk term of insufficient consistency with irreversible costs, the system can identify candidate routes that appear uncertain and are difficult to correct once executed. As a result, the navigation agent can reject high-risk routes before actually entering the wrong room, dead end, or repetitive area, and generate targeted self-correcting decisions.

[0130] S6: When the candidate set is empty, a self-correction action is output based on the self-correction decision; when the candidate set is not empty, a target route is selected and the corresponding action is output. Finally, by executing the self-correction action or the corresponding action, new action data is obtained and written into the navigation memory as training sub-data for the generative visual imagination model. This step includes the following sub-steps: S61: When the verified candidate set is empty, output a self-correction action based on the self-correction decision; Specifically, if the candidate set is validated If empty, it means that none of the current candidate forward route assumptions have been verified by the future perspective, or that their error risk exceeds the preset risk threshold. In this case, the navigation agent does not directly execute the original candidate route, but instead uses the self-correcting decision generated by S52. Output self-calibration action. Self-calibration action It can be represented as:

[0131] in, This indicates the self-correction action of the output; This represents the function that generates the self-correcting action.

[0132] In this implementation, the self-correction actions include reselecting a route, reverting to the previous safe node, turning around to observe in place, advancing in small steps to obtain supplementary visual evidence, stopping, and conservatively advancing. For example, if the primary reason for rejection is inconsistent language goals, a route reselection action is output; if the primary reason for rejection is conflict in historical memory, a revert action is output; if the primary reason for rejection is insufficient reliability of future view generation or unreasonable visual transfer, an in-place observation or small-step verification action is output; if the primary reason for rejection is excessively high irreversible costs, a stop or conservatively advancing action is output.

[0133] S62: When the verified candidate set is not empty, based on the navigation backbone model confidence, consistency verification results and route execution cost, select the target candidate route from the verified candidate set and output the corresponding next navigation action. Specifically, the navigation agent determines the outcome by verifying the candidate set. Is it empty? If it is not empty, it means that there exists at least one candidate forward-looking route hypothesis that has been verified by a future perspective and the error risk does not exceed the preset risk threshold.

[0134] for Each candidate forward route hypothesis The navigation agent comprehensively considers the confidence of the navigation backbone model, the consistency score between visual and language history, error risk, and irreversible cost to calculate the target route selection score. :

[0135] in, Indicates the candidate forward-looking route hypothesis Target route selection score; This indicates the current navigation backbone model in the navigation state. The following hypotheses regarding candidate forward-looking routes Confidence level; to This indicates the weight of each item.

[0136] Subsequently, by verifying the candidate set Select target candidate route :

[0137] The corresponding action output from the target candidate route can be represented as:

[0138] in, This indicates the corresponding navigation action output; This represents the action decoding function.

[0139] S63: After executing the self-correction action or the corresponding action, new action data is acquired and written into the navigation memory as training sub-data of the generative visual imagination model; wherein, the action data serves as an auxiliary supervision label for training or updating the generative visual imagination model.

[0140] Specifically, the action data is subdivided into future perspective evidence packages, consistency verification results, rejection reasons, visual language historical consistency scores, error risks, rejection reasons, output actions, and new actual visual observation data.

[0141] After executing the navigation action, action sequence, or self-correction action, the navigation agent acquires new actual visual observation data. :

[0142] in, Indicates the execution of the output action. The actual visual observation data obtained later; express.

[0143] Subsequently, the navigation agent will include future-perspective evidence from this round of reasoning. Consistency verification results Visual language historical consistency score Error risk Reasons for refusal Output Action and new actual visual observation data Write to navigation memory:

[0144] in, This represents the navigation memory at the current moment; This indicates the updated navigation memory.

[0145] Furthermore, the navigation agent will memorize navigation information. As auxiliary supervisory labels, sub-data is constructed for training or updating generative visual imagination models. Generative visual imagination models are trained by using sub-data as auxiliary supervisory labels. Optimize the entire model to complete the closed loop.

[0146]

[0147] in, This represents the sub-data used to train or update generative visual imagination models; This represents the current visual observation data when generating a future visual imagined view; Indicates historical navigation trajectory; Indicates the candidate forward-looking route hypothesis; Indicates a future-oriented anchor point; This represents a generated visual vision of the future.

[0148] For candidate routes that have been validated and executed, if actual observations... Its future visual imagination view If the difference is small, it is treated as a positive sample; if the actual observation differs greatly from the future visual imagination, or if the route is rejected, the reason for rejection is written as an auxiliary label into the sub-data to prompt the generative visual imagination model to reduce the same type of erroneous imagination in similar situations.

[0149] Through the above steps, the navigation agent can output navigation actions validated by the future perspective when reliable candidate routes exist, and output self-correcting actions when all candidate routes are unreliable, thereby avoiding forcibly executing incorrect routes under high-risk conditions. Simultaneously, by writing the future perspective evidence package, visual language history consistency score, error risk, rejection reasons, and actual observation results into the navigation memory, and using the rejection reasons as sub-data to train or update the generative visual imagination model, the system forms a closed loop from future visual imagination, route validation, self-correcting decision-making to model feedback updates. This closed loop can progressively improve the reliability of future perspective predictions and reduce repetitive errors in subsequent navigation tasks.

[0150] Corresponding to the aforementioned embodiments of the VLN inference self-correction method based on future visual imagination verification, this application also provides embodiments of the VLN inference self-correction device based on future visual imagination verification.

[0151] Figure 2 This is a block diagram illustrating a VLN inference self-correction device based on future visual imagination verification according to an exemplary embodiment. The device includes: The candidate route generation module 1 is used to acquire current visual observation data, historical navigation trajectory and natural language navigation instructions, parse the natural language navigation instructions into multiple sub-target predicates that can be verified by future perspectives, and generate multiple candidate forward route hypotheses based on the current navigation state. The future perspective evidence construction module 2 is used to construct a generative visual imagination model for each candidate prospective route hypothesis, determine the corresponding future perspective anchor point to generate a future visual imagination view, extract semantic structure information from the view, calculate the reliability of future perspective generation, and obtain a future perspective evidence package. Consistency verification module 3 is used to verify the consistency of the future perspective evidence package with the sub-target predicate, the current visual observation data and the historical navigation trajectory respectively, and obtain the visual language historical consistency score; Error risk calculation module 4 is used to generate reliability and irreversible cost of candidate routes based on the visual language historical consistency score, and to calculate the error risk of candidate prospective route assumptions. The candidate segmentation and self-correction module 5 is used to segment the candidate prospective route hypothesis into a set of candidates that pass verification and a set of candidates that are rejected based on the error risk, and to generate rejection reasons and self-correction decisions for the rejected candidate prospective route hypotheses. The navigation decision output module 6 is used to output a self-correction action based on the self-correction decision when the candidate set is empty, and to select a target route and output the corresponding action when the candidate set is not empty. Finally, by executing the self-correction action or the corresponding action, new action data is obtained and written into the navigation memory as training sub-data of the generative visual imagination model.

[0152] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the VLN inference self-correction method based on future visual imagination verification as described above. Figure 3 The diagram shown is a hardware structure diagram of any data processing-capable device, including a VLN inference self-correction device based on future visual imagination verification provided by an embodiment of the present invention. (Except for...) Figure 3 In addition to the processor and memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0153] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the VLN inference self-correction method based on future visual imagination verification as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0154] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0155] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A VLN inference self-correction method based on future visual imagination verification, characterized in that, include: The system acquires current visual observation data, historical navigation trajectories, and natural language navigation instructions. It parses the natural language navigation instructions into multiple sub-target predicates that can be verified from a future perspective and generates multiple candidate forward route hypotheses based on the current navigation state. For each candidate prospective route hypothesis, a generative visual imagination model is constructed, and a future view anchor point corresponding to it is determined to generate a future visual imagination view. Semantic structure information is extracted from the view, and the reliability of future view generation is calculated to obtain a future view evidence package. The future perspective evidence package is verified for consistency with the sub-target predicate, current visual observation data and historical navigation trajectory to obtain a visual language historical consistency score. Based on the visual language historical consistency score, the reliability and irreversible cost of candidate routes are generated, and the error risk of the candidate prospective route assumption is calculated. Based on the error risk, candidate prospective route hypotheses are divided into a set of candidates to be verified and a set of candidates to be rejected, and rejection reasons and self-correction decisions are generated for the rejected candidate prospective route hypotheses. When the candidate set is empty, a self-correction action is output based on the self-correction decision. When the candidate set is not empty, a target route is selected from it and the corresponding action is output. Finally, by executing the self-correction action or the corresponding action, new action data is obtained and written into the navigation memory as training sub-data of the generative visual imagination model.

2. The VLN inference self-correction method based on future visual imagination verification according to claim 1, characterized in that, The system acquires current visual observation data, historical navigation trajectories, and natural language navigation instructions. It parses the natural language navigation instructions into multiple sub-target predicates that can be verified from a future perspective, and generates multiple candidate forward route hypotheses based on the current navigation state, including: Acquire current visual observation data, historical navigation trajectory, and natural language navigation commands to obtain a representation of the current navigation status; Based on the current navigation state representation, the natural language navigation instructions are semantically parsed to obtain a set of sub-target predicates that can be verified from a future perspective; Based on the current navigation state representation and the sub-target predicate set, multiple candidate action or candidate route nodes are generated. For each of the multiple candidate action or candidate route nodes, a look-ahead extension is performed to obtain multiple candidate look-ahead route hypotheses.

3. The VLN inference self-correction method based on future visual imagination verification according to claim 1, characterized in that, For each candidate prospective route hypothesis, a generative visual imagination model is constructed to determine its corresponding future viewpoint anchor point, generating a future visual imagination view. Semantic structure information is extracted from the view, and the reliability of the future viewpoint generation is calculated to obtain a future viewpoint evidence package, including: For each candidate forward route hypothesis, predict the future location, future orientation, and future observable range after executing the candidate forward route hypothesis, and obtain the future viewpoint anchor point; Based on the future perspective anchor point, a generative visual imagination model is constructed, and the current visual observation data, historical navigation trajectory, candidate forward route hypothesis and future perspective anchor point are input into the generative visual imagination model to generate a future visual imagination view corresponding to the candidate forward route hypothesis. Semantic analysis is performed on the imagined future visual view to extract target objects, visual landmarks, region types, passable areas, and spatial relationships, thereby obtaining future semantic structure information. Based on the source, geometric continuity, and semantic clarity of the imagined future visual view, the generation reliability is calculated, and the candidate forward route hypothesis, future perspective anchor point, imagined future visual view, future semantic structure information, and generation reliability are combined into a future perspective evidence package.

4. The VLN inference self-correction method based on future visual imagination verification according to claim 1, characterized in that, The future perspective evidence package is then compared with the sub-target predicate, current visual observation data, and historical navigation trajectory for consistency verification, resulting in a visual language historical consistency score, including: By matching future semantic structure information with multiple sub-target predicates, language target consistency is obtained; Based on the current visual observation data, the candidate prospective route hypothesis, and the future viewpoint anchor, it is determined whether the future visual imagined view can be reasonably transferred from the current visual observation data through the candidate prospective route hypothesis, thus obtaining visual transfer consistency. Based on the historical navigation trajectory, visited areas and observed landmarks, it is determined whether the future semantic structure information conflicts with the historical navigation memory, thus obtaining the consistency of historical memory. Based on the consistency of language goals, visual transfer, and historical memory, a visual-language historical consistency score is obtained for the corresponding candidate prospective route hypothesis.

5. The VLN inference self-correction method based on future visual imagination verification according to claim 1, characterized in that, Based on the visual language historical consistency score, reliability and irreversible costs of candidate routes are generated, and the error risk of candidate prospective route assumptions is calculated, including: Based on the route length, deviation from the target, repeated visits, backtracking difficulty, collision risk, restricted area risk, and task time delay assumed by the candidate prospective routes, calculate the irreversible cost. Based on the visual language historical consistency score, a reliability score is generated, and the error risk of the corresponding candidate prospective route assumption is calculated in conjunction with the irreversible cost.

6. The VLN inference self-correction method based on future visual imagination verification according to claim 1, characterized in that, Based on the aforementioned error risk, candidate prospective route hypotheses are divided into a set of candidates to be validated and a set of candidates to be rejected. Rejection reasons and self-correcting decisions are generated for the rejected candidate prospective route hypotheses, including: The error risk is compared with a preset risk threshold. Candidate prospective route hypotheses with error risks not exceeding the preset risk threshold are added to the verified candidate set, while candidate prospective route hypotheses with error risks exceeding the preset risk threshold are added to the rejection candidate set. For the candidate prospective route hypothesis in the rejection candidate set, the rejection reasons are determined based on its language goal consistency, visual transfer consistency, historical memory consistency, generation reliability and irreversible cost, and a self-correcting decision is generated based on the rejection reasons.

7. The VLN inference self-correction method based on future visual imagination verification according to claim 1, characterized in that, When the candidate set is empty, a self-correction action is output based on the self-correction decision. When the candidate set is not empty, a target route is selected and the corresponding action is output. Finally, by executing the self-correction action or the corresponding action, new action data is obtained and written into the navigation memory as training sub-data for the generative visual imagination model, including: When the candidate set that has passed verification is empty, a self-correction action is output based on the self-correction decision; When the verified candidate set is not empty, based on the navigation backbone model confidence, consistency verification results and route execution cost, a target candidate route is selected from the verified candidate set, and the corresponding next navigation action is output. After executing the self-correction action or the corresponding action, new action data is acquired and written into the navigation memory as training sub-data for the generative visual imagination model; wherein, the action data serves as auxiliary supervision labels for training or updating the generative visual imagination model.

8. A VLN inference self-correction device based on future visual imagination verification, characterized in that, include: The candidate route generation module is used to acquire current visual observation data, historical navigation trajectories and natural language navigation instructions, parse the natural language navigation instructions into multiple sub-target predicates that can be verified by future perspectives, and generate multiple candidate forward route hypotheses based on the current navigation state. The future perspective evidence construction module is used to construct a generative visual imagination model for each candidate prospective route hypothesis, determine the corresponding future perspective anchor point to generate a future visual imagination view, extract semantic structure information from the view, calculate the reliability of future perspective generation, and obtain a future perspective evidence package. The consistency verification module is used to verify the consistency of the future perspective evidence package with the sub-target predicate, the current visual observation data and the historical navigation trajectory, respectively, and obtain a visual language historical consistency score. The error risk calculation module is used to generate reliability and irreversible cost of candidate routes based on the visual language historical consistency score, and to calculate the error risk of candidate prospective route assumptions. The candidate segmentation and self-correction module is used to segment candidate prospective route hypotheses into a set of candidates to be verified and a set of candidates to be rejected based on the error risk, and to generate rejection reasons and self-correction decisions for the rejected candidate prospective route hypotheses. The navigation decision output module outputs a self-correction action based on the self-correction decision when the candidate set is empty, and selects a target route and outputs the corresponding action when the candidate set is not empty. Finally, it obtains new action data by executing the self-correction action or the corresponding action and writes it into the navigation memory as training sub-data for the generative visual imagination model.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction performs the method as described in any one of claims 1-7.

Images