Visual navigation model training method, system and equipment and readable storage medium
The visual navigation model training method addresses the lack of generalization in existing indoor navigation technologies by using offline reinforcement learning to predict navigation actions from video frames, enabling efficient and adaptable indoor navigation without maps or sensors.
Patent Information
- Application Number
- CN202410983062.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-22
AI Technical Summary
Existing 3D indoor visual navigation methods lack generalization ability, requiring additional training or environmental interaction when migrating to new scenarios, hindering rapid deployment in real navigation tasks.
A visual navigation model training method that utilizes offline reinforcement learning to predict navigation actions based on video frames and target images, without relying on maps or sensors, by determining action types, integrating video frames and observation images, and using a pre-trained model to output navigation decisions.
Enables autonomous navigation in complex indoor environments by understanding the layout and reaching target objects using offline video input, demonstrating strong generalization and scalability without retraining.
Smart Images

Figure CN119068231A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a visual navigation model training method, system, device and readable storage medium. Background Art
[0002] In the 3D indoor visual navigation task, it is necessary to solve the problem of how to navigate to the target or destination by only observing the indoor scene features in the absence of a global map.
[0003] Traditional solutions can be divided into three main branches. The first is to achieve navigation with the help of relatively mature modular visual models such as target detection, key point matching, depth estimation, etc., assisted by simultaneous positioning and mapping technology and planning algorithms. The second is to imitate and learn navigation strategies from existing navigation data sets in indoor scenes. The third is to allow the intelligent agent to be directly placed in the environment for reinforcement learning, and learn to navigate to the target location through exploration and utilization after multiple trial and error.
[0004] Although these methods have demonstrated satisfactory navigation capabilities in many scenarios, their common problem is the lack of generalization ability. When migrating to new scenarios, additional model training or additional interaction with the environment is required, which is not conducive to rapid deployment in real navigation tasks. Summary of the invention
[0005] The main purpose of the present invention is to provide a visual navigation model training method, system, device and readable storage medium, which do not rely on maps, depth cameras, various position and posture sensors, and can complete the task of understanding the indoor layout and navigating to the target location based on a given target object image based only on offline video and target image input.
[0006] In order to achieve the above objectives, the present application provides the following technical solutions:
[0007] According to a first aspect of an embodiment of the present application, a visual navigation model training method is provided, the method comprising:
[0008] Determine the action type in each video frame based on the training video;
[0009] Input all video frames and corresponding action types, target images and current observation images into a pre-trained visual navigation model to output navigation decision information; the target image is a video frame in the training video selected by the user, and the target image includes a navigation target;
[0010] Based on all video frames and corresponding action types, the target image, the current observed image, and the navigation decision information, a visual navigation model is trained through an offline reinforcement learning algorithm so that the visual navigation model predicts the navigation action of reaching the navigation target in the specified image based on the specified image.
[0011] Optionally, the step of inputting all video frames and corresponding action types, target images, and current observation images into a pre-trained visual navigation model to output navigation decision information includes:
[0012] Input all video frames and corresponding action types into the pre-trained visual navigation model to obtain a representation sequence;
[0013] Processing the current observation image and the target image through a visual encoder, and generating an observation hidden state in combination with a corresponding action type;
[0014] Integrate the representation sequence of each currently observed image and the observed hidden state to obtain a cyclic hidden state;
[0015] The cyclic hidden states of all currently observed images are aggregated and input into a multilayer perceptron to output navigation decision information, which includes quality parameters of state-action pairs, end signal parameters and predicted guidance actions.
[0016] Optionally, the step of inputting all video frames and corresponding action types into a pre-trained visual navigation model to obtain a representation sequence includes:
[0017] Calling the pre-trained visual navigation model, processing all video frames and corresponding action types at a time step, and obtaining a video frame action sequence;
[0018] Position encoding is performed on the video frame action sequence in combination with the hidden state to obtain a position encoding result;
[0019] The position encoding result is input into a bidirectional multi-head attention module to generate a representation sequence.
[0020] Optionally, the processing of the current observation image and the target image by a visual encoder and generating an observation hidden state in combination with a corresponding action type includes:
[0021] Inputting the current observation image and the target image into the visual encoder to extract visual features;
[0022] The visual features and the corresponding action types are combined to generate an observed hidden state corresponding to each action type.
[0023] Optionally, based on all video frames and corresponding action types, the target image, the current observation image, and the navigation decision information, a visual navigation model is trained by an offline reinforcement learning algorithm, including:
[0024] All video frames, corresponding action types and binary rewards constitute the training dataset;
[0025] For each video frame in the training data set, the visual navigation model is trained through an offline reinforcement learning algorithm based on the target image, the current observation image and the set action threshold to update the navigation decision information.
[0026] Optionally, determining the action type in each video frame based on the training video includes:
[0027] Inputting the video frame sequence of the training video into the optical flow model to obtain an optical flow map corresponding to two adjacent video frames;
[0028] Filter out dominant vectors from all optical flow maps based on set amplitudes;
[0029] An average direction of the dominant vector is calculated, and an action type is determined based on the average direction, the action type comprising discrete navigation actions in different directions.
[0030] Optionally, the method further comprises:
[0031] Inputting the scene video into the visual navigation model for initialization;
[0032] Inputting a designated image into the visual navigation model to identify a navigation target in the designated image;
[0033] The visual navigation model predicts a navigation strategy to reach a navigation target in the specified image.
[0034] According to a second aspect of an embodiment of the present application, a visual navigation model training system is provided, the system comprising:
[0035] An action type module, used to determine the action type in each video frame based on the training video;
[0036] A training reasoning module is used to input all video frames and corresponding action types, target images and current observation images into a pre-trained visual navigation model to output navigation decision information; the target image is a video frame in the training video selected by the user, and the target image includes a navigation target;
[0037] The offline reinforcement learning training module is used to train the visual navigation model through an offline reinforcement learning algorithm based on all video frames and corresponding action types, the target image, the current observation image, and the navigation decision information, so that the visual navigation model predicts the navigation action of reaching the navigation target in the specified image based on the specified image.
[0038] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.
[0039] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described in the first aspect above.
[0040] In summary, the embodiments of the present application provide a visual navigation model training method, system, device and readable storage medium, by determining the action type in each video frame based on the training video; inputting all video frames and the corresponding action type, target image and current observation image into the pre-trained visual navigation model to output navigation decision information; the target image is a video frame in the training video selected by the user, and the target image includes a navigation target; based on all video frames and the corresponding action type, the target image, the current observation image, and the navigation decision information, the visual navigation model is trained through an offline reinforcement learning algorithm, so that the visual navigation model predicts the navigation action of reaching the navigation target in the specified image according to the specified image. It does not rely on maps, depth cameras, and various position and posture sensors, and can complete the task of understanding the indoor layout and navigating to the target position according to a given target object image only based on offline video and target image input. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0042] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.
[0043] Figure 1 A flow chart of a visual navigation model training method provided in an embodiment of the present application;
[0044] Figure 2 A schematic diagram of a 3D visual indoor navigation framework based on offline video training provided in an embodiment of the present application;
[0045] Figure 3 A flowchart for determining navigation behavior in an input video by using an optical flow model provided in an embodiment of the present application;
[0046] Figure 4 A video navigation decision network structure diagram provided for an embodiment of the present application;
[0047] Figure 5 A diagram of a multi-layer perceptron network architecture provided in an embodiment of the present application;
[0048] Figure 6 A residual network structure diagram provided in an embodiment of the present application;
[0049] Figure 7 A diagram of the Transformer network architecture based on the attention mechanism provided in the embodiment of the present application;
[0050] Figure 8 A block diagram of a visual navigation model training system provided in an embodiment of the present application;
[0051] Fig. 9 A structural diagram of an electronic device provided in an embodiment of the present application is shown;
[0052] Fig.10 A diagram showing a computer-readable storage medium provided by an embodiment of the present application.
[0053] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0056] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0057] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0058] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0059] Inspired by human navigation behavior, the embodiment of the present application proposes a more natural "video navigation" context learning (ICL) paradigm. The goal of video navigation is to find the target navigation object from the input video and complete the navigation task. The input video is a video of random walking in the target indoor scene from a first-person perspective.
[0060] During the model training process, the embodiment of the present application accesses video data of first-person traversal of different scenes to train the contextual visual navigation strategy in an offline manner. Compared with the traditional strategy based only on the current observation input, this visual navigation strategy has stronger generalization ability and can be directly extended to new scenes without retraining. It also has stronger large-scale scalability and can be applied to navigation tasks in more scenes by simply expanding the scale of the video data set without changing other processes and methods.
[0061] Figure 1 A visual navigation model training method provided in an embodiment of the present application is shown, the method comprising:
[0062] Step 101: determining the action type in each video frame based on the training video;
[0063] Step 102: inputting all video frames and corresponding action types, target images and current observation images into a pre-trained visual navigation model to output navigation decision information; the target image is a video frame in the training video selected by the user, and the target image includes a navigation target;
[0064] Step 103: Based on all video frames and corresponding action types, the target image, the current observed image, and the navigation decision information, a visual navigation model is trained through an offline reinforcement learning algorithm so that the visual navigation model predicts the navigation action of reaching the navigation target in the specified image based on the specified image.
[0065] In a possible implementation, in step 101, determining the action type in each video frame based on the training video includes:
[0066] The video frame sequence of the training video is input into the optical flow model to obtain an optical flow map corresponding to two adjacent video frames; the dominant vector is screened out from all the optical flow maps based on a set amplitude; the average direction of the dominant vector is calculated, and the action type is determined based on the average direction, and the action type includes discrete navigation actions in different directions.
[0067] In a possible implementation, in step 102, inputting all video frames and corresponding action types, target images, and current observation images into a pre-trained visual navigation model to output navigation decision information includes:
[0068] All video frames and corresponding action types are input into a pre-trained visual navigation model to obtain a representation sequence; the current observation image and the target image are processed by a visual encoder, and an observation hidden state is generated in combination with the corresponding action type; the representation sequence and the observation hidden state of each current observation image are integrated to obtain a cyclic hidden state; the cyclic hidden states of all current observation images are aggregated and input into a multi-layer perceptron to output navigation decision information, wherein the navigation decision information includes quality parameters of state-action pairs, end signal parameters and predicted guidance actions.
[0069] In a possible implementation, inputting all video frames and corresponding action types into a pre-trained visual navigation model to obtain a representation sequence includes:
[0070] The pre-trained visual navigation model is called to process all video frames and corresponding action types at a time step to obtain a video frame action sequence; the video frame action sequence is combined with a hidden state to perform position encoding to obtain a position encoding result; the position encoding result is input into a bidirectional multi-head attention module to generate a representation sequence.
[0071] In a possible implementation, processing the current observation image and the target image by a visual encoder and generating an observation hidden state in combination with a corresponding action type includes:
[0072] The current observation image and the target image are input into the visual encoder to extract visual features; the visual features and the corresponding action types are combined to generate an observation hidden state corresponding to each action type.
[0073] In a possible implementation, in step 103, based on all video frames and corresponding action types, the target image, the current observation image, and the navigation decision information, a visual navigation model is trained by an offline reinforcement learning algorithm, including:
[0074] All video frames, corresponding action types and binary rewards constitute a training data set; for each video frame in the training data set, the visual navigation model is trained through an offline reinforcement learning algorithm based on the target image, the current observation image and the set action threshold to update the navigation decision information.
[0075] In a possible implementation, the method further includes: inputting a scene video into the visual navigation model for initialization; inputting a designated image into the visual navigation model to identify a navigation target in the designated image; and the visual navigation model predicting a navigation strategy for reaching the navigation target in the designated image.
[0076] The embodiment of the present application combines traditional path planning methods and reinforcement learning algorithms. By estimating the relative position of the intelligent agent, an abstract sparse topology graph of the scene is established based on key positions, and the connection relationship is defined according to the actual physical distance between the nodes, a hierarchical navigation strategy is formed to decompose long-distance navigation problems into navigation problems of short-term goals.
[0077] The training and application scenarios provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0078] In the application scenario of the embodiment of the present application, a camera mounted on a mobile device is used to surround a set scene for a set duration at a set frame rate to collect a number of video frames, which will be used as basic data for subsequent processing and training. For example, an RGB camera mounted on a smart car is used to surround a closed indoor home scene for 30 seconds at a frame rate of 30 frames per second, and a total of 900 frames of video are collected.
[0079] Next, the above video frames are processed. In the first stage, each two adjacent frames are input into the optical flow model to predict the navigation action. In the second stage, a video frame is randomly selected as the target image, and the scene object described by the target image is used as the navigation target. All video frames (as background information of the environment), target images and the current observation image (i.e., current frame) are input into the decision model, which will use this information to output the final navigation strategy.
[0080] In the third stage, the model is trained using offline reinforcement learning BCQ, and the decision model learns how to select the optimal navigation action based on the current observation and target image. The navigation strategy output by the decision model will guide the mobile device to navigate in the indoor environment until it reaches the scene object described by the target image. Usually, there is a reward mechanism in reinforcement learning to evaluate the behavior of the agent. In this scenario, the reward mechanism is to give a positive reward when successfully navigating to the scene object described by the target image, otherwise a negative reward or zero reward is given. As the training progresses, the decision model will continuously optimize its strategy to better adapt to different indoor environments and target images.
[0081] In this way, mobile devices can achieve autonomous navigation in complex indoor environments, find and reach target objects. This method has broad application potential in areas such as robot navigation and automated home systems.
[0082] Figure 2 The overall framework schematic diagram of the visual navigation decision-making provided by the embodiment of the present application is shown as follows:
[0083] Phase 1: Determine navigation behavior in the input video through an optical flow model.
[0084] Figure 3A flowchart of determining navigation behavior in an input video by an optical flow model is shown, specifically including:
[0085] Step 1: Transform the video sequences f1, f2, ... T Input the pre-trained optical flow prediction model GMFlow to obtain the optical flow map corresponding to adjacent video frames. By comparing two consecutive frames in the video sequence, the displacement of each pixel is calculated to generate the optical flow map corresponding to adjacent video frames.
[0086] Optical flow refers to the apparent change in the pixel intensity pattern caused by the movement of an object in an image sequence. It describes the motion pattern of the surface points of an object in an image as it moves in a scene. That is, the horizontal and vertical displacement of each pixel from one image to another. Adjacent video frames refer to two consecutive frames in a video, such as the first frame and the second frame, the second frame and the third frame, etc. By comparing two adjacent video frames, the displacement of each pixel between the two frames is calculated. The optical flow prediction model GMFlow outputs an optical flow map, in which each vector represents the displacement of the corresponding pixel from the first frame to the second frame.
[0087] Step 2: Filter out dominant vectors from the optical flow map to determine the action behavior in the video. For example, filter out the vectors with the top 10% magnitude in the optical flow map as representative dominant vectors.
[0088] Dominant vectors have different meanings in different fields, but usually refer to vectors that have a greater influence or a decisive role in a set of vectors. In mathematics and physics, dominant vectors refer to vectors that dominate in a vector space, and they determine the main direction or characteristics of the space.
[0089] Step 3: Calculate the average direction of the dominant vector and add action labels to determine the discrete navigation actions of left, right, and straight.
[0090] Discrete action Discrete action refers to the limited number of well-defined action options that an agent can take when interacting with its environment.
[0091] The second stage: This stage is the forward reasoning process of the decision model.
[0092] A video frame is randomly selected from the video segment as the target image, the scene object described by the image is used as the navigation target, all video frames are used as context information, and the target image and the current observation image are jointly input into the decision model to output the final navigation strategy.
[0093] Figure 4 The structure and workflow of the visual navigation decision network based on video input are shown. The network is trained through reinforcement learning to achieve indoor navigation. All video frames and predicted actions are input into the visual navigation decision model, and the decision model is trained through batch-constrained Q-Learning (BCQ) offline reinforcement learning algorithm, so that the decision model can output reasonable navigation actions based on current observations. Specifically, it includes:
[0094] Step 1: All video frames f and predicted actions a are used as input data into the visual navigation decision model. These input data represent the video information observed by the agent from the first-person perspective and the actions predicted based on this information.
[0095] Step 2: Combine the video frame-action sequence with the hidden state and perform positional encoding. At time step t, the complete video frame-action sequence combined with the hidden state h is processed by positional encoding. The hidden state is calculated using the model parameters and input data.
[0096] Position encoding is used in sequence models to provide the position information of each element (e.g., video frame) in the sequence.
[0097] Step 3: Input the position encoded result into the Bidirectional Multi-Head Self Attention module to obtain the representation sequence.
[0098] The bidirectional multi-head attention module is able to process information and capture long-range dependencies within the sequence. The bidirectional multi-head attention module is a component in the visual navigation decision model, not an independent module. It is embedded in the architecture of the entire visual navigation decision model to process input data, especially serialized data such as video frames and action sequences. It works in conjunction with other components (such as position encoding, visual encoder, multimodal multi-head cross attention module, and multi-layer perceptron) to complete the navigation task.
[0099] Step 4: Input the current observation image o and the given target image g into the visual encoder for processing to extract visual features; the visual features are combined with the predicted navigation actions to generate the observation hidden state.
[0100] After the current observation image o and the given target image g are processed by the visual encoder, they are combined with the navigation action to generate the observation hidden state. For example, three navigation actions are fused separately to obtain three observation hidden states. If there are multiple possible navigation actions, multiple observation hidden states can be generated. For example, left, right, and straight actions can generate three different observation hidden states respectively.
[0101] Step 5: The representation sequence and the observed hidden state are jointly input into the Multi-Modal Multi-Head Cross Attention module to obtain the Token Recurrence, which is then used in the next iteration.
[0102] The multimodal multi-head cross-attention module can handle the information interaction between different modalities (such as visual information and action information), enhance the model's ability to integrate different types of data, and generate cyclic hidden states, which will participate in the next iteration process.
[0103] Step 6: All the cyclic hidden states are aggregated to form a comprehensive state representation. This aggregated state representation is input into a multi-layer perceptron (MLP) to output decision information; the decision information includes: Q value (evaluating the quality of the current state-action pair, that is, predicting the cumulative reward that may be obtained after taking a certain action), end signal d (indicating whether the agent should end the current task or continue navigation), and strategy π (guiding the agent to choose the next action based on the current state and the learned strategy).
[0104] Phase 3: Use the offline reinforcement learning algorithm BCQ to train the visual navigation decision model.
[0105] The training dataset consists of navigation videos in different environments, navigation actions predicted by optical flow, and binary rewards. The BCQ algorithm allows the model to learn from a static dataset that has never interacted with the environment. BCQ restricts the set of action choices to ensure that policy updates only occur on actions with higher probabilities. The agent learns how to navigate to the target location in an indoor environment by observing the environment (video frames) and predicting actions. In this way, the agent can gradually improve its decision-making ability to adapt to different indoor environments and tasks.
[0106] The reinforcement learning design, offline reinforcement learning training process and strategy network architecture involved in the embodiments of this application are as follows:
[0107] Part I: Reinforcement Learning Design. Partially Observable Markov Decision Processes (POMDPs) provide a powerful and general mathematical framework that captures the essence of sequential decision making and can be represented as a seven-tuple (S, A, P, R, Ω, O, γ), where:
[0108] 1.S represents the state space, which is the set of all possible states s.
[0109] 2.A represents the action space, which is the set of all possible actions a.
[0110] 3.P represents the state transition probability, P(s′|s, a) represents the probability of transitioning to state s′ after taking action a in state s.
[0111] 4. R represents the reward function, and R(s, a, s′) represents the reward obtained after taking action a in state s and transitioning to state s′.
[0112] 5.Ω represents the observation space, the set of all possible observations o.
[0113] 6. O represents the conditional observation probability, where O(o|s′, a) represents the probability of observing o after taking action a to transition to state s′.
[0114] 7.γ represents the discount factor, γ∈[0,1).
[0115] In the embodiment of the present application, all RGB images observed by the agent constitute the observation space, the set of navigation actions of moving forward, turning left, turning right, or stopping constitutes the action space, the reward function indicates whether the goal is achieved, and the overall goal of reinforcement learning R(θ) is defined as the expectation of the discounted reward and the T-step navigation task γ:
[0116]
[0117] For deep reinforcement learning, a policy is a mapping from state to action, denoted as π(a|s), which represents the probability of taking action a in state s. The goal of reinforcement learning is to learn the optimal policy π that maximizes the expected cumulative reward. * :
[0118]
[0119] Where R is the reward function, π is the optimization strategy, T represents the length of a training set, and γ∈[0, 1) is the discount factor used to adjust the impact of future rewards on current decisions.
[0120] In order to solve the optimal strategy, the embodiment of the present application uses a value function, which satisfies the Bellman equation. The present invention uses a value function V(s), which satisfies the Bellman equation defined as follows:
[0121]
[0122] Through the Bellman equation, dynamic programming or iterative methods can be used to solve the optimal value function V * .
[0123] Part II: Offline reinforcement training process design.
[0124] Offline reinforcement learning emphasizes policy learning based on static interaction data without further interaction with the environment, and static data sets lack optimality guarantees. In this case, heterogeneous online reinforcement learning and imitation learning, which are closely related to offline reinforcement learning, are not applicable. Imitation learning is difficult to learn robust policies from limited and incomplete trajectories under the condition of restricting environmental interactions, and heterogeneous online reinforcement learning has the problem of policy offset between the data distribution of the data set and the data distribution of the current policy. Batch-constrained Q-Learning (BCQ), as an offline reinforcement learning method, effectively alleviates the inference error problem caused by the inconsistency between the data distribution in the experience pool and the current policy, and can learn robust navigation policies from suboptimal navigation data.
[0125] In the embodiment of the present application, the navigation videos in different environments, the navigation actions predicted by the optical flow and the binary rewards constitute an offline reinforcement learning training data set, which is trained using the BCQ algorithm, and the derivation and update method is as follows:
[0126]
[0127] The above binary reward value indicates whether the task is completed, 0 or 1. Usually in reinforcement learning, there is a reward mechanism to evaluate the behavior of the agent. In this scenario, the reward mechanism is to give positive rewards when successfully navigating to the scene object described by the target image, otherwise negative rewards or zero rewards are given. As training progresses, the decision model will continuously optimize its strategy to better adapt to different indoor environments and target images.
[0128] The reward setting The action selection set complies with the following constraints:
[0129]
[0130] The core of batch constrained Q learning is to use the Bellman equation to update Where g is the target image, o is the current observed image, and a is the predicted action type. Represents the i-th trajectory in the dataset, representing each independent task or scenario considered during training. Each trajectory contains a series of states, actions, and rewards used to train and update the agent's strategy.
[0131] At the same time, the constraints of the action selection set are added on the basis of the Bellman equation. The minimum selection threshold β is set, and the policy update is limited to the action set whose normalized probability is greater than the threshold.
[0132] Part III: Deep learning network architecture. The embodiment of the present application adopts a Transformer network as the core backbone network structure, a residual network as a visual feature extractor, and a multi-layer perceptron (MLP) as an output head for offline reinforcement learning to output a state action value function Q, a termination signal d, and a decision network for a strategy π.
[0133] (1) Multilayer Perceptron MLP. The output head uses a fully connected neural network consisting of one input layer, one output layer, and two hidden layers. The input layer contains n D Each of these two hidden linear layers has an n neuron dimensional feature space, the output layer will be n neuron Neurons mapped to out D neurons. LeakyReLU activation function is used between each linear layer. Figure 5 The multi-layer perceptron network architecture in the embodiment of the present application is shown, including:
[0134] 1. Input layer: The dimension of this layer is equal to the state space dimension (n D ), i.e., the state observation of the agent.
[0135] 2. Hidden Layer 1: A fully connected layer followed by a LeakyReLU activation function.
[0136] 3. Hidden Layer 2: Another fully connected layer, also followed by a LeakyReLU activation function.
[0137] 4. Output layer: A fully connected layer whose dimension is equal to the action space dimension of the actor network (out D ), or a single output neuron of the critic network.
[0138] (2) Residual Network. Residual Network (ResNet) uses skip connections to allow input signals to bypass one or more layers and pass directly to the output, effectively alleviating the gradient vanishing and gradient exploding problems in deep neural networks. Figure 6 The 18-layer ResNet shown acts as a feature extractor to convert each input video frame into a lower-dimensional visual representation for subsequent deep feature extraction.
[0139] (3) Transformer network. Transformer has shown strong modeling capabilities in long sequence understanding and prediction. In the embodiment of the present application, a special Transformer network Bert is used to further extract the causal relationship of the sequence representation obtained by the residual network and then form the final navigation strategy. Its structure is as follows: Figure 7The architecture of the model starts with an input layer that accepts n D Data of the specified dimension.
[0140] First, encode the input data as follows:
[0141] X′=X+PositionalEncoding(X)
[0142] The input of the Transformer is X, where X∈R N×C×D ; N represents the batch size, C represents the length of the sequence, which is the state in the example of the embodiment of the present application; D is the input dimension. The calculation formula for each element (i, j) is:
[0143]
[0144] Next, X' is passed to Y∈R through the Transformer encoder N×C×D :
[0145] Y = TransformerEncoder(X′)
[0146] The Transformer encoder layer consists of 2 layers. Each of them consists of two main sub-layers: a multi-head self-attention mechanism with nhead heads and a positional feed-forward network. The hidden size of the feed-forward network is set to nneuron, which defines the size of the intermediate feature space of each layer of the Transformer encoder.
[0147] Specifically, the Transformer encoder integrates a multi-head self-attention mechanism. For each head h, the embodiment of the present application has a learned weight set W Q h , W K h , W V h Used to calculate queries, keys, and values.
[0148] For each input X', its query, key, and value are first calculated. In the Transformer model, the attention representation is calculated by assigning a weight to each position in the input sequence to represent the input sequence. This weight is calculated through the self-attention mechanism, which treats each position in the input sequence as a query, key, and value to calculate the similarity between them. Then, by applying the weight of each position to the elements in the input sequence, a global representation can be calculated that captures the important features and dependencies in the input sequence.
[0149]
[0150] W Q h , W K h , W V h represent the learnable weight matrices of query, key, and value respectively, and X' is the input data.
[0151] When the query, key and value of each head are calculated, the attention mechanism representation is calculated as:
[0152]
[0153] Q, K, V represent the learned representations of query, key, and value, respectively, and D is the given scaling weight.
[0154] Afterwards, the embodiment of the present application connects the attention representations of all heads and calculates them through a linear layer:
[0155] MultiHead(Q,K,V)=concat(Attention1,...,Attention H ) O
[0156] In the above matrix, Attention_i represents the attention value output by the i-th head, and concat represents the concatenation operation. After obtaining the output of the multi-head attention mechanism, the embodiment of the present application passes this result matrix Y through the feedforward neural network to generate a new output, represented as Z∈R N×C×D :
[0157] Z=FFN(Y)
[0158] Finally, after mapping the multi-layer perceptron output head, the decision network output, the state action value function Q and the termination signal d are used for offline reinforcement learning, and the output navigation strategy π is used to perform navigation tasks in a given environment. The embodiment of the present application adds an additional hidden state on the basis of the Bert structure to capture sequence history information and participate in loop iteration.
[0159] In summary, the first stage uses an optical flow model, which takes every two adjacent frames as input to predict navigation actions. In the second stage, a video frame is randomly selected from the captured video segment as the target image, and the scene object described by the target image is used as the navigation target. All video frames (as background information of the environment), the target image and the current observation image (i.e., the current frame) are input into the decision model, and the decision model will use this information to output the final navigation strategy. Further, the model is trained using BCQ. During the training process, the decision model will learn how to select the optimal navigation action based on the current observation and target image. The navigation strategy output by the decision model will guide the mobile device to navigate in the indoor environment until it reaches the scene object described by the target image.
[0160] Phase 4: Application.
[0161] Step 1: Deploy the trained visual navigation decision model to a mobile device with an RGB camera in a new indoor environment.
[0162] Step 2: Input the recorded indoor walking video to initialize the algorithm.
[0163] Step 3: Driven by the algorithm, the mobile device continuously interacts with the environment, continuously observes the current image through the camera, and outputs a navigation strategy based on the decision model in combination with the target image, and finally navigates to the target location.
[0164] The embodiment of the present application proposes an innovative video navigation solution, which for the first time realizes the completion of navigation tasks without additional training by using only video clips in unfamiliar environments. As the first method that does not rely on traditional maps, depth cameras or position and posture sensors, it can be easily deployed to mobile devices by using only offline videos and target images as input to achieve first-person 3D visual indoor navigation. After inputting the indoor scene video, it quickly understands the indoor layout and navigates to the target object image position specified by the user. In addition, by combining the optical flow model with the offline reinforcement learning framework, it breaks through the limitations of traditional offline reinforcement learning, allowing the system to learn and optimize navigation strategies only from first-person observation images, significantly improving the flexibility and adaptability of the navigation method.
[0165] In summary, the embodiment of the present application provides a visual navigation model training method, which determines the action type in each video frame based on the training video; inputs all video frames and the corresponding action type, target image and current observation image into the pre-trained visual navigation model to output navigation decision information; the target image is a video frame in the training video selected by the user, and the target image includes a navigation target; based on all video frames and the corresponding action type, the target image, the current observation image, and the navigation decision information, the visual navigation model is trained through an offline reinforcement learning algorithm, so that the visual navigation model predicts the navigation action of reaching the navigation target in the specified image according to the specified image. It does not rely on maps, depth cameras, and various position and posture sensors. It can complete the task of understanding the indoor layout and navigating to the target position according to a given target object image based on offline video and target image input.
[0166] Based on the same technical concept, the present application embodiment also provides a visual navigation model training system, such as Figure 8 As shown, the system comprises:
[0167] An action type module 801 is used to determine the action type in each video frame based on the training video;
[0168] The training reasoning module 802 is used to input all video frames and corresponding action types, target images and current observation images into the pre-trained visual navigation model to output navigation decision information; the target image is a video frame in the training video selected by the user, and the target image includes a navigation target;
[0169] The offline reinforcement learning training module 803 is used to train the visual navigation model through an offline reinforcement learning algorithm based on all video frames and corresponding action types, the target image, the current observation image, and the navigation decision information, so that the visual navigation model predicts the navigation action of reaching the navigation target in the specified image based on the specified image.
[0170] The present application also provides an electronic device corresponding to the method provided in the above embodiment. Fig. 9 , which shows an electronic device diagram provided by some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the method provided by any of the aforementioned embodiments of the present application when running the computer program.
[0171] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0172] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.
[0173] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.
[0174] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.
[0175] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Fig.10The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided in any of the aforementioned embodiments is executed.
[0176] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.
[0177] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.
[0178] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.
[0179] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
[0180] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.
Claims
1. A visual navigation model training method, characterized in that: The method comprises: Determine the action type in each video frame based on the training video; Input all video frames and corresponding action types, target images and current observation images into a pre-trained visual navigation model to output navigation decision information; the target image is a video frame in the training video selected by the user, and the target image includes a navigation target; Based on all video frames and corresponding action types, the target image, the current observed image, and the navigation decision information, a visual navigation model is trained through an offline reinforcement learning algorithm so that the visual navigation model predicts the navigation action of reaching the navigation target in the specified image based on the specified image.
2. The method according to claim 1, characterized in that The method of inputting all video frames and corresponding action types, target images, and current observation images into a pre-trained visual navigation model to output navigation decision information includes: Input all video frames and corresponding action types into the pre-trained visual navigation model to obtain a representation sequence; Processing the current observation image and the target image through a visual encoder, and generating an observation hidden state in combination with a corresponding action type; Integrate the representation sequence of each currently observed image and the observed hidden state to obtain a cyclic hidden state; The cyclic hidden states of all currently observed images are aggregated and input into a multilayer perceptron to output navigation decision information, which includes quality parameters of state-action pairs, end signal parameters and predicted guidance actions.
3. The method according to claim 1, characterized in that All video frames and corresponding action types are input into the pre-trained visual navigation model to obtain a representation sequence, including: Calling the pre-trained visual navigation model, processing all video frames and corresponding action types at a time step, and obtaining a video frame action sequence; Position encoding is performed on the video frame action sequence in combination with the hidden state to obtain a position encoding result; The position encoding result is input into a bidirectional multi-head attention module to generate a representation sequence.
4. The method according to claim 1, characterized in that The processing of the current observation image and the target image by the visual encoder and generating the observation hidden state in combination with the corresponding action type includes: Inputting the current observation image and the target image into the visual encoder to extract visual features; The visual features and the corresponding action types are combined to generate an observed hidden state corresponding to each action type.
5. The method according to claim 1, characterized in that Based on all video frames and corresponding action types, the target image, the current observation image, and the navigation decision information, a visual navigation model is trained by an offline reinforcement learning algorithm, including: All video frames, corresponding action types and binary rewards constitute the training dataset; For each video frame in the training data set, the visual navigation model is trained through an offline reinforcement learning algorithm based on the target image, the current observation image and the set action threshold to update the navigation decision information.
6. The method according to claim 1, characterized in that Determine the action type in each video frame based on the training video, including: Inputting the video frame sequence of the training video into the optical flow model to obtain an optical flow map corresponding to two adjacent video frames; Filter out dominant vectors from all optical flow maps based on set amplitudes; An average direction of the dominant vector is calculated, and an action type is determined based on the average direction, the action type comprising discrete navigation actions in different directions.
7. The method according to claim 1, characterized in that The method further comprises: Inputting the scene video into the visual navigation model for initialization; Inputting a designated image into the visual navigation model to identify a navigation target in the designated image; The visual navigation model predicts a navigation strategy to reach a navigation target in the specified image.
8. A visual navigation model training system, characterized in that: The system comprises: An action type module, used to determine the action type in each video frame based on the training video; A training reasoning module is used to input all video frames and corresponding action types, target images and current observation images into a pre-trained visual navigation model to output navigation decision information; the target image is a video frame in the training video selected by the user, and the target image includes a navigation target; The offline reinforcement learning training module is used to train the visual navigation model through an offline reinforcement learning algorithm based on all video frames and corresponding action types, the target image, the current observation image, and the navigation decision information, so that the visual navigation model predicts the navigation action of reaching the navigation target in the specified image based on the specified image.
9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1-7.
Citation Information
Patent Citations
Self-supervised learning and navigation method for autonomous mobile robot in complex scene
CN113156959A
Indoor navigation method, indoor navigation device, indoor navigation equipment and storage medium
CN113984052A
Self-supervised visual language navigation pre-training method and device and storage medium
CN116168333A
Visual navigation method, device and equipment and readable storage medium
CN117152412A
Visual navigation method and device, electronic equipment and storage medium
CN117213496A
Cited By
Visual motion control method and device for automobile automatic charging system, storage medium and program product
CN120395823A
