A Target-Driven Navigation Method and System Based on Hierarchical Relationships and State Regularization

Abstract

This invention belongs to the field of visual navigation technology and discloses a target-driven navigation method and system based on hierarchical relationships and state regularization. The target-driven navigation method includes the following steps: Step 1, acquiring the name of the item to be found and a global observation image, and detecting and acquiring the appearance features, bounding box positions, confidence levels, and label information of various preset target items in the global observation image; Step 2, based on the data information acquired in Step 1, using a pre-trained visual navigation network for navigation to find the item to be found. The technical solution of this invention can solve the problems of difficult navigation in a home environment where the target is not visible and the deadlock problem caused by the high correlation between navigation states in the existing technology.

Description

Technical Field

[0001] This invention belongs to the field of visual navigation technology, and specifically relates to a target-driven navigation method and system based on hierarchical relationships and state regularization. Background Technology

[0002] Home service robots represent a new direction for the home-based elderly care industry, and item finding is a fundamental task for them. The complex spatial layout, diverse room types, and concealed locations of objects in home environments present significant challenges to the navigation of home service robots.

[0003] Visual navigation, as a fundamental task in robotics and artificial intelligence, has received increasing attention in recent years. Currently, existing visual navigation methods can be divided into two categories: map-based visual navigation methods and deep reinforcement learning-based visual navigation methods. Map-based methods typically divide the navigation task into three steps: mapping, localization, and path planning, utilizing pre-built or in-process environmental maps for navigation. However, building these maps is resource-intensive, and even when maps are constructed, they are not suitable for completely new, unseen environments. With the widespread application of deep learning in end-to-end localization and exploration, deep reinforcement learning-based visual navigation methods have gradually been applied to visual navigation tasks and have achieved remarkable success. These methods use visual observation as input, treating the policy network as a black box, and directly training it through reinforcement learning algorithms to predict the agent's next action and guide the agent's interaction with the environment.

[0004] Goal-driven visual navigation is a navigation method based on deep reinforcement learning. This task requires an agent to find a specified target within a finite step size. No environmental map information is provided during navigation; the only information the agent can obtain is a egocentric RGB image. For visible targets, the agent can quickly and easily navigate to the designated location. However, for invisible targets, achieving accurate and fast navigation is extremely challenging. Furthermore, because reinforcement learning-based goal navigation tasks are generally modeled based on the first-order Markov assumption, there are no explicit constraints on the correlation between navigation states. The high correlation between navigation states is a major cause of agent deadlock (interpretive, i.e., the agent remains stuck in a certain place without taking other effective actions to autonomously escape). Summary of the Invention

[0005] The purpose of this invention is to provide a target-driven navigation method and system based on hierarchical relationships and state regularization to solve one or more of the aforementioned technical problems. The technical solution provided by this invention proposes a "from large to small" target hierarchy relationship and designs a state regularization method to impose explicit constraints on the correlation between navigation states. This can solve the problems of difficulty in navigating without visible targets in home environments and the deadlock problem caused by high correlation between navigation states in existing technologies.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides a target-driven navigation method based on hierarchical relationships and state regularization, comprising the following steps:

[0008] Step 1: Obtain the name of the item to be searched and the global observation image, and detect and obtain the appearance features, bounding box positions, confidence scores and label information of various target items in the global observation image;

[0009] Step 2: Based on the data obtained in Step 1, a pre-trained visual navigation network is used for navigation to find the item to be found.

[0010] A further improvement of the present invention is that, in step 1, the step of detecting and acquiring the appearance features, bounding box positions, confidence levels, and label information of various preset target objects in the global observation image specifically includes:

[0011] The DINO target detection model is used to detect and obtain the appearance features, bounding box positions, confidence levels, and label information of various target objects in the global observation image.

[0012] A further improvement of the present invention is that the visual navigation network includes:

[0013] The local feature embedding module is used to acquire local features;

[0014] The global feature embedding module is used to obtain global features;

[0015] The visual attention model adopts the classic Transformer architecture; the visual attention model is used to encode the spatial relationship between the detected target and the observation area; wherein, the local features are used as keys and values, and the global features are used as queries;

[0016] The reinforcement learning navigation module includes a Long Short-Term Memory (LSTM) network and an asynchronous dominant actor-critic network (ACRC) for learning navigation strategies to guide the agent's actions at each step. The LTM network takes as input the previous state, the previous action, and the decoder output of the current visual attention model, and outputs the current state. The current state is input to the ACRC. In the ACRC, the Actor network generates the strategy, and the Critic network evaluates the strategy. They continuously update collaboratively through a complementary mechanism to generate the optimal strategy to guide the agent's actions.

[0017] A further improvement of the present invention is that,

[0018] The overall operation process of the local feature embedding module is represented as follows:

[0019] LE=concat(ReLu(Linear(Α)),target info );

[0020] In the formula, A represents the local appearance features of the target; Linear(·) is the linear transformation operation; ReLU(·) is the activation function, defined as ReLU(x) = max(0,x), used to perform nonlinear transformation operations; concat is the vector concatenation operation; target info For other spatial information of the target, LE is the desired local feature embedding.

[0021] A further improvement of the present invention is that,

[0022] The overall operation process of the global feature embedding module is represented as follows:

[0023] GE = ReLu(Conv) 1×1 (X))+PE;

[0024] In the formula, X represents the global image appearance feature; Conv 1×1 (·) is a convolution operation with a filter size of 1×1; Linear(·) is a linear transformation operation; ReLu(·) is an activation function, defined as ReLu(x) = max(0,x), used to perform nonlinear transformation operations; PE is positional encoding; GE is the desired global feature embedding.

[0025] A further improvement of the present invention is that the attention function of the visual attention model decoder is expressed as follows:

[0026]

[0027] In the formula, G represents global feature embedding; L represents local feature embedding; and d is the dimension of the mapped global appearance feature channel.

[0028] A further improvement of the present invention is that, in the asynchronous advantage actor-critic network, the reward function is set as follows:

[0029]

[0030] In the formula, R(s,a) is the reward R under state s and action a; R partial For partial rewards; R partial Rewards for achieving the goal; R publish As punishment.

[0031] A further improvement of the present invention is that the training steps of the visual navigation network include:

[0032] The Dijkstra shortest path algorithm is used to generate optimal action commands, which are then used as a human expert's experience. A visual attention network is trained through supervised imitation learning to force the association between input features and navigation signals, resulting in a pre-trained network. The loss function used in the pre-trained network is...

[0033]

[0034] In the formula, CrossEntropy is the cross-entropy loss function; a and These are the model-predicted action and the optimal action instruction, respectively.

[0035] The weights of the pre-trained model are loaded as the initial weights of the visual navigation network. The observed image and target information at the current time are input, processed by the visual attention module, and then input to the long short-term memory network. The state representation output by the long short-term memory network is used as the input to the asynchronous dominant actor-critic network. The Actor network in the asynchronous dominant actor-critic network outputs actions to guide the agent's navigation, and the loss of the Actor network and the Critic network is calculated based on the value output by the Critic network in the asynchronous dominant actor-critic network.

[0036] During each training step, the loss function value is calculated based on the total loss function, and backpropagation is performed starting from the loss function value. A stochastic gradient descent optimizer is used to optimize the network parameters based on the gradient information obtained from backpropagation, thereby guiding the neural network to achieve accurate navigation based on the input environmental information.

[0037] The expression for the total loss function is as follows:

[0038] L=λ0L policy +λ1L value +λ2L sr ;

[0039] In the formula, Lpolicy It is a strategy loss; L value Value loss; L sr λ0, λ1, and λ2 are the state regularization loss; λ0, λ1, and λ2 are the weighting coefficients used to balance the three losses.

[0040]

[0041] In the formula, ξ is the unregularized threshold; s t Let s be the state at time t. i Let be the i-th navigation state in a round; ρ(·) is the formula for calculating the Pearson correlation coefficient, defined as... Cov(·) is the covariance calculation operation, and Var(·) is the variance calculation operation. T represents the round length.

[0042] This invention provides a target-driven navigation system based on hierarchical relationships and state regularization, comprising:

[0043] The data acquisition module is used to acquire the name of the item to be searched and the global observation image, and to detect and acquire the appearance features, bounding box positions, confidence levels and label information of a variety of preset target items in the global observation image;

[0044] The navigation module is used to navigate using a pre-trained visual navigation network based on the data information obtained by the data acquisition module, in order to find the item to be searched.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The target-driven navigation method based on hierarchical relationships and state regularization provided by this invention proposes a "parent-child" target hierarchy in the visual navigation network model of target-driven navigation. This addresses the navigation difficulties caused by the complexity of home environments and the often small and invisible size of targets. Furthermore, a state regularization loss function is designed to impose explicit constraints on the correlation between navigation states, thus resolving the agent deadlock problem caused by high state correlation. In summary, the technical solution provided by this invention can solve the problem of finding items in home environments and improve navigation success rates. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art are briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0048] Figure 1This is a flowchart illustrating a target-driven navigation method based on hierarchical relationships and state regularization provided in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of the overall structure of the target-driven visual navigation network in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the local feature embedding module in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the global feature embedding module in an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the reinforcement learning navigation module in an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the navigation results in an embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0055] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0056] The present invention will now be described in further detail with reference to the accompanying drawings:

[0057] Please see Figure 1 The present invention provides a target-driven navigation method based on hierarchical relationships and state regularization, which specifically includes the following steps:

[0058] Step 1: Obtain the name of the item to be found and the global observation image; detect the appearance features, bounding box positions, confidence scores, and label information of various target items in the global observation image; among them, the DINO target detection model can be used for detection.

[0059] Step 2: Based on the data obtained in Step 1, a pre-trained visual navigation network is used for navigation to find the item to be found.

[0060] Please see Figure 2 In this embodiment of the invention, the visual navigation network includes:

[0061] The local feature embedding module is used to encode local features of instances, including appearance features of items of interest, bounding box positions, confidence scores, and label information; among them, the extraction of appearance features and spatial information of items is achieved by the DINO model fine-tuned in the AI2Thor home environment;

[0062] The global feature embedding module is used to encode the global features of the input observed image and add the position encoding to the visual global features, so that the agent focuses on a specific region of the observed image in each global feature; specifically, for example, the global features of the observed image are extracted by a ResNet50 network pre-trained on the ImageNet dataset, and the position encoding refers to the encoding settings in the classic Transformer model.

[0063] The visual attention model employs the classic Transformer architecture to encode the spatial relationship between detected targets and observed regions. Local features are used as keys and values, and since each global feature corresponds to a region of the observed image, the global features are used as queries and input to the decoder. Finally, the decoder output is used as the input to the navigation module.

[0064] The reinforcement learning navigation module, comprising a Long Short-Term Memory (LSTM) network and an Asynchronous Advantage Actor-Critic (A3C) network, is used to learn navigation policies to guide each action of the agent. The LSTM network takes the previous state and action as input, along with the output of the current visual attention decoder, and outputs the current state, which is then fed into the A3C network. In the A3C network, the Actor network generates the policy, and the Critic network evaluates it. They work together in a complementary manner to continuously update and generate the optimal policy to guide the agent's actions. Building upon the classic A3C algorithm, we add a partial reward for finding a parent item of the target item to encourage the agent to prioritize finding related parent items, thus improving navigation success rate. Furthermore, we add a state regularization loss to the model's loss function to constrain the correlation between navigation states, mitigating deadlock to some extent.

[0065] As specifically exemplified in this embodiment of the invention, the implementation process of the target-driven navigation method may include the following steps:

[0066] Step 1: Obtain images and target object bounding box locations of all reachable locations of the AI2Thor simulation environment; Optionally, the collected images should have RGB color channels and a pixel size of 300×300; The dataset contains a total of 120 rooms, divided into four room types: kitchen, living room, bedroom, and bathroom. The first 20 rooms of each type are assigned to the training set, and the last 10 rooms to the test set, resulting in a training set of 80 rooms and a test set of 40 rooms.

[0067] Step 2: Input the collected labeled image data into the DINO model for fine-tuning, and then use the fine-tuned model to perform detection to obtain the appearance information and spatial information (item label, bounding box, confidence score, etc.) of the target object;

[0068] Step 3: Construct a goal-driven visual navigation network framework, including a local feature embedding module, a global feature embedding module, and an A3C reinforcement learning navigation module.

[0069] Please see Figure 3 In this embodiment of the invention, the local feature embedding module first maps the 300×256-dimensional appearance features to 300×249 dimensions, and then concatenates them with 300×7 target information (300×1 label information, 300×4 bounding box information, 300×1 confidence information, and 300×1 target one-hot information), finally obtaining 300×256 local features, which serve as the keys and values ​​of the visual attention encoder. The overall operation process is represented as follows:

[0070] LE=concat(ReLu(Linear(Α)),target info );

[0071] In the formula, A represents the local appearance features of the target, Linear(·) is a linear transformation operation, ReLU(·) is an activation function defined as ReLU(x) = max(0,x), which performs a non-linear transformation operation, concat is the vector concatenation operation, and target info For other spatial information of the target, LE is the desired local feature embedding.

[0072] Please see Figure 4In this embodiment of the invention, the global feature embedding module first extracts the global image appearance features into a 7×7×512 vector using an RseNet50 network, then maps it to a 7×7×256 vector through a convolutional layer and a non-linear activation layer. Simultaneously, it uses a sinusoidal function to calculate the absolute position code with the same dimension as the global feature vector. Finally, it adds the absolute position code point-by-point to the 7×7×256 feature vector to obtain the final global feature, which serves as the position query for the visual attention decoder. The overall operation process is represented as follows:

[0073] GE = ReLu(Conv) 1×1 (X))+PE;

[0074] In the formula, X represents the global image appearance feature, and Conv 1×1 (·) is a convolution operation with a filter size of 1×1, Linear(·) is a linear transformation operation, ReLu(·) is the above activation function that performs a nonlinear transformation operation, PE is position encoding, and GE is the desired global feature embedding.

[0075] If we abbreviate local feature embedding as L and global feature embedding as G, then the attention function of the final visual attention model decoder is expressed as follows:

[0076]

[0077] In the formula, d is the dimension of the global appearance feature channel after mapping.

[0078] Please see Figure 5 In this embodiment of the invention, the navigation algorithm adopts the standard Asynchronous Advantageous Actor-Critic (A3C) architecture. The navigation policy network takes the current visual representation, the action of the previous time step, and the embedding of the state of the previous time step as joint inputs, and outputs the probability distribution and value of the action. The agent selects the action with the highest probability according to the predicted policy, and uses the predicted value to train the navigation policy network. In goal-driven navigation tasks, the reward for reinforcement learning is usually set as follows: the agent receives a small penalty (usually -0.01) for each step taken, and a large reward (usually +5.00) when the agent successfully finds the goal.

[0079] Further optimized, an additional partial reward is proposed on top of the basic reward. This partial reward is set as follows: when the agent detects a parent item of the target item in the environment, a partial reward is given. The partial reward value is obtained by calculating the relative spatial distance between parent and child items in the AI2Thor environment (based on Euclidean distance). When multiple parent items are visible, only the reward with the highest probability (i.e., closest to the target) is selected. To avoid parent items having an excessive impact on the agent's navigation to the child target, a scaling factor k∈(0,1) is set, and the obtained spatial similarity value is multiplied by the scaling factor k to obtain the final partial reward. Simultaneously, to encourage the agent to explore different parent items in the room as much as possible before successfully locating the target, a partial reward is only given when the agent detects a certain type of parent item for the first time. The final reward is set as follows:

[0080]

[0081] In the formula, R(s,a) is the reward R under state s and action a. partial As part of the reward, R partial For the target reward, R publish As punishment.

[0082] In this embodiment of the invention, the training steps of the visual navigation network include:

[0083] 1) Dijkstra's shortest path algorithm is used to generate optimal action commands as a human expert experience. A visual attention network is trained under supervised learning through imitation, forcing the association between input features and navigation signals to ensure convergence in subsequent network training. The loss function of the pre-trained network is...

[0084]

[0085] In the formula, CrossEntropy is the cross-entropy loss function, and a and These are the model-predicted action and the optimal action instruction, respectively.

[0086] 2) After obtaining the pre-trained model, the end-to-end visual navigation network is trained. First, the weights of the pre-trained model are loaded and updated in the parameter dictionary of the visual navigation network. Finally, the updated parameter dictionary is loaded into the current model as the initial weights for the visual navigation model. The observed image and target information at the current time are input, processed by the visual attention module, and then input to the Long Short-Term Memory (LSTM) network. The network input state representation serves as the input to the A3C network. The Actor network then outputs actions to guide the agent's navigation, and the loss of both networks is calculated using the output value of the Critic network.

[0087] Since goal-driven navigation tasks based on reinforcement learning are generally modeled using the first-order Markov assumption, but without explicit constraints on the correlation between navigation states, and the high correlation between navigation states is a major cause of agent deadlock (i.e., the agent is continuously trapped in a certain place without taking other effective actions to autonomously escape), this invention addresses the problems in the modeling process based on the first-order Markov assumption by imposing explicit constraints on the correlation between navigation states: The correlation between navigation states in each round is calculated. Since navigation failure is only related to positive correlation, negative correlation is not considered, and the correlation coefficient between positively correlated states is forced to be 0. The mean value is calculated to obtain the loss, and finally, the loss is obtained by multiplying it by a scaling factor and adding it to the policy loss and value loss. According to the first-order Markov property, the current state is only related to the state at the previous time step. Therefore, an unregularized threshold ξ is defined here. If the time interval between two states is less than the threshold ξ, correlation regularization between consecutive states is not applied. The set of states after the round is defined as follows: In the formula, T is the round length. The state regularization loss function is then defined as:

[0088]

[0089] In the formula, ξ is the unregularized threshold, and s t Let s be the state at time t. i Let be the i-th navigation state in a round, and ρ(·) be the formula for calculating the Pearson correlation coefficient, defined as:

[0090]

[0091] In the formula, Cov(·) is the covariance calculation operation, and Var(·) is the variance calculation operation.

[0092] The final total loss is expressed as:

[0093] L=λ0L policy +λ1L value +λ2L sr ;

[0094] In the formula, L policy It is a strategy loss, L value The loss is the value loss, and λ0, λ1, and λ2 are the weighting coefficients that balance the three losses.

[0095] Another example is that, during each training step, backpropagation is performed starting from the loss function value; a stochastic gradient descent optimizer is used to optimize the network parameters based on the gradient information obtained from backpropagation, thereby guiding the neural network to achieve precise navigation based on the input environmental information. Specifically, this embodiment of the invention operates on a computer with the PyTorch framework and the AI2Thor simulation environment. The computer system is Linux and requires a graphics driver capable of visualizing the AI2Thor environment's graphical interface. This embodiment of the invention, trained on a Gold 6626R@2.90GHz CPU, 8GB of memory, and two NVIDIA GeForce RTX3090 GPUs, takes approximately 4 days. Figure 6 As shown, the navigation result is illustrated.

[0096] In summary, due to the complex spatial layout, diverse room types, and concealed target locations in home environments, reinforcement learning algorithms struggle to achieve correct navigation under sparse rewards. Therefore, this invention proposes a hierarchical "parent-child" relationship. By calculating the Euclidean distance between parent and child items, the proximity between items is determined, and this distance serves as a partial reward to guide the agent, encouraging it to start with larger, more easily discovered items, effectively improving navigation success rate. Furthermore, addressing the issue that reinforcement learning-based navigation tasks typically model using first-order Markov assumptions without explicitly constraining the correlation between navigation states, this invention proposes a state regularization loss function. By calculating the correlation between navigation states, it forces the correlation coefficient between positively correlated states to be zero, thereby eliminating state correlation and partially resolving agent deadlock, thus improving navigation success rate. The reward and loss functions proposed in this invention are plug-and-play modules that do not alter the end-to-end nature of the model and require no other dependencies, demonstrating broad application prospects.

[0097] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0098] In another embodiment of the present invention, a target-driven navigation system based on hierarchical relationships and state regularization is provided, comprising:

[0099] The data acquisition module is used to acquire the name of the item to be searched and the global observation image, and to detect and acquire the appearance features, bounding box positions, confidence levels and label information of a variety of preset target items in the global observation image;

[0100] The navigation module is used to navigate using a pre-trained visual navigation network based on the data information obtained by the data acquisition module, in order to find the item to be searched.

[0101] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A target-driven navigation method based on hierarchical relationships and state regularization, characterized in that, Includes the following steps: Step 1: Obtain the name of the item to be searched and the global observation image, and detect and obtain the appearance features, bounding box positions, confidence scores and label information of various target items in the global observation image; Step 2: Based on the data obtained in Step 1, a pre-trained visual navigation network is used for navigation to locate the item to be found; The visual navigation network includes: The local feature embedding module is used to acquire local features; The global feature embedding module is used to obtain global features; The visual attention model adopts the classic Transformer architecture; the visual attention model is used to encode the spatial relationship between the detected target and the observation area; wherein, the local features are used as keys and values, and the global features are used as queries; The reinforcement learning navigation module includes a Long Short-Term Memory (LSTM) network and an asynchronous dominant actor-critic network (ACRC) network, used to learn navigation strategies to guide the agent's actions at each step. The LTM network takes as input the previous state, the previous action, and the decoder output of the current visual attention model, and outputs the current state. The current state is input to the ACRC network. In the ACRC network, the Actor network generates the strategy, and the Critic network evaluates the strategy. The two networks continuously update collaboratively through a complementary approach to generate the optimal strategy to guide the agent's actions. In the asynchronous advantage actor-critic network, the reward function is set as follows: ； In the formula, R ( s , a ) is in state s and actions a The following rewards R ; R partial Partial reward; R target Rewards for achieving goals; R publish As punishment; The training steps of the visual navigation network include: The Dijkstra shortest path algorithm is used to generate optimal action commands, which are then used as a human expert's experience. A visual attention network is trained through supervised imitation learning to force the association between input features and navigation signals, resulting in a pre-trained network. The loss function used in the pre-trained network is... ； In the formula, The cross-entropy loss function; and These are the model-predicted action and the optimal action instruction, respectively. The weights of the pre-trained model are loaded as the initial weights of the visual navigation network. The observed image and target information at the current time are input, processed by the visual attention module, and then input to the long short-term memory network. The state representation output by the long short-term memory network is used as the input to the asynchronous dominant actor-critic network. The Actor network in the asynchronous dominant actor-critic network outputs actions to guide the agent's navigation, and the loss of the Actor network and the Critic network is calculated by the value output by the Critic network in the asynchronous dominant actor-critic network. During each training step, the loss function value is calculated based on the total loss function, and backpropagation is performed starting from the loss function value. A stochastic gradient descent optimizer is used to optimize the network parameters based on the gradient information obtained from backpropagation, thereby guiding the neural network to achieve accurate navigation based on the input environmental information. The expression for the total loss function is as follows: ； In the formula, L policy It is a strategic loss; L value Value loss; It is the state regularization loss; λ 0, λ 1, λ 2 is the weighting coefficient used to balance the three types of losses; ； In the formula, ξ The threshold is unregularized; s t for t The state at any given moment, s i For the first round i Each navigation status; ρ ( The formula for calculating the Pearson correlation coefficient is defined as follows: Cov( For covariance calculation, Var( This is an operation to calculate variance; , T The length of the round.

2. The target-driven navigation method according to claim 1, characterized in that, Step 1, specifically including the step of detecting and acquiring the appearance features, bounding box positions, confidence levels, and label information of various target objects in the global observation image, includes: The DINO target detection model is used to detect and obtain the appearance features, bounding box positions, confidence levels, and label information of various target objects in the global observation image.

3. The target-driven navigation method according to claim 1, characterized in that, The overall operation process of the local feature embedding module is represented as follows: ； In the formula, A For the local appearance features of the target; Linear( ) represents a linear transformation operation; ReLU( ) is the activation function, defined as , is used to perform nonlinear transformation operations; concat is the vector concatenation operation; For other spatial information of the target, Embed the desired local features.

4. The target-driven navigation method according to claim 1, characterized in that, The overall operation process of the global feature embedding module is represented as follows: ； In the formula, X For global image appearance features; Conv1 1( () is a filter with a size of 1 1. Convolution operation; Linear( ) represents a linear transformation operation; ReLU( ) is the activation function, defined as , used to perform nonlinear transformation operations; For position encoding; Embed the desired global features.

5. The target-driven navigation method according to claim 1, characterized in that, The attention function of the visual attention model decoder is expressed as follows: ； In the formula, G Indicates global feature embedding; L Represents local feature embedding; d This represents the channel dimension of the mapped global appearance features.

6. A target-driven navigation system based on hierarchical relationships and state regularization, characterized in that, The target-driven navigation method for implementing any one of claims 1 to 5 includes: The data acquisition module is used to acquire the name of the item to be searched and the global observation image, and to detect and acquire the appearance features, bounding box positions, confidence levels and label information of a variety of preset target items in the global observation image; The navigation module is used to navigate using a pre-trained visual navigation network based on the data information obtained by the data acquisition module, in order to find the item to be searched.

Images