An object navigation method based on object bias correction and directed attention map

Abstract

The present invention relates to an object navigation method based on object bias correction and directed attention graph, comprising the following steps: S1, a robot acquires an original image at the current moment in real time; S2, extracts a global image feature I of a visual input t and local object features S t ; S3, get the adaptive object attention map G through the multi-head attention score generation module v ; S4, get the object attention weight G at the current moment t ; S5, get the final encoding of the object branch #imgabs0 #S6, get the final encoding of the image branch through multi-head attention #imgabs1 #S7, aggregate the three branches of image, object and past action to get the feature representation H t ; S8, predict the probability of action execution; S9, strengthen the probability of outputting the stop action when the target object is detected; S10, select the action with the highest probability to explore the scene. Compared with the prior art, the present invention has the advantages of solving the problem of attention bias, strengthening the utilization of information fusion, better distinguishing branch types, and improving the rationality of branch feature aggregation.

Description

Technical Field

[0001] The present invention relates to the field of robot visual semantic navigation, and in particular to an object navigation method based on object bias correction and directed attention graph. Background Art

[0002] In the object navigation task, the intelligent agent (robot) navigates to the user-specified target in an indoor environment based on first-person visual observations (usually RGB images captured by its onboard camera). This is a basic and complete task for achieving artificial intelligence goals, which requires the intelligent robot to understand its visual input, infer its current position, reason about the target position, plan the trajectory, and perform an action at each step. The development of the object navigation field has also led to the flourishing development of fields such as visual question answering (requiring the robot to navigate to the specified object and answer questions) and visual language navigation (requiring the robot to navigate in an indoor environment according to the instructions of the instructor). Therefore, object navigation has attracted the attention of more and more researchers and has spawned a large number of works trying to solve this problem.

[0003] Classic map-based visual navigation methods have been studied for many years. These methods explicitly decompose the navigation task into a set of subtasks, namely mapping, localization, planning, and motion control. Although these methods have achieved considerable success in recent years, their modular design has fundamental limitations that hinder their widespread application. An important limitation is that they are susceptible to sensor noise, which accumulates and propagates throughout the entire process from mapping to motion control, making these methods less robust in complex environments. More importantly, they require a lot of scene exploration and manual operations, which makes them difficult to integrate with other downstream artificial intelligence tasks such as visual recognition, question answering, and scene captioning.

[0004] Observing the recent success of learning-based methods in related tasks, there has been a large amount of work applying learning methods to object navigation tasks. Learning-based methods usually take visual input and user-specified goals and the best action at each timestamp as the input and output of the intelligent agent. Unlike traditional methods, learning-based methods infer solutions directly from data, so they do not require much manual engineering and serve as the basis for new AI-driven visual navigation tasks. However, learning-based object navigation methods also face a crucial problem: how to effectively represent visual input to infer the connection between the current observation and the goal.

[0005] In order to solve this problem, more and more researchers have begun to use methods such as target detection to extract high-level information from the intelligent agent's field of view, trying to use high-level semantic information to better guide the robot's movement. The SP model first proposed to use the graph network to learn the prior knowledge related to objects in the scene; the MJOLNIR model integrates the position and semantic information of the detected objects into the graph convolutional network based on the SP model; the ORG model reasonably uses the relationship between the object's visual features, semantic features and spatial features to improve the adaptability of the object feature application; the CKR model uses cross-modal knowledge reasoning to apply external knowledge graph information to the navigation of the intelligent agent; the HOZ model starts from the scene as a whole, proposes an object area map, and integrates the scene into a hierarchical graph structure.

[0006] Today, object navigation based on high-order semantics is still booming, but researchers need to start from practical problems, find the difficulties of object navigation methods from data sets to the real world, and truly apply object navigation methods to the physical world. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an object navigation method based on object bias correction and directed attention map.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] An object navigation method based on object bias correction and directed attention map comprises the following steps:

[0010] S1. During the object navigation process, the robot obtains the original image at the current moment as visual input in real time;

[0011] S2. Extract global image features of visual input through ResNet18 I t , and extract the local object features S of the visual input through Faster RCNN t ;

[0012] S3, based on global image feature I t As a query, the local object feature S t As the key value, the adaptive object attention map C is obtained through the multi-head attention score generation module v ;

[0013] S4, the adaptive object attention map C v The weighted addition of the fixed object attention map is used to obtain the object attention weight G at the current moment. t ;

[0014] S5. Using object attention weight G tFor local object features S t Weighted to get the final encoding of the object branch

[0015] S6: Use the object semantic representation D weighted by the object attention weight as the query, and the global image information as the key value to obtain the final encoding of the image branch through multi-head attention.

[0016] S7, redistribute and aggregate the energy of the three branches of image, object and past action to obtain the feature representation H t ;

[0017] S8, using LSTM recurrent neural network and A3C reinforcement learning method to predict the probability of action execution;

[0018] S9, increasing the probability of outputting a stop action when a target object is detected;

[0019] S10. The robot selects the action with the highest probability to explore the scene.

[0020] In step S2, the local object feature S t By object visual features Object location features Object Confidence and target object indication position Stitched together.

[0021] The step S3 specifically includes the following steps:

[0022] S31, setting a confidence threshold, and setting the object features below the confidence threshold to 0 through confidence filtering to obtain the filtered object features

[0023] S32, through the one-hot encoding and two layers of fully connected layers to obtain the object index encoding feature OI, the global image feature I t After global average pooling, the target index is encoded OI p Stitching to obtain global image features containing target object information Then we have:

[0024]

[0025] Where M is the number of pixels in the feature map, p represents the target object, and OI p The index code of the target object p in the object index coding feature OI, Concat means concatenating two vectors into one;

[0026] S33, using global image features containing target object information As a query, the filtered object features As a key, use the multi-head attention score generator to get the adaptive object attention map G v , then:

[0027]

[0028]

[0029]

[0030] Among them, NH and HD are the number of heads and hidden dimensions of the multi-head attention score generator, respectively. are all learnable dimension mapping parameters, and The global image features containing the target object information are And the filtered local object features Mapped to the same dimension HD, Aggregate the sub-graphs calculated by each head into an adaptive object attention graph G v .

[0031] The step S4 is specifically as follows:

[0032] Define a learnable fixed-size matrix represents a fixed object attention map, and extracts edge weights from the fixed object attention map according to the object p being searched The unbiased object attention weight G is obtained by weighting the fixed object attention map and the adaptive object attention map t , then:

[0033]

[0034] Among them, ω n With ω v are all learnable weights.

[0035] The step S5 is specifically as follows:

[0036] In order to balance the amount of computation and information integrity, two fully connected layers with ReLU nonlinearity are used to local object features S t Perform dimensionality reduction to obtain the object feature S after dimensionality reduction t ′, using unbiased object attention weight G t For the object feature S after dimensionality reduction t The features of each object q in ′ are weighted to obtain the final encoding of the object branch Then we have:

[0037]

[0038]

[0039] Where N is the total number of objects, is the low-dimensional feature of the qth object, is the unbiased attention weight of the qth object.

[0040] The step S6 specifically includes the following steps:

[0041] S61. Using unbiased object attention weight G t The object index encoding feature OI is progressively weighted and aggregated to obtain the weighted object semantic representation D, then:

[0042]

[0043] in, represents the indicator function, N is the total number of objects, OI q The index code of the qth object in the object index coding feature OI, confidence filtering The semantic features of objects whose confidence is less than a certain threshold are set to 0;

[0044] S62, introducing the one-dimensional pixel index code PI into the global image features, we have:

[0045] I′ t =δ(δ(I t W1)W2)+PI

[0046] Among them, I′ t is a global image feature with pixel position identification, δ represents the ReLU nonlinear activation function, and W1 and W2 are learnable dimension mapping parameters;

[0047] S63, using the weighted object semantic representation D as the query, the global image feature I′ with pixel position identification t As the key value, the final encoding of the image branch is obtained through the multi-head attention network

[0048] The step S7 is specifically as follows:

[0049] Define a vector R = {r1, r2, r3} containing three learnable parameters, and transform the global image branch Local object branch and the past action branch PA are aggregated into a feature representation H through vector R t , then:

[0050]

[0051] Among them, F pw Represents point-wise convolution.

[0052] The step S8 is specifically as follows:

[0053] Use two layers of fully connected layers to represent the feature H t The refined data is used as the input of the LSTM network. The LSTM network learns the temporal relationship of the robot in the process of exploring the environment, and outputs an action probability vector that integrates past actions and the current environment. The actions include going straight, turning left, turning right, raising the head, lowering the head and stopping. If the robot outputs a stop action, it means that the robot has found the object and navigated to the location of the object.

[0054] The confidence filter threshold is 0.6.

[0055] In step S9, when the robot detects a target object, the target detection confidence is multiplied by a fixed coefficient and then multiplied by the probability of the stop action in the probability vector to achieve explicit enhancement of the stop action trigger.

[0056] Compared with the prior art, the present invention has the following advantages:

[0057] 1. The present invention proposes a directed object attention graph representation, which effectively solves the object attention bias problem that occurs in object navigation by intelligent agents, and enables neural networks to have a clearer understanding of the intrinsic relationship between objects.

[0058] 2. Based on the directed object attention map, the present invention designs the cross-attention of the global image branch and the local object branch, that is, unbiased image attention and unbiased object attention. This cross-attention method enhances the rationality of the attention allocation of global image features and object features.

[0059] 3. The present invention designs an adaptive energy allocation branch feature aggregation method that requires almost no additional computational effort. This method effectively enhances the network's ability to distinguish different branches and improves the data distribution represented by the aggregated features.

[0060] 4. The present invention was experimented on the AI2-Thor dataset, and the SR, SPL, and SAE indicators were improved by 7.4%, 8.1%, and 17.6% respectively compared with the most advanced methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Flow chart of the method of the present invention.

[0062] Figure 2 This is the directed object attention map calculation process of the method of the present invention.

[0063] Figure 3 It is the adjacency matrix of the fixed attention graph learned by the method of the present invention.

[0064] Figure 4 It is the adjacency matrix of the adaptive attention graph learned by the method of the present invention.

[0065] Figure 5 The figure shows the change of the success rate indicator SR on the training set and the test set during the training process of the method of the present invention, wherein Figure (5a) shows the change of the success rate indicator SR on the training set, and Figure (5b) shows the change of the success rate indicator SR on the test set.

[0066] Figure 6 The present invention's method is compared with past methods in guiding intelligent agents to find objects in the AI2-Thor simulation environment.

[0067] Figure 7 These are the four scenes in the AI2-Thor dataset and the objects contained in each scene. DETAILED DESCRIPTION

[0068] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0069] The present invention provides an object navigation method based on object bias correction and directed attention graph. The method uses the object directed attention graph to solve the object attention bias problem in the robot object navigation process, reasonably distributes the object attention according to the unbiased relationship between objects, and obtains the most advanced results in the experiment of AI2-Thor data set. The method specifically includes the following steps:

[0070] S1. The object attention bias problem of robots when performing purely visual object navigation in unknown environments is proposed;

[0071] The object attention bias problem in step S1 is caused by two reasons:

[0072] (1) Endogenous reasons: The network itself prefers objects with richer visual features.

[0073] (2) Exogenous causes: the differences in the frequencies that can be detected for each object in the data set.

[0074] This method mainly starts from endogenous factors and does not change the number of objects in the dataset.

[0075] S2, using ResNet18 and Faster RCNN to extract global image information and local object information of visual input;

[0076] In step S2, ResNet18 extracts the global image feature I from the original image t , Faster RCNN extracts object features S from the original image t , which is composed of the visual features of the object Object location features Object Confidence and target object indication position Stitched together.

[0077] S3, using global image information as query and local object information as key value, obtain adaptive object attention map through multi-head attention score generation module;

[0078] In step S3, in order to eliminate the interference of low-confidence objects containing more noise, confidence filtering is used to set the object features below a certain confidence threshold to 0. The filtered object features are expressed as

[0079] After filtering, we first obtain the object index encoding feature OI through one-hot encoding and two-layer fully connected layers. Because different attention allocation strategies are required when searching for different targets, the global image feature I t After global average pooling, the target index is encoded OI p Stitching to get the global image features after stitching Then we have:

[0080]

[0081] Among them, M represents the number of pixels in the feature map, p represents the target object, and OI p It represents the index code of the pth object (target object) in OI, and Concat means concatenating two vectors into one.

[0082] The global image features after splicing As a query, the filtered object features As a key, use the multi-head attention score generator to get the adaptive object attention map G v , then:

[0083]

[0084]

[0085]

[0086] Among them, NH and HD represent the number of heads and hidden dimensions of the multi-head attention score generator, respectively. and The global image features are and local object features Mapped to the same dimension HD, Aggregate the sub-graphs calculated by each head into an adaptive object attention graph G v .

[0087] S4, weighted addition of the adaptive object attention map and the fixed object attention map to obtain the object attention weight at the current moment;

[0088] In step S4, a learnable fixed-size matrix is ​​defined represents a fixed object attention map, N represents the total number of objects, and since the defined object attention map is directed, G n It is asymmetric.

[0089] Extract edge weights from the fixed object attention graph according to the object p being searched p is the end point of the edge in the directed graph. The unbiased object attention weight G is obtained by weighted addition of the fixed object attention map and the adaptive object attention map. t , then:

[0090]

[0091] Among them, ω n With ω v These are all learnable weights.

[0092] S5, using the object attention weight to weight the local information of the object to obtain the final encoding of the object branch;

[0093] In step S5, in order to balance the amount of computation and information integrity, two fully connected layers with ReLU nonlinearity are used to calculate the local object features S t Perform dimensionality reduction to obtain the object feature S after dimensionality reduction t ′, using the unbiased object attention weight G calculated in step S4 t For the object feature S after dimensionality reduction t The features of each object q in ′ are weighted, then:

[0094]

[0095] in, is the low-dimensional feature of the qth object, is the unbiased attention weight of the qth object.

[0096] S6, using the object semantics weighted by the object attention weight as the query, and the global image information as the key value, the final encoding of the image branch is obtained through multi-head attention;

[0097] In step S6, we first use the unbiased object attention weight G t The object index encoding feature OI is weighted and aggregated to obtain the weighted object semantic representation D, then:

[0098]

[0099] in, Refers to indicator function, confidence filter The semantic features representing objects with confidence less than a certain threshold will be set to 0.

[0100] Then the one-dimensional pixel index encoding PI is introduced into the global image features:

[0101] I′ t =δ(δ(I t W1)W2)+PI

[0102] Among them, δ refers to the ReLU nonlinear activation, W1 and W2 are learnable dimension mapping parameters used to reduce the dimension of global image features, and PI provides a position identifier for each pixel of the global image feature.

[0103] Finally, the weighted object semantic representation D is used as the query, and the global image feature I′ with pixel position identification is t As the key value, the final encoding of the image branch is obtained through the multi-head attention network Then we have:

[0104]

[0105]

[0106]

[0107] S7, redistribute and re-aggregate the energy of the three branches of image, object and past action;

[0108] In step S7, a vector R={r1, r2, r3} containing three learnable parameters is defined, the global image branch Local object branch The three branches of the past action branch PA are aggregated into a feature representation H through the vector R t , then:

[0109]

[0110] Among them, Fpw Refers to point-wise convolution, which reduces the feature dimension.

[0111] S8, using LSTM recurrent neural network and A3C reinforcement learning method to predict the probability of action execution;

[0112] S9, using the stop action reminder module to enhance the probability of outputting a stop action when the target object is detected;

[0113] In step S9, when the intelligent agent detects the target object, the target detection confidence is multiplied by a fixed coefficient and then multiplied by the probability of the stop action in the probability vector to achieve explicit enhancement of the stop action trigger.

[0114] S10. The robot selects the action with the highest probability to explore the scene.

[0115] Example

[0116] This embodiment provides an object navigation method based on object bias correction and directed attention map. The framework diagram of the method is as follows: Figure 1 As shown, the specific steps include:

[0117] S1. Use ResNet18 and Faster RCNN to extract global image information and local object information of visual input;

[0118] S2, using global image information as query and local object information as key value, obtain adaptive object attention map through multi-head attention score generation module;

[0119] S3, weighted addition of the adaptive object attention map and the fixed object attention map to obtain the object attention weight at the current moment;

[0120] S4, using the object attention weight to weight the local information of the object to obtain the final encoding of the object branch;

[0121] S5, using the object index encoding feature weighted by the object attention weight as the query, the global image information as the key value, and obtaining the final encoding of the image branch through multi-head attention;

[0122] S6, redistribute and re-aggregate the energy of the three branches of image, object and past action;

[0123] S7, using LSTM recurrent neural network and A3C reinforcement learning method to predict the probability of action execution;

[0124] S8. Use the stop action reminder module to enhance the probability of outputting a stop action when the target object is detected.

[0125] S9. The robot selects the action with the highest probability to explore the scene.

[0126] In step S1, the intelligent agent first reads the main perspective image of the current scene through the forward-shooting camera, and then converts the image data into the numpy format that is easy to process in python. In order to input the neural network built with pytorch, the image data in numpy format is converted into the tensor format in pytorch.

[0127] When training the model, the main-view images captured in all possible states of the intelligent agent in the scene are taken out, and the global image information and local object information are extracted using the pre-trained ResNet18 and Faster RCNN respectively. The above two features in each scene are stored in HDF5 format files. In this way, during training, the intelligent agent only needs to read the HDF5 file to obtain the global image and local object features of the current state, and the intelligent agent does not need to constantly interact with the environment, which consumes a lot of time.

[0128] In step S2, Figure 2 The detailed process of calculating the object attention map is shown. Object features need to pass through a confidence filter to remove the noise caused by objects with low confidence. After experimental exploration, the optimal threshold of the confidence filter is 0.6. The global image features need to be spliced ​​with the index code of the target object after global average pooling. The concatenation vector method is used here instead of feature addition because the dimension of the image feature (512) and the dimension of the target index code (64) are quite different, so the concatenation method can better utilize the feature information. The multi-head attention score generation module essentially calculates the similarity between each local object feature embedding and the global image feature embedding, so as to obtain the importance weight of each object that varies according to different visual images.

[0129] In step S3, the fixed object attention map is represented by a learnable N×N matrix, which continuously learns the essential relationship between objects during training and does not change when used for testing. So in theory, the more comprehensive and realistic the training scene is, the more reasonable the fixed object attention map will be. Figure 3 and Figure 4 The adjacency matrices of the fixed object attention map and the adaptive object attention map are visualized respectively. It can be found that the weight of the self-connected edge in the fixed object attention map is the largest, that is, the attention to the target object is the strongest. Figure 4Each column of the adjacency matrix of the adaptive object attention map in represents the adaptive object attention when randomly extracting a frame when searching for such objects. It can be found that the adaptive object attention is generally focused on one object. This is because the objects in the field of view at each moment are very sparse relative to all objects. This sparsity leads to an extreme tilt of attention to the most important objects. However, since the weighted average of the fixed attention map and the adaptive attention map is used, the weight assigned to the fixed attention map learned by the neural network is 0.96, and the weight assigned to the adaptive attention map is 0.04. Therefore, the concentration of adaptive attention does not lead to extreme attention of the overall object, but will have a good effect of strengthening the weight of important objects.

[0130] In step S4, in order to balance the subsequent computational effort of the network and the information richness of the features, a fully connected layer with ReLU nonlinearity is used to reduce the dimension of local object features from 518 dimensions to 64 dimensions, and then each object feature is multiplied by the corresponding object attention weight to obtain the final object feature output.

[0131] In step S5, the global image feature needs to add position coding to each pixel. In this method, 1D position coding is used. This is because the size of the global image feature map is only 7×7, so only 1D coding can also learn two-dimensional position information, so there is no need to use 2D or relative position coding. Object semantic information also needs to be de-noised through confidence filtering after attention allocation. The confidence threshold selected here is the same as in step S2, which is 0.6.

[0132] In step S6, each branch has a learnable parameter to adjust the data distribution of the branch features. All branches are connected in a splicing manner. Note that after being spliced ​​into a feature vector, it is necessary to go through two layers of fully connected layers to refine the information before it can be input into the LSTM network.

[0133] In step S7, the refined features are passed through the LSTM network to learn the temporal relationship of the intelligent agent in the process of exploring the environment, thereby outputting an action probability vector that integrates previous actions and the environment. There are 6 actions in total: go straight, turn left, turn right, raise your head, lower your head, and stop. If the intelligent agent outputs a stop action, it means that it believes it has found the object and navigated to the location of the object.

[0134] In step S8, since the goal of this task is to find the target object, when the intelligent agent detects the target object in the field of view, the probability of ending the action is enhanced according to the confidence of detecting the target object.

[0135] In step S9, the robot will take the action with the highest probability in the action probability vector to proceed to the next step of scene exploration.

[0136] This paper proposes a directed object attention graph method to effectively solve the problem of object attention bias in the object navigation process. Based on the object attention graph, two cross-attention strategies, unbiased adaptive object attention and adaptive image attention, are proposed. They allow the neural network to reasonably allocate attention to object features and image features. In addition, an adaptive energy allocation method is proposed to optimize the aggregation of branch features. Experiments were conducted on the AI2-Thor dataset. Figure 7 Shows the scenes and objects in the dataset. Figure 5 In the figure, we can see that during the reinforcement learning training process, the success rate curves of our method on both the training set and the test set converge smoothly. Figure 6 The actual effect of the method in guiding the navigation of intelligent agents in the AI2-Thor simulation environment is demonstrated. It can be found that the direction of rotation and the timing of turning straight are very important for the success of navigation. These two decisions are mainly determined by the intelligent agent's understanding of the scene when the key frame contains multiple objects in the field of view. The method based on the directed object attention map can provide the intelligent agent with a more reasonable and unbiased attention allocation, so it can guide it to make correct decisions at the key frame. This method has improved SR, SPL and SAE indicators by 7.4%, 8.1% and 17.6% compared with advanced methods such as HOZ. This method has unique advantages both in terms of improving the effect of the method and guiding future research.

[0137] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. An object navigation method based on object bias correction and directed attention graph, characterized in that: The following steps are involved: S1. During the object navigation process, the robot obtains the original image at the current moment as visual input in real time; S2. Extract global image features of visual input through ResNet18 I t , and extract the local object features S of the visual input through Faster RCNN t ; S3, based on global image feature I t As a query, the local object feature S t As the key value, the adaptive object attention map G is obtained through the multi-head attention score generation module v ; S4, the adaptive object attention map G v The weighted addition of the fixed object attention map is used to obtain the object attention weight G at the current moment. t ; S5. Using object attention weight G t For local object features S t Weighted to get the final encoding of the object branch S6: Use the object semantic representation D weighted by the object attention weight as the query, and the global image information as the key value to obtain the final encoding of the image branch through multi-head attention. S7, redistribute and aggregate the energy of the three branches of image, object and past action to obtain the feature representation H t ; S8, using LSTM recurrent neural network and A3C reinforcement learning method to predict the probability of action execution; S9, increasing the probability of outputting a stop action when a target object is detected; S10. The robot selects the action with the highest probability to explore the scene.

2. The object navigation method based on object bias correction and directed attention map according to claim 1, characterized in that: In step S2, the local object feature S t By object visual features Object location features Object Confidence and target object indication position Stitched together.

3. The object navigation method based on object bias correction and directed attention map according to claim 1, characterized in that: The step S3 specifically includes the following steps: S31, setting a confidence threshold, and setting the object features below the confidence threshold to 0 through confidence filtering to obtain the filtered object features S32, through the one-hot encoding and two layers of fully connected layers to obtain the object index encoding feature OI, the global image feature I t After global average pooling, the target index is encoded OI p Stitching to obtain global image features containing target object information Then we have: Where M is the number of pixels in the feature map, p represents the target object, and OI p The index code of the target object p in the object index coding feature OI, Concat means concatenating two vectors into one; S33, using global image features containing target object information As a query, the filtered object features As a key, use the multi-head attention score generator to get the adaptive object attention map G v , then: Among them, NH and HD are the number of heads and hidden dimensions of the multi-head attention score generator, respectively. are all learnable dimension mapping parameters, and The global image features containing the target object information are And the filtered local object features Mapped to the same dimension HD, Aggregate the sub-graphs calculated by each head into an adaptive object attention graph G v .

4. The object navigation method based on object bias correction and directed attention map according to claim 1, characterized in that: The step S4 is specifically as follows: Define a learnable fixed-size matrix represents a fixed object attention map, and extracts edge weights from the fixed object attention map according to the object p being searched The unbiased object attention weight G is obtained by weighting the fixed object attention map and the adaptive object attention map t , then: Among them, ω n With ω v are all learnable weights.

5. The object navigation method based on object bias correction and directed attention map according to claim 1, characterized in that: The step S5 is specifically as follows: In order to balance the amount of computation and information integrity, two fully connected layers with ReLU nonlinearity are used to local object features S t Perform dimensionality reduction to obtain the object feature S after dimensionality reduction t ′, using unbiased object attention weight G t For the object feature S after dimensionality reduction t The features of each object q in ′ are weighted to obtain the final encoding of the object branch Then we have: Where N is the total number of objects, is the low-dimensional feature of the qth object, is the unbiased attention weight of the qth object.

6. The object navigation method based on object bias correction and directed attention map according to claim 5, characterized in that: The step S6 specifically includes the following steps: S61. Using unbiased object attention weight G t The object index encoding feature OI is progressively weighted and aggregated to obtain the weighted object semantic representation D, then: in, represents the indicator function, N is the total number of objects, OIq is the index code of the qth object in the object index code feature OI, and the confidence filter The semantic features of objects whose confidence is less than a certain threshold are set to 0; S62, introducing the one-dimensional pixel index code PI into the global image features, we have: AND' t =δ(δ(I t W1)W2)+PI Among them, I′ t is a global image feature with pixel position identification, δ represents the ReLU nonlinear activation function, and W1 and W2 are learnable dimension mapping parameters; S63, using the weighted object semantic representation D as a query, the global image feature I′ with pixel position identification t As the key value, the final encoding of the image branch is obtained through the multi-head attention network 7. The object navigation method based on object bias correction and directed attention map according to claim 1, characterized in that: The step S7 is specifically as follows: Define a vector R = {r1, r2, r3} containing three learnable parameters, and transform the global image branch Local object branch and the past action branch PA are aggregated into a feature representation H through vector R t , then: Among them, F pw Represents point-wise convolution.

8. The object navigation method based on object bias correction and directed attention map according to claim 7, characterized in that: The step S8 is specifically as follows: Use two layers of fully connected layers to represent the feature H t The refined data is used as the input of the LSTM network. The LSTM network learns the temporal relationship of the robot in the process of exploring the environment, and outputs an action probability vector that integrates past actions and the current environment. The actions include going straight, turning left, turning right, raising the head, lowering the head and stopping. If the robot outputs a stop action, it means that the robot has found the object and navigated to the location of the object.

9. The object navigation method based on object bias correction and directed attention map according to claim 3 or 6, characterized in that: The threshold of the confidence filter is 0.

6.

10. The object navigation method based on object bias correction and directed attention map according to claim 1, characterized in that: In step S9, when the robot detects a target object, the target detection confidence is multiplied by a fixed coefficient and then multiplied by the probability of the stop action in the probability vector to achieve explicit enhancement of the stop action trigger.

Images