Visual navigation method and system based on fault-tolerant target positioning and medium

Abstract

The invention belongs to the technical field of visual navigation, and particularly relates to a visual navigation method and system based on fault-tolerant target localization and a medium, and the method comprises the following steps: generating appearance and position similarity among all stored target detection records by using an appearance and position similarity calculation module; performing clustering analysis on the detection records with different similarities by using a clustering module; based on a clustering result, generating a masked attention matrix by using a sparse self-attention module, and updating a detection record; a target cross attention module is used, and the embedded representation of the target object is used as a query vector to carry out attention operation on the updated detection record; and finally, generating a specific agent moving or steering action by using a strategy module. According to the model disclosed by the invention, the navigation success rate and the navigation efficiency of the intelligent agent are improved by reducing the interference and misguidance of non-target class objects on positioning of a real target.

Description

Technical Field

[0001] The present application belongs to the field of visual navigation technology, and specifically relates to a visual navigation method, system and medium based on fault-tolerant target positioning. Background Art

[0002] Visual navigation technology is a method that uses real-time visual input from an agent or device to achieve autonomous navigation. It analyzes image data from the first-person perspective, identifies environmental features and target objects, and guides the agent to reach the predetermined target accurately and efficiently. This technology is widely used in robotics, unmanned vehicles, and augmented reality. However, in actual scenarios, confusing objects that are similar to the target object but of different types are often likely to appear, which interferes with and misleads the positioning of the target. Therefore, fault-tolerant target positioning capabilities need to be taken into consideration in the design of navigation models.

[0003] There are currently two main methods for target positioning of visual navigation agents: time domain convolutional network type and attention mechanism type. The former uses multi-scale convolution kernels to perform convolution operations on the stored target detection records, which is suitable for fixed-length detection record memory buffer pools. The latter uses a multi-head attention mechanism, which can comprehensively analyze detection records of different capacity sizes and refine the directional information of the generated target. However, the existing methods do not consider how to identify the difference between objects of different categories but similar in appearance to the target type in the scene and avoid interference of such objects in target positioning. This results in insufficient navigation efficiency of the agent or even navigation to the wrong target. Summary of the invention

[0004] In response to the above technical problems, the present invention provides a visual navigation method based on fault-tolerant target positioning. By using clustering to comprehensively compare and analyze the information of potential target objects detected during the navigation process, and utilizing a sparse self-attention mechanism to avoid interference of confusing targets on real targets, the target positioning fault-tolerant capability of the intelligent body is enhanced, and the navigation efficiency and success rate are improved.

[0005] To achieve the above object, the technical solution of the present invention is as follows: A visual navigation method based on fault-tolerant target positioning comprises the following steps: S1. Appearance and position similarity calculation: Generate the appearance and position similarity between all target detection records stored up to the current moment, generate the query vector, key vector and value vector as well as the initial attention matrix; S2. Clustering: Use the query vector and key vector generated in step S1 to calculate the distance between the detection records and perform cluster analysis on these records; S3. Sparse self-attention calculation: Use the clustering result obtained in step S2 to perform a mask operation on the initial attention matrix generated in step S1, and obtain the updated matrix to perform sparse self-attention calculation on the value vector obtained in step S1; S4. Target cross attention calculation: Use the embedded representation of the target object as the query vector and perform cross attention calculation on the sparse self-attention result output in step S3, so as to reduce or eliminate the influence of similar objects that are easy to cause confusion on locating the real target; S5. Action generation: Generate specific agent movement or turning actions based on recursive neural networks and fully connected neural networks.

[0006] Preferably, step S1 uses the target object detection records collected by the intelligent agent through the target detector during the navigation process , get the position query vector and position key vector , : ; ; in, L2 norm normalization; is the plane coordinate of the target detection box of the i-th target object detected in the image; Contains the relative spatial position coordinates of the agent when the i-th target object is detected, as well as the rotation and pitch angles of the camera; is a learnable parameter matrix; , They are used to calculate the location similarity respectively; Use the target object detection records collected by the agent during navigation to obtain the appearance query vector and appearance key vector , : ; ; in, is the appearance feature of the i-th target object detected; is a learnable parameter matrix; , They are the query vector and key vector used to calculate the appearance similarity respectively; Calculate the initial attention matrix : ; in is the sigmoid activation function; A learnable parameter for balancing position and appearance similarity, T is the transpose.

[0007] Preferably, in step S1, the target object detection records collected by the agent during navigation are used to obtain a value vector for comprehensively refining the target information. : ; in, is the appearance feature of the i-th target object detected; is a learnable parameter matrix; It is a value vector that integrates the position and appearance information of the i-th detected object.

[0008] Preferably, the clustering step in step S2 is as follows: S21. Calculate the distance between any two target object detection records ; ; S22. Under the set maximum neighbor distance Epsilon and minimum number of neighbors MinPts hyperparameters, the DBSCAN algorithm is applied to cluster all detection records.

[0009] Preferably, the sparse self-attention calculation steps in step S3 are as follows: S31. Perform mask operation on the initial attention matrix to obtain the updated attention matrix : ; S32. Use the masked attention matrix to detect the recorded value vector Perform sparse self-attention calculation to obtain updated detection records : ; in Used to convert the input into a probability distribution, ensuring that each element of the output is non-negative and the sum is 1; Refers to the dimension size of the key vector; This is the updated test record.

[0010] Preferably, step S4 target cross attention calculation: Use the target category semantic vector obtained from the Glove dataset Query the updated test records Perform cross-attention calculations to reduce or even remove the influence of confusing objects with low similarity to the real target, and obtain a directional vector that is not easily disturbed by false detection. : ; in is the learnable parameter mapping matrix; is the confidence vector of the detected target object; is the normalized attention weight vector; is a vector concatenation operation; is the number of heads in the multi-head attention calculation; It is the directional vector that ultimately performs the fault-tolerant target positioning function.

[0011] Preferably, the action generation steps in step S5 are as follows: S51. Global features of images extracted using multi-layer convolutional networks and the action feature vector performed by the agent at the last moment , combined with the directivity vector , generates the state vector of the agent at the current moment : ; in, is a fully connected layer; It is the Layernorm normalization layer; S52. Generate the next action that the agent will perform: ; ; in is a long short-term memory recurrent neural network; is the hidden state and cell state vector of the agent at the previous moment; is the output state, hidden state and cell state vector at the current moment; is a multi-layer perceptron network; is the action vector to be executed next.

[0012] A visual navigation system based on fault-tolerant target positioning is used to implement the visual navigation method based on fault-tolerant target positioning described in the present application.

[0013] A storage medium, when running on a computer, executes the steps of the visual navigation method based on fault-tolerant target positioning described in the present application.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The model disclosed in the present invention improves the navigation success rate and navigation efficiency of the intelligent agent by reducing the interference and misleading of non-target category objects on the positioning of the real target. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1An overall schematic diagram of a visual navigation method based on fault-tolerant target positioning disclosed in an embodiment of the present invention; Figure 2 Schematic diagram of a sparse self-attention calculation module in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Embodiment 1: The present invention provides a visual navigation method based on fault-tolerant target positioning, such as Figure 1 As shown in the figure, the model uses clustering to comprehensively compare and analyze the information of potential target objects detected during navigation, and takes advantage of the sparse self-attention mechanism to avoid the interference of confusing targets on the real targets, thereby improving the target positioning tolerance of the intelligent agent and improving the navigation efficiency and success rate.

[0018] A visual navigation method based on error-tolerant target positioning includes the following steps: (1) Appearance and position similarity calculation: Generate the appearance and position similarity between all target detection records stored up to the current moment, generate the query vector, key vector, value vector and initial attention matrix.

[0019] The specific method of step (1) is as follows: (1.1) Use the target object detection records collected by the target detector during the agent's navigation process , obtain the location query vector and location key vector used to calculate the cluster distance in the subsequent step (2.1) , : ; ; in, is L2 norm normalization; is the plane coordinate of the target detection box of the i-th target object detected in the image; Contains the relative spatial position coordinates of the agent when the i-th target object is detected, as well as the rotation and pitch angles of the camera; is a learnable parameter matrix; , are used to calculate the location similarity respectively.

[0020] (1.2) Use the target object detection records collected by the agent during navigation to obtain the appearance query vector and appearance key vector used to calculate the cluster distance in the subsequent step (2.1) , : ; ; in, is the appearance feature of the i-th target object detected; is a learnable parameter matrix; , They are the query vector and key vector used to calculate the appearance similarity.

[0021] (1.3) Use the target object detection records collected by the agent during navigation to obtain a value vector for comprehensive and refined target information : ; in, is the appearance feature of the i-th target object detected; is a learnable parameter matrix; It is a value vector that integrates the position and appearance information of the i-th detected object.

[0022] (1.4) Using the query vector and key vector pairs obtained in (1.1) and (1.2), calculate the initial attention matrix : ; in is the sigmoid activation function; Learnable parameters for balancing location and appearance similarity.

[0023] (2) Clustering: Use the query vector and key vector generated in step (1) to calculate the distance between the detection records, and use the DBSCAN algorithm to perform cluster analysis on these records; The specific method of step (2) is as follows: (2.1) Use the distance generated in (1) to calculate the distance between any two target object detection records : ; in is the sigmoid activation function; Keep the same as step (1.4).

[0024] (2.2) Using the distance obtained in (2.1) ,When the maximum neighbor distance Epsilon and the minimum number of neighbors MinPts are set, the DBSCAN algorithm is applied to cluster all the detection records.

[0025] (3) Sparse self-attention calculation: Use the clustering result obtained in step (2) to perform a mask operation on the initial attention matrix generated in step (1) to obtain the updated matrix and perform sparse self-attention calculation on the value vector obtained in step (1); Figure 2 As shown, similar objects belonging to different categories can be identified and distinguished in the sparse attention matrix, thus avoiding the interference of false targets on real targets.

[0026] The specific method of step (3) is as follows: (3.1) According to the clustering results obtained in (2.2), the initial attention matrix in (1.4) is masked to obtain the updated attention matrix : .

[0027] (3.2) Using the masked attention matrix in (3.1), perform sparse self-attention calculation on the detection record value vector obtained in (1.3) to obtain the updated detection record : ; in Used to convert the input into a probability distribution, ensuring that each element of the output is non-negative and the sum is 1; Refers to the dimension size of the key vector; This is the updated test record.

[0028] (4) Target cross-attention calculation: Use the embedded representation of the target object as the query vector and perform cross-attention calculation on the sparse self-attention result output in step (3), thereby reducing or eliminating the impact of similar objects that are easy to cause confusion on locating the true target.

[0029] The specific method of step (4) includes: Use the target category semantic vector obtained from the Glove dataset Query the updated test records obtained in step (3.2) Perform cross-attention calculations to reduce or even remove the influence of confusing objects with low similarity to the real target, and obtain a directional vector that is not easily disturbed by false detection. : ; in is the learnable parameter mapping matrix; is the confidence vector of the detected target object; is the normalized attention weight vector; is a vector concatenation operation; is the number of heads in the multi-head attention calculation; It is the directional vector that ultimately performs the fault-tolerant target positioning function.

[0030] (5) Action generation: Generate specific intelligent agent movement or turning actions based on recursive neural networks and fully connected neural networks.

[0031] The specific method of step (5) is as follows: (5.1) Using multi-layer convolutional networks to extract global features of images and the action feature vector performed by the agent at the last moment , combined with the directivity vector obtained in step (4) , generates the state vector of the agent at the current moment : ; in, is a fully connected layer; It is the Layernorm normalization layer.

[0032] (5.2) Using (5.1) the state vector is obtained And the historical state information of the agent, generate the action that the agent will perform next: ; ; in is a long short-term memory recurrent neural network; is the hidden state and cell state vector of the agent at the previous moment; is the output state, hidden state and cell state vector at the current moment; is a multi-layer perceptron network; is the action vector to be executed next.

[0033] Embodiment 2: The present invention also provides an embodiment of a visual navigation system based on fault-tolerant target positioning, using the navigation method described in the above embodiment, the navigation system comprises: (1) Appearance and position similarity calculation module: Generates the appearance and position similarity between all target detection records stored up to the current moment, generates the query vector, key vector, value vector and initial attention matrix; (2) Clustering module: Use the query vector and key vector generated in step (1) to calculate the distance between the detection records, and use the DBSCAN algorithm to perform cluster analysis on these records; (3) Sparse self-attention calculation module: Use the clustering result obtained in step (2) to perform a mask operation on the initial attention matrix generated in step (1) to obtain the updated matrix. Perform sparse self-attention calculation on the value vector obtained in step (1). Figure 2 That is a schematic diagram of a target feature generation module in an embodiment of the present invention; (4) Target cross-attention calculation module: Use the embedded representation of the target object as the query vector and perform cross-attention calculation on the sparse self-attention result output in step (4), thereby reducing or eliminating the influence of similar objects that are easy to cause confusion on locating the true target; (5) Action generation module: Generates specific intelligent agent movement or turning actions based on recursive neural networks and fully connected neural networks.

[0034] The method embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0035] The system is constructed to run the method of the present application. Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0036] In order to evaluate the effectiveness of the model proposed in this invention, we designed the following experiment: The experimental environment is based on the Ai2Thor and RoboThor platforms. In Ai2Thor, there are 30 different rooms, 20 of which are used to train the model, 5 are used to verify the model performance, and the remaining 5 are used for the final test. On the RoboThor platform, there are 75 floors, of which we selected 60 floors as the training set, 5 floors for the verification stage, and the other 10 floors for test evaluation. In this way, we can fully verify the actual application effect of the model.

[0037] The experimental results use SR and SPL to evaluate the performance of the model, which are the most commonly used evaluation indicators in visual navigation. In order to reflect the navigation ability of the intelligent agent in large scenes and long paths, we use the optimal path experimental results of the starting position to the target end point as shown in Tables 1 and 2.

[0038] TAMSA and NTWA in Table 1 and Table 2 are commonly used target localization methods in end-to-end visual navigation. From the results in the table, it can be seen that the model of the present invention is better than the other two methods in navigating to unseen objects, which fully proves the effectiveness of the model proposed in the present invention.

[0039] Table 1 Ai2Thor experimental results .

[0040] Table 2 RoboThor experimental results .

[0041] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual navigation method based on fault-tolerant target positioning, characterized in that: The following steps are involved: S1. Appearance and position similarity calculation: Generate the appearance and position similarity between all target detection records stored up to the current moment, generate the query vector, key vector and value vector as well as the initial attention matrix; S2. Clustering: Use the query vector and key vector generated in step S1 to calculate the distance between the detection records and perform cluster analysis on the detection records; S3. Sparse self-attention calculation: Use the clustering result obtained in step S2 to perform a mask operation on the initial attention matrix generated in step S1 to obtain an updated matrix, and perform sparse self-attention calculation on the value vector obtained in step S1; S4. Target cross attention calculation: Use the embedded representation of the target object as the query vector and perform cross attention calculation on the sparse self-attention result output in step S3; S5. Action generation: Generate specific agent movement or turning actions based on recursive neural networks and fully connected neural networks.

2. The visual navigation method based on fault-tolerant target positioning according to claim 1 is characterized in that: Step S1 uses the target object detection records collected by the agent during navigation through the target detector , get the position query vector and position key vector , : ； ； in, is L2 norm normalization; is the plane coordinate of the target detection box of the i-th target object detected in the image; Contains the relative spatial position coordinates of the agent when the i-th target object is detected, as well as the rotation and pitch angles of the camera; is a learnable parameter matrix; , They are used to calculate the location similarity respectively; Use the target object detection records collected by the agent during navigation to obtain the appearance query vector and appearance key vector , : ; ; in, is the appearance feature of the i-th target object detected; is a learnable parameter matrix; , They are the query vector and key vector used to calculate the appearance similarity respectively; Calculate the initial attention matrix : ； in is the sigmoid activation function; Learnable parameters for balancing location and appearance similarity.

3. The visual navigation method based on fault-tolerant target positioning according to claim 1, characterized in that: In step S1, the target object detection records collected by the agent during navigation are used to obtain a value vector for comprehensive and refined target information : ； in, is the appearance feature of the i-th target object detected; is a learnable parameter matrix; It is a value vector that integrates the position and appearance information of the i-th detected object.

4. The visual navigation method based on fault-tolerant target positioning according to claim 1, characterized in that: The clustering steps in step S2 are as follows: S21. Calculate the distance between any two target object detection records ; ; S22. Under the set maximum neighbor distance Epsilon and minimum number of neighbors MinPts hyperparameters, the DBSCAN algorithm is applied to cluster all detection records.

5. The visual navigation method based on fault-tolerant target positioning according to claim 1, characterized in that: The sparse self-attention calculation steps in step S3 are as follows: S31. Perform mask operation on the initial attention matrix to obtain the updated attention matrix : ； S32. Use the masked attention matrix to detect the recorded value vector Perform sparse self-attention calculation to obtain updated detection records : ； in Used to convert the input into a probability distribution, ensuring that each element of the output is non-negative and the sum is 1; Refers to the dimension size of the key vector; This is the updated test record.

6. The visual navigation method based on fault-tolerant target positioning according to claim 1, characterized in that: Step S4 target cross attention calculation: Use the target category semantic vector obtained from the Glove dataset Query the updated test records Perform cross-attention calculations to reduce or even remove the influence of confusing objects with low similarity to the real target, and obtain a directional vector that is not easily disturbed by false detection. : ； in is the learnable parameter mapping matrix; is the confidence vector of the detected target object; is the normalized attention weight vector; is a vector concatenation operation; is the number of heads in the multi-head attention calculation; It is the directional vector that ultimately performs the fault-tolerant target positioning function.

7. The visual navigation method based on fault-tolerant target positioning according to claim 1, characterized in that: Step S5: Action generation steps are as follows: S51. Global features of images extracted using multi-layer convolutional networks and the action feature vector performed by the agent at the last moment , combined with the directivity vector , generates the state vector of the agent at the current moment : ； in, is a fully connected layer; It is the Layernorm normalization layer; S52. Generate the next action that the agent will perform: ； ； in is a long short-term memory recurrent neural network; is the hidden state and cell state vector of the agent at the previous moment; is the output state, hidden state and cell state vector at the current moment; is a multi-layer perceptron network; is the action vector to be executed next.

8. A visual navigation system based on fault-tolerant target positioning, characterized in that: Used to implement the visual navigation method based on fault-tolerant target positioning as described in any one of claims 1-7.

9. A storage medium, characterized in that: When the storage medium is run on a computer, the steps in the visual navigation method based on fault-tolerant target positioning according to any one of claims 1 to 7 are executed.

Images