Visual language navigation method and system based on monocular camera and language instruction

Abstract

The invention discloses a visual language navigation method and system based on a monocular camera and a language instruction, and the method comprises the steps: S1, feature field construction: employing a 3DGS to construct a 3DGS feature field by employing visual information obtained by a monocular RGB-D camera; s2, implicit partial complementation: inferring the representation of a missing region through a context feature relationship to generate a complete feature map; s3, waypoint prediction: generating an aerial view feature map based on the 3DGS feature field, and predicting nearby navigable waypoints through a waypoint predictor; s4, based on active perception of uncertainty, when the robot is uncertain in navigation decision, more visual information is acquired and missing visual information is supplemented by rotating a camera; and S5, constructing a topological map based on the constructed panoramic feature map and the predicted path points, and carrying out navigation decision making by using a panoramic visual language navigation planning model in combination with a language instruction. The problem that monocular vision information is incomplete is effectively solved, and the success rate and efficiency of navigation are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot navigation, and in particular relates to a visual language navigation method and system based on a monocular camera and language instructions. Background Art

[0002] With the rapid development of artificial intelligence and robotics, vision-language navigation (VLN) has become a research hotspot. VLN requires robots to navigate in unknown environments based on natural language instructions. Existing VLN methods mostly assume that robots have access to panoramic visual information. However, in practical applications, robots are typically equipped with a monocular RGB-D camera with a limited field of view, resulting in incomplete information and thus poor navigation performance. Therefore, achieving efficient vision-language navigation using monocular vision has become a pressing technical challenge. Summary of the Invention

[0003] The main purpose of this invention is to solve the problem of incomplete information under monocular vision conditions, and propose a visual language navigation method and system based on a monocular camera and language instructions to improve the accuracy and efficiency of navigation.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The visual language navigation method based on a monocular camera and language instructions includes the following steps:

[0006] S1, feature field construction, using the visual information obtained by the monocular RGB-D camera, 3DGS is used to construct the 3DGS feature field;

[0007] S2, implicit partial completion, addresses the problem of incomplete monocular visual information by inferring the representation of the missing area through contextual feature relationships to generate a complete feature map;

[0008] S3, waypoint prediction, based on the 3DGS feature field, generates a bird's-eye view feature map and predicts nearby navigable waypoints through the waypoint predictor;

[0009] S4, active perception based on uncertainty: when the robot is uncertain about the navigation decision, it rotates the camera to obtain more visual information and supplement the missing visual information;

[0010] S5. Navigation decision: Based on the constructed panoramic feature map and predicted path points, a topological map is constructed, and combined with language instructions, a panoramic visual language navigation planning model is used to make navigation decisions.

[0011] Furthermore, step S1 is specifically as follows:

[0012] The visual information obtained by the monocular RGB-D camera is used to extract the feature map, which is then mapped to the 3D world coordinate system to generate point cloud data.

[0013] The point cloud data is converted into a 3D Gaussian scattering feature field. Each Gaussian point contains coordinates, color, scale, rotation, transparency, and feature information. The coordinates, color, and features are derived from the point cloud data. The scale and transparency are preset to 2.5 cm and 1, respectively, and no rotation is set.

[0014] Furthermore, step S2 is specifically as follows:

[0015] Render feature maps of 12 different viewing angles based on the 3DGS feature field, with each viewing angle spaced 30° apart;

[0016] Each rendered feature map is input into the MAE encoder to generate the corresponding potential representation, which is then decoded into a complete feature map by the decoder and trained using the auxiliary loss function. The auxiliary loss function is:

[0017]

[0018] in, represents the loss function, p represents the masked or incomplete patch index, S is the set of all p, that is, all masked or incomplete patch indices, x′ p is a defective patch that requires model recovery, and x p It is a non-incomplete patch.

[0019] Furthermore, step S3 is specifically as follows:

[0020] Generate a BEV feature map based on 3DGS feature field rendering; input the BEV feature map into the waypoint prediction model to predict nearby navigable waypoints;

[0021] The waypoint prediction model consists of two UNets connected in series. The first UNet completes the relatively incomplete BEV, and the second UNet predicts a heatmap based on the completed BEV to indicate the probability that each point in the bird's-eye view is a waypoint.

[0022] The discrete path points are extracted by the non-maximum suppression method and converted into coordinates in the world coordinate system;

[0023] The predicted waypoints are used to update the topological map for subsequent navigation decisions.

[0024] Furthermore, step S4 is specifically as follows:

[0025] When the robot is uncertain about its navigation decision, it evaluates the reliability of the prediction and determines whether it needs to obtain more visual information. If the prediction is unreliable, it rotates the camera to obtain more visual information.

[0026] By actively rotating the camera to obtain visual information, the robot re-makes navigation decisions, thereby improving navigation accuracy. The evaluation basis for the prediction reliability is the normalized entropy of the probability p predicted by the panoramic visual language navigation planning model, which is calculated as follows:

[0027]

[0028] Among them, N is the number of candidate path points, p i Represents the probability of the i-th path point predicted by the model;

[0029] If the normalized entropy H(p) of the probability distribution p is greater than the preset threshold τ, the uncertainty of the model prediction is considered high, and the camera is only rotated in the direction predicted by the model, without the robot actually moving. During the rotation process, the model obtains new visual information and updates the feature field, and the newly introduced features improve the model's prediction. For directions that the model has already observed, obtaining new information is not helpful for the model's prediction, and the robot rotates the camera and moves in the predicted direction.

[0030] The present invention also includes a visual language navigation system based on a monocular camera and language instructions, the system including a feature field construction module, an implicit part completion module, a waypoint prediction module, an active perception strategy module and a navigation decision module;

[0031] The feature field construction module uses the visual information obtained by the monocular RGB-D camera and adopts 3DGS to construct the 3DGS feature field;

[0032] The implicit partial completion module addresses the problem of incomplete monocular visual information by inferring the representation of the missing area through contextual feature relationships to generate a complete feature map;

[0033] The waypoint prediction module generates a bird's-eye view feature map based on the 3DGS feature field and predicts nearby navigable waypoints through the waypoint predictor;

[0034] Active perception strategy module, which rotates the camera to obtain more visual information and supplement missing visual information when the robot is uncertain about the navigation decision;

[0035] The navigation decision module builds a topological map based on the constructed panoramic feature map and predicted path points, and uses the panoramic visual language navigation planning model in combination with language instructions to make navigation decisions.

[0036] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0037] The present invention effectively solves the problem of incomplete monocular visual information through 3DGS feature field construction, implicit partial completion, path point prediction and active perception strategy of uncertainty, improves the accuracy and efficiency of navigation, and provides an efficient and practical solution for the actual deployment of visual language navigation models. It has broad application prospects in actual robot navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is an overall schematic diagram of the present invention;

[0039] Figure 2 It is a schematic diagram of implicit partial completion;

[0040] Figure 3 It is a schematic diagram of the waypoint prediction;

[0041] Figure 4 This is an actual machine deployment example of the system of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0043] Example

[0044] like Figure 1 As shown, the present invention, a visual language navigation method based on a monocular camera and language instructions, includes the following steps:

[0045] S1. Feature field construction: Using the visual information obtained by the monocular RGB-D camera, the 3DGS feature field is constructed using 3DGS technology. In this embodiment, specifically:

[0046] The visual information obtained by the monocular RGB-D camera is used to extract the feature map, which is then mapped to the 3D world coordinate system to generate point cloud data.

[0047] The point cloud data is converted into a 3D Gaussian scattering (3DGS) feature field. Each Gaussian point contains coordinates, color, scale, rotation, transparency, and feature information. The coordinates, color, and features are derived from the point cloud data. The scale and transparency are preset to 2.5 cm and 1, respectively, and no rotation is set. This 3DGS feature field is used to render the panoramic feature map and the bird's-eye view feature map and subsequent navigation decisions.

[0048] S2, implicit partial completion, to solve the problem of incomplete monocular visual information, the representation of the missing area is inferred through the context feature relationship to generate a complete feature map; in this embodiment, Figure 2 As shown, specifically:

[0049] Render feature maps of 12 different viewing angles based on the 3DGS feature field, with each viewing angle spaced 30° apart;

[0050] Each rendered feature map is input into the MAE (Masked Autoencoder) encoder to generate the corresponding potential representation, which is then decoded into a complete feature map by the decoder and trained using the auxiliary loss function. The auxiliary loss function is specifically:

[0051]

[0052] in, represents the loss function, p represents the masked or incomplete patch index, S is the set of all p, that is, all masked or incomplete patch indices, x′ p is a defective patch that requires model recovery, and x p It is a non-incomplete patch.

[0053] S3, waypoint prediction, based on the 3DGS feature field, generates a bird's eye view (BEV) feature map, and predicts nearby navigable waypoints through the waypoint predictor; in this embodiment, Figure 3 As shown, specifically:

[0054] Generate a bird's-eye view (BEV) feature map based on 3DGS feature field rendering; input the BEV feature map into the waypoint prediction model to predict nearby navigable waypoints;

[0055] The waypoint prediction model consists of two serially connected UNets. The first UNet completes the relatively incomplete BEV, and the second UNet predicts a heatmap based on the completed BEV to represent the probability that each point in the bird's-eye view is a waypoint. Discrete waypoints are extracted using the non-maximum suppression (NMS) method and converted to coordinates in the world coordinate system.

[0056] The predicted waypoints are used to update the topological map for subsequent navigation decisions.

[0057] S4. Active perception based on uncertainty: When the robot is uncertain about its navigation decision, it rotates the camera to obtain more visual information and supplement the missing visual information. In this embodiment, specifically:

[0058] When the robot is uncertain about its navigation decision, it evaluates the reliability of the prediction and determines whether it needs to obtain more visual information. If the prediction is unreliable, it rotates the camera to obtain more visual information.

[0059] By actively rotating the camera to obtain visual information, the robot re-makes navigation decisions, thereby improving navigation accuracy. The evaluation basis for the prediction reliability is the normalized entropy of the probability p predicted by the panoramic visual language navigation planning model, which is calculated as follows:

[0060]

[0061] Among them, N is the number of candidate path points, p i Represents the probability of the i-th path point predicted by the model;

[0062] If the normalized entropy H(p) of the probability distribution p is greater than the preset threshold τ, the uncertainty of the model prediction is considered high, and the camera is only rotated in the direction predicted by the model, without the robot actually moving. During the rotation process, the model obtains new visual information and updates the feature field, and the newly introduced features improve the model's prediction. For directions that the model has already observed, obtaining new information is not helpful for the model's prediction, and the robot rotates the camera and moves in the predicted direction.

[0063] S5. Navigation decision: Based on the constructed panoramic feature map and predicted path points, a topological map is constructed, and combined with language instructions, a panoramic visual language navigation planning model is used to make navigation decisions.

[0064] In another embodiment, a visual language navigation system based on a monocular camera and language instructions is provided. The system adopts the visual language navigation method of the above embodiment and includes a feature field construction module, an implicit partial completion module, a waypoint prediction module, an active perception strategy module, and a navigation decision module.

[0065] The feature field construction module uses the visual information obtained by the monocular RGB-D camera and adopts 3DGS technology to construct the 3DGS feature field;

[0066] The implicit partial completion module addresses the problem of incomplete monocular visual information by inferring the representation of the missing area through contextual feature relationships to generate a complete feature map;

[0067] The waypoint prediction module generates a bird's-eye view (BEV) feature map based on the 3DGS feature field and predicts nearby navigable waypoints through the waypoint predictor;

[0068] Active perception strategy module, which rotates the camera to obtain more visual information and supplement missing visual information when the robot is uncertain about the navigation decision;

[0069] The navigation decision module builds a topological map based on the constructed panoramic feature map and predicted path points, and uses the panoramic visual language navigation planning model in combination with language instructions to make navigation decisions.

[0070] like Figure 4 The figure shows a schematic diagram of an actual machine deployment example of the system in this embodiment.

[0071] It should also be noted that, in this specification, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or apparatus comprising the element.

[0072] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual language navigation method based on a monocular camera and language instructions, characterized in that: The following steps are involved: S1, feature field construction, using the visual information obtained by the monocular RGB-D camera, 3DGS is used to construct the 3DGS feature field; S2, implicit partial completion, addresses the problem of incomplete monocular visual information by inferring the representation of the missing area through contextual feature relationships to generate a complete feature map; S3, waypoint prediction, based on the 3DGS feature field, generates a bird's-eye view feature map and predicts nearby navigable waypoints through the waypoint predictor; S4, active perception based on uncertainty: when the robot is uncertain about the navigation decision, it rotates the camera to obtain more visual information and supplement the missing visual information; S5. Navigation decision: Based on the constructed panoramic feature map and predicted path points, a topological map is constructed, and combined with language instructions, a panoramic visual language navigation planning model is used to make navigation decisions.

2. The visual language navigation method based on a monocular camera and language instructions according to claim 1, characterized in that: Step S1 is specifically as follows: The visual information obtained by the monocular RGB-D camera is used to extract the feature map, which is then mapped to the 3D world coordinate system to generate point cloud data. The point cloud data is converted into a 3D Gaussian scattering feature field. Each Gaussian point contains coordinates, color, scale, rotation, transparency, and feature information. The coordinates, color, and features are derived from the point cloud data. The scale and transparency are preset to 2.5 cm and 1, respectively, and no rotation is set.

3. The visual language navigation method based on a monocular camera and language instructions according to claim 2, characterized in that: Step S2 is specifically as follows: Render feature maps of 12 different viewing angles based on the 3DGS feature field, with each viewing angle spaced 30° apart; Each rendered feature map is input into the MAE encoder to generate the corresponding potential representation, which is then decoded into a complete feature map by the decoder and trained using the auxiliary loss function. The auxiliary loss function is: in, represents the loss function, p represents the masked or incomplete patch index, S is the set of all p, that is, all masked or incomplete patch indices, x′ p is a defective patch that requires model recovery, and x p It is a non-incomplete patch.

4. The visual language navigation method based on a monocular camera and language instructions according to claim 2, characterized in that: Step S3 is specifically as follows: Generate a BEV feature map based on 3DGS feature field rendering; input the BEV feature map into the waypoint prediction model to predict nearby navigable waypoints; The waypoint prediction model consists of two UNets connected in series. The first UNet completes the relatively incomplete BEV, and the second UNet predicts a heatmap based on the completed BEV to indicate the probability that each point in the bird's-eye view is a waypoint. The discrete path points are extracted by the non-maximum suppression method and converted into coordinates in the world coordinate system; The predicted waypoints are used to update the topological map for subsequent navigation decisions.

5. The visual language navigation method based on a monocular camera and language instructions according to claim 1, characterized in that: Step S4 is specifically as follows: When the robot is uncertain about its navigation decision, it evaluates the reliability of the prediction and determines whether it needs to obtain more visual information. If the prediction is unreliable, it rotates the camera to obtain more visual information. By actively rotating the camera to obtain visual information, the robot re-makes navigation decisions, thereby improving navigation accuracy. The evaluation basis for the prediction reliability is the normalized entropy of the probability p predicted by the panoramic visual language navigation planning model, which is calculated as follows: Where N is the number of candidate path points, p i Represents the probability of the i-th path point predicted by the model; If the normalized entropy H(p) of the probability distribution p is greater than the preset threshold τ, the uncertainty of the model prediction is considered high, and the camera is only rotated in the direction predicted by the model, without the robot actually moving. During the rotation process, the model obtains new visual information and updates the feature field, and the newly introduced features improve the model's prediction. For directions that the model has already observed, obtaining new information is not helpful for the model's prediction, and the robot rotates the camera and moves in the predicted direction.

6. A visual language navigation system based on a monocular camera and language instructions, characterized by: The system includes a feature field construction module, an implicit part completion module, a waypoint prediction module, an active perception strategy module, and a navigation decision module; The feature field construction module uses the visual information obtained by the monocular RGB-D camera and adopts 3DGS to construct the 3DGS feature field; The implicit partial completion module addresses the problem of incomplete monocular visual information by inferring the representation of the missing area through contextual feature relationships to generate a complete feature map; The waypoint prediction module generates a bird's-eye view feature map based on the 3DGS feature field and predicts nearby navigable waypoints through the waypoint predictor; Active perception strategy module, which rotates the camera to obtain more visual information and supplement missing visual information when the robot is uncertain about the navigation decision; The navigation decision module builds a topological map based on the constructed panoramic feature map and predicted path points, and uses the panoramic visual language navigation planning model in combination with language instructions to make navigation decisions.

7. The visual language navigation system based on a monocular camera and language instructions according to claim 6, characterized in that: 3DGS is used to construct the 3DGS feature field, specifically: The visual information obtained by the monocular RGB-D camera is used to extract the feature map, which is then mapped to the 3D world coordinate system to generate point cloud data. The point cloud data is converted into a 3D Gaussian scattering feature field. Each Gaussian point contains coordinates, color, scale, rotation, transparency, and feature information. The coordinates, color, and features are derived from the point cloud data. The scale and transparency are preset to 2.5 cm and 1, respectively, and no rotation is set.

8. The visual language navigation system based on a monocular camera and language instructions according to claim 7, characterized in that: The specific workflow of the implicit partial completion module is as follows: Render feature maps of 12 different viewing angles based on the 3DGS feature field, with each viewing angle spaced 30° apart; Each rendered feature map is input into the MAE encoder to generate the corresponding potential representation, which is then decoded into a complete feature map by the decoder and trained using the auxiliary loss function. The auxiliary loss function is: in, represents the loss function, p represents the masked or incomplete patch index, S is the set of all p, that is, all masked or incomplete patch indices, x′ p is a defective patch that requires model recovery, and x p It is a non-incomplete patch.

9. The visual language navigation system based on a monocular camera and language instructions according to claim 7, characterized in that: The workflow of the waypoint prediction module is as follows: Generate a BEV feature map based on 3DGS feature field rendering; input the BEV feature map into the waypoint prediction model to predict nearby navigable waypoints; The waypoint prediction model consists of two UNets connected in series. The first UNet completes the relatively incomplete BEV, and the second UNet predicts a heatmap based on the completed BEV to indicate the probability that each point in the bird's-eye view is a waypoint. The discrete path points are extracted by the non-maximum suppression method and converted into coordinates in the world coordinate system; The predicted waypoints are used to update the topological map for subsequent navigation decisions.

10. The visual language navigation system based on a monocular camera and language instructions according to claim 6, characterized in that: The active perception strategy module is specifically: When the robot is uncertain about its navigation decision, it evaluates the reliability of the prediction and determines whether it needs to obtain more visual information. If the prediction is unreliable, it rotates the camera to obtain more visual information. By actively rotating the camera to obtain visual information, the robot re-makes navigation decisions, thereby improving navigation accuracy. The evaluation basis for the prediction reliability is the normalized entropy of the probability p predicted by the panoramic visual language navigation planning model, which is calculated as follows: Where N is the number of candidate path points, p i Represents the probability of the i-th path point predicted by the model; If the normalized entropy H(p) of the probability distribution p is greater than the preset threshold τ, the uncertainty of the model prediction is considered high, and the camera is only rotated in the direction predicted by the model, without the robot actually moving. During the rotation process, the model obtains new visual information and updates the feature field, and the newly introduced features improve the model's prediction. For directions that the model has already observed, obtaining new information is not helpful for the model's prediction, and the robot rotates the camera and moves in the predicted direction.

Images