Visual navigation method and device for mobile robot

Abstract

The invention provides a visual navigation method and device for a mobile robot, and relates to the technical field of artificial intelligence, and the method comprises the steps: respectively inputting an image target, a point target and a navigation instruction in a topological map of an environment where the mobile robot is currently located and a current observation image of the mobile robot into a first condition encoder; obtaining a current observation feature, a target image similar feature, a point target feature and a navigation instruction feature; then inputting the image into a second condition encoder to obtain a mask fusion feature; inputting the mask fusion feature, the time step feature and the random Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed; and finally, driving the mobile robot to move based on the to-be-executed track sequence data. According to the visual navigation method for the mobile robot, the navigation success rate and the farthest navigation distance of the mobile robot are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a mobile robot visual navigation method and device. Background Art

[0002] Multimodal target-driven visual navigation of mobile robots refers to generating movement trajectories based on pre-acquired topological maps to track the exploration paths of known environments.

[0003] Existing mobile robot visual navigation approaches either independently design the observation image feature extraction branch and the image target feature extraction branch to perform multimodal target navigation tasks, or employ contrastive learning to align image target and language command features into the same feature space to perform multimodal target navigation tasks. However, these approaches fail to achieve the complementary fusion of multimodal targets in different scenarios, limiting the overall performance of multimodal target visual navigation tasks and resulting in low navigation success rates and short navigation distances for mobile robots. Summary of the Invention

[0004] The present invention provides a mobile robot visual navigation method and device, which are used to solve the technical problems of low navigation success rate and short navigation distance of mobile robots in the prior art.

[0005] The present invention provides a mobile robot visual navigation method, comprising the following steps: Obtain the current observation image of the mobile robot; Obtaining image targets, point targets, and navigation instructions in a topological map of the environment in which the mobile robot is currently located; Inputting the current observation image, the image target, the point target and the navigation instruction into a first conditional encoder respectively, and obtaining the current observation feature, the target image similarity feature, the point target feature and the navigation instruction feature output by the first conditional encoder; Inputting the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder; Inputting the mask fusion feature, the time step feature and the random Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder; The mobile robot is driven to move based on the trajectory sequence data to be executed.

[0006] According to a mobile robot visual navigation method provided by the present invention, the step of obtaining an image target, a point target, and a navigation instruction for each node in a topological map of an environment in which the mobile robot is currently located comprises: Determining an image target and a point target for each node in the topological map based on continuous image frames collected by the mobile robot in the current environment and local mileage records corresponding to the image frames; The coordinate transformation is performed on the point target of each node in the topological map to determine the navigation instruction.

[0007] According to a mobile robot visual navigation method provided by the present invention, the first conditional encoder includes a first encoder, a second encoder, a third encoder and a fourth encoder; The first encoder is used to extract the current observation feature according to the current observation image; The second encoder is used to extract similar features of the target image based on the spliced ​​image; wherein the spliced ​​image is obtained by channel splicing the current observation image and the image target; The third encoder is used to extract point target features based on the point target; The fourth encoder is used to extract navigation instruction features according to the navigation instruction.

[0008] According to a mobile robot visual navigation method provided by the present invention, after inputting the current observation image, the image target, the point target, and the navigation instruction into a first conditional encoder respectively and obtaining the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature output by the first conditional encoder, the method further includes: Inputting the target image similarity feature into an image similarity feature decoder, and obtaining the time distance and relative yaw angle output by the image similarity feature decoder; The time distance is used to determine the distance of the image target relative to the current observation image, and the relative yaw angle is used to determine the orientation of the image target relative to the current observation image.

[0009] According to a mobile robot visual navigation method provided by the present invention, the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature are input into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder, including: Position encoding is performed on the target image similarity features, the point target features, and the navigation instruction features to obtain a multimodal feature sequence corresponding to the target image similarity features, the point target features, and the navigation instruction features; Determining the mask feature sequence based on a screening result of one or more features in the multimodal feature sequence; The current observation feature and the mask feature sequence are input into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder.

[0010] According to a mobile robot visual navigation method provided by the present invention, the mask fusion feature, the time step feature and the randomly sampled Gaussian noise are input into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder, including: Inputting the mask fusion feature, the time step feature and the random Gaussian noise into a trajectory decoder to obtain denoised Gaussian noise; Repeat the iterative steps for a preset number of times to obtain the final Gaussian noise; Using the final random Gaussian noise as the trajectory sequence data to be executed; The iterative step includes: Obtain predicted noise based on the mask fusion feature, the time step feature, and the denoised Gaussian noise obtained in the previous iteration; Obtaining Gaussian noise obtained in this round of iteration based on the mask fusion feature, the time step feature, and the predicted noise; The random Gaussian noise used in the first round of iteration is randomly sampled from the standard Gaussian distribution.

[0011] The present invention also provides a mobile robot visual navigation device, comprising the following modules: A first acquisition module is used to acquire the current observation image of the mobile robot; A second acquisition module is used to acquire image targets, point targets and navigation instructions in a topological map of the environment where the mobile robot is currently located; a feature extraction module, configured to input the current observation image, the image target, the point target, and the navigation instruction into a first conditional encoder, respectively, and obtain the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature output by the first conditional encoder; a feature fusion module, configured to input the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder; A prediction module, configured to input the mask fusion feature, the time step feature, and the random Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder; A navigation module is used to drive the mobile robot to move based on the trajectory sequence data to be executed.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements any one of the above-described mobile robot visual navigation methods.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described mobile robot visual navigation methods.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned mobile robot visual navigation methods.

[0015] The present invention provides a mobile robot visual navigation method, which obtains the current observation image of the mobile robot; obtains image targets, point targets and navigation instructions in the topological map of the mobile robot's current environment; then inputs the current observation image, image target, point target and navigation instruction into a first conditional encoder respectively, and obtains the current observation feature, target image similarity feature, point target feature and navigation instruction feature output by the first conditional encoder; then inputs the current observation feature, target image similarity feature, point target feature and navigation instruction feature into a second conditional encoder, and obtains the mask fusion feature output by the second conditional encoder, thereby using a multimodal target mask mechanism. And the diffusion strategy's perception ability of different guidance conditions realizes the effective fusion of different modal features, breaks through the performance bottleneck of existing multimodal target navigation methods in different scenarios, and obtains more effective features; then the mask fusion features, time step features and random Gaussian noise are input into the trajectory decoder to obtain the trajectory sequence data to be executed output by the trajectory decoder, and realizes the accurate prediction of the movement trajectory of the mobile robot in the current environment; finally, based on the trajectory sequence data to be executed, the mobile robot is driven to move, realizing the effective tracking of the topological map of the mobile robot's current environment, and improving the navigation success rate and the maximum navigation distance of the mobile robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is one of the flow charts of a mobile robot visual navigation method provided by the present invention.

[0018] Figure 2 This is a flowchart of the navigation model training process provided by the present invention.

[0019] Figure 3It is a schematic diagram of a navigation instruction self-labeling method provided by the present invention.

[0020] Figure 4 3 is a comparative schematic diagram of the image target supervision method provided by the present invention.

[0021] Figure 5 This is the second flow chart of a mobile robot visual navigation method provided by the present invention.

[0022] Figure 6 It is a schematic diagram of the flow of the model provided by the present invention during actual prediction.

[0023] Figure 7 This is a schematic diagram of a visualization example of the mobile robot visual navigation method provided by the present invention during simulation testing.

[0024] Figure 8 It is a structural schematic diagram of a mobile robot visual navigation device provided by the present invention.

[0025] Figure 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The goal of multimodal target-driven visual navigation is to use the pre-acquired topological map as a priori clues about the environment to track the exploration path of a known environment. The key challenges of this task are how to effectively construct similar features between the current observation image and the target image, and how to complement the advantages of different modal targets in different scenarios to guide downstream actions or trajectory generation tasks. Based on the differences in target modalities, navigation tasks can be subdivided into command navigation tasks, image target navigation tasks, and multimodal target navigation tasks. However, when performing the three navigation tasks in existing technologies, the following three challenges still exist: First, command navigation tasks require input of the current observed image and the corresponding annotated commands, followed by model training. Previous work on visual language navigation has used public datasets with fully annotated commands for training and testing, or conducted research in established indoor simulation environments. However, existing outdoor navigation datasets lack command labels that match the current observed image and future trajectory sequences. Furthermore, using visual language alignment methods to annotate large-scale navigation data is expensive, and the annotated commands can only be aligned with the observed image, rather than associated with the trajectory sequence or target state, limiting the development of command navigation tasks.

[0028] Second, the image target navigation task directly outputs the trajectory sequence to be executed in the future by inputting the target image and the current observation image. In the prior art, there are two specific methods for completing the image target navigation task. One is to establish the relationship between the observation image, target image, and action sequence through various fusion mechanisms of the target image and the observation image, using the implicit supervision method of the action generation task and end-to-end learning of similar features between images. The other is to use the explicit supervision method of temporal distance regression, using the time frame interval constraint to improve the navigation model's ability to extract image similar features. However, both of the above methods have different degrees of similarity feature ambiguity and lack effective supervision of the orientation features of the target image relative to the observation image. The ambiguity of the similarity features of the image target makes it impossible for the trajectory sequence generation model to clearly determine the orientation information of the image target relative to the current observation image, which in turn affects the robot's navigation performance in scenarios such as forks in the road.

[0029] Third, multimodal target navigation tasks utilize information from different modalities (such as image targets, point targets, and navigation commands) to guide the model to output a trajectory sequence or action toward the target. Two existing approaches address this task. One involves independently designing the observation image feature extraction branch and the image target feature extraction branch. This involves first training the navigation model for the image target, then replacing the image target feature extraction branch with the point target and command extraction network, each fine-tuned on a dataset. This approach cannot uniformly implement navigation models driven by all three modalities, is complex to deploy, and can only accept a single modality target input at a time. Another approach employs contrastive learning to align image target and language command features into the same feature space. While this approach addresses the lack of language command labels in datasets, it transforms the problem of multiple inputs for different modal targets into a single output trajectory sequence into a single input for aligned latent features into a single output trajectory sequence. This cross-modal alignment approach offers the advantage of unifying the navigation tasks driven by different modalities into a common navigation model, but it can only accept a single modality target at a time, preventing the complementary integration of multimodal targets in different scenarios. Both of the above methods ignore to some extent the differences in navigation performance of different modal targets in different scenarios, thereby limiting the overall performance of multimodal target visual navigation tasks.

[0030] In order to solve the problems of low navigation success rate and short navigation distance of mobile robots caused by the lack of instruction annotation data, fuzzy similar features of image targets, and neglected differences in navigation performance of targets of different modalities, the present invention proposes a mobile robot visual navigation method and device to improve the navigation success rate and maximum navigation distance of the mobile robot.

[0031] The following combination Figures 1 to 9 A mobile robot visual navigation method and device of the present invention are described.

[0032] Figure 1 This is one of the flow charts of a mobile robot visual navigation method provided by the present invention, such as Figure 1 As shown, the method includes the following steps: Step 101: Acquire the current observation image of the mobile robot; Specifically, a mobile robot is driven to explore an unknown environment, obtaining a topological map of the environment and the robot's current observation image. The topological map consists of a series of nodes and edges. Nodes are composed of the robot's current observation image and Global Navigation Satellite System (GNSS) information (including latitude, longitude, and yaw angle) of the robot's current exploration position in the environment. Edges are represented by Boolean operators (0 indicates unreachable, 1 indicates reachable). The source node represents the starting point of the robot's exploration, and the target node is the destination point at the robot's first collision.

[0033] Step 102: Obtain image targets, point targets, and navigation instructions in a topological map of the environment in which the mobile robot is currently located; Optionally, the acquiring of image targets, point targets, and navigation instructions in a topological map of an environment in which the mobile robot is currently located includes: Determining an image target and a point target for each node in the topological map based on continuous image frames collected by the mobile robot in the current environment and local mileage records corresponding to the image frames; The coordinate transformation is performed on the point target of each node in the topological map to determine the navigation instruction.

[0034] Specifically, based on the above embodiment, a complete exploration path is constructed using the above topological map information. The continuous image frames collected by the mobile robot in its current environment are the image frames of each node, which are defined as image targets, and the two-dimensional coordinates of each node are defined as point targets. By aligning the image frames and the local odometry records corresponding to the image frames, image targets and point targets at the same instant can be obtained.

[0035] There are several ways to obtain navigation instructions. For example, you can convert the coordinates of each point target in the topological map to obtain navigation instructions; you can also directly input the topological map into a visual language instruction model to obtain navigation instructions output by the visual language instruction model; you can also directly obtain navigation instructions based on the instructions issued by the real-time remote control of the human handle. After obtaining navigation instructions through these methods, they are mapped to corresponding instruction indexes, providing basic information for subsequent extraction of navigation instruction features.

[0036] Step 103: Input the current observation image, the image target, the point target, and the navigation instruction into a first conditional encoder respectively, and obtain the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature output by the first conditional encoder; Optionally, the first conditional encoder includes a first encoder, a second encoder, a third encoder and a fourth encoder; The first encoder is used to extract the current observation feature according to the current observation image; The second encoder is used to extract similar features of the target image based on the spliced ​​image; wherein the spliced ​​image is obtained by channel splicing the current observation image and the image target; The third encoder is used to extract point target features based on the point target; The fourth encoder is used to extract navigation instruction features according to the navigation instruction.

[0037] Specifically, Figure 2 This is a flowchart of the navigation model training process provided by the present invention. Figure 2 A navigation model designed in an embodiment of the present invention includes a first conditional encoder, a second conditional encoder, and a trajectory decoder, wherein the first conditional encoder further includes a first encoder, a second encoder, a third encoder, and a fourth encoder.

[0038] In an embodiment of the present invention, the first encoder is an observation image encoder based on EfficientNet (EN), which is used to extract current observation features based on the current observation image; The second encoder is an image similarity feature encoder based on EN, which is used to extract similar features of the target image based on the spliced ​​image, where the spliced ​​image is obtained by channel splicing of the current observation image and the image target; The third encoder is a point target encoder based on a multi-layer perceptron (MLP), which is used to extract point target features based on the point target. The fourth encoder is an instruction encoder based on MLP, which is used to extract navigation instruction features according to the navigation instruction.

[0039] The input dimension of the observation image encoder is [batch, 3, 640, 360], the input dimension of the image similarity feature encoder is [batch, 6, 640, 360], the input dimension of the point target encoder is [batch, 2], and the input dimension of the instruction encoder is [batch, 1]. However, the dimensions of the output current observation features, target image similarity features, point target features, and navigation instruction features are all [batch, 256].

[0040] To improve the performance of the navigation model, the first encoder in the navigation model can also be pre-supervised trained. The training dataset can be real data collected from campuses, residential areas, and parks, or simulated data from the Town01 scene in the CARLA simulator. In the collected topological map, image targets and point targets at the same instant can be annotated by aligning the image frame and local odometry. However, navigation command annotation is difficult to achieve with existing technologies due to the lack of command data that matches the current observed image and future trajectory sequence.

[0041] In order to solve the problem of lack of navigation instruction annotation data, an embodiment of the present invention proposes to use the relative coordinates of the target point to reversely annotate the navigation instructions, expand the navigation instruction annotation data, improve the model training effect, and further improve the success rate of the mobile robot in executing the instruction navigation task.

[0042] Figure 3 This is a schematic diagram of a navigation instruction self-labeling method provided by the present invention. Figure 3 As shown, navigation instructions are obtained through the instruction self-labeling method. Figure 3 The three trajectory sequences in are composed of 8 points, each of which is the coordinate of the current position O of the mobile robot [ , ]definition, represents the sequence number of the point, All possible target points Randomly select between the future [1, 16] points of point O, that is, The blue arrow indicates the direction of the robot at point O, and the red arrow indicates the direction between point O and the target point. The direction of the line, the angle between the two directions is used definition.

[0043] when , that is, the target point At a certain point in the trajectory sequence, ,like Figure 3 As shown in points A, B and C; when , that is, when the target point is outside the trajectory sequence range, ,like Figure 3 As shown in point D in the figure. is converted to the range of [-180°, 180°]. The positive and negative and size of are divided into 9 intervals, from left to right: [-180°, -35°], [-35°, -20°], [-20°, -8°], [-8°, -2°], [-2°, 2°], [2°, 8°], [8°, 20°], [20°, 35°], [35°, 180°] , mapping each interval into an instruction list in turn "Turn left 35°", "Turn left 20°", "Turn left 8°", "Turn left 2°", "Go straight", "Turn right 2°", "Turn right 8°", "Turn right 20°", "Turn right 35°" , the instruction index [0, 1, 2, 3, 4, 5, 6, 7, 8] is obtained from the instruction list, and the mapped instruction index is the navigation instruction.

[0044] By labeling image targets, point targets and navigation instructions, a training data set is obtained, and the first encoder in the navigation model is pre-trained to improve the performance of the first encoder. Then, the first conditional encoder is used to extract features of different modalities for data of different modalities, providing a basis for the subsequent effective fusion of multimodal features, realizing the full utilization of multimodal features, and thus improving the performance of the navigation model.

[0045] Optionally, after inputting the current observation image, the image target, the point target, and the navigation instruction into a first conditional encoder respectively and obtaining the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature output by the first conditional encoder, the method further includes: Inputting the target image similarity feature into an image similarity feature decoder, and obtaining the time distance and relative yaw angle output by the image similarity feature decoder; The time distance is used to determine the distance of the image target relative to the current observation image, and the relative yaw angle is used to determine the orientation of the image target relative to the current observation image.

[0046] Specifically, Figure 4 is a comparative schematic diagram of the image target supervision method provided by the present invention, such as Figure 4 As shown in the prior art, the image target navigation task often uses the implicit supervision method of trajectory sequence generation task when extracting similar features between the current observation image and the target image, such as Figure 4 As shown in (a) in , or using explicit supervision of time distance regression, such as Figure 4 As shown in (b) in the figure. These all have different degrees of feature ambiguity, which makes it impossible for the model to effectively determine the accurate orientation information of the image target relative to the current observation image, and thus causes the downstream trajectory decoder to be unable to correctly generate the trajectory sequence leading to the image target. In view of this, the embodiment of the present invention proposes an explicit supervision method that combines the time distance and relative yaw angle between images, such as Figure 4 As shown in (c) in the figure.

[0047] In the traditional temporal distance supervision method, Indicates the number of frame intervals between the current observation image and the target image in the image sequence, Indicates that the target image is on the right side of the current observation image. Indicates that the target image is in front of the current observation image. It means that the target image is on the left side of the current observation image, but the frame interval number of the three is 3. The features after the temporal distance constraint of the downstream branch can only determine the temporal distance of the target image relative to the observation image, but cannot distinguish its relative orientation information.

[0048] In the improved approach of combining time distance with relative yaw angle, the yaw angle information of the image frame can be obtained from the mileage information collected by the mobile robot in its current environment. By adding a yaw angle constraint on the target image relative to the observation image, the relative orientation of the target image can be distinguished based on the sign and magnitude of the relative yaw angle.

[0049] Specifically, Figure 4 The blue arrow in the image represents the normal direction of the current observation image plane, and the red arrow represents the normal direction of the target image plane. A negative value indicates the right side. A value close to 0 indicates the front. A positive value indicates the left side. The value is converted to the range of [-180°, 180°] according to The positive and negative and size of can be divided into 9 intervals, from left to right: [-180°, -65°], [-65°, -30°], [-30°, -10°], [-10°, -2°], [-2°, 2°], [2°, 10°], [10°, 30°], [30°, 65°], [65°, 180°] , mapping each interval into the index of the relative yaw angle in turn ,at this time, , combined with discrete time distances to form label pairs [ ], which can be used to determine the distance and orientation of the image target relative to the current observation image.

[0050] The embodiment of the present invention introduces a relative yaw angle constraint to explicitly supervise similar features between images, thereby enhancing the image target navigation's ability to perceive the target image orientation and improving the navigation success rate of the mobile robot's image target navigation task.

[0051] Based on the above embodiment, before using the navigation model, in order to introduce the relative yaw angle constraint, the navigation model needs to be additionally supervised and trained. During the training phase, the training data set includes a noisy trajectory sequence and the corresponding added noise labels, that is, by inputting the noisy trajectory sequence and the historical observation image, the model can output the corresponding predicted noise. The historical observation image can be real data collected from campuses, residential areas and parks, or simulated data of the Town01 scene in the simulator CARLA. In order to achieve explicit supervision constraints on time distance and relative yaw angle, during the model training phase, the three-layer fully connected network MLP (i.e., image similarity feature decoder) is used as an independent branch to perform label matching on the features fused from the historical observation image and the corresponding target image. , ] explicit supervision constraints, while when using the navigation model for trajectory prediction, the predicted [ , ] information to judge the distance and direction of the image target and realize the perceptual feedback of the environment.

[0052] In addition, during the model training phase, since the time distance, relative yaw angle and prediction noise need to be jointly supervised and optimized, the loss function of the navigation model provided by the present invention is composed of the image target similarity metric loss and noise estimation loss It consists of two parts.

[0053] Among them, the image target similarity measurement loss Divided into time step distance loss and yaw loss Two parts.

[0054] When constructing the training dataset, the time step distance labels are randomly sampled in the interval [1, 16], and the data distribution is balanced, so the MSE loss function is used for calculation. When the time step distance labels are randomly generated, the data sample frequency of the yaw angle index [0, 1, 2, 3, 4, 5, 6, 7, 8] presents a Gaussian distribution with a mean of 4 (indicating straight ahead). , so calculate The weighted MSE loss function is used. The formula is described as follows: Among them, is the scale of the unified loss value, The value is 2, the weight The value is determined according to the inverse of the proportion of each yaw angle index sample in the total samples in the training set. The problem of sample imbalance is alleviated by weighting, thereby improving the prediction accuracy of the model.

[0055] Noise estimation loss MSE loss is also used, and the formula is as follows: The noise estimation model is defined as , are model parameters. is the multimodal conditional feature after position encoding, is the time step, is the true value noise, is the noise trajectory with added noise.

[0056] The total loss function The formula is as follows: In the formula, the weight coefficient Set it according to the order of magnitude of the actual loss value during training. The value is The purpose is to weight the two loss values ​​to the same order of magnitude and sum them. During the training phase, the total loss function Gradient backpropagation is performed to achieve joint supervised optimization of temporal distance, relative yaw angle, and prediction noise.

[0057] Step 104: input the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder; Optionally, inputting the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder includes: Position encoding is performed on the target image similarity features, the point target features, and the navigation instruction features to obtain a multimodal feature sequence corresponding to the target image similarity features, the point target features, and the navigation instruction features; Determining the mask feature sequence based on a screening result of one or more features in the multimodal feature sequence; The current observation feature and the mask feature sequence are input into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder.

[0058] Specifically, Figure 5This is the second flow chart of a mobile robot visual navigation method provided by the present invention, such as Figure 5 shown.

[0059] The first encoder (observation image encoder based on EN) is used to Encode and get the current observation characteristics ; Image Target and the current observation image Perform channel cascade to obtain a spliced ​​image, and use the second encoder (image similarity feature encoder based on EN) to independently encode the spliced ​​image to obtain the image target image similarity feature ; Using the three-layer fully connected network with the same structure, the third encoder (point target encoder based on MLP) and the fourth encoder (command encoder based on MLP) are respectively and navigation instructions Encode them separately to get point target features and navigation command features ; The position encoding of the above-mentioned target image similarity features, point target features and navigation instruction features can be performed on the current observation features. On the basis of the above, we further obtain the multimodal feature sequences corresponding to the target image similarity features, point target features and navigation instruction features [ , , , ], where each element has the shape [batch, 256].

[0060] Design a mask list based on whether the three modal targets are involved in driving the navigation task = {[0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1], [0, 1, 1, 0], [0, 1, 0, 1], [0, 0, 1, 1], [0, 1, 1, 1]}, Seven different combinations can be expressed, including unimodal targets, bimodal targets, and trimodal targets.

[0061] Among them, 0 means that the feature corresponding to the position does not participate in the driving navigation task (the first 0 in the mask list represents the current observation feature , does not participate in the screening), 1 indicates that the feature corresponding to the position participates in driving the navigation task. For example, if the screening result of one or more features in the multimodal feature sequence is [0, 1, 0, 1], then the target image similarity features and navigation instruction features in the mask feature sequence participate in driving the navigation task, and the point target features do not participate in driving the navigation task. That is, the mask feature sequence at this time includes a bimodal target.

[0062] Randomly select masks during model pre-training ,in , thereby improving the generalization performance of the model. In the actual prediction stage of the model, different masks are selected according to the needs of the current navigation environment. Combined with the current observation features.

[0063] In the embodiment of the present invention, the second conditional encoder is a Transformer encoder composed of a multi-head self-attention mechanism, which performs mask fusion on the mask feature sequence after positional encoding (PE) to obtain the mask fusion feature [ , , , ], through this multimodal target mask mechanism, the second conditional encoder can uniformly implement seven different conditional features such as single modality, dual modality and multimodality, such as Figure 5 As shown in (a) in .

[0064] The formula is described as follows: in, Defined as a channel cascade operation, represents the mask fusion feature after position encoding, Represents the parameters of the corresponding model.

[0065] The embodiments of the present invention achieve an effective fusion of the advantages of different modal targets by utilizing a multimodal target mask mechanism and the ability to perceive different guidance conditions with the help of a diffusion strategy, thus breaking through the performance bottleneck of existing multimodal target navigation methods in different scenarios.

[0066] Step 105: Input the mask fusion feature, time step feature, and random Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder; Optionally, inputting the mask fusion feature, the time step feature, and the randomly sampled Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder includes: Inputting the mask fusion feature, the time step feature and the random Gaussian noise into a trajectory decoder to obtain denoised Gaussian noise; Repeat the iterative steps for a preset number of times to obtain the final Gaussian noise; Using the final random Gaussian noise as the trajectory sequence data to be executed; The iterative step includes: Obtain predicted noise based on the mask fusion feature, the time step feature, and the denoised Gaussian noise obtained in the previous iteration; Obtaining Gaussian noise obtained in this round of iteration based on the mask fusion feature, the time step feature, and the predicted noise; The random Gaussian noise used in the first round of iteration is randomly sampled from the standard Gaussian distribution.

[0067] Specifically, Figure 6 This is a flow chart of the actual prediction process of the model provided by the present invention. Figure 6 As shown, in the trajectory decoder part, the input mask fusion feature is [ , , , ], on this basis, add time step features , due to the time step characteristics No mask is needed, so the first time step is added to occupy 0 to obtain the conditional feature for[ , , , , ]. Then at the current Add time step encoding at all times Conditional features of post-cascade It can be expressed as: Among them, the trajectory decoder is adapted from the Transformer decoder. The specific network structure includes multi-head causal attention network layer, layer normalization, multi-head cross attention network layer, layer normalization, feedforward neural network layer, layer normalization and prediction output layer, such as Figure 5 As shown in (b) in .

[0068] like Figure 6 As shown in the figure, after adding the time step feature on the basis of the original mask fusion feature, the conditional feature is obtained, and then the conditional feature and random Gaussian noise are input into the trajectory decoder to obtain the denoised Gaussian noise; Repeat the iterative steps for a preset number of times to obtain the final Gaussian noise; The final random Gaussian noise is used as the trajectory sequence data to be executed; The iterative steps include: Based on the original mask fusion feature in the current Add time step encoding at all times Conditional features obtained by post-cascade And the denoised Gaussian noise obtained in the previous iteration to obtain the predicted noise; Based on conditional features And predict the noise to obtain the Gaussian noise obtained in this round of iteration; The random Gaussian noise used in the first round of iteration is randomly sampled from the standard Gaussian distribution.

[0069] Trajectory sequence of random Gaussian noise through trajectory decoder Perform autoregressive encoding, combine the fusion of mask list and multi-head cross attention mechanism multimodal features, and output the added noise during the training phase through the forward denoising and reverse denoising process of the diffusion strategy. In actual prediction, the random Gaussian noise is gradually denoised and restored to the trajectory sequence through the preset K rounds of iterations. , and outputs the time distance and relative yaw angle to improve the accuracy of the trajectory sequence, thereby increasing the success rate and maximum distance of subsequent robot navigation movements.

[0070] Step 106: Drive the mobile robot to move based on the trajectory sequence data to be executed.

[0071] Specifically, the trajectory sequence data to be executed is input into the robot controller, and control signals such as the linear velocity and angular velocity of the mobile robot are output to drive the chassis of the mobile robot to complete the navigation or exploration task.

[0072] In the embodiment of the present invention, a proportional plus derivative controller (PPDC), a pure pursuit controller (PPC) or a model predictive controller (MPC) can be selected as the controller of the robot, and the trajectory sequence data to be executed is input. , the robot controller calculates the linear velocity of the driving robot chassis and angular velocity The navigation control robot tracks the target point in the action sequence based on the control signal.

[0073] Based on the above embodiments, Figure 7 This is a schematic diagram of a visualization example of the mobile robot visual navigation method provided by the present invention during simulation testing. Figure 7 As shown in the figure, when deployed in a real environment, the physical distance between nodes obtained based on the topological map is approximately 4.10m in the simulation environment and approximately 1.80m in the real environment. The image target is defined by the observed image of each node; the point target is defined by converting the latitude and longitude of each node and the robot's current position into local relative coordinates. However, the point target has a positioning error. The single-point positioning error of the Scoutcar's onboard GNSS device at the loss time of 0s is calibrated to 1.8m. Taking into account the obstruction of buildings and trees and calculation errors, the positioning error of the point target is approximately estimated to be 8m. Therefore, in the Town07 scene, the relative coordinates of the point targets obtained are perturbed by 8m using Gaussian noise to simulate the drift of positioning in the real environment. The navigation command uses a wireless handle to manually send accurate command indexes in real time, which can be used to replace the visual language model.

[0074] The mobile robot starts from the source node and tracks a series of intermediate nodes in sequence. When the time distance of the tracked node is less than or equal to the threshold 3, it switches to the next child node as the tracking target. The loop is executed until it reaches the target node. This process is defined as a complete navigation test. If the robot deviates from the path during the test, manual intervention will be performed to correct the robot's position and direction, and the number of manual interventions will be recorded. At the same time, the farthest navigation distance of the mobile robot at the first manual intervention will be recorded. Figure 7 It can be seen from the figure that the terrain that the robot traverses when driving navigation. The methods in which navigation instructions participate in driving enable the mobile robot to reach the target node from the source node, verifying that the navigation instructions have a strong anti-interference effect.

[0075] The embodiments of the present invention improve the success rate of image target navigation by adding explicit supervision of yaw angle constraints. The multimodal target mask mechanism enables the navigation model to reduce the number of manual interventions and increase the maximum navigation distance, thereby effectively improving the mobile robot's tracking ability of the topological map.

[0076] The present invention provides a mobile robot visual navigation method, which obtains the current observation image of the mobile robot; obtains image targets, point targets and navigation instructions in the topological map of the mobile robot's current environment; then inputs the current observation image, image target, point target and navigation instruction into a first conditional encoder respectively, and obtains the current observation feature, target image similarity feature, point target feature and navigation instruction feature output by the first conditional encoder; then inputs the current observation feature, target image similarity feature, point target feature and navigation instruction feature into a second conditional encoder, and obtains the mask fusion feature output by the second conditional encoder, thereby using a multimodal target mask mechanism. And the diffusion strategy's perception ability of different guidance conditions realizes the effective fusion of different modal features, breaks through the performance bottleneck of existing multimodal target navigation methods in different scenarios, and obtains more effective features; then the mask fusion features, time step features and random Gaussian noise are input into the trajectory decoder to obtain the trajectory sequence data to be executed output by the trajectory decoder, and realizes the accurate prediction of the movement trajectory of the mobile robot in the current environment; finally, based on the trajectory sequence data to be executed, the mobile robot is driven to move, realizing the effective tracking of the topological map of the mobile robot's current environment, and improving the navigation success rate and the maximum navigation distance of the mobile robot.

[0077] A mobile robot visual navigation device provided by the present invention is described below. The mobile robot visual navigation device described below and the mobile robot visual navigation method described above can be referenced to each other.

[0078] Based on any of the above embodiments, Figure 8 This is a structural diagram of a mobile robot visual navigation device provided by the present invention. Figure 8 The embodiment of the present invention provides a mobile robot visual navigation device, including a first acquisition module 801, a second acquisition module 802, a feature extraction module 803, a feature fusion module 804, a prediction module 805 and a navigation module 806, wherein: The first acquisition module 801 is used to acquire the current observation image of the mobile robot; the second acquisition module 802 is used to acquire the image target, point target and navigation instruction in the topological map of the environment where the mobile robot is currently located; the feature extraction module 803 is used to input the current observation image, the image target, the point target and the navigation instruction into the first conditional encoder respectively, and obtain the current observation feature, target image similarity feature, point target feature and navigation instruction feature output by the first conditional encoder; the feature fusion module 804 is used to input the current observation feature, the target image similarity feature, the point target feature and the navigation instruction feature into the second conditional encoder, and obtain the mask fusion feature output by the second conditional encoder; the prediction module 805 is used to input the mask fusion feature, time step feature and random Gaussian noise into the trajectory decoder, and obtain the trajectory sequence data to be executed output by the trajectory decoder; the navigation module 806 is used to drive the mobile robot to move based on the trajectory sequence data to be executed.

[0079] The present invention provides a mobile robot visual navigation device, which obtains the current observation image of the mobile robot; obtains image targets, point targets and navigation instructions in the topological map of the mobile robot's current environment; then inputs the current observation image, image target, point target and navigation instruction into a first conditional encoder respectively, and obtains the current observation feature, target image similarity feature, point target feature and navigation instruction feature output by the first conditional encoder; then inputs the current observation feature, target image similarity feature, point target feature and navigation instruction feature into a second conditional encoder, and obtains the mask fusion feature output by the second conditional encoder, thereby using a multimodal target mask mechanism. And the diffusion strategy's perception ability of different guidance conditions realizes the effective fusion of different modal features, breaks through the performance bottleneck of existing multimodal target navigation methods in different scenarios, and obtains more effective features; then the mask fusion features, time step features and random Gaussian noise are input into the trajectory decoder to obtain the trajectory sequence data to be executed output by the trajectory decoder, and realizes the accurate prediction of the movement trajectory of the mobile robot in the current environment; finally, based on the trajectory sequence data to be executed, the mobile robot is driven to move, realizing the effective tracking of the topological map of the mobile robot's current environment, and improving the navigation success rate and the maximum navigation distance of the mobile robot.

[0080] Figure 9 An example of a physical structure diagram of an electronic device is shown below. Figure 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the mobile robot visual navigation method, which includes: Obtain the current observation image of the mobile robot; Obtaining image targets, point targets, and navigation instructions in a topological map of the environment in which the mobile robot is currently located; Inputting the current observation image, the image target, the point target and the navigation instruction into a first conditional encoder respectively, and obtaining the current observation feature, the target image similarity feature, the point target feature and the navigation instruction feature output by the first conditional encoder; Inputting the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder; Inputting the mask fusion feature, the time step feature and the random Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder; The mobile robot is driven to move based on the trajectory sequence data to be executed.

[0081] Furthermore, the logic instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0082] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the mobile robot visual navigation method provided by the above methods, which includes: Obtain the current observation image of the mobile robot; Obtaining image targets, point targets, and navigation instructions in a topological map of the environment in which the mobile robot is currently located; Inputting the current observation image, the image target, the point target and the navigation instruction into a first conditional encoder respectively, and obtaining the current observation feature, the target image similarity feature, the point target feature and the navigation instruction feature output by the first conditional encoder; Inputting the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder; Inputting the mask fusion feature, the time step feature and the random Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder; The mobile robot is driven to move based on the trajectory sequence data to be executed.

[0083] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the mobile robot visual navigation method provided by the above methods, the method comprising: Obtain the current observation image of the mobile robot; Obtaining image targets, point targets, and navigation instructions in a topological map of the environment in which the mobile robot is currently located; Inputting the current observation image, the image target, the point target and the navigation instruction into a first conditional encoder respectively, and obtaining the current observation feature, the target image similarity feature, the point target feature and the navigation instruction feature output by the first conditional encoder; Inputting the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder; Inputting the mask fusion feature, the time step feature and the random Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder; The mobile robot is driven to move based on the trajectory sequence data to be executed.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0085] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0086] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0087] It should also be noted that the terms "target," "first," and "second," etc., used in the present invention are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein. Furthermore, the terms "first" and "second" generally distinguish objects of the same type, and do not limit the number of objects. For example, the first object can be one or more.

[0088] In the embodiments of the present application, "determine B based on A" means that the factor A must be considered when determining B. It is not limited to "B can be determined based on A alone", and should also include: "determine B based on A and C", "determine B based on A, C and E", "determine C based on A, and further determine B based on C", etc. It can also include taking A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; for example, "when A meets the second condition, determine B", etc.; for example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor in determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0089] In the present invention, the term "plurality" refers to two or more, and other quantifiers are similar to it.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A mobile robot visual navigation method, characterized in that: include: Obtain the current observation image of the mobile robot; Obtaining image targets, point targets, and navigation instructions in a topological map of the environment in which the mobile robot is currently located; Inputting the current observation image, the image target, the point target and the navigation instruction into a first conditional encoder respectively, and obtaining the current observation feature, the target image similarity feature, the point target feature and the navigation instruction feature output by the first conditional encoder; Inputting the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder; Inputting the mask fusion feature, the time step feature and the random Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder; The mobile robot is driven to move based on the trajectory sequence data to be executed.

2. The mobile robot visual navigation method according to claim 1, characterized in that: The step of obtaining image targets, point targets, and navigation instructions in a topological map of an environment in which the mobile robot is currently located includes: Determining an image target and a point target for each node in the topological map based on continuous image frames collected by the mobile robot in the current environment and local mileage records corresponding to the image frames; The coordinate transformation is performed on the point target of each node in the topological map to determine the navigation instruction.

3. The mobile robot visual navigation method according to claim 1, characterized in that: The first conditional encoder includes a first encoder, a second encoder, a third encoder and a fourth encoder; The first encoder is used to extract the current observation feature according to the current observation image; The second encoder is used to extract similar features of the target image based on the spliced ​​image; wherein the spliced ​​image is obtained by channel splicing the current observation image and the image target; The third encoder is used to extract point target features based on the point target; The fourth encoder is used to extract navigation instruction features according to the navigation instruction.

4. The mobile robot visual navigation method according to claim 1, characterized in that: After inputting the current observation image, the image target, the point target, and the navigation instruction into a first conditional encoder respectively and obtaining the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature output by the first conditional encoder, the method further includes: Inputting the target image similarity feature into an image similarity feature decoder, and obtaining the time distance and relative yaw angle output by the image similarity feature decoder; The time distance is used to determine the distance of the image target relative to the current observation image, and the relative yaw angle is used to determine the orientation of the image target relative to the current observation image.

5. The mobile robot visual navigation method according to claim 1, characterized in that: The step of inputting the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder includes: Position encoding is performed on the target image similarity features, the point target features, and the navigation instruction features to obtain a multimodal feature sequence corresponding to the target image similarity features, the point target features, and the navigation instruction features; Determining the mask feature sequence based on a screening result of one or more features in the multimodal feature sequence; The current observation feature and the mask feature sequence are input into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder.

6. The mobile robot visual navigation method according to claim 1, characterized in that: Inputting the mask fusion feature, the time step feature, and the randomly sampled Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder, comprising: Inputting the mask fusion feature, the time step feature and the random Gaussian noise into a trajectory decoder to obtain denoised Gaussian noise; Repeat the iterative steps for a preset number of times to obtain the final Gaussian noise; Using the final random Gaussian noise as the trajectory sequence data to be executed; The iterative step includes: Obtain predicted noise based on the mask fusion feature, the time step feature, and the denoised Gaussian noise obtained in the previous iteration; Obtaining Gaussian noise obtained in this round of iteration based on the mask fusion feature, the time step feature, and the predicted noise; The random Gaussian noise used in the first round of iteration is randomly sampled from the standard Gaussian distribution.

7. A mobile robot visual navigation device, characterized in that: include: A first acquisition module is used to acquire the current observation image of the mobile robot; A second acquisition module is used to acquire image targets, point targets and navigation instructions in a topological map of the environment where the mobile robot is currently located; a feature extraction module, configured to input the current observation image, the image target, the point target, and the navigation instruction into a first conditional encoder, respectively, and obtain the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature output by the first conditional encoder; a feature fusion module, configured to input the current observation feature, the target image similarity feature, the point target feature, and the navigation instruction feature into a second conditional encoder to obtain a mask fusion feature output by the second conditional encoder; A prediction module, configured to input the mask fusion feature, the time step feature, and the random Gaussian noise into a trajectory decoder to obtain trajectory sequence data to be executed output by the trajectory decoder; A navigation module is used to drive the mobile robot to move based on the trajectory sequence data to be executed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the mobile robot visual navigation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the mobile robot visual navigation method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the mobile robot visual navigation method according to any one of claims 1 to 6 is implemented.

Images