Robotic vision navigation method and system

Abstract

The application provides a kind of robot vision navigation method and system, it is related to robot navigation technical field, the method comprises: the coordinate information of navigation target corresponding to target robot is input into long scale target guide module in navigation planning model, and target guide feature is obtained;Current visual observation information of target robot is input into local feature representation module in navigation planning model, and the local visual feature and local environmental risk feature of local area are obtained;Target guide feature and local visual feature are input into local target decision module in navigation planning model, and current local subgoal is obtained;According to the coordinate information of current visual observation information and current local subgoal, initial navigation trajectory is generated, and according to local environmental risk feature, initial navigation trajectory is optimized, to navigate control target robot according to optimized navigation trajectory.The application realizes to improve the accuracy, safety and executability of navigation.

Description

Technical Field

[0001] This invention relates to the field of robot navigation technology, and in particular to a robot vision navigation method and system. Background Technology

[0002] Vision-based robot navigation is a key technology in the field of robotics. Its core objective is to use the robot's current visual observations and target information obtained by the robot's camera to generate the robot's future waypoints, thereby enabling the robot to navigate autonomously in unknown environments.

[0003] Current technologies typically input the robot's current visual observation information and the coordinates of its corresponding final navigation target directly into an end-to-end network, which then implicitly outputs the robot's predicted trajectory. However, in real-world scenarios, the current visual observation information suffers from limited field of view, complex perspective relationships, and occlusion interference. Furthermore, the navigation target is often far away. Directly using the trajectory predicted by this end-to-end network for robot navigation control can easily lead to deviations from the navigation target and makes it difficult to guarantee the safety and feasibility of navigation. Summary of the Invention

[0004] This invention provides a robot visual navigation method and system to address the shortcomings of existing technologies that use end-to-end networks to directly predict motion trajectories for robot navigation control, which can easily lead to deviations from the navigation target and make it difficult to guarantee the safety and executability of navigation. The invention generates navigation trajectories with target consistency, local safety, and practical executability, thereby improving the accuracy, safety, and executability of navigation.

[0005] This invention provides a robot visual navigation method, comprising: The coordinate information of the navigation target corresponding to the target robot is input into the long-scale target guidance module in the navigation planning model to obtain the target guidance features; the target guidance features are used to characterize the geometric relationship features between the navigation target and the target robot and each grid in the local area corresponding to the target robot; The current visual observation information of the target robot is input into the local feature representation module in the navigation planning model to obtain the local visual features and local environmental risk features of the local area; The target guidance features and the local visual features are input into the local target decision module in the navigation planning model to obtain the current local sub-target within the local area; Based on the current visual observation information and the coordinate information of the current local sub-target, an initial navigation trajectory is generated, and based on the local environmental risk characteristics, the initial navigation trajectory is optimized to obtain an optimized navigation trajectory; Based on the optimized navigation trajectory, the target robot is subjected to navigation control. The navigation planning model is trained based on the historical driving data of the sample robot. The historical driving data includes the sample visual observation information acquired by the sample robot at the sample time, the coordinate information of the sample navigation target, the sample depth information corresponding to the sample visual observation information, and the sample driving trajectory.

[0006] This invention provides a robot visual navigation method, wherein the coordinate information of the navigation target corresponding to the target robot is input into a long-scale target guidance module in a navigation planning model to obtain target guidance features, including: The coordinate information of the navigation target is encoded to obtain coordinate encoding features; The coordinate encoding features are input into the multilayer perceptron layer of the long-scale target guidance module to obtain target coordinate features; the target coordinate features are used to characterize the geometric relationship between the navigation target and the target robot. A relative target feature is constructed based on the directional angle of each grid pointing to the navigation target and the distance between each grid and the navigation target; the relative target feature is used to characterize the geometric relationship between the navigation target and each grid. The relative target features and the target coordinate features are input into the fusion layer of the long-scale target guidance module to obtain the target guidance features.

[0007] According to the present invention, a robot visual navigation method is provided, wherein encoding the coordinate information of the navigation target to obtain coordinate encoded features includes: The coordinate information of the navigation target is normalized to obtain normalized target coordinate information; By applying sine and cosine functions to each element of the normalized target coordinate information at multiple frequencies, the sine coding result and cosine coding result corresponding to each frequency are obtained. The normalized target coordinate information, the sine encoding result, and the cosine encoding result are concatenated to obtain the coordinate encoding feature.

[0008] According to the present invention, a robot visual navigation method is provided, wherein the current visual observation information of the target robot is input into the local feature representation module of the navigation planning model to obtain the local visual features and local environmental risk features of the local region, including: The current visual observation information is input into the visual coding layer of the local feature representation module to obtain image features; The image features are input into the frustum transformation layer of the local feature representation module to obtain frustum features; Based on the intrinsic parameters of the target robot's camera and the transformation matrix between the camera's coordinate system and the target robot's coordinate system, the visual cone feature is projected onto each grid in the local region to obtain the local visual feature. The local visual features are input into the accessibility prediction layer of the local feature representation module to obtain the local environmental risk features.

[0009] According to the present invention, a robot visual navigation method is provided, wherein the step of inputting the target guidance features and the local visual features into the local target decision module in the navigation planning model to obtain the current local sub-target within the local region includes: The target guidance features and the local visual features are input into the fusion layer of the local target decision module to obtain local spatial features; The local spatial features are input into the local target decision layer of the local target decision module to obtain a local sub-target sequence; the local sub-target sequence includes at least one local sub-target located within the local region; If the navigation target is located outside the local area, the effective local sub-target that is closest to the target robot in the local sub-target sequence is determined as the current local sub-target; If the navigation target is located within the local area, the navigation target is identified as the current local sub-target.

[0010] According to the present invention, a robot visual navigation method is provided, wherein generating an initial navigation trajectory based on the current visual observation information and the coordinate information of the current local sub-target includes: The current visual observation information is input into the visual encoding module in the trajectory prediction model to obtain a visual feature sequence; The coordinate information of the current local sub-target is input into the feature embedding module in the trajectory prediction model to obtain the target embedding features; The target embedded features and the visual feature sequence are input into the feature interaction module of the trajectory prediction model to obtain interactive features; The interactive features are input into the prediction module of the trajectory prediction model to obtain the initial navigation trajectory.

[0011] The present invention provides a robot visual navigation method, wherein optimizing the initial navigation trajectory based on the local environmental risk characteristics to obtain an optimized navigation trajectory includes: Based on the local environmental risk characteristics, determine the obstacle probability value for each of the grids; Based on the obstacle probability value, each grid is divided into an obstacle grid or a free grid; the obstacle probability value of the obstacle grid is greater than the obstacle probability threshold, and the obstacle probability value of the free grid is less than or equal to the obstacle probability threshold. For each grid cell within the local area, calculate a first distance between the grid cell and the nearest obstacle grid cell, and a second distance between the grid cell and the nearest free grid cell, and obtain the sign distance of the grid cell based on the difference between the first distance and the second distance; A local cost map is generated based on the symbolic distance of each of the grids; Based on the local cost map, the initial navigation trajectory is iteratively corrected to obtain the optimized navigation trajectory.

[0012] According to the present invention, a robot vision navigation method is provided, wherein the step of iteratively correcting the initial navigation trajectory based on the local cost map to obtain the optimized navigation trajectory includes: Map each waypoint in the initial navigation trajectory to a grid in the local area, and determine the total cost of the initial navigation trajectory based on the cost value of the grid mapped to each waypoint in the local cost map. Based on the spatial gradient of the total value relative to the initial navigation trajectory, the update step size, and the cost constraint weight, the waypoints in the initial navigation trajectory are iteratively corrected to obtain the optimized navigation trajectory.

[0013] According to the present invention, a robot visual navigation method is provided, wherein the training steps of the navigation planning model include: Based on the sample depth information, obstacle region evidence is generated within the sample local area of ​​the sample robot; Based on the driving trajectory located within the sample local area in the sample driving trajectory, evidence of passable area within the sample local area is generated; By fusing the obstacle area evidence and the passable area evidence, a passability label and a valid area mask are generated; the valid area mask is used to identify unknown or known areas. Based on the driving trajectory located within the local region of the sample driving trajectory, generate local sub-target sequence labels; The sample robot's visual observation information and the coordinate information of the sample navigation target at the sample time are input into the initialization model to obtain the sample local environmental risk characteristics and sample local sub-target sequence output by the initialization model. The first loss value is determined based on the deviation between the local environmental risk characteristics of the sample and the accessibility label in the known area; The second loss value is determined based on the deviation between the sample local sub-target sequence and the local sub-target sequence label; The initialization model is trained based on the first loss value and the second loss value to obtain the navigation planning model.

[0014] The present invention also provides a robot vision navigation system, comprising: The first feature extraction unit is used to input the coordinate information of the navigation target corresponding to the target robot into the long-scale target guidance module in the navigation planning model to obtain target guidance features; the target guidance features are used to characterize the geometric relationship features between the navigation target and the target robot and each grid in the local area corresponding to the target robot; The second feature extraction unit is used to input the current visual observation information of the target robot into the local feature representation module in the navigation planning model to obtain the local visual features and local environmental risk features of the local area; The sub-target generation unit is used to input the target guidance features and the local visual features into the local target decision module in the navigation planning model to obtain the current local sub-target within the local area; An initial trajectory generation unit is used to generate an initial navigation trajectory based on the current visual observation information and the coordinate information of the current local sub-target; An optimization unit is used to optimize the initial navigation trajectory based on the local environmental risk characteristics to obtain an optimized navigation trajectory; A control unit is used to perform navigation control on the target robot according to the optimized navigation trajectory; The navigation planning model is trained based on the historical driving data of the sample robot. The historical driving data includes the sample visual observation information acquired by the sample robot at the sample time, the coordinate information of the sample navigation target, the sample depth information corresponding to the sample visual observation information, and the sample driving trajectory.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot visual navigation method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot visual navigation method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot visual navigation method as described above.

[0018] The robot visual navigation method and system provided by this invention obtains target guidance features by geometrically modeling the coordinate information of the navigation target through a long-scale target guidance module, establishing an effective connection between long-distance target information and local motion decisions. This overcomes the technical defect that relying solely on current local observations for decision-making can easily lead to short-sighted behavior. Based on this, a local feature representation module converts the current visual observation information in the perspective image space into local visual features in the robot coordinate system and predicts local environmental risk features, enabling the target robot to explicitly perceive the surrounding spatial layout and potential risks. Furthermore, a local target decision module fuses the target guidance features and local visual features to determine the current local sub-target, transforming the long-distance target navigation task into an executable motion task within a local range. Local environmental risk features are used to constrain and optimize the initial navigation trajectory, ensuring that the final optimized navigation trajectory remains consistent with the navigation target direction while avoiding obstacle areas and high-risk areas. This effectively ensures the target robot's target consistency, local safety, and trajectory executability during navigation in the absence of high-precision maps and strong prior environmental information, achieving correct, safe, stable, and executable autonomous robot navigation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts of the robot visual navigation method provided by the present invention.

[0021] Figure 2 This is the second flowchart of the robot visual navigation method provided by the present invention.

[0022] Figure 3 This is the third flowchart of the robot visual navigation method provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the process for predicting local environmental risk characteristics provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the initial navigation trajectory prediction process provided by the present invention.

[0025] Figure 6 This is a schematic diagram of the local cost map generation process provided by the present invention.

[0026] Figure 7 This is one of the simulation diagrams of the robot visual navigation method provided by the present invention.

[0027] Figure 8 This is the second simulation diagram of the robot visual navigation method provided by the present invention.

[0028] Figure 9 This is a schematic diagram of the training process for the navigation planning model provided by the present invention.

[0029] Figure 10 This is a schematic diagram of the robot vision navigation system provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] All actions involving the acquisition of signal information or data in this application are carried out in accordance with the relevant data protection laws and regulations of the locality and with the authorization of the relevant device owner.

[0032] Traditional robot navigation methods typically rely on lidar, global positioning systems, pre-built environmental maps, or manually designed path planning rules. While they can achieve good navigation results in structured, static, and known environments, they often face problems such as high map building and maintenance costs, accumulation of positioning errors, high sensor costs, and insufficient adaptability to environmental changes in open, dynamic, or unstructured environments.

[0033] With the development of deep learning, learning-based visual navigation methods have gradually become an important direction for robot navigation. These methods typically use an end-to-end network to learn the mapping relationship between visual observations and actions from the robot's past driving experience. Thus, even in the absence of high-precision maps, they can implicitly output the robot's predicted trajectory based on the robot's current visual observation information and the coordinates of its corresponding final navigation target.

[0034] However, such methods mainly rely on neural networks to implicitly learn navigation behavior from data. In real-world scenarios, current visual observation information suffers from problems such as limited field of view, complex perspective relationships, and occlusion interference, making it difficult to directly form a local spatial representation suitable for navigation decisions. This results in the robot's insufficient understanding of the surrounding spatial layout and potential risks, making it difficult to guarantee the safety and feasibility of navigation.

[0035] To enhance spatial understanding in visual navigation, some techniques have introduced spatial representations such as local maps, occupancy grids, semantic maps, or bird's-eye view (BEV) on top of end-to-end networks. Compared to making decisions directly in the forward-view image space, these local spatial representations can describe the surrounding environment structure in the robot's coordinate system, making them more suitable for drivability assessment and local path planning. However, these local spatial representations typically only cover a limited area around the robot and are insufficient to solve the problem of navigating distant targets independently. In actual navigation tasks, the location of the navigation target is often outside the robot's current field of vision. If the robot makes reactive decisions based solely on current visual observation information, there is a lack of effective connection between distant target information and local motion decisions. This can easily lead to short-sighted behavior at intersections, forks in the road, occluded areas, or long routes, resulting in a direction of movement inconsistent with the distant target.

[0036] Furthermore, even if end-to-end learning methods can generate future path points or trajectories based on visual observations and target information, their output lacks local risk constraints and executability guarantees, and may not meet the requirements of local drivability, obstacle avoidance, and actual robot motion execution. In unknown environments or complex obstacle scenarios, the directly predicted trajectory may still approach obstacles, cross high-risk areas, or deviate from drivable areas, thus affecting the safety and stability of navigation.

[0037] Therefore, in the absence of high-precision maps and strong prior environmental information, how to enable robots to generate navigation trajectories with target consistency, local safety, and practical feasibility when the navigation target is outside the current field of view or when there is ambiguity in local observation remains a technical problem that urgently needs to be solved in this field.

[0038] In view of this, this application provides a robot visual navigation method, which offers a visual navigation decision-making framework based on dual-scale guidance. The dual-scale framework comprises two levels: long-scale target guidance and local-scale motion decision-making. Long-scale target guidance expresses the direction and distance of a distant navigation target relative to the robot, as well as the geometric relationship between the navigation target and its local spatial position. Local-scale motion decision-making understands the local spatial structure around the robot based on current visual observations and generates short-term executable navigation trajectories. These two levels are connected through local sub-targets, transforming the long-distance target navigation task into an executable navigation intent within the current local area. Furthermore, the trajectory is constrained and optimized using local environmental risk characteristics. This enables the generation of navigation trajectories with target consistency, local safety, and practical executability even in the absence of high-precision maps and strong prior environmental information, thereby improving the accuracy, safety, and executability of navigation.

[0039] Figure 1 This is one of the flowcharts of the robot visual navigation method provided by the present invention. Figure 2 This is the second flowchart of the robot visual navigation method provided by the present invention.

[0040] The method provided in this application can be applied to the field of autonomous robot navigation technology, especially in mobile robot navigation scenarios in open, dynamic or unstructured environments where there is a lack of high-precision maps or global positioning support. It can generate local sub-targets through the fusion decision of long-scale target guidance modeling and local spatial representation, and optimize the predicted trajectory based on local environmental risk characteristics, thereby outputting an optimized navigation trajectory that takes into account target consistency, local safety and executability. It can effectively prevent the robot from deviating from the navigation target during long-distance navigation and reduce the risk of the trajectory approaching obstacles or crossing high-risk areas.

[0041] like Figure 1 As shown, the method includes steps 110, 120, 130, 140 and 150.

[0042] Step 110: Input the coordinate information of the navigation target corresponding to the target robot into the long-scale target guidance module in the navigation planning model to obtain the target guidance features; the target guidance features are used to characterize the geometric relationship features between the navigation target and the target robot and each grid in the local area corresponding to the target robot.

[0043] The target robot is the mobile robot currently performing the navigation task, which is equipped with a visual sensor for acquiring environmental images, such as a forward-looking camera. This embodiment does not specifically limit this.

[0044] The navigation objective is the target location that the robot needs to reach in the current navigation task. It can be specified in advance by the user or issued by the upper-level task planning system.

[0045] The coordinate information of the navigation target, or simply target information, refers to the position coordinates of the navigation target in the robot coordinate system, used to quantitatively describe the spatial relationship between the navigation target and the target robot. In one possible implementation, the coordinate information of the navigation target is the two-dimensional target coordinates in the robot coordinate system, specifically denoted as: , among them This represents the forward distance of the navigation target relative to the current position of the target robot. This represents the lateral offset of the navigation target relative to the current position of the target robot. The robot coordinate system can be based on the current position of the target robot as the origin, the forward direction of the target robot as the first coordinate axis, and the lateral direction of the target robot as the second coordinate axis. In this way, distant navigation targets can be unified into the target robot's own coordinate system, which facilitates the construction of target guidance features and local navigation decisions, without relying on a global positioning system or a pre-built global map.

[0046] A navigation planning model is a machine learning model used to make navigation decisions based on the coordinate information of the navigation target and the visual observation information of the target robot. It includes at least a long-scale target guidance module, a local feature representation module, and a local target decision module. Among them, the long-scale target guidance module is a functional module in the navigation planning model specifically used for long-scale target guidance modeling of navigation target information at a distance.

[0047] The navigation planning model is trained based on the historical driving data of the sample robot. Here, the sample robot is a mobile robot operating and collecting data in a real-world environment. It can be the same robot as the target robot, or it can be another robot with the same or similar sensor configuration. The historical driving data includes the sample robot's visual observation information, the coordinate information of the sample navigation target, the sample depth information, and the sample driving trajectory acquired at each sample moment. Specifically, the sample visual observation information consists of image information collected by the sample robot's visual sensors at each sample moment during the historical operation; the coordinate information of the sample navigation target is the coordinate information of the navigation target in the robot's coordinate system at each sample moment during the historical operation; the sample depth information is the depth measurement data corresponding to the sample visual observation information, which can be used to generate local spatial supervision information during the training phase; and the sample driving trajectory is the actual trajectory traversed by the sample robot during the historical operation, which can be used to generate local sub-targets and navigation trajectory supervision information during the training phase.

[0048] Target guidance features are the output features of the long-scale target guidance module. They are used to characterize the geometric relationship between the navigation target and the target robot, as well as the geometric relationship between the navigation target and each grid within a local area. These geometric relationship features include directional and distance relationship features. In other words, the target guidance features can express the direction and distance of the navigation target relative to the target robot at the overall level, and also express the directional and distance relationships between the positions of each grid within a local area and the navigation target at the local spatial level.

[0049] A local region refers to a finite spatial range extending forward and to the left and right sides of the target robot, based on its current position in the robot coordinate system. A grid refers to a spatial unit obtained by discretizing the local region according to a preset spatial resolution. Each grid has definite position coordinates in the robot coordinate system and is used to describe the spatial attributes of different positions within the local region in discrete space. It should be noted that the coverage area of ​​the local region and the grid resolution can be dynamically determined based on the target robot's speed, camera field of view, and navigation scenario. For example, the local region can be set to cover an area of ​​20 meters in front of the target robot and 20 meters laterally, and the grid resolution can be set to 0.1 meters, corresponding to a 200×200 grid representation. This embodiment does not impose specific limitations on this.

[0050] like Figure 2 As shown, when performing navigation control on a target robot, the position coordinates of the navigation target in the robot coordinate system are first acquired in real time to obtain the coordinate information of the navigation target. Then, the coordinate information of the navigation target is input into the long-scale target guidance module in the navigation planning model, so that the long-scale target guidance module outputs the target guidance features.

[0051] It should be noted that when acquiring target guidance features through the long-scale target guidance module, the coordinate information of the navigation target can be feature-encoded to obtain a feature representation expressing the geometric relationship between the navigation target and the target robot. Simultaneously, based on the spatial relationship between the position coordinates of each grid within a local area and the coordinate information of the navigation target, a feature representation expressing the geometric relationship between each grid and the navigation target is constructed. Then, the two types of feature representations are fused to obtain the target guidance features. Specifically, the feature encoding method can employ position encoding, multi-layer perceptron (MLP) encoding, or a combination of both. The feature fusion method can employ feature concatenation, weighted fusion, or attention fusion, etc., and this embodiment does not specifically limit these methods.

[0052] Furthermore, the target guidance features here are not directly used to output control commands, but rather serve as target conditions for local navigation decisions. Since the navigation target is often located outside the target robot's current field of vision, the target robot cannot perceive the directional constraints of the navigation target when making decisions solely based on current visual observation information. However, through long-scale target guidance modeling, the direction and distance information of the navigation target are transformed into guidance signals aligned with the local spatial region. This allows the local decision-making process to continuously perceive the direction of the distant navigation target while understanding the local environmental structure, enabling the target robot to maintain a consistent motion trend with the navigation target even when the navigation target is invisible or there is ambiguity in local observation.

[0053] As can be seen from the above, by modeling the geometric relationship of the coordinate information of the navigation target through the long-scale target guidance module, the navigation target information at a distance can be transformed into target guidance features that can be perceived by the local decision-making process. This effectively establishes a connection between the long-distance target information and the local motion decision-making, and overcomes the technical defect that relying solely on current local observations for reactive decision-making can easily lead to short-sighted behavior.

[0054] Step 120: Input the current visual observation information of the target robot into the local feature representation module in the navigation planning model to obtain the local visual features and local environmental risk features of the local area.

[0055] Here, the current visual observation information refers to the image information collected by the target robot's visual sensors at the current moment. It reflects the visual information of the target robot's current environment, including the road structure ahead, the distribution of obstacles, and scene textures. This current visual observation information can be a monocular red-green-blue (RGB) image collected by the target robot's front-facing camera, or it can be a binocular image, a red-green-blue-depth (RGB-D) image, or a multi-frame image sequence, etc. This embodiment does not specifically limit it in this way.

[0056] The local feature representation module is a functional module in the navigation planning model responsible for constructing a local spatial representation in the robot's coordinate system from the current visual observation information.

[0057] Local visual features are local spatial representations output by the local feature representation module. They are visual features obtained by transforming the environmental structure information contained in the current visual observation information into a space aligned with the local region in the robot coordinate system.

[0058] Local environmental risk features are features that are further predicted by the local feature representation module based on local visual features, and are used to characterize the accessibility or obstacle risk level of each grid in a local area. Their specific form can be a accessibility probability map, an obstacle probability map, or other forms of local environmental risk representation. This embodiment does not specifically limit this.

[0059] like Figure 2 As shown, after obtaining the current visual observation information of the target robot, the current visual observation information of the target robot can be input into the local feature representation module to obtain local visual features through the local feature representation module.

[0060] It should be noted that when obtaining local visual features through the local feature representation module, image features can first be extracted from the current visual observation information through the visual feature extraction network. Then, combined with the imaging parameters of the visual sensor and the spatial projection relationship, the image features are transformed from the image space to the bird's eye view (BEV) space corresponding to the local area in the robot coordinate system to obtain the local visual features.

[0061] Then, the accessibility of each grid in the local area is predicted based on the local visual features by the local feature representation module, and the local environmental risk features are obtained.

[0062] As can be seen from the above, the local feature representation module converts the current visual observation information in the perspective image space into local visual features in the robot coordinate system and explicitly predicts local environmental risk features, enabling the target robot to explicitly perceive the surrounding spatial layout and local risks. This not only provides environmental structural constraints for the generation of local sub-targets, but also provides risk constraints for the optimization of navigation trajectories.

[0063] Step 130: Input the target guidance features and the local visual features into the local target decision module in the navigation planning model to obtain the current local sub-target within the local area.

[0064] Here, the local target decision module is a functional module in the navigation planning model responsible for integrating target guidance features and local visual features and making local navigation intent decisions.

[0065] The current local sub-target is an intermediate navigation target located within the local area, output by the local target decision module. It represents the intermediate navigation intent that the target robot is suitable to execute within the current local area.

[0066] like Figure 2 As shown, after obtaining the target guidance features and local visual features, these features can be input into the local target decision module to obtain the current local sub-target within the local area. Specifically, the local target decision module can first fuse the target guidance features and local visual features to obtain a fused feature that simultaneously contains environmental structure information and target direction information, and then predict the current local sub-target within the local area based on this fused feature. The feature fusion method can be feature concatenation, element-wise fusion, or attention fusion, etc.; the representation of the current local sub-target can be the target point coordinates within the local area, a local target heatmap, a local target probability distribution, or multiple ordered local target points, etc., which are not specifically limited in this embodiment.

[0067] It should be noted that the decision-making process for the current local sub-target is constrained by two aspects of information: firstly, the environmental structure and spatial layout within the local area expressed by the local visual features, which ensures that the current local sub-target is located within a reasonable area reachable by the target robot; secondly, the navigation target direction and distance information expressed by the target guidance features, which ensures that the current local sub-target is aligned with the direction of the distant navigation target. Therefore, the long-distance target navigation task is decomposed into executable motion tasks within the local area. That is, the target robot does not need to plan the complete path to the navigation target all at once, but only needs to move towards the current local sub-target at each decision moment, and continuously update the current local sub-target as the target robot moves, thus gradually approaching the navigation target.

[0068] As can be seen from the above, by integrating target guidance features and local visual features through the local target decision module to make local sub-target decisions, the information of distant targets can be transformed into a navigation intent that can be executed within the current local range. This effectively alleviates the directional uncertainty caused by the robot relying solely on local observations in ambiguous scenarios such as intersections, branching points, and occluded areas, and improves the target consistency and navigation stability in long-distance visual navigation.

[0069] Step 140: Generate an initial navigation trajectory based on the current visual observation information and the coordinate information of the current local sub-target, and optimize the initial navigation trajectory based on the local environmental risk characteristics to obtain an optimized navigation trajectory.

[0070] like Figure 2 As shown, after obtaining the current local sub-target, the current visual observation information and the coordinate information of the current local sub-target can be used as the conditional input for trajectory prediction to generate the initial navigation trajectory.

[0071] The initial navigation trajectory here is a trajectory composed of multiple future waypoints, which reflects the trajectory trend of the target robot moving towards the current local sub-target in a short time. Each future waypoint can be represented as a position coordinate in the robot's coordinate system. The number of future waypoints can be set according to the target robot's speed, control frequency, and navigation scenario. This embodiment does not specifically limit this.

[0072] It should be noted that using the coordinates of the current local sub-target as the condition for trajectory generation, compared to directly using a distant navigation target as the condition for trajectory prediction, allows the generated trajectory to match the environmental structure within the current local area, avoiding trajectory prediction deviations caused by the navigation target being too far away or invisible.

[0073] Furthermore, in unknown environments or complex obstacle scenarios, the initial navigation trajectory may still approach obstacles, traverse high-risk areas, or deviate from passable areas. Therefore, the initial navigation trajectory can be optimized based on the local environmental risk characteristics obtained in step 120 to obtain an optimized navigation trajectory.

[0074] Specifically, when optimizing the initial navigation trajectory, the risk cost of each grid in the local area can be determined based on the local environmental risk characteristics. Areas with obstacles, impassable areas, or areas close to obstacles are assigned a higher cost, while areas that are passable are assigned a lower cost. Then, using the initial navigation trajectory as the initial value for optimization, the waypoints in the initial navigation trajectory are modified under the constraint of risk cost. This allows the modified trajectory to maintain the movement trend toward the current local sub-target while avoiding areas with higher cost, thus obtaining an optimized navigation trajectory.

[0075] It should be noted that, since the optimization process starts with the initial navigation trajectory, the optimized trajectory retains the target direction and overall movement trend of the initial trajectory. Simultaneously, due to the introduction of constraints based on local environmental risk characteristics, the optimized trajectory can avoid obstacle areas and high-risk areas. In other words, the generation and optimization of the initial navigation trajectory form a collaborative relationship: the trajectory generation process ensures the consistency between the trajectory and the current local sub-target and navigation target direction, while the trajectory optimization process, based on this, ensures the local safety and practical feasibility of the trajectory. Together, they ensure that the final output optimized navigation trajectory balances target consistency, safety, and feasibility.

[0076] As can be seen from the above, by introducing constraint optimization based on local environmental risk characteristics in the trajectory generation process, the defects of lack of local risk constraints and executability guarantee in end-to-end trajectory prediction results are overcome. This allows the final trajectory to stay away from obstacles and high-risk areas while maintaining its direction toward the current local sub-target, thereby improving the safety, stability and actual executability of the navigation trajectory.

[0077] Step 150: Perform navigation control on the target robot according to the optimized navigation trajectory.

[0078] like Figure 2 As shown, after obtaining the optimized navigation trajectory, the optimized navigation trajectory can be input into the controller of the target robot, and the controller will output the control signal of the target robot to perform navigation control on the target robot.

[0079] For example, during navigation control, the controller can calculate the linear velocity, angular velocity, or other chassis control commands of the target robot based on one or more future waypoints in the optimized navigation trajectory, driving the target robot to move along the optimized navigation trajectory. The controller can be a proportional-derivative controller, a pure tracking controller, or a model predictive controller, etc., and this embodiment does not specifically limit its application. For example, on a low-speed mobile robot platform, a proportional-derivative controller can be used to generate linear velocity and angular velocity control quantities based on the target waypoint error.

[0080] It should be noted that during the closed-loop execution process, the target robot can repeatedly execute steps 110 to 150 at a fixed frequency. That is, at each decision moment, it reacquires the current visual observation information and the coordinate information of the navigation target, updates the target guidance features, local visual features, and local environmental risk features, re-determines the current local sub-target, and generates and optimizes the navigation trajectory. Since the local spatial understanding and trajectory optimization are performed again based on the latest current visual observation information at each moment, this method can respond in real time to changes in the local environment, perception errors, and changes in obstacle distribution, further ensuring the stability of the navigation process.

[0081] The method provided in this embodiment obtains target guidance features by geometrically modeling the coordinate information of the navigation target through a long-scale target guidance module, establishing an effective connection between long-distance target information and local motion decisions. This overcomes the technical defect that relying solely on current local observations for decision-making can easily lead to short-sighted behavior. Based on this, a local feature representation module converts the current visual observation information in the perspective image space into local visual features in the robot coordinate system and predicts local environmental risk features, enabling the target robot to explicitly perceive the surrounding spatial layout and potential risks. Furthermore, a local target decision module fuses the target guidance features and local visual features to determine the current local sub-target, transforming the long-distance target navigation task into an executable motion task within a local range. Local environmental risk features are used to constrain and optimize the initial navigation trajectory, ensuring that the final optimized navigation trajectory remains consistent with the navigation target direction while avoiding obstacle areas and high-risk areas. This effectively ensures the target robot's target consistency, local safety, and trajectory executability during navigation in the absence of high-precision maps and strong prior environmental information, achieving correct, safe, stable, and executable autonomous robot navigation.

[0082] In some embodiments, inputting the coordinate information of the navigation target corresponding to the target robot into the long-scale target guidance module in the navigation planning model to obtain target guidance features includes: A relative target feature is constructed based on the directional angle of each grid pointing to the navigation target and the distance between each grid and the navigation target; the relative target feature is used to characterize the geometric relationship between the navigation target and each grid. The coordinate information of the navigation target is encoded to obtain coordinate encoding features; The coordinate encoding features are input into the multilayer perceptron layer of the long-scale target guidance module to obtain target coordinate features; the target coordinate features are used to characterize the geometric relationship between the navigation target and the target robot. The relative target features and the target coordinate features are input into the fusion layer of the long-scale target guidance module to obtain the target guidance features.

[0083] Figure 3 This is the third flowchart of the robot visual navigation method provided by the present invention.

[0084] like Figure 3 As shown, the long-scale target guidance modeling process, which utilizes the long-scale target guidance module to perform long-scale target guidance modeling on the coordinate information of the navigation target, specifically includes the processes of encoding the coordinate information of the navigation target, obtaining target coordinate features, constructing relative target features, and feature fusion. The specific execution steps are as follows: First, the coordinate information of the navigation target is encoded to obtain coordinate encoded features. This encoding can employ position encoding or other coordinate feature mapping methods, etc., and this embodiment does not specifically limit this approach.

[0085] In one possible implementation, encoding the coordinate information of the navigation target to obtain coordinate encoding features includes: The coordinate information of the navigation target is normalized to obtain normalized target coordinate information; By applying sine and cosine functions to each element of the normalized target coordinate information at multiple frequencies, the sine coding result and cosine coding result corresponding to each frequency are obtained. The normalized target coordinate information, the sine encoding result, and the cosine encoding result are concatenated to obtain the coordinate encoding feature.

[0086] Optionally, during the coordinate information encoding process, the coordinate information of the navigation target in the target robot's coordinate system can be encoded first. Normalization is performed to obtain normalized target coordinate information. This is to eliminate the difference in the absolute values ​​of the coordinate information of navigation targets in different scenarios, and to avoid the adverse effects of excessively large coordinate value range on encoding and network training.

[0087] Then, in order to enhance the ability to express target distance and direction information, Fourier Positional Embedding (FoPE) can be used to encode the normalized target coordinate information.

[0088] Specifically, sine and cosine functions are applied element-wise to the normalized target coordinate information at multiple frequencies, and the normalized target coordinate information itself, the sine coding results corresponding to each frequency, and the cosine coding results are concatenated to obtain the coordinate coding features. An exemplary expression of this coding process is as follows: ]; in, Represents normalized target coordinate information; Indicates the number of frequency bands; and These represent the sine and cosine functions applied element-wise to the normalized target coordinate information, respectively. [·] represents the coordinate encoded features obtained by concatenation; [·] represents the concatenation operation. The number of frequency bands is also specified. The settings can be adjusted according to the required accuracy of the coordinate encoding; however, this embodiment does not impose any specific limitations on this.

[0089] It should be noted that by introducing periodic functions of multiple frequencies to encode the normalized target coordinate information, low-dimensional coordinate information can be mapped to a high-dimensional feature space. This makes the target positions with similar coordinate values ​​but corresponding to different navigation directions more distinguishable in the encoded feature space, thereby enhancing the ability to express the distance and direction information of the navigation target and helping to alleviate the spectral bias problem that exists when the neural network directly learns low-dimensional coordinate input.

[0090] In this embodiment, by combining normalization processing with multi-frequency sine and cosine coding, the coordinate information of the navigation target is transformed into coordinate coding features with high-dimensional expressive capabilities, which significantly enhances the ability of the target coordinate features to express the direction and distance information of the navigation target.

[0091] Furthermore, after obtaining the coordinate-encoded features, these features can be input into the Multi-Layer Perceptron (MLP) layer in the long-scale target guidance module to obtain the target coordinate features. This MLP layer consists of multiple sequentially connected fully connected layers, which perform nonlinear transformations on the coordinate-encoded features to express the position, orientation, and distance information of the navigation target relative to the target robot in a compact feature space; that is, to characterize the geometric relationship between the navigation target and the target robot.

[0092] Furthermore, considering that encoding only the coordinate information of the navigation target itself is insufficient to fully describe the geometric relationship between different grid positions in a local area and distant navigation targets, this embodiment can also construct relative target features that are spatially aligned with the local area when acquiring target guidance features.

[0093] Specifically, for any grid position within a local area It can calculate the azimuth angle of the grid position pointing to the navigation target. and the distance between the grid position and the navigation target. And thereby construct the relative target representation corresponding to the grid position. : ; in, For distance normalization parameters, and These are the direction angles. The sine and cosine values, It is the hyperbolic tangent function.

[0094] After performing the above calculations on all grid positions within a local area, the data is input into a multilayer perceptron layer to obtain the relative target features aligned with the local spatial area.

[0095] It should be noted that using a combination of the sine and cosine values ​​of the direction angle to represent direction information avoids the problem of numerical discontinuity at angle boundaries, thereby improving the stability of direction information learning. Furthermore, compressing the distance between each grid cell and the navigation target to a bounded range using the hyperbolic tangent function eliminates the influence of differences in the distance dimensions of the navigation target under different scenarios, improving the adaptability of the navigation planning model to navigation targets with different distance scales. Therefore, relative target features can provide each grid cell position within a local area with its directional and distance relationships relative to the navigation target. This allows the local decision-making process to not only know the coordinates of the navigation target but also perceive the relative geometric relationships between different grid cell positions and the navigation target within the local area.

[0096] Finally, relative target features and target coordinate features The input is fed into the fusion layer of the long-scale target guidance module to obtain target guidance features. The obtained target guidance features include the overall geometric relationship between the navigation target and the target robot expressed by the target coordinate features, as well as the positional geometric relationship between each grid and the navigation target in the local area expressed by the relative target features. This enables the local target decision module to provide multi-granular target direction guidance.

[0097] The calculation formula for the target-guided feature is as follows: ; in, The target feature fusion function executed by the fusion layer can be implemented by convolutional layers after feature concatenation, weighted fusion, or attention fusion, etc. This embodiment does not specifically limit this.

[0098] In this embodiment, by constructing target guidance features, the target guidance features can express the direction and distance of the navigation target at the overall level, and also express the geometric relationship between the local space and the navigation target at the grid granularity. This provides spatially perceptive target constraints for local sub-target decision-making, thereby improving the target robot's ability to distinguish directions in local decision-making under different target directions.

[0099] In some embodiments, inputting the current visual observation information of the target robot into the local feature representation module of the navigation planning model to obtain the local visual features and local environmental risk features of the local region includes: The current visual observation information is input into the visual coding layer of the local feature representation module to obtain image features; The image features are input into the frustum transformation layer of the local feature representation module to obtain frustum features; Based on the intrinsic parameters of the target robot's camera and the transformation matrix between the camera's coordinate system and the target robot's coordinate system, the visual cone feature is projected onto each grid in the local region to obtain the local visual feature. The local visual features are input into the accessibility prediction layer of the local feature representation module to obtain the local environmental risk features.

[0100] Figure 4 This is a schematic diagram of the process for predicting local environmental risk characteristics provided by the present invention.

[0101] like Figure 3 and Figure 4 As shown, the local feature representation module can acquire local visual features and local environmental risk features through processes such as visual encoding, BEV projection, local visual feature generation, and accessibility prediction. Specific implementation steps include: First, the current visual observation information is input into the visual encoding layer of the local feature representation module to extract image features containing local environmental structure information. Here, the visual encoding layer can use an EfficientNet-B0 network as the image encoder, which has strong image feature extraction capabilities with a small number of parameters and is suitable for deployment on mobile robot platforms with limited computing resources. In other embodiments, the visual encoding layer can also use other image feature extraction networks such as a Residual Network (ResNet) or a Vision Transformer (ViT), and this embodiment does not specifically limit this approach.

[0102] It should be noted that, because the current visual observation information is in perspective image space, the actual spatial scale corresponding to different pixel regions in the image is not consistent, and there are problems such as limited field of view and occlusion interference. Directly making navigation decisions on the image plane is easily affected by perspective relationships. However, by transforming image features into a local space in the robot coordinate system, local visual features can describe the environmental structure within a local area in a spatially aligned manner in the robot coordinate system, which is more suitable for drivability assessment and local path planning. Furthermore, this local spatial representation does not require the target robot to build a complete global map, nor does it require long-term maintenance of global environmental consistency. Instead, it provides local spatial priors for the current navigation decision, avoiding the problems of high map building and maintenance costs and localization error accumulation in traditional methods. Moreover, this local spatial representation can provide drivable areas and environmental structure information for the generation of local sub-objectives, and can provide a risk constraint basis for trajectory safety optimization.

[0103] Therefore, after obtaining the image features, this embodiment can further perform BEV projection on the image features. The specific implementation steps include: First, the image features are input into the frustum transformation layer of the local feature representation module to elevate the image features into a depth-dependent frustum space. This means associating a set of depth distributions with each pixel location of the image features in the depth dimension, thus obtaining the frustum features. While preserving the semantic information of the image, the frustum features introduce depth distribution information along the camera's line of sight, providing a spatial depth basis for the projection from the image space to the local region space.

[0104] Then, based on the intrinsic parameters of the target robot's camera and the transformation matrix between the camera's coordinate system and the target robot's coordinate system, the frustum features are projected or aggregated into each BEV grid in the local area to obtain local visual features, also known as local BEV features.

[0105] The camera's intrinsic parameters include imaging parameters such as the camera's focal length and principal point, while the transformation matrix includes the rotation and translation relationship between the camera coordinate system and the robot coordinate system.

[0106] It should be noted that the process described above, which involves lifting image features to the view frustum and then projecting and aggregating them to the local grid, can be achieved using the Lift-Splat-Shoot (LSS) projection method. This projection process can be represented as follows: ; in, Indicates current visual observation information; This represents the visual feature extraction process performed by the visual coding layer; Indicates the camera's internal parameters; The transformation matrix represents the coordinate system between the camera and the target robot. This represents the feature projection process from image space to local region space. It represents local visual features.

[0107] It should be noted that the projection method described above eliminates the need for additional depth sensors. It utilizes only current visual observation information and camera parameters to complete the feature transformation from the perspective image space to the local region space in the robot coordinate system, reducing the hardware dependence on depth sensors. Furthermore, the projected local visual features are aligned with the grid spaces within the local region, enabling an explicit description of the environmental structure within the local region centered on the target robot.

[0108] After obtaining the local visual features, these features can be input into the accessibility prediction layer of the local feature representation module to predict the accessibility of each grid cell within the local area, thus obtaining the local environmental risk features. This prediction process can be represented as follows: ; ; in, This represents the accessibility prediction layer; This represents the predicted feasibility logical value. This represents the normalization exponential function, used to transform feasible logical values ​​into a probability distribution; It represents the probability distribution of passability of each grid cell within a local area, which is also a representation of the local environmental risk characteristics.

[0109] The accessibility prediction layer here can be implemented using a convolutional decoder, an upsampling decoder, or a U-shaped network (UNet) structure to recover a high spatial resolution accessibility probability map from local visual features. This embodiment does not specifically limit this.

[0110] In this embodiment, the current visual observation information in the perspective image space is converted into local visual features aligned with the space of each grid cell in the local region through a visual coding layer, a view frustum transformation layer, and spatial projection based on camera intrinsic parameters and transformation matrix. This effectively eliminates the interference of perspective relationships and viewpoint distortion on the understanding of local space. Furthermore, the accessibility prediction layer explicitly estimates the accessibility risk of each grid cell in the local region, providing environmental structural information and risk constraints with clear spatial semantics for the generation of local sub-targets and optimization of navigation trajectories.

[0111] In some embodiments, inputting the target guidance features and the local visual features into the local target decision module of the navigation planning model to obtain the current local sub-target within the local area includes: The target guidance features and the local visual features are input into the fusion layer of the local target decision module to obtain local spatial features; The local spatial features are input into the local target decision layer of the local target decision module to obtain a local sub-target sequence; the local sub-target sequence includes at least one local sub-target located within the local region; If the navigation target is located outside the local area, the effective local sub-target that is closest to the target robot in the local sub-target sequence is determined as the current local sub-target; If the navigation target is located within the local area, the navigation target is identified as the current local sub-target.

[0112] Optionally, in order to enable local visual features to perceive target orientation, the target can be guided to the features first. and local visual features The input is fed into the fusion layer of the local target decision module for target condition feature fusion to obtain local spatial features. This fusion process can be represented as follows: ; in, This represents the target condition feature fusion function executed by the fusion layer. This represents the local spatial features obtained after fusion. The resulting local spatial features include not only the environmental structure and traversable area information within the current local area, but also the direction and spatial relationship information of distant navigation targets.

[0113] Then, the local spatial features are input into the local target decision layer of the local target decision module to obtain the local sub-target sequence. The process can be represented as follows: ; in, This represents the local target decision-making level; Indicates the number of predicted local sub-objectives; Represents the first in the robot coordinate system Each local sub-target can be represented as two-dimensional coordinates within a local region. The sequence of local sub-targets describes the phased navigation intentions of the target robot as it moves from its current position toward the navigation target. The local sub-targets in the sequence are arranged in order of increasing distance from the target robot, thus decomposing the long-distance target navigation task into multiple phased motion targets within a local region.

[0114] Furthermore, after obtaining the local sub-target sequence, the current local sub-target can be determined based on the positional relationship between the navigation target and the local area.

[0115] Specifically, when the navigation target is outside the local area, i.e., before it has entered the target robot's current local planning range, the nearest valid local sub-target in the local sub-target sequence is determined as the current local sub-target. Here, a valid local sub-target refers to a local sub-target located within the local area and meeting the prediction validity condition; correspondingly, local sub-targets outside the local area or whose prediction validity does not meet the condition are not considered candidates for the current local sub-target. Selecting the nearest valid local sub-target ensures that the current local sub-target best matches the target robot's current observable environment structure, guaranteeing the executability of the motion task within a short timeframe.

[0116] When the navigation target is located within a local area, that is, when the target robot is close to the navigation target and the navigation target has entered the local planning range, there is no need to use intermediate local sub-targets for transition. The navigation target is directly identified as the current local sub-target to guide the target robot to move directly towards the navigation target, avoiding detours caused by introducing intermediate targets.

[0117] In this embodiment, the target guidance features are injected into the local visual features through the fusion layer to obtain local spatial features. The local target decision layer predicts the local sub-target sequence, and then the current local sub-target is adaptively determined by combining the positional relationship between the navigation target and the local area. This allows the target robot to steadily advance according to the phased navigation intention when the navigation target is far away, and to directly reach the target when it is close to the navigation target, thus taking into account the target consistency of long-distance navigation and the arrival accuracy of short-distance navigation.

[0118] In some embodiments, generating an initial navigation trajectory based on the current visual observation information and the coordinate information of the current local sub-target includes: The current visual observation information is input into the visual encoding module in the trajectory prediction model to obtain a visual feature sequence; The coordinate information of the current local sub-target is input into the feature embedding module in the trajectory prediction model to obtain the target embedding features; The target embedded features and the visual feature sequence are input into the feature interaction module of the trajectory prediction model to obtain interactive features; The interactive features are input into the prediction module of the trajectory prediction model to obtain the initial navigation trajectory.

[0119] Figure 5 This is a schematic diagram of the initial navigation trajectory prediction process provided by the present invention; for example... Figure 5 As shown, during the initial navigation trajectory prediction process, the current visual observation information is first input into the visual encoding module in the trajectory prediction model to obtain a visual feature sequence. This visual encoding module can employ a ViT backbone network, a convolutional neural network, or other image encoding networks to encode the current visual observation information into a visual feature sequence composed of multiple image feature tokens. This embodiment does not specifically limit this approach.

[0120] Furthermore, the coordinate information of the current local sub-target is input into the feature embedding module of the trajectory prediction model to obtain target embedding features. Specifically, the feature embedding module first uses a Fourier position encoder to encode the coordinate information of the current local sub-target, obtaining an encoded vector containing position frequency information, and then performs feature mapping through a multilayer perceptron to obtain the target embedding features. This target embedding feature can be injected as conditional information into the visual feature sequence, enabling the trajectory prediction model to generate a trajectory moving towards the current local sub-target based on the current visual observation information.

[0121] Then, the target embedding features and visual feature sequences are input into the feature interaction module of the trajectory prediction model to obtain interactive features. This feature interaction module can be implemented using a Transformer encoding layer or a self-attention module. It uses a self-attention mechanism to model the interaction between each image feature token in the visual feature sequence and the target embedding features, thereby capturing visual features related to the current local sub-target within the global receptive field and establishing the conditional relationship between the current visual observation information and the current local sub-target.

[0122] Finally, the interaction features are input into the prediction module of the trajectory prediction model to obtain the initial navigation trajectory. Specifically, the prediction module may include a pooling layer and a trajectory prediction head. After the interaction features are aggregated by feature mean pooling in the pooling layer, the trajectory prediction head outputs the initial navigation trajectory composed of H future waypoints. This initial trajectory represents the trajectory information of the mobile robot moving towards a local sub-target in a short time period, and the specific formula is as follows: ; in, This indicates the trajectory prediction process performed by the trajectory prediction model; Indicates current visual observation information; Indicates the current local sub-target; The first term in the coordinate system of the target robot A future waypoint, which can be represented by two-dimensional coordinates; This represents the predicted total number of future waypoints. The settings can be adjusted according to the target robot's speed, control frequency, and navigation scenario. For example, it can be set to 8, etc. This embodiment does not specifically limit this.

[0123] It should be noted that during the training of the trajectory prediction model, navigation trajectory labels can be constructed first based on the travel trajectories of the sample robot that are located after the sample time. These navigation trajectory labels can be constructed based on the real waypoints in the travel trajectories after the sample time, and are used to supervise the prediction of the initial navigation trajectory of the sample.

[0124] After obtaining the supervision information, the visual observation information of the sample robot at the sample time and the coordinate information of the sample local sub-targets determined in the sample local area determined by the position of the sample robot at the sample time can be input into the initial trajectory generation model to obtain the initial navigation trajectory of the sample. Then, the trajectory generation loss value is determined based on the deviation between the initial navigation trajectory of the sample and the navigation trajectory label.

[0125] For example, suppose the navigation track label is The trajectory generation loss value can be calculated using the following mean squared error loss: ; in, Represents the first in the initial navigation trajectory of the sample One waypoint; Indicates the first in the navigation track label One waypoint; Indicates the total number of waypoints; This represents the trajectory generation loss value.

[0126] Finally, the initial trajectory generation model is trained based on the trajectory generation loss value to obtain the trajectory prediction model.

[0127] In this embodiment, through the cascaded structure of the visual encoding module, feature embedding module, feature interaction module and prediction module, the current visual observation information is interactively modeled and future waypoints are predicted based on the current local sub-target as a condition, so that the generated initial navigation trajectory not only matches the current local environment structure, but also maintains the motion trend toward the current local sub-target.

[0128] In some embodiments, optimizing the initial navigation trajectory based on the local environmental risk characteristics to obtain an optimized navigation trajectory includes: Based on the local environmental risk characteristics, determine the obstacle probability value for each of the grids; Based on the obstacle probability value, each grid is divided into an obstacle grid or a free grid; the obstacle probability value of the obstacle grid is greater than the obstacle probability threshold, and the obstacle probability value of the free grid is less than or equal to the obstacle probability threshold. For each grid cell within the local area, calculate a first distance between the grid cell and the nearest obstacle grid cell, and a second distance between the grid cell and the nearest free grid cell, and obtain the sign distance of the grid cell based on the difference between the first distance and the second distance; A local cost map is generated based on the symbolic distance of each of the grids; Based on the local cost map, the initial navigation trajectory is iteratively corrected to obtain the optimized navigation trajectory.

[0129] Figure 6 This is a schematic diagram of the local cost map generation process provided by the present invention; for example... Figure 6 As shown, during trajectory optimization, the obstacle probability value of each grid cell within the local area can be determined first based on the local environmental risk characteristics. For example, when the local environmental risk characteristics are represented as a passability probability map, the corresponding obstacle probability value can be calculated based on the passability probability of each grid cell; when the local environmental risk characteristics are directly represented as an obstacle probability map, the obstacle probability value of each grid cell can be directly read. This embodiment does not specifically limit this.

[0130] Then, each grid is thresholded based on the obstacle probability value to obtain the obstacle region mask. For the grid positions within a local region... Its division formula can be expressed as follows: ; in, The obstacle probability threshold; This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. Indicates grid position The obstacle region mask value, where a value of 1 corresponds to an obstacle grid and a value of 0 corresponds to a free grid. It should be noted that the obstacle probability threshold... The value can be set according to the safety requirements of the navigation scenario. The smaller the value, the more conservative the obstacle judgment strategy. For example, it can be set to 0.3. This embodiment does not make specific limitations on this.

[0131] Furthermore, since the obstacle grid and free grid obtained through thresholding are discrete binary representations, they are difficult to directly use for continuous trajectory optimization. Therefore, for each grid position within a local region... It can calculate the first distance between the grid cell and the nearest obstacle grid cell. And the second distance between the grid and the nearest free grid. And based on the difference between the first distance and the second distance, the sign distance of the grid is obtained: ; Then, a local cost map is generated based on the signed distance of each grid cell, where the grid cell positions are... The corresponding cost can be expressed as follows: ; in, This is a parameter used to control the rate at which the cost value decreases as the distance to the obstacle changes; It is an exponential function; Indicates grid position The cost value in the local cost map. In this way, the obstacle grid and its neighboring areas have a large cost value, while the passable areas far away from the obstacle grid have a small cost value. The cost value changes continuously with the distance between the grid and the obstacle grid, thus forming a local cost map suitable for continuous trajectory optimization and ensuring that the local cost map can provide effective gradient guidance at all points in the local region.

[0132] Finally, using the initial navigation trajectory as the initial value for optimization, the initial navigation trajectory is iteratively corrected based on the local cost map. This corrected trajectory maintains the movement trend toward the current local sub-target while gradually moving toward passable areas with lower cost in the local cost map, thus obtaining the optimized navigation trajectory.

[0133] Figure 7 This is one of the simulation diagrams of the robot visual navigation method provided by the present invention.

[0134] For example, such as Figure 7As shown, before optimization based on the local cost map, the initial navigation trajectory may be close to areas with high cost value or road boundaries in the local cost map; however, after iterative correction based on the local cost map, the optimized navigation trajectory can move towards traversable areas with lower cost value and maintain the trend of movement towards the current local sub-target.

[0135] In this embodiment, by sequentially transforming local environmental risk features into discrete partitions of obstacle grids and free grids, a symbolic distance based on the difference between the first distance and the second distance, and a continuous local cost map, the transformation from discrete risk perception results to continuous optimizable risk representation is realized. This enables the initial navigation trajectory to be iteratively corrected under continuous cost constraints, effectively ensuring that the optimized navigation trajectory stays away from obstacle areas, and improving the safety and practical feasibility of the navigation trajectory.

[0136] In some embodiments, the step of iteratively correcting the initial navigation trajectory based on the local cost map to obtain the optimized navigation trajectory includes: Map each waypoint in the initial navigation trajectory to a grid in the local area, and determine the total cost of the initial navigation trajectory based on the cost value of the grid mapped to each waypoint in the local cost map. Based on the spatial gradient of the total value relative to the initial navigation trajectory, the update step size, and the cost constraint weight, the waypoints in the initial navigation trajectory are iteratively corrected to obtain the optimized navigation trajectory.

[0137] Optionally, since each waypoint in the initial navigation trajectory is represented as a continuous coordinate in the target robot's coordinate system, while the local cost map is defined on a discrete grid in a local region, a coordinate mapping function can be used first. Each waypoint in the initial navigation trajectory is mapped to a corresponding BEV grid within a local region. The total cost of the initial navigation trajectory is determined based on the cost value of the grid mapped to each waypoint in the local cost map. The specific calculation method is as follows: ; in, , Indicates the initial navigation trajectory; Represents the first in the initial navigation trajectory One waypoint; Waypoints The grid positions within the mapped local area; Indicates the position of the grid. Cost value in the local cost map; This represents the total number of waypoints in the initial navigation trajectory; This represents the total value of the initial navigation trajectory. This total value comprehensively reflects the risk level of the areas traversed by the trajectory as a whole. The higher the total value, the closer the trajectory is to obstacle areas or high-risk areas.

[0138] Then, based on the spatial gradient of the total agent value relative to the trajectory, the update step size, and the cost constraint weight, the waypoints in the initial navigation trajectory are iteratively corrected.

[0139] Let the first The trajectory obtained from the next iteration correction is The calculation formula for iterative correction is as follows: ; in, Indicates the first The trajectory obtained after the next iteration correction; This indicates the update step size, used to control the magnitude of the adjustment of each waypoint position in each iteration; This represents the cost constraint weight, which is used to control the intensity of the influence of local cost map constraints on trajectory correction. Representing the trajectory The spatial gradient of the total agency value relative to the position of each waypoint in the trajectory indicates the waypoint adjustment direction that causes the total agency value to decrease the fastest.

[0140] It should be noted that, since the cost value in the local cost map changes continuously with the distance between the grid and the obstacle grid, the spatial gradient of the total cost value relative to the trajectory naturally points away from the obstacle region. Therefore, during the iterative correction process, waypoints in the trajectory that are close to the obstacle region will gradually move towards the passable region with lower cost value along the gradient descent direction. After several iterations until a preset iteration termination condition is met, the optimized navigation trajectory can be obtained. The iteration termination condition can be that the number of iterations reaches a preset number, or that the change in the total cost value between two adjacent iterations is less than a preset change threshold, etc. This embodiment does not specifically limit this.

[0141] Update step size With cost constraint weights The settings can be adjusted according to the convergence speed and stability requirements of trajectory correction.

[0142] Figure 8 This is the second simulation diagram of the robot visual navigation method provided by the present invention.

[0143] For example, such as Figure 8As shown, in scenarios such as intersections, forks in the road, or where local observations are ambiguous, the same or similar current visual observation information may correspond to navigation targets in different directions. Because target guidance features are injected into the local spatial features, the local target decision layer can generate different local sub-target sequences under different navigation target conditions: when the navigation target is to the left of the target robot, the local sub-target sequence can guide the target robot to move to the left branch; when the navigation target is to the right of the target robot, the local sub-target sequence can guide the target robot to move to the right branch. This achieves decision disambiguation in intersection branch scenarios, improving target consistency and navigation stability in long-distance visual navigation.

[0144] In this embodiment, since the optimization process starts from the initial navigation trajectory, the optimized trajectory can retain the target direction and overall movement trend of the original trajectory. At the same time, due to the introduction of local cost graph constraints, the trajectory is gradually guided to a passable area away from the obstacle area, so that the optimized navigation trajectory can stay away from the obstacle area and high-risk area, thereby improving the local safety and actual executability of the navigation trajectory.

[0145] In some embodiments, the training steps of the navigation planning model include: Based on the sample depth information, obstacle region evidence is generated within the sample local area of ​​the sample robot; Based on the driving trajectory located within the sample local area in the sample driving trajectory, evidence of passable area within the sample local area is generated; By fusing the obstacle area evidence and the passable area evidence, a passability label and a valid area mask are generated; the valid area mask is used to identify unknown or known areas. Based on the driving trajectory located within the local region of the sample driving trajectory, generate local sub-target sequence labels; The sample robot's visual observation information and the coordinate information of the sample navigation target at the sample time are input into the initialization model to obtain the sample local environmental risk characteristics and sample local sub-target sequence output by the initialization model. The first loss value is determined based on the deviation between the local environmental risk characteristics of the sample and the accessibility label in the known area; The second loss value is determined based on the deviation between the sample local sub-target sequence and the local sub-target sequence label; The initialization model is trained based on the first loss value and the second loss value to obtain the navigation planning model.

[0146] Here, the sample moment refers to the historical moment selected from the historical driving data of the sample robot to construct a single training sample.

[0147] The sample local region refers to the local region determined based on the position of the sample robot at the sample time.

[0148] The initialization model refers to the navigation planning model before training, whose model parameters are to be optimized. Its network structure is consistent with the navigation planning model.

[0149] It should be noted that manually annotating passable or obstacle areas within a sample's local region frame by frame is costly, and the significant differences in road morphology, obstacle distribution, and visual appearance across different environments further increase the difficulty of model training and deployment. Therefore, this embodiment utilizes the sample depth information and trajectory acquired by the sample robot at the sample time point to automatically generate the supervision information required for training.

[0150] Figure 9 This is a schematic diagram of the training process for the navigation planning model provided by the present invention; for example... Figure 9 As shown, obstacle region evidence is first generated within the sample's local area based on the sample depth information. Specifically, for the sample depth information corresponding to the sample time, the intrinsic parameters of the robot's camera and the transformation matrix between the camera's coordinate system and the robot's coordinate system are combined to convert the depth pixels into a 3D point cloud in the robot's coordinate system. Then, the 3D point cloud is filtered by spatial range and height, and points that meet the obstacle height condition are projected onto each grid within the sample's local area to obtain obstacle region evidence. This obstacle region evidence is used to represent areas within the sample's local area that have a high probability of being impassable or have a high collision risk.

[0151] Simultaneously, based on the driving trajectory within the sample local area in the sample driving trajectory, evidence of traversable areas within the sample local area is generated. Specifically, since the area actually traversed by the sample robot can usually be considered a traversable area, for the sample time, the sample driving trajectory of the sample robot, i.e., the complete driving trajectory, can be obtained first; Then, using the pose of the sample robot at the sample time, the trajectory points in the sample driving trajectory are transformed to the coordinate system of the sample robot at the sample time. The trajectory points or trajectory segments that fall into the local area of ​​the sample are then selected and projected onto each BEV grid in the local area of ​​the sample. Furthermore, in order to improve the stability of the tag, the width of the sample robot's body or the safe passage width it covers can be appropriately expanded according to the width of the sample robot or the safe passage width to finally obtain evidence of the passable area.

[0152] Then, obstacle area evidence and passable area evidence are fused to generate passability labels and effective area masks. Specifically, the fused passability labels can be used to identify passable, impassable, and unknown areas. Passable areas are determined by passable area evidence, while impassable areas are determined by obstacle area evidence. Areas with insufficient depth observation, uncovered driving trajectories, or unreliable evidence are designated as unknown areas. The effective area mask is used to identify whether each grid belongs to an unknown or known area. The known areas are those with relatively reliable supervision information, i.e., passable and impassable areas.

[0153] It should be noted that in actual data, due to factors such as momentary occlusion by dynamic obstacles and depth measurement noise, the same grid may be simultaneously marked by obstacle area evidence and traversable area evidence, resulting in evidence conflict. For such conflicting areas, since the areas covered by the sample robot's historical trajectories have been physically traversed, their evidence reliability is relatively high, so traversable area evidence can be prioritized. In addition, temporal consistency and local neighborhood consistency can be combined for processing, that is, examining whether obstacle evidence for the same grid appears stably at multiple sample times (or historical times), and whether the evidence distribution of neighboring grids is consistent with that grid. If the obstacle evidence is consistent in both temporal and neighborhood aspects, the grid is marked as an impassable area; otherwise, it is considered noise and set as an unknown area, which is ignored during training through effective area masking, thereby reducing the impact of depth noise, dynamic obstacles, or momentary occlusion on label quality.

[0154] Furthermore, local sub-target sequence labels are generated based on the driving trajectories located within the sample local region of the sample driving trajectory. Specifically, for local sub-target sequence labels, several intermediate points can be sampled along the driving trajectories located within the sample local region in order of distance or time, and used as local sub-target sequence labels; when the length of the driving trajectory located within the sample local region is insufficient, it can be supplemented using the last reachable point or sample navigation target, and the actual valid supervision items are marked using a valid sub-target mask.

[0155] After obtaining the above-mentioned supervision information, the sample robot's visual observation information and the coordinate information of the sample navigation target at the sample time are input into the initialization model to obtain the sample local environmental risk characteristics and sample local sub-target sequence output by the initialization model.

[0156] Then, based on the deviation between the local environmental risk characteristics of the sample and the accessibility label within the known area, a first loss value is determined. For example, the first loss value can be calculated using masked cross-entropy loss: ; in, This represents the grid position within the local sample region predicted by the initial model. The passability logical value at the location, which is also the local environmental risk characteristic of the sample; Indicates the accessibility label at the grid position. The label value at the location; Indicates the effective area mask at the grid position. The value at the given location is 1 for known regions and 0 for unknown regions; Represents the cross-entropy loss function; This represents the first loss value.

[0157] Here, by introducing an effective region mask, the initialization model learns only within known regions with relatively reliable supervision information, thereby reducing the interference of unknown regions and noisy labels on the training process.

[0158] Simultaneously, a second loss value is determined based on the deviation between the sample's local sub-target sequence and its label. For example, let the sample's local sub-target sequence be... The corresponding local sub-target sequence label is The second loss value can then be expressed as follows: ; in, Represents the first sub-target sequence in the sample. A local sub-target; Indicates the corresponding first Local sub-target labels; Indicates the first The weights corresponding to each local sub-objective; Represents the square of the Euclidean distance; This represents the second loss value.

[0159] It should be noted that, The weights are dynamically determined based on the distances between each local sub-target and the sample robot. For example, local sub-targets that are closer to the sample robot can be given larger weights to enhance their constraint on the current local motion decision; while local sub-targets that are farther from the sample robot can be given smaller weights so that they are mainly used to maintain the overall directional consistency.

[0160] Finally, the initialization model is trained based on the first loss value and the second loss value to obtain the navigation planning model. For example, the first loss value and the second loss value can be weighted and summed to obtain the total loss value. The model parameters of the initialization model are then updated through backpropagation based on the total loss value. This training process is repeated until a preset convergence condition is met. If the total loss value converges or the preset number of training epochs is reached, the initialization model that meets the convergence condition is determined as the navigation planning model. The weighting coefficients corresponding to each loss value can be set according to the importance of each training task; this embodiment does not specifically limit this.

[0161] In this embodiment, by automatically generating local spatial weak supervision labels using sample depth information and sample driving trajectory, the automatic construction of training supervision information is realized, which significantly reduces the cost of manually annotating local passable areas frame by frame. Furthermore, by jointly training the initialization model using the first loss value and the second loss value, the navigation planning model can simultaneously learn local environmental risk prediction and local sub-target decision-making capabilities. Moreover, the two share the underlying feature representation and constrain each other, which effectively improves the training efficiency and prediction consistency of the navigation planning model and enhances the feasibility of large-scale training and deployment of the navigation planning model in real-world environments.

[0162] The robot vision navigation system provided by the present invention is described below. The robot vision navigation system described below can be referred to in correspondence with the robot vision navigation method described above.

[0163] Figure 10 This is a schematic diagram of the robot vision navigation system provided by the present invention; as shown below. Figure 10 As shown, the system includes: The first feature extraction unit 1010 is used to input the coordinate information of the navigation target corresponding to the target robot into the long-scale target guidance module in the navigation planning model to obtain target guidance features; the target guidance features are used to characterize the geometric relationship features between the navigation target and the target robot and each grid in the local area corresponding to the target robot; The second feature extraction unit 1020 is used to input the current visual observation information of the target robot into the local feature representation module in the navigation planning model to obtain the local visual features and local environmental risk features of the local area; The sub-target generation unit 1030 is used to input the target guidance features and the local visual features into the local target decision module in the navigation planning model to obtain the current local sub-target within the local area; The initial trajectory generation unit 1040 is used to generate an initial navigation trajectory based on the current visual observation information and the coordinate information of the current local sub-target; The optimization unit 1050 is used to optimize the initial navigation trajectory based on the local environmental risk characteristics to obtain an optimized navigation trajectory; The control unit 1060 is used to perform navigation control on the target robot according to the optimized navigation trajectory; The navigation planning model is trained based on the historical driving data of the sample robot. The historical driving data includes the sample visual observation information acquired by the sample robot at the sample time, the coordinate information of the sample navigation target, the sample depth information corresponding to the sample visual observation information, and the sample driving trajectory.

[0164] The system provided in this embodiment obtains target guidance features by geometrically modeling the coordinate information of the navigation target through a long-scale target guidance module. This establishes an effective connection between long-distance target information and local motion decisions, overcoming the technical defect that relying solely on current local observations for decision-making can easily lead to short-sighted behavior. Based on this, a local feature representation module converts the current visual observation information in the perspective image space into local visual features in the robot coordinate system and predicts local environmental risk features, enabling the target robot to explicitly perceive the surrounding spatial layout and potential risks. Furthermore, a local target decision module fuses the target guidance features and local visual features to determine the current local sub-target, transforming the long-distance target navigation task into an executable motion task within a local range. Local environmental risk features are used to constrain and optimize the initial navigation trajectory, ensuring that the final optimized navigation trajectory remains consistent with the navigation target direction while avoiding obstacle areas and high-risk areas. This effectively ensures the target robot's target consistency, local safety, and trajectory executability during navigation in the absence of high-precision maps and strong prior environmental information, achieving correct, safe, stable, and executable autonomous robot navigation.

[0165] The system provided by this invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0166] The device embodiments described above are merely illustrative. The layers described as separate components may or may not be physically separated, and the components shown as layers may or may not be physical layers; that is, they may be located in one place or distributed across multiple network layers. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0167] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot visual navigation method, characterized in that, include: The coordinate information of the navigation target corresponding to the target robot is input into the long-scale target guidance module in the navigation planning model to obtain the target guidance features; The target guidance feature is used to characterize the geometric relationship between the navigation target and the target robot and the grids within the local area corresponding to the target robot; The current visual observation information of the target robot is input into the local feature representation module in the navigation planning model to obtain the local visual features and local environmental risk features of the local area; The target guidance features and the local visual features are input into the local target decision module in the navigation planning model to obtain the current local sub-target within the local area; Based on the current visual observation information and the coordinate information of the current local sub-target, an initial navigation trajectory is generated, and based on the local environmental risk characteristics, the initial navigation trajectory is optimized to obtain an optimized navigation trajectory; Based on the optimized navigation trajectory, the target robot is subjected to navigation control. The navigation planning model is trained based on the historical driving data of the sample robot. The historical driving data includes the sample visual observation information acquired by the sample robot at the sample time, the coordinate information of the sample navigation target, the sample depth information corresponding to the sample visual observation information, and the sample driving trajectory.

2. The robot vision navigation method according to claim 1, characterized in that, The step of inputting the coordinate information of the navigation target corresponding to the target robot into the long-scale target guidance module in the navigation planning model to obtain target guidance features includes: The coordinate information of the navigation target is encoded to obtain coordinate encoding features; The coordinate encoding features are input into the multilayer perceptron layer of the long-scale target guidance module to obtain target coordinate features; the target coordinate features are used to characterize the geometric relationship between the navigation target and the target robot. A relative target feature is constructed based on the directional angle of each grid pointing to the navigation target and the distance between each grid and the navigation target; the relative target feature is used to characterize the geometric relationship between the navigation target and each grid. The relative target features and the target coordinate features are input into the fusion layer of the long-scale target guidance module to obtain the target guidance features.

3. The robot vision navigation method according to claim 2, characterized in that, The process of encoding the coordinate information of the navigation target to obtain coordinate encoding features includes: The coordinate information of the navigation target is normalized to obtain normalized target coordinate information; By applying sine and cosine functions to each element of the normalized target coordinate information at multiple frequencies, the sine coding result and cosine coding result corresponding to each frequency are obtained. The normalized target coordinate information, the sine encoding result, and the cosine encoding result are concatenated to obtain the coordinate encoding feature.

4. The robot visual navigation method according to any one of claims 1-3, characterized in that, The step of inputting the current visual observation information of the target robot into the local feature representation module of the navigation planning model to obtain the local visual features and local environmental risk features of the local region includes: The current visual observation information is input into the visual coding layer of the local feature representation module to obtain image features; The image features are input into the frustum transformation layer of the local feature representation module to obtain frustum features; Based on the intrinsic parameters of the target robot's camera and the transformation matrix between the camera's coordinate system and the target robot's coordinate system, the visual cone feature is projected onto each grid in the local region to obtain the local visual feature. The local visual features are input into the accessibility prediction layer of the local feature representation module to obtain the local environmental risk features.

5. The robot visual navigation method according to any one of claims 1-3, characterized in that, The step of inputting the target guidance features and the local visual features into the local target decision module of the navigation planning model to obtain the current local sub-target within the local area includes: The target guidance features and the local visual features are input into the fusion layer of the local target decision module to obtain local spatial features; The local spatial features are input into the local target decision layer of the local target decision module to obtain a local sub-target sequence; the local sub-target sequence includes at least one local sub-target located within the local region; If the navigation target is located outside the local area, the effective local sub-target that is closest to the target robot in the local sub-target sequence is determined as the current local sub-target; If the navigation target is located within the local area, the navigation target is identified as the current local sub-target.

6. The robot visual navigation method according to any one of claims 1-3, characterized in that, The step of generating an initial navigation trajectory based on the current visual observation information and the coordinate information of the current local sub-target includes: The current visual observation information is input into the visual encoding module in the trajectory prediction model to obtain a visual feature sequence; The coordinate information of the current local sub-target is input into the feature embedding module in the trajectory prediction model to obtain the target embedding features; The target embedded features and the visual feature sequence are input into the feature interaction module of the trajectory prediction model to obtain interactive features; The interactive features are input into the prediction module of the trajectory prediction model to obtain the initial navigation trajectory.

7. The robot visual navigation method according to any one of claims 1-3, characterized in that, The step of optimizing the initial navigation trajectory based on the local environmental risk characteristics to obtain an optimized navigation trajectory includes: Based on the local environmental risk characteristics, determine the obstacle probability value for each of the grids; Based on the obstacle probability value, each grid is divided into an obstacle grid or a free grid; the obstacle probability value of the obstacle grid is greater than the obstacle probability threshold, and the obstacle probability value of the free grid is less than or equal to the obstacle probability threshold. For each grid cell within the local area, calculate a first distance between the grid cell and the nearest obstacle grid cell, and a second distance between the grid cell and the nearest free grid cell, and obtain the sign distance of the grid cell based on the difference between the first distance and the second distance; A local cost map is generated based on the symbolic distance of each of the grids; Based on the local cost map, the initial navigation trajectory is iteratively corrected to obtain the optimized navigation trajectory.

8. The robot vision navigation method according to claim 7, characterized in that, The step of iteratively correcting the initial navigation trajectory based on the local cost map to obtain the optimized navigation trajectory includes: Map each waypoint in the initial navigation trajectory to a grid in the local area, and determine the total cost of the initial navigation trajectory based on the cost value of the grid mapped to each waypoint in the local cost map. Based on the spatial gradient of the total value relative to the initial navigation trajectory, the update step size, and the cost constraint weight, the waypoints in the initial navigation trajectory are iteratively corrected to obtain the optimized navigation trajectory.

9. The robot visual navigation method according to any one of claims 1-3, characterized in that, The training steps of the navigation planning model include: generating obstacle region evidence within the sample local area of ​​the sample robot based on the sample depth information; Based on the driving trajectory located within the sample local area in the sample driving trajectory, evidence of passable area within the sample local area is generated; By fusing the obstacle area evidence and the passable area evidence, a passability label and a valid area mask are generated; the valid area mask is used to identify unknown or known areas. Based on the driving trajectory located within the local region of the sample driving trajectory, generate local sub-target sequence labels; The sample robot's visual observation information and the coordinate information of the sample navigation target at the sample time are input into the initialization model to obtain the sample local environmental risk characteristics and sample local sub-target sequence output by the initialization model. The first loss value is determined based on the deviation between the local environmental risk characteristics of the sample and the accessibility label in the known area; The second loss value is determined based on the deviation between the sample local sub-target sequence and the local sub-target sequence label; The initialization model is trained based on the first loss value and the second loss value to obtain the navigation planning model.

10. A robot vision navigation system, characterized in that, include: The first feature extraction unit is used to input the coordinate information of the navigation target corresponding to the target robot into the long-scale target guidance module in the navigation planning model to obtain target guidance features; The target guidance feature is used to characterize the geometric relationship between the navigation target and the target robot and the grids within the local area corresponding to the target robot; The second feature extraction unit is used to input the current visual observation information of the target robot into the local feature representation module in the navigation planning model to obtain the local visual features and local environmental risk features of the local area; The sub-target generation unit is used to input the target guidance features and the local visual features into the local target decision module in the navigation planning model to obtain the current local sub-target within the local area; An initial trajectory generation unit is used to generate an initial navigation trajectory based on the current visual observation information and the coordinate information of the current local sub-target; An optimization unit is used to optimize the initial navigation trajectory based on the local environmental risk characteristics to obtain an optimized navigation trajectory; A control unit is used to perform navigation control on the target robot according to the optimized navigation trajectory; The navigation planning model is trained based on the historical driving data of the sample robot. The historical driving data includes the sample visual observation information acquired by the sample robot at the sample time, the coordinate information of the sample navigation target, the sample depth information corresponding to the sample visual observation information, and the sample driving trajectory.

Images