Pure visual navigation method for leg-foot robot fusing image guidance and image understanding

Abstract

The invention discloses a pure visual navigation method for a legged robot fusing image guidance and image understanding. An image guiding part of the method adopts a target navigation planning algorithm based on a Transform architecture, outputs a relative navigation target point with a target image as guidance, and guides a robot to complete key path selection; in the image understanding part, a lightweight semantic segmentation and obstacle avoidance algorithm based on an improved Yov5 model is adopted, a passable area is obtained based on a semantic segmentation result and a preset passable semantic table, coordinates of obstacle avoidance target points are distributed and output at the position of the passable area in a visual angle, and a robot completes path following and obstacle avoidance tasks by means of semantic understanding. According to the method disclosed by the invention, the robot can perform a navigation task in a complex environment including an outdoor environment by only depending on a small number of target images, so that the generalization problem of an original pure end-to-end vision algorithm is effectively solved, and a long-distance navigation task in a complex scene is realized.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a purely visual navigation method for legged robots that integrates image guidance and image understanding. Background Technology

[0002] Legged robots, due to their unique configuration, possess strong terrain adaptability and have enormous application potential in various complex environments. Autonomous navigation capability is the core of the intelligent application of legged robots in various complex task scenarios. Autonomous navigation technology refers to a technological system in which a robot, relying on various sensors to acquire information about its environment and its own state, can reach a designated target location or move along a given task trajectory without human intervention. Autonomous navigation technology has evolved rapidly, gradually developing from relying on geometric environmental information to being based on image and semantic information, and then combining with large language models, becoming one of the research hotspots in the field of embodied intelligence.

[0003] Early mainstream methods for autonomous navigation in legged robots were geometric navigation based on spatial information. Geometric navigation methods collect geometric data of the environment using sensors (such as LiDAR and accelerometers), and construct navigation space models based on this data, such as grid maps and topological maps. Given a starting point and a target point, global path planning algorithms, such as A* and Dijkstra's algorithm, are used to obtain the geometrically shortest path. Local planning algorithms, such as TEB and DWA, are used to perform obstacle avoidance while tracking the path. Geometric navigation methods are suitable for static and spatially regular environments. However, as robot applications become more complex, the limitations of geometric navigation become increasingly apparent, especially in dynamic or unknown environments, or when localization is unavailable. Relying solely on geometric information for path planning leads to low success rates.

[0004] With the development of edge computing power and artificial intelligence technology in robotics, autonomous navigation technology for robots is evolving towards intelligence. Vision-based autonomous navigation methods are emerging.

[0005] Pure vision-based autonomous navigation refers to a technology where robots rely solely on visual sensors to perceive their environment and generate motion decision commands through end-to-end modular algorithms to complete navigation, obstacle avoidance, and target arrival tasks. This technology does not depend on geometric or positioning information obtained from devices such as lidar and accelerometers. It simulates humans relying on purely visual information to gain a semantic understanding of the environment for navigation. Specifically, this semantic understanding includes understanding terrain, environmental memory, and traversable areas, helping robots make more flexible and intelligent navigation decisions in complex and unfamiliar environments.

[0006] Currently, domestic and international research on purely visual navigation methods focuses on purely visual image-based target navigation. A typical characteristic of purely visual target navigation is that, given image information of the target point, the robot infers the potential positional relationship between the target image information and the current image information, and navigates to the target point based on this potential positional relationship. For example, Zhu et al. from the Stanford team proposed a target-driven visual navigation method (Target-Driven Visual Navigation in Indoor Scenes) based on reinforcement learning and target view images as input. This method combines the current viewpoint and target image as input, employs end-to-end reinforcement learning to output speed, enabling the robot to navigate autonomously based on target image commands. Li et al. from Georgia Tech developed a camera-view-based visual servoing method (Learning View and TargetInvariant Visual Servoing), enabling the robot to complete navigation tasks and avoid obstacles in unknown environments by establishing a visual-motor mapping through visual information, relying solely on a camera. These studies provide theoretical support for purely visual navigation technology for robots in unknown and dynamic environments and improve the autonomous navigation and obstacle avoidance capabilities of robots in various environments.

[0007] Domestic research mainly focuses on navigation algorithms that integrate reinforcement learning and visual information, guided by target images. Xing Jianchuan et al. (CN117288205A) proposed a pure visual navigation method based on autonomous learning. They extracted features from the image using ResNet-50 and fed them into the A3C network, continuously optimizing the loss function to achieve navigation based on target image information. Sun Chongkun et al. (CN120351913A) proposed a visual navigation algorithm based on X-corner detection and matching in images, realizing a method for visual navigation based on calculating pixel coordinates and physical coordinates using predetermined features.

[0008] Despite some progress in existing pure visual image-based target navigation technologies, domestic research largely focuses on similar indoor scenarios. This is because pure visual image-based target navigation relies on model training and environmental interaction with large amounts of data, while simulation environments are mostly indoors, making it impossible to transfer to complex outdoor scenarios. International research also faces performance degradation issues in zero-shot, multi-scenario tasks due to environmental changes and dynamic lighting. Furthermore, current image-based target navigation technologies rely on end-to-end target point output, requiring high environmental generalization capabilities, which can easily lead to low navigation success rates and unstable navigation performance.

[0009] To address these issues, this paper proposes a pure vision-based navigation method for legged robots that integrates image guidance and image understanding. The image guidance component employs a target navigation planning algorithm based on the Transformer architecture, outputting relative navigation target points guided by the target image to guide the robot in selecting critical paths. The image understanding component utilizes a lightweight semantic segmentation obstacle avoidance algorithm based on an improved Yolov5 model. Based on the semantic segmentation results and a pre-defined navigable semantic table, navigable regions are obtained. The relative obstacle avoidance target point coordinates are output within the navigable region's position in the viewpoint, allowing the robot to perform path following and obstacle avoidance tasks through semantic understanding. This combination of image guidance and image understanding leverages the lightweight and data-driven architecture advantages of end-to-end algorithms, while explicit semantic segmentation for image understanding supplements and corrects the generalization limitations of end-to-end algorithms. This method enables robots to navigate in complex environments, including outdoor environments, relying only on a limited number of target images, effectively compensating for the generalization limitations of traditional pure end-to-end vision algorithms and achieving long-distance navigation in complex scenarios. Summary of the Invention

[0010] In light of the aforementioned existing methods and their problems, this invention proposes the following technical solution: a pure vision navigation method for legged robots that integrates image guidance and image understanding, comprising the following steps: (1) Collect image sequence-trajectory coordinate datasets obtained when the robot performs navigation tasks; (2) Design an end-to-end target navigation planning algorithm with image guidance characteristics, segment the dataset in step (1) and train it, design the loss function and optimize the network parameters; (3) A lightweight semantic segmentation model is obtained by improving the Yolov5 network structure for object detection. The distribution of passable regions is obtained by combining the segmentation results from the current perspective with semantic rules. The coordinates of the desired target point are obtained by using the passable region distribution calculation module to complete image understanding. Semantic obstacle avoidance is achieved by tracking the coordinates. (4) Based on the real-time estimated distance output by the target navigation planning algorithm in step (2), the image guidance-image understanding upper state machine is designed as the switching basis. The upper state machine is used to freely switch between step (2) or step (3) to output the target point. (5) Collect first-person images in the navigation scene as key image guidance information; (6) Deploy the methods of steps (2), (3), and (4) onto the robot; Based on the target image in the navigation scene given in step (5), the robot uses the upper state machine selection strategy of step (4), the target navigation planning algorithm of step (2) and the semantic segmentation obstacle avoidance algorithm of step (3) to obtain the real-time relative target point coordinates, performs calculations on the real-time relative target point coordinates, and outputs angular velocity and linear velocity to achieve target point tracking, thereby completing the long-distance navigation task in the outdoor scene.

[0011] Further, step (1) specifically involves: collecting image sequence-trajectory coordinate datasets obtained in actual scenarios using unmanned vehicles, legged robots, quadruped robots, and drones in autonomous navigation or remote control states. The dataset includes first-person perspective RGB images and the robot's relative pose in each image.

[0012] Further, step (2) specifically involves: designing and training a Transformer-based target navigation planning algorithm. The target navigation planning algorithm uses an end-to-end architecture to obtain the relationship between multiple image frame sequences and the target image, extracts features, and outputs target points guided by the target image. The input to the target navigation planning algorithm network consists of three parts: the target image, the current image, and historical images. The output consists of two parts: the first part is the estimated future trajectory represented by relative coordinates. By tracking this future trajectory, the robot estimates the target point on the trajectory and can reach the location of the target image. The second part is the estimated distance between the current image and the target image, which is used for the design of the upper-level state machine. During the network training phase, a complete image sequence-trajectory coordinate dataset in the dataset is divided into four parts: target image and coordinates, current image and coordinates, historical image sequence and historical coordinate sequence, and future coordinate sequence. The first three parts are the input data of the target navigation planning algorithm, and the last part is used as the network's output ground truth. In the two parts of the output results of the target navigation planning algorithm, the estimated future trajectory is compared with the actual future trajectory in the dataset to obtain the first part of the error used for backpropagation of the network, and the estimated distance is compared with the actual distance between the current image and the target image in the dataset to obtain the second part of the error. The errors of the two parts are combined and backpropagated to train the neural network.

[0013] Further, step (3) specifically involves: designing and training a semantic segmentation obstacle avoidance algorithm, which is divided into two parts. First, the original target detection Yolov5 network structure is improved by adding a semantic segmentation component to obtain a lightweight semantic segmentation model. The CitySpace dataset is used for training to obtain a real-time semantic segmentation strategy network. Based on the output of the semantic segmentation model and combined with preset semantic rules, a passable area is obtained. Then, the passable area distribution calculation module is used to output the robot's expected target point coordinates and perform coordinate following to complete the semantic obstacle avoidance planning.

[0014] Further, step (4) specifically involves: designing an image-guided image understanding upper-level state machine. The upper-level state machine determines the algorithm to be used at the current moment based on the real-time estimated distance output by the target navigation planning algorithm in step (2). If the estimated distance is small, the target navigation planning algorithm in step (2) is used to output the target point; if the estimated distance is large, the semantic segmentation obstacle avoidance algorithm in step (3) is used to obtain the distribution of passable areas and output the target point.

[0015] Further, step (5) specifically involves: using a camera on the robot or an external camera to collect first-person target images in the navigation environment as key image guidance for pure visual navigation.

[0016] Furthermore, the coordinates of the desired target point are in a two-dimensional coordinate system with the location of the robot camera as the origin, where the positive direction of the horizontal coordinate is the direction directly in front of the camera's viewpoint, and the positive direction of the vertical coordinate is the direction directly to the right of the camera's viewpoint.

[0017] Furthermore, the output of the target navigation planning algorithm is a sequence of target points containing 5 discrete coordinate points. Depending on the specific scenario, any one of these 5 target points can be used as the target point tracked in step (6).

[0018] Furthermore, the target navigation planning algorithm adopts the EfficientNet-B0 image coding architecture with 5M parameters, and then uses position coding to feed the high-dimensional features of the image into a Transformer block with 4 attention heads, and then outputs the distance estimate and discrete target point sequence.

[0019] Furthermore, the semantic segmentation obstacle avoidance algorithm first uses a lightweight semantic segmentation model to obtain each semantic region, then obtains the passable region based on predefined passable semantic categories, and finally uses the passable region calculation module to calculate the coordinates of the desired target point that can bypass impassable objects.

[0020] The beneficial effects of this invention are that it integrates two navigation methods: image-guided target navigation and semantic obstacle avoidance based on image semantic understanding. It fully leverages the implicit image guidance and explicit semantic segmentation in the navigation planning process, improving the safety and success rate of long-distance navigation for legged robots in outdoor environments facing changing conditions and dynamic lighting. To address the excellent mobility of legged robots in various complex terrains, this paper proposes a Transformer-based target navigation planning algorithm. The network is trained using a large dataset of robots of various shapes in outdoor environments, improving its generalization ability in end-to-end navigation for legged robots. Furthermore, this paper proposes a lightweight semantic segmentation-based obstacle avoidance algorithm. This algorithm, based on an improvement of Yolov5, allows for direct output of the planned target point after a small amount of computation following the segmentation results. The network is well-suited to the edge computing hardware of legged robots and can meet the real-time semantic segmentation requirements. In summary, the target navigation planning algorithm and the lightweight semantic segmentation-based obstacle avoidance planning algorithm in this invention provide a solid foundation for pure vision-based long-distance navigation planning for legged robots in complex outdoor environments. Attached Figure Description

[0021] Figure 1 This is an overall flowchart of the present invention; Figure 2 This is a flowchart of the target navigation planning algorithm of the present invention; Figure 3 This is a schematic diagram of the training results of the target navigation planning algorithm of the present invention; Figure 4 This is a flowchart of the semantic segmentation obstacle avoidance algorithm of the present invention; Figure 5 This is an accuracy graph of the lightweight semantic segmentation algorithm of this invention; Figure 6 This is an image showing the output of the lightweight semantic segmentation algorithm of this invention. Figure 7 This is a diagram showing the extracted passable region after semantic segmentation according to the present invention. Figure 8 This is a schematic diagram of the upper-level state machine of the image-guided image understanding invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings.

[0023] Legged robots are widely used due to their superior terrain adaptability. However, in complex outdoor environments, most current navigation solutions require positioning information from LiDAR and accelerometers or pre-built 3D maps. Pure vision navigation methods mostly employ implicit image guidance or explicit semantic segmentation techniques. There are few solutions that consider using both implicit image guidance and explicit semantic segmentation simultaneously. This limits the reliability and flexibility of pure vision path planning solutions under long-distance outdoor path planning conditions and does not fully demonstrate the generalization and superiority of pure vision path planning methods.

[0024] To address the aforementioned problems, this invention proposes a purely visual navigation method for legged robots that integrates image guidance and image understanding. The overall flowchart is as follows: Figure 1 As shown, it can be divided into three parts: target navigation planning algorithm module, semantic obstacle avoidance planning algorithm module, and image guidance-image understanding upper-level state machine module. Figure 1 This is an overall flowchart of the invention, reflecting the information flow during network deployment. Specific training and model details are provided later. The following section details the steps involved in implementing this method: S1. Dataset Collection Steps. Using Isaac Sim and Carla software, along with various robots (autonomous vehicles, legged robots, quadruped robots, and drones), RGB images and six-DOF pose information were collected in both virtual and real-world scenarios using first-person cameras to obtain an image sequence-trajectory coordinate dataset. This dataset helps the model learn the potential correlation between images and trajectories without online interaction.

[0025] The image sequence-trajectory coordinate dataset contains RGB images that correspond one-to-one with six-degree-of-freedom pose information. Each pose information is the robot pose at the moment the RGB image is captured. This pose is taken as the origin from the center of the camera's viewpoint and is used for training and testing the target navigation planning algorithm in subsequent steps.

[0026] S2. Design and train an end-to-end target navigation planning algorithm based on Transformer. Design an algorithm that adopts an end-to-end architecture, acquires visual features between multiple image frame sequences and the target image, and finally outputs the coordinates of the target point guided by the target image. Then, train the target navigation planning algorithm using the dataset from step S1.

[0027] The significant advantage of this target navigation planning algorithm is that it can output the target point based solely on the current image information and a small amount of target image guidance. It does not require the robot to build a map in the navigation environment in advance, nor does it rely on the robot's positioning information during the navigation process.

[0028] The flowchart of the target navigation planning algorithm is as follows: Figure 2 As shown. The input to the target navigation planning algorithm consists of three parts: the target image, the current image, and historical images. The network output consists of two parts. The first part is the estimated future trajectory represented by relative coordinates. By tracking this future trajectory, the robot estimates the target point on the trajectory and can reach the location of the target image. The second part is the estimated distance between the current image and the target image, which is used for the design of the state machine in step S4.

[0029] S21. Obtain the input to the target navigation planning algorithm and extract image features. Divide a complete image sequence-trajectory coordinate dataset into four parts: target image and coordinates, current image and coordinates, historical image sequence and historical coordinate sequence, and future coordinate sequence. The first three parts are the input data for the target navigation planning algorithm, and the last part, the future coordinate sequence, is the expected output ground truth of the algorithm.

[0030] Definition of historical image sequence: The dataset contains RGB images acquired by the camera, each with a width and height of 224 pixels and 3 channels. The current time step is defined as... In the current Time, define the historical sequence of images observed by the robot as input. for: in Let P represent the observed image at time t, and let P represent the length of the model's historical frames, selecting the length of the historical image sequence. .

[0031] Current image definition: Define the current image as , representing the RGB image captured by the camera at the current moment.

[0032] Target image definition: Define the target image at time t as The target image is selected directly from the dataset in step S1.

[0033] Six frames, including the historical image sequence and the current image, are input into the EfficientNet-B0 feature extraction module to obtain six one-dimensional feature vectors of length 512. These vectors serve as the historical / current image feature vectors. The sequence contains several known image features from both the current and historical images during the robot navigation process, defined as follows: in This represents the extracted image features at time t.

[0034] The current image and the target image are concatenated according to their channel counts to obtain a 224*224*6 input image. The purpose of this process is to directly compare the pixel relationships between the current and target images at each pixel location, extracting these relationship features in subsequent steps. Since EfficientNet-B0 defaults to 3 input image channels, to ensure training stability and input channel adaptation, the weights of the original 3-channel input convolutional layer are copied to obtain a 6-channel input convolutional layer, resulting in the feature network EfficientNet-B0*. The concatenated 224*224*6 image is then input into EfficientNet-B0*, resulting in a one-dimensional vector of length 512. This vector is the current / target image feature vector, defined as... .

[0035] S22. After obtaining two feature vectors in step S21, these two feature vectors are concatenated to obtain a complete set of 7 feature vectors with a length of 512. Then, position encoding is performed according to the timestamp order of each feature vector to obtain the following encoded vector: In the above formula This represents the extracted image features at time t, where the position codes are 0, 1, 2, 3, 4, 5, and 6. Position codes 0-5 correspond to the historical / current image feature vectors, and position code 6 corresponds to the current / target image feature vector. The purpose of position coding is to assign an initial temporal relationship to each vector before inputting it into the Transformer layer, i.e., the features. The target image information should be placed at the end of the sequence, while the features... Similarly, following the chronological order, a temporal logic of historical image - current image - future image is formed.

[0036] S23. After obtaining the feature vector after position encoding, the feature vector is input into four identical self-attention mechanism layers containing four attention heads, i.e., four Transformer layers. This step extracts the correlation information of image features at different times in the feature vector, and finally inputs a feature vector of length 32.

[0037] S24. After obtaining the final feature vector, map each feature vector to a 1-dimensional vector using a fully connected network layer to obtain the current image. With target image Estimated distance Simultaneously, the final feature vector is mapped to a 3x5 matrix through a fully connected network layer, representing the predicted 5 target points. The sequence of these 5 target points constitutes the estimated future trajectory. The output sequence of these target points is defined as follows: for: in This indicates that the target point output sequence contains information on 5 target points, and the information on each target point... Represented as: in Represents the target point at time t. The relative X coordinate, Represents the target point at time t. The relative Y coordinate, Represents the target point at time t. relative angle .

[0038] S25. Based on the above network structure, network training is performed, and the network... Overall mapping of time It can be defined as: in Input the historical sequence of observed images. This represents the observed image at time t. This represents the target image at time t.

[0039] Define network loss It is divided into three parts: trajectory loss error, distance regression loss, and trajectory smoothing loss. in , , The weighting coefficients for trajectory loss error, distance regression loss, and trajectory smoothing loss are 0.5, 0.5, and 0.3, respectively. Trajectory loss error is the difference between the estimated future trajectory output by the algorithm and the actual future trajectory in the dataset. The trajectory loss error represents the deviation between the estimated trajectory and the actual trajectory. The Huber loss function used in this invention is more robust to outliers than L2 regression.

[0040] The distance regression loss is derived from the estimated distance. The difference between the current image and the target image in the dataset: in Based on the results of the previous distance estimation, the hyperparameters Setting this parameter to 0.3 smooths the distance estimate using historical data, thus improving the stability of the estimate.

[0041] The trajectory smoothing loss, used to reduce high-frequency oscillations, is beneficial for conforming to the dynamic constraints of legged robots and encourages speeds within a reasonable range; it is defined as: This smoothing loss encourages velocity and acceleration to remain within a reasonable range, preventing the distribution of each trajectory from deviating too much from the desired trajectory.

[0042] Figure 3 This is a schematic diagram illustrating the training results of a target navigation planning network. In the left image, the green dot represents the robot's current position, the red dot represents the target image position, the purple trajectory represents the actual trajectory in the dataset, including the robot's orientation information, and the blue trajectory represents the model's predicted trajectory. The predicted trajectory contains five target points; generally, the third target point located in the middle is taken as the final target point, and its relative coordinates are denoted as . ,in The x-axis is relative. The vertical axis represents the relative vertical axis. A schematic diagram of the target navigation planning algorithm training results is shown below. Figure 3 As shown, the model predicts the future trajectory, which, based on image guidance information, enables the robot to complete navigation tasks and reach the target point.

[0043] S3. Design and train a semantic segmentation obstacle avoidance algorithm. This algorithm consists of two parts: a lightweight semantic segmentation model and a drivable region distribution calculation module. The lightweight semantic segmentation model improves the original Yolov5 network structure for object detection by adding a semantic segmentation component, resulting in a lightweight semantic segmentation algorithm. It is trained using datasets such as CitySpace to obtain an efficient real-time semantic segmentation policy network. The drivable region distribution calculation module, based on the segmentation results from the current viewpoint and combined with preset drivable and impassable semantic rules, obtains drivable regions. Based on the distribution of drivable regions in the current viewpoint, it outputs the coordinates of the robot's desired target point and performs coordinate following to achieve semantic obstacle avoidance.

[0044] The desired target point coordinates are in a two-dimensional coordinate system with the location of the robot camera as the origin, where the positive direction of the horizontal coordinate is the direction directly in front of the camera and the positive direction of the vertical coordinate is the direction directly to the right of the camera.

[0045] The advantage of this algorithm is that it can compensate for the distribution offset problem of the target navigation planning algorithm in step S2 during long-distance navigation. It uses explicit semantic segmentation results for obstacle avoidance, effectively improving the overall safety and success rate of the navigation algorithm. The flowchart of the semantic segmentation obstacle avoidance algorithm is as follows: Figure 4 As shown.

[0046] S31. Input the current real-time image into the lightweight semantic segmentation model. The real-time image size is 224*224*3. The lightweight semantic segmentation network is an improvement on the Yolov5 object detection network. The feature vectors of the original network's 16th, 19th, and 22nd feature concatenation layers are extracted separately and output for 19 semantic categories using the SegMaskPSP semantic detection head. The CitySpace and Yolov datasets are used for training, and the training results are as follows. Figure 5 As shown in the figure. The semantic segmentation pixel accuracy obtained on the test set is 94.9%, and the IoU (Intersection over Union) ratio is 73.3%, which meets the accuracy requirements of semantic segmentation and can obtain the semantic segmentation result based on real-time image input. The semantic segmentation output result is shown in the figure. Figure 6 As shown, the results indicate that the semantic segmentation model obtained the correct semantic regions based on the image information.

[0047] S32. After obtaining the semantic segmentation image, the passable regions are determined according to predefined semantic rules. The semantic rules are: "road," "sidewalk," and "terrain" are defined as passable regions, while the remaining 16 semantic types are defined as impassable regions. Passable regions are represented in white, and impassable regions are represented in black, as shown below. Figure 7 As shown.

[0048] S33. Select 5 equally sized rectangular regions in the robot's frontal view area, such as... Figure 6 As shown, calculate the area ratio of the passable area in each of the five rectangular regions. The passable area ratio is denoted from left to right as follows: ,and First, normalize the horizontal direction of each rectangle in the field of view to... Where -1 represents left, -0.5 represents slightly to the left, 0 represents center, 0.5 represents slightly to the right, and 1 represents right. Therefore, the horizontal centers of the five rectangles are: To avoid extreme values ​​and make subsequent calculations of the passage ratio more robust, power-law enhancement is used to obtain the directional weights. .

[0049] in Taking 1.5 makes the overall traffic flow ratio smoother. The following formula calculates the relative x-coordinate of the target point. .

[0050] When the traffic flow ratio of the left rectangle is small and the traffic flow ratio of the right rectangle is large... The result will be biased towards a positive value, leading to a target point on the right. This guides the robot to turn right, avoiding areas with limited passage on the left. Next, the relative ordinate of the target point will be calculated. .

[0051] in This indicates the traffic flow ratio in each area. This represents the maximum forward distance, taken as 5m. This represents the minimum forward distance, taken as 1m. Multiply the maximum forward distance by the average passage ratio of the five rectangles, then... Take the maximum value to obtain the vertical coordinate of the target point. This calculation indicates that when the road is generally clear, the overall target point is farther away, encouraging the robot to move forward. When there are many obstacles, a closer target point is chosen, allowing the robot to move at a slower speed, but ultimately reaching the minimum value. Prevent robot navigation strategies from becoming overly cautious.

[0052] Thus, the target point coordinates output by the semantic segmentation obstacle avoidance algorithm are obtained. Including the relative x-axis relative ordinate .

[0053] S4. Design the image-guided image understanding upper-level state machine. This state machine determines the appropriate algorithm to use at the current moment based on the real-time estimated distance output by the target navigation planning algorithm in step S2. When the distance is small, the target point is output using the target navigation planning algorithm in step S2; when the estimated distance is small... When the value is large, the semantic segmentation obstacle avoidance algorithm in step S3 is used to obtain the distribution of passable areas and output the target points.

[0054] The advantage of this image-guided image understanding upper-level state machine is that it can use the estimated distance output by the target navigation planning algorithm to select between the target points output in steps S2 and S3, ensuring that a better strategy is adopted and improving the success rate of navigation tasks.

[0055] The state machine is based on the estimated distance in step S24. Real-time determination of the coordinates output by the semantic segmentation obstacle avoidance algorithm to select the current position. Or the target point coordinates output by the target navigation planning algorithm The specific decision-making process of the state machine is as follows: Figure 8As shown, the estimated distance between the current image output by the target navigation planning algorithm and the first frame of the current target image sequence is defined as dist_to_start, and the estimated distance between the current image output by the target navigation planning algorithm and the last frame of the current target image sequence is defined as dist_to_end. When dist_to_start is less than the threshold of 4m for 10 consecutive time steps, it indicates that the robot has approached the starting point of the current target image sequence, and the state machine needs to be switched to the state output by the target navigation planning algorithm. At the same time as switching the state, the robot's linear velocity is reduced. When dist_to_end is less than the threshold of 2m for 10 consecutive time steps, it indicates that the robot has approached the ending point of the current target image sequence, and the state machine needs to be switched to the state output by the semantic segmentation obstacle avoidance algorithm. At the same time as switching the state, the robot's linear velocity is reduced, and the next target image sequence is loaded.

[0056] S5. Using the robot's camera or a handheld camera, collect first-person target images of the navigation environment. For example, in an urban road scene, image sequences in key areas (such as intersections, path selection areas, etc.) serve as key image guidance for purely visual navigation. These multiple image sequences are provided as target images to the target navigation planning algorithm in step S2, and also serve as an important basis for the upper-level state machine to select a strategy. That is, based on the target image sequences, the algorithm in step S2 is used in areas requiring target image guidance, while the algorithm in step S3 is used in areas where only obstacle avoidance is required.

[0057] S6. Deploy the algorithms onto the physical robot, including the image-guided image understanding upper-level state machine in step S4, the image-guided target navigation planning algorithm in step S2, and the image understanding-based semantic segmentation obstacle avoidance algorithm in step S3. Each of these algorithms can select a strategy in real time based on the real-time image and the target image in the navigation scene given in step S5, and provide the coordinates of the navigation target point. After obtaining the target point coordinates through the algorithms in step S3 or S5, they need to be converted into the target angular velocity and linear velocity of the robot's actuators. The specific calculation formulas are as follows: in This represents the angle of the target point relative to the robot, calculated using the two-parameter arctangent function. For the linear velocity gain, set it to 0.5, then set the linear velocity... Limited to 0 and between, Take 2 m / s, For the angular velocity gain, set it to 1.5, then limit the angular velocity to... and between, The angular velocity is set to 2 rad / s. The angular and linear velocities are then output to the low-level execution module of the legged robot to achieve target point tracking. Once the algorithm detects that the robot has reached the last frame of the target image sequence, the program automatically stops outputting angular and linear velocities, indicating that the final target point has been reached and the navigation task is complete, thus fulfilling the long-distance mapless navigation task in an outdoor scene.

[0058] In summary, the target navigation planning algorithm and semantic segmentation obstacle avoidance algorithm proposed in this invention provide a reference for long-distance navigation planning tasks of legged robots in outdoor environments using pure vision. By fusing the two algorithms through an upper-level state machine, the implicit image guidance and explicit semantic segmentation play a full role in the planning and navigation process. This improves the safety and success rate of long-distance navigation of legged robots in outdoor environments with changing environments and dynamic lighting conditions, without prior maps or explicit localization. It provides a reference for the research on the fusion of implicit image-guided end-to-end navigation and explicit semantic segmentation obstacle avoidance algorithms, and also provides a solid foundation for long-distance navigation planning of legged robots in complex outdoor environments using pure vision.

[0059] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A purely visual navigation method for legged robots that integrates image guidance and image understanding, characterized in that, Includes the following steps: (1) Collect image sequence-trajectory coordinate datasets obtained when the robot performs navigation tasks; (2) Design an end-to-end target navigation planning algorithm with image guidance characteristics, segment the dataset in step (1) and train it, design the loss function and optimize the network parameters; (3) A lightweight semantic segmentation model is obtained by improving the Yolov5 network structure for object detection. The distribution of passable regions is obtained by combining the segmentation results from the current perspective with semantic rules. The coordinates of the desired target point are obtained by using the passable region distribution calculation module to complete image understanding. Semantic obstacle avoidance is achieved by tracking the coordinates. (4) Based on the real-time estimated distance output by the target navigation planning algorithm in step (2), the image guidance-image understanding upper state machine is designed as the switching basis. The upper state machine is used to freely switch between step (2) or step (3) to output the target point. (5) Collect first-person images in the navigation scene as key image guidance information; (6) Deploy the methods of steps (2), (3), and (4) onto the robot; Based on the target image in the navigation scene given in step (5), the robot uses the upper state machine selection strategy of step (4), the target navigation planning algorithm of step (2) and the semantic segmentation obstacle avoidance algorithm of step (3) to obtain the real-time relative target point coordinates, performs calculations on the real-time relative target point coordinates, and outputs angular velocity and linear velocity to achieve target point tracking, thereby completing the long-distance navigation task in the outdoor scene.

2. The pure vision navigation method for legged robots that integrates image guidance and image understanding according to claim 1, characterized in that, The specific step (1) is as follows: collect image sequence-trajectory coordinate datasets obtained in actual scenarios by unmanned vehicles, legged robots, quadruped robots, and drones in autonomous navigation or remote control states. The dataset includes first-person perspective RGB images and the robot's relative pose in each image.

3. The pure vision navigation method for legged robots that integrates image guidance and image understanding according to claim 1, characterized in that, The specific steps (2) are as follows: designing and training a target navigation planning algorithm based on Transformer. The target navigation planning algorithm adopts an end-to-end architecture to obtain the relationship between multiple image frame sequences and the target image, extracts features, and outputs the target point guided by the target image. The input to the target navigation planning algorithm network consists of three parts: the target image, the current image, and historical images. The output is divided into two parts. The first part is the estimated future trajectory in relative coordinates. By tracking this future trajectory, the robot can estimate the target point on the trajectory and reach the location of the target image. The second part is the estimated distance between the current image and the target image. This distance is used for the design of the upper-level state machine. During the network training phase, a complete image sequence-trajectory coordinate dataset is divided into four parts: target image and coordinates, current image and coordinates, historical image sequence and historical coordinate sequence, and future coordinate sequence. The first three parts are the input data of the target navigation planning algorithm, and the last part is used as the network's output ground truth. The target navigation planning algorithm outputs two parts of results: the estimated future trajectory is compared with the actual future trajectory in the dataset to obtain the first part of the error used for backpropagation of the network; the estimated distance is compared with the actual distance between the current image and the target image in the dataset to obtain the second part of the error. The errors of the two parts are combined and backpropagated to train the neural network.

4. The pure vision navigation method for legged robots that integrates image guidance and image understanding according to claim 1, characterized in that, The specific steps (3) are as follows: design and train a semantic segmentation obstacle avoidance algorithm, which is divided into two parts. First, improve the original target detection Yolov5 network structure, add a semantic segmentation component, obtain a lightweight semantic segmentation model, and train it using the CitySpace dataset to obtain a real-time semantic segmentation strategy network. Based on the output of the semantic segmentation model, combined with the preset semantic rules, obtain the passable area. Then, use the passable area distribution calculation module to output the robot's expected target point coordinates and perform coordinate following to complete the semantic obstacle avoidance planning.

5. The pure vision navigation method for legged robots that integrates image guidance and image understanding according to claim 1, characterized in that, The specific steps (4) are as follows: design an image-guided image understanding upper-level state machine. The upper-level state machine determines the algorithm to be used at the current moment based on the real-time estimated distance output by the target navigation planning algorithm in step (2). If the estimated distance is small, the target navigation planning algorithm in step (2) is used to output the target point; if the estimated distance is large, the semantic segmentation obstacle avoidance algorithm in step (3) is used to obtain the distribution of passable areas and output the target point.

6. The pure vision navigation method for legged robots that integrates image guidance and image understanding according to claim 1, characterized in that, The specific step (5) is as follows: using a camera on the robot or an external camera to collect first-person target images in the navigation environment as key image guidance for pure visual navigation.

7. The pure vision navigation method for legged robots that integrates image guidance and image understanding according to claim 1, characterized in that, The coordinates of the desired target point are in a two-dimensional coordinate system with the location of the robot camera as the origin, where the positive direction of the horizontal coordinate is the direction directly in front of the camera's viewpoint, and the positive direction of the vertical coordinate is the direction directly to the right of the camera's viewpoint.

8. The pure vision navigation method for legged robots that integrates image guidance and image understanding according to claim 1, characterized in that, The output of the target navigation planning algorithm is a sequence of target points containing 5 discrete coordinate points. Depending on the specific scenario, any one of these 5 target points can be used as the target point to be tracked in step (6).

9. The pure vision navigation method for legged robots that integrates image guidance and image understanding according to claim 1, characterized in that, The target navigation planning algorithm adopts the EfficientNet-B0 image coding architecture with 5M parameters, and then uses position coding to feed the high-dimensional features of the image into a Transformer block with 4 attention heads. The output is then the distance estimate and the discrete target point sequence.

10. The pure vision navigation method for legged robots that integrates image guidance and image understanding according to claim 1, characterized in that, The semantic segmentation obstacle avoidance algorithm first uses a lightweight semantic segmentation model to obtain each semantic region, then obtains the passable region based on the predefined passable semantic categories, and finally uses the passable region calculation module to calculate the coordinates of the desired target point that can bypass impassable objects.

Images