A robot path planning method and system

Abstract

This invention relates to the field of mobile robot path planning, specifically to a robot path planning method and system. The method first constructs a global semantic map by receiving natural language commands and 2D navigation map images from the user. Then, it uses a Visual Language Model (VLM) to extract semantic information from the map and commands, generating a series of macroscopic landmarks. After parsing the landmarks into a global coordinate sequence, it uses each landmark as a temporary target point and runs a Dynamic Window Method (DWA) for local path planning and real-time obstacle avoidance. When an unreachable landmark is detected, a replanning mechanism is triggered, and the VLM regenerates the landmark sequence. This invention combines the macroscopic semantic understanding capability of VLM with the local real-time obstacle avoidance advantage of DWA, alleviating the problem of traditional DWA algorithms easily getting trapped in local optima. It also supports natural language human-computer interaction, significantly improving the robot's navigation performance and user experience in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of mobile robot path planning, specifically relating to a robot path planning method and system. Background Technology

[0002] Path planning technology for mobile robots is one of the core technologies for achieving autonomous robot navigation. Its goal is to plan a collision-free optimal path from the starting point to the destination for the robot in an environment with obstacles. Depending on the planning scope and the method of information acquisition, path planning technology is generally divided into two categories: global path planning and local path planning.

[0003] Global path planning, given a known environment map, pre-computes a globally optimal path from the starting point to the destination. Commonly used global path planning algorithms include A* algorithm, Dijkstra's algorithm, RRT (Rapid Expanding Random Tree) algorithm, and JPS (Jump Point Search) algorithm. Among them, A* algorithm, by introducing a heuristic function, can significantly improve search efficiency while ensuring optimality, and is one of the most widely used global path planning algorithms. However, global path planning algorithms rely on a pre-built static environment map, cannot handle dynamic changes in the environment and unknown obstacles, and have a large computational load, making it difficult to meet real-time requirements.

[0004] Local path planning relies on sensors onboard the robot (such as LiDAR and vision sensors) to perceive the surrounding environment in real time and dynamically generate motion commands to avoid unknown or dynamic obstacles. A representative local path planning algorithm is the Dynamic Window Approach (DWA). Proposed by Fox et al. in 1997, the core idea of ​​DWA is to sample multiple possible velocity combinations in the robot's velocity space (linear velocity v and angular velocity ω), simulate the robot's trajectory over a future period, and then score each trajectory using a predefined evaluation function, selecting the trajectory with the highest score as the motion command for the current moment. The DWA algorithm has advantages such as high computational efficiency, strong real-time performance, and effective avoidance of dynamic obstacles, and is widely used in various mobile robot platforms.

[0005] However, traditional DWA algorithms have a fundamental flaw: they rely solely on local cost maps and target point information for decision-making, lacking an understanding of the global environmental structure. This makes DWA algorithms prone to getting trapped in local optima in complex environments. Specifically, they may hesitate in long corridors due to the inability to determine the correct direction; oscillate in front of symmetrical obstacles due to similar evaluation function scores; or become trapped in U-shaped areas or dead ends. These problems severely limit the application of DWA algorithms in complex indoor environments.

[0006] To address the problem that traditional DWA algorithms are prone to getting trapped in local optima, academia and industry have proposed various improvement schemes, which can be mainly divided into the following two categories:

[0007] The first category involves improving the evaluation function of the DWA algorithm. For example, the degree of global path following can be added as a new evaluation metric to the trajectory evaluation of DWA, allowing the robot to consider the direction of the global path during local planning. Another example is balancing multiple objectives such as obstacle avoidance, speed, and target orientation by adjusting the weights of various terms in the evaluation function. However, these methods still rely on pre-generated global paths, and the robot will still encounter difficulties when the global path is blocked.

[0008] The second category combines global planning algorithms with DWA algorithms. A globally optimal path is generated using a global planning algorithm (such as A*, JPS, Dlite, etc.), and this path is discretized into a series of critical path points. The DWA algorithm is then guided to sequentially track these sub-objectives, avoiding blind searching. For example, the TWA-DWA algorithm proposed by Fuzhou University improves the A* algorithm to generate a global path and then combines it with the DWA algorithm for local obstacle avoidance, thus improving the efficiency and stability of path planning.

[0009] Although the above methods alleviate the local optima problem of the DWA algorithm to some extent, they still have the following three shortcomings:

[0010] First, traditional global planning algorithms generate paths based solely on geometric costs (such as distance and time), lacking a semantic understanding of the environment. These algorithms fail to recognize semantically meaningful environmental features such as doors, corridor corners, open areas, and densely populated areas, resulting in unintelligent selection of guide points. For example, a traditional algorithm might plan a path through a densely populated area or select the closest doorway from multiple doorways that is actually blocked.

[0011] Second, when the environment changes significantly or the global path is blocked, the global planning algorithm needs to be re-run frequently, resulting in high computational overhead. Especially in dynamic environments, the global path may fail frequently, causing the robot to stop frequently and wait for replanning, which seriously affects navigation efficiency.

[0012] Third, users find it difficult to specify macro-level travel strategies using natural language. Traditional path planning systems typically require users to input precise coordinates or select preset target points, and cannot understand natural language instructions such as prioritizing routes along walls, avoiding densely populated areas, or going around tables, resulting in a poor human-computer interaction experience.

[0013] In recent years, with the rapid development of deep learning technology, Vision-Language Models (VLMs) have made groundbreaking progress. VLMs possess powerful visual semantic understanding and common-sense reasoning capabilities, enabling them to identify objects in images, understand spatial relationships, and generate high-level plans that conform to language instructions. For example, models such as GPT-4V, Gemini, and LLaVA can generate detailed task plans and step descriptions based on input images and text instructions. Existing research has applied VLMs to high-level decision-making tasks such as task planning and sub-objective decomposition for robots, achieving good results.

[0014] However, directly outputting end-to-end motion commands from a VLM faces several challenges. First, VLMs have high inference latency, typically on the order of seconds or even tens of seconds, which cannot meet the real-time requirements of robot motion control (which usually requires frequencies above ten hertz). Second, the motion space output by a VLM is usually discrete, making it difficult to generate continuous motion control commands. Finally, VLMs lack strict safety guarantees and may generate motion commands that could lead to collisions or other hazards.

[0015] Therefore, how to organically combine the macroscopic semantic understanding capability of VLM with the real-time obstacle avoidance advantage of traditional local planning algorithms, so as to give full play to the high-level planning and semantic understanding capabilities of VLM while ensuring the real-time performance and safety of local motion control, is a key technical problem that urgently needs to be solved in the field of robot path planning. Summary of the Invention

[0016] The purpose of this invention is to provide a robot path planning method and a robot path planning system to solve the problems mentioned in the background art.

[0017] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0018] A robot path planning method includes the following steps:

[0019] Retrieve the starting point text description, the target point text description, and the corresponding 2D navigation map image;

[0020] The starting point text, target point text, and 2D navigation map image are input into a visual language model to generate macroscopic road sign information, which includes one or more intermediate road sign points.

[0021] The macroscopic road sign information is parsed into a road sign coordinate sequence in the global coordinate system;

[0022] Starting from the current robot pose, and taking the next unreached landmark in the landmark coordinate sequence as the temporary target point, the dynamic window method is used to generate local motion control commands.

[0023] When the robot reaches the current temporary target point, it updates the temporary target point to the next unreached landmark, until all landmarks are reached;

[0024] When the current temporary target point is detected to be unreachable, a replanning mechanism is triggered, and information is re-inputted into the visual language model to generate new macroscopic road sign information;

[0025] The visual language model adopts a multimodal large model architecture, including a visual encoder, modal connectors and a language model backbone, and is fine-tuned by a navigation instruction dataset to enable it to identify key waypoints from map images and generate road sign sequences.

[0026] Furthermore, the macroscopic road sign information includes structured text descriptions and their corresponding global coordinates. The structured text descriptions contain semantic information, including region names, landmark objects, or turn instructions.

[0027] Furthermore, the evaluation function of the dynamic window method includes a landmark orientation guidance term, which is used to encourage the robot to choose a motion trajectory toward the current target landmark.

[0028] Furthermore, the replanning mechanism includes: acquiring obstacle detection results output by sensors in real time, projecting the obstacle detection results onto the global semantic map in the form of obstacle semantic labels to generate an updated local semantic map image; inputting the current robot pose, remaining landmark descriptions, and the updated local semantic map image into a visual language model, and generating an alternative landmark sequence from the current position to the target point based on the thought chain prompting method.

[0029] Furthermore, the visual encoder adopts the SigLIP, CLIP, or ViT model, the modal connector adopts the Q-Former, linear projection layer, or cross-attention layer, and the language model backbone adopts the Phi-2, Llama-2-7B, or Mistral-7B model.

[0030] Furthermore, the navigation instruction dataset is constructed in the following way: starting point-target point pairs are randomly sampled in a simulation environment, a reference path is generated using a global path planning algorithm, and the reference path is converted into natural language instructions and road sign sequence descriptions using a large language model, forming a triplet of map image, text instructions, and road sign sequence.

[0031] Furthermore, the evaluation function of the dynamic window method also includes a path following guidance term, which measures the vertical distance between the end position of the simulated trajectory and the line connecting the current target landmark and the next landmark, encouraging the robot to follow the macroscopic path.

[0032] Furthermore, the thought chain prompting method includes guiding the visual language model to analyze the reasons for the obstruction, finding alternative channels, and generating a structured road sign sequence.

[0033] Furthermore, the conditions under which the current temporary target point is unreachable include: the robot cannot approach the target landmark within a continuously preset time period, all sampling trajectories have a collision risk, and the sensor detects that the channel is completely blocked.

[0034] This application also discloses an electronic device, including:

[0035] At least one processor; and

[0036] A memory communicatively connected to the at least one processor; wherein,

[0037] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the robot path planning method of the present invention described above.

[0038] This application also discloses a robot path planning system, including:

[0039] The map building module is used to build and store 2D navigation maps of the environment;

[0040] The human-computer interaction module is used to receive the starting point and target point descriptions input by the user in text form;

[0041] The VLM road sign generation module deploys a pre-trained visual language model, which takes map images and text commands as input and outputs macroscopic road sign information.

[0042] The landmark parsing and maintenance module is used to parse the VLM output into a global coordinate sequence and maintain the current target landmark status;

[0043] The DWA local planning module generates motion control commands in real time, guided by the current target landmark.

[0044] The replanning trigger module monitors the local planning status. When it detects that the current target landmark is unreachable, it generates an updated local semantic map image and triggers the VLM to regenerate the landmark.

[0045] The motion execution module receives control commands and drives the robot to move.

[0046] Beneficial Effects: This invention alleviates the local optima problem of the DWA algorithm. Traditional DWA algorithms rely solely on local cost maps and target point information for decision-making, making them prone to oscillations or dead zones in complex environments. This invention introduces macroscopic landmark guidance through VLM (Virtual Mapping Model), decomposing long-distance navigation tasks into several short sub-tasks that are easier to plan locally. In each sub-task, the DWA algorithm only needs to focus on obstacle avoidance and motion control within a local range of a few meters, avoiding the problem of getting trapped in local optima due to a lack of foresight. Experimental results show that in typical scenarios such as U-shaped obstacles and long corridors, the method of this invention reduces the number of times it gets trapped in local optima by 91.7% compared to the traditional DWA algorithm.

[0047] This invention endows the path planning system with semantic understanding capabilities. Traditional global planning algorithms only plan paths based on geometric distances and cannot understand semantic concepts such as doorways, corridors, and open areas. This invention utilizes the visual semantic understanding capabilities of Visual Learning Models (VLMs) to identify structured features in the environment and generate a sequence of landmarks that conforms to the semantic structure of the environment. For example, VLMs can identify doorways as key nodes connecting two areas, thus using them as important landmarks; it can understand corridors as long, narrow passageways, thus generating landmarks along the corridors. This makes the planned paths more consistent with human cognitive habits and more intelligent and rational.

[0048] Supporting natural language human-computer interaction, this invention allows users to specify starting and ending points through natural language descriptions, and even add macroscopic walking preferences, such as prioritizing routes along walls, avoiding densely populated areas, and circling around tables. The VLM (Visual Learning Machine) can understand these natural language instructions and incorporate them into the waypoint generation process. This greatly enhances the user-friendliness of human-computer interaction, lowers the barrier to entry for users, and enables the robot to better serve ordinary users.

[0049] Balancing global optimality and local real-time performance, this invention employs a hierarchical architecture. The Virtual Model (VLM) handles low-frequency macroscopic landmark updates (typically on the order of seconds), while the Dynamical Object Assist (DWA) handles high-frequency local motion control (typically 10Hz-50Hz). This design leverages the powerful reasoning and semantic understanding capabilities of the VLM while ensuring the real-time performance requirements of local obstacle avoidance. Compared to a pure end-to-end VLA model, the motion control latency of this invention is reduced by more than an order of magnitude, meeting the demands of high-speed robot movement and real-time obstacle avoidance.

[0050] Possessing strong adaptability to dynamic environments, this invention designs a robust dynamic replanning mechanism. When changes in the local environment render the original landmarks unreachable, the system can quickly trigger replanning, invoking the VLM (Virtual Mapping Model) to re-understand the local environment and adjust subsequent landmarks. Compared to traditional global replanning methods, the replanning in this invention only requires updating the remaining landmark sequence, resulting in lower computational overhead and faster response time. Experimental results show that the navigation success rate of this invention in dynamic environments is 10.3 percentage points higher than the A*+DWA fusion algorithm.

[0051] With excellent scalability and versatility, the technical solution of this invention does not depend on a specific robot platform or sensor type, and can be easily ported to various mobile robots, including wheeled robots, tracked robots, and humanoid robots. Furthermore, the VLM model can improve its performance in different environments by continuously adding new training data, demonstrating excellent scalability. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0053] Figure 2 This is a system architecture block diagram according to an embodiment of the present invention;

[0054] Figure 3 This is an example image of VLM macro-roadmark generation in an embodiment of the present invention, where (a) is the original 2D navigation map and (b) is the roadmark annotation result output by VLM;

[0055] Figure 4 This is a path comparison diagram between the embodiment of the present invention and the traditional DWA algorithm in a U-shaped obstacle scene. Detailed Implementation

[0056] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0057] This invention provides a robot path planning method, such as... Figure 1 As shown, the steps include:

[0058] Step 1: Construct a global semantic map. Use laser SLAM (Simultaneous Localization and Mapping) or visual SLAM technology to build a 2D grid map or topological map of the robot's working environment. The map resolution should be 2cm-10cm / pixel. Add semantic annotations to the completed map, including: area attributes (e.g., rooms, corridors, doorways, elevator lobbies, meeting rooms, offices, break rooms, etc.), static obstacles (e.g., walls, furniture, pillars, equipment, etc.), passable areas, and prohibited areas. Store the annotated semantic map in the robot's local storage or on a cloud server.

[0059] Step 2: Receive user commands and visual input. The human-computer interaction module receives the user's starting point and destination descriptions in text or voice format, such as "starting from the elevator and heading to the general manager's office." Simultaneously, it retrieves the corresponding 2D navigation map image from the map storage unit. If the user inputs a voice command, it is first converted into a text command using speech recognition technology.

[0060] Step 3: VLM Macro Landmark Generation. The starting point text, target point text, and 2D navigation map image obtained in Step 2 are input into a pre-trained Visual Language Model (VLM). VLM extracts features from the map image using a visual encoder, fuses visual and textual features through a modal connector, and then performs spatial reasoning and semantic understanding through the language model backbone, outputting a series of semantically meaningful macro landmarks. These macro landmarks include, but are not limited to, key turning points (such as corridor T-junctions, corridor corners), area crossing points (such as doorways, passageway entrances, elevator entrances), and semantic landmarks (such as next to a sofa, to the left of a pillar, at the entrance to the tea room).

[0061] Step 4: Landmark Sequence Parsing and Coordinate Transformation. The macroscopic landmark information output by the VLM is parsed to extract the structured text description and corresponding image coordinates for each landmark. Then, through a pre-established mapping relationship between image coordinates and the global coordinate system, the image coordinates are converted to coordinates in the global coordinate system used for robot navigation, forming a landmark coordinate sequence. ,in For the target point, to This serves as the intermediate guiding point.

[0062] Step 5: Local Path Planning and Execution (DWA). Starting from the robot's current pose (x, y, θ), and using the next unreachable landmark in the landmark coordinate sequence as the temporary target point, the Dynamic Window Method (DWA) is run for local path planning. The DWA algorithm samples multiple combinations of linear velocity v and angular velocity ω in the velocity space to simulate the robot's trajectory over a future time interval Δt. Then, each simulated trajectory is scored using an evaluation function, and the velocity combination corresponding to the trajectory with the highest score is selected as the motion control command for the current moment. Finally, the motion control command is sent to the robot's motion execution module to drive the robot to move towards the current temporary target point.

[0063] Step 6: Landmark Arrival Judgment and Update. Monitor the robot's current position in real time and calculate the Euclidean distance between the robot's current position and the current temporary target landmark. When this distance is less than a preset arrival threshold (usually 0.2m-0.5m), the landmark is considered reached. Then, the next landmark in the landmark coordinate sequence is used as the new temporary target point, and the process returns to Step 5 to continue local path planning. If all landmarks in the landmark sequence have been reached, the navigation task ends.

[0064] Step 7: Dynamic Replanning Triggered. During the execution of the DWA algorithm, the local planning status is monitored in real time. If any of the following conditions occur, the current temporary target point is determined to be unreachable, triggering the replanning mechanism:

[0065] Within a continuously preset time period (usually 3s-10s), the distance between the robot and the current temporary target landmark did not decrease;

[0066] All trajectories sampled by the DWA algorithm are at risk of collision and cannot generate effective motion control commands.

[0067] The sensor detected that the current passage to the temporary target point is completely blocked by an obstacle.

[0068] After triggering replanning, the system first acquires real-time obstacle detection results from LiDAR, visual sensors, etc., via the environmental perception module, extracting the global coordinates, dimensions, and channel blockage status information of the obstacles. Then, this information is projected onto a pre-built global semantic map in the form of corresponding obstacle semantic labels, generating an updated local semantic map image centered on the robot's current position within a 15m radius. Finally, the robot's current pose, descriptions of the remaining landmark sequences, and the updated local semantic map image are re-inputted into the VLM. The VLM combines the global map and real-time environmental perception information to regenerate a new macroscopic landmark sequence from the current position to the target point. Then, it returns to step 4 to parse and transform the coordinates of the new landmark sequence, continuing the navigation task.

[0069] Furthermore, the visual language model in step 3 adopts a lightweight multimodal large model architecture, including three parts: a visual encoder, a modal connector, and a language model backbone. The visual encoder uses models such as SigLIP, CLIP, or ViT to extract visual features from map images; the modal connector uses Q-Former, linear projection layers, or cross-attention layers to convert visual features into feature representations that the language model can understand; and the language model backbone uses lightweight large language models such as Phi-2, Llama-2-7B, or Mistral-7B for spatial reasoning and generating signpost sequences.

[0070] Furthermore, the visual language model is trained using a navigation instruction fine-tuning dataset. The method for constructing the navigation instruction fine-tuning dataset involves randomly sampling a large number of start-point-target point pairs in simulation environments such as Gazebo and CoppeliaSim. Traditional global path planning algorithms (such as A* and JPS) are used to generate reference paths. Then, a large language model (such as GPT-4) is used to convert the reference paths into natural language instructions and corresponding landmark sequence descriptions, forming a map image-text instruction-landmark sequence triplet dataset. Efficient parameter fine-tuning methods such as LoRA (low-rank adaptation) are used to train the VLM, enabling it to generate high-quality landmark sequences from map images and text instructions.

[0071] Furthermore, the macroscopic road sign information output by VLM in step 3 includes two formats: one is a structured text description combined with coordinate points, such as 1. Go straight to the T-junction of the corridor (x1, y1); 2. Turn right into the east corridor (x2, y2); 3. Go straight to the entrance of the general manager's office (x3, y3); the other is to directly output the coordinates of the path points marked on the map image.

[0072] Furthermore, the evaluation function of the DWA algorithm in step 5 is...

[0073] ;

[0074] in: For the road sign orientation guidance item, the angle between the direction of the end of the simulated trajectory and the direction of the current target road sign is measured. The smaller the angle, the higher the score. For obstacle avoidance, the distance between the simulated trajectory and the nearest obstacle is measured; the greater the distance, the higher the score. For the speed category, robots are encouraged to choose higher movement speeds; the faster the speed, the higher the score. For path following guidance, it measures the vertical distance between the end position of the simulated trajectory and the line connecting the current target landmark and the next landmark. The smaller the distance, the higher the score. Let be the weight coefficients of each item, satisfying .

[0075] Furthermore, the replanning mechanism in step 7 incorporates a Chain of Thought (CoT) prompting method. When replanning is triggered, the system inputs structured prompts into the Virtual Learning Model (VLM), guiding it to progressively analyze the current obstacles, find alternative paths, and generate a new sequence of signposts. For example, the format of the prompts is as follows:

[0076] The robot is currently located at coordinates [X, Y] with an orientation angle of [θ]. The original plan was to proceed to [remaining road sign description], but an obstacle has been detected blocking the path ahead. Analyze the current scene image and consider the following questions: What could be the cause of the blockage? Are there any other passable paths nearby? Please provide an alternative path from the current location to the target point, and output the road sign description and corresponding coordinates in a structured format.

[0077] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0078] Example 1

[0079] This embodiment provides a robot path planning method based on the fusion of visual language models and dynamic windowing, applied to a wheeled mobile robot platform in an indoor office environment. The robot is equipped with a 16-line LiDAR, RGB cameras, an IMU (Inertial Measurement Unit), and a wheel encoder, and features an NVIDIA Jetson Xavier NX edge computing platform, capable of meeting real-time computing requirements.

[0080] The method in this embodiment includes the following steps:

[0081] Step S101: Environmental Map Construction and Semantic Labeling. A 2D grid map of the office environment is constructed using laser SLAM technology. Specifically, the robot is controlled to move slowly within the office environment. The laser radar collects point cloud data of the surrounding environment at a frequency of 10Hz, while the IMU and wheel encoder provide pose estimation for the robot. Simultaneous localization and map construction are performed using the GMAPping algorithm, generating a 2D grid map with a resolution of 5cm / pixel.

[0082] Perform manual semantic annotation on the completed raster map. Use annotation tools to mark the following semantic elements on the map:

[0083] Areas: Elevator lobby, main corridor, east corridor, west corridor, meeting room A, meeting room B, general manager's office, finance office, tea room, restrooms;

[0084] Key points: doorways of each room, T-junctions and corners of the corridor, and elevator entrances;

[0085] Static obstacles: walls, columns, desks, sofas, filing cabinets, water dispensers.

[0086] The labeled semantic map is stored in the robot's local storage unit in PNG format, and a mapping relationship between image coordinates and the global coordinate system is established.

[0087] Step S102, VLM Model Preparation and Fine-tuning: This embodiment uses a SigLIP visual encoder (400M parameters) combined with a Phi-2 language model (2.7B parameters) as the basic VLM architecture, connecting the visual and language modalities through a Q-Former. The SigLIP visual encoder is responsible for extracting visual features from the map image, and the Q-Former converts the visual features into 256 learnable query vectors, which are then input into the Phi-2 language model for inference.

[0088] A fine-tuning dataset of navigation instructions was constructed. The aforementioned office environment was reproduced in the Gazebo simulation environment, and 5000 sets of start-point-target point pairs were randomly sampled. The A* algorithm was used to generate a globally optimal path for each start-point-target point pair, and then GPT-4 was used to convert the path into natural language instructions and corresponding landmark sequences. For example, for the path from the elevator to the general manager's office, the GPT-4-generated description was: "Start from the elevator, go straight for 12 meters to the T-junction of the corridor, turn right into the east corridor, go straight for 8 meters past the tea room entrance, and turn left into the general manager's office." This resulted in a fine-tuning dataset containing 5000 triples of map images, text instructions, and landmark sequences.

[0089] The VLM was trained using a LoRA parameter-efficient fine-tuning method. During training, most parameters of the visual encoder and language model backbone were frozen, and only the Q-Former and the LoRA adapter in the language model were trained. The training parameters were set to a batch size of 8, a learning rate of 2e-4, and 10 training epochs. After training, the model weights were exported and deployed to the robot's edge computing platform.

[0090] Step S103: Receive user instructions and obtain map. The user inputs a text instruction via the robot's touchscreen, such as "From the elevator to the general manager's office." After receiving the user instruction, the human-computer interaction module retrieves a 2D navigation map image of the office environment from its local storage unit.

[0091] Step S104: The VLM generates a macroscopic signpost sequence. The user's text instructions, such as "from the elevator entrance to the general manager's office," and a 2D navigation map image are input into the fine-tuned VLM. The VLM first extracts features from the map image using a SigLIP visual encoder, then converts the visual features into query vectors using Q-Former, which are then input into the Phi-2 language model. The Phi-2 language model combines the text instructions and visual features for spatial reasoning, outputting the following structured signpost information: "Go straight for 12 meters from the starting point (elevator entrance) to the T-junction between the main corridor and the east corridor."

[0092] Turn right into the east corridor; walk straight along the east corridor for 8 meters, passing the entrance to the tea room; turn left into the general manager's office.

[0093] At the same time, VLM outputs the pixel coordinates of the above four key points in the map image: (200, 300), (400, 300), (400, 500), (350, 550).

[0094] Step S105: Landmark Sequence Parsing and Coordinate Transformation. The landmark parsing and maintenance module parses the landmark information output by the VLM, extracting the text description and pixel coordinates of each landmark. Then, based on the pre-established mapping relationship between image coordinates and the global coordinate system (1 pixel... (The map origin is located at the top left corner). The pixel coordinates are converted to coordinates in the global coordinate system. Specifically, in this example:

[0095] .

[0096] Forming a sequence of landmark coordinates ,in The target point.

[0097] Step S106: DWA Local Path Planning and Execution. Starting from the current robot pose (0.0m, 0.0m, 0.0rad), and using the first landmark... (10.0m, 15.0m) is a temporary target point. The DWA algorithm is used for local path planning.

[0098] The parameters of the DWA algorithm are set as follows: linear velocity sampling range v ∈ [0, 0.5] m / s; angular velocity sampling range ω ∈ [-1.0, 1.0] rad / s; velocity resolution... , Simulation time Evaluation function weights , , .

[0099] The formula for calculating the evaluation function is:

[0100] ;

[0101] in:

[0102] , The direction angle of the current target road sign. This is the orientation angle at the end of the simulated trajectory;

[0103] , To simulate the distance between each point on the trajectory and the nearest obstacle;

[0104] , This represents the maximum linear velocity.

[0105] The DWA algorithm samples all possible velocity combinations in the velocity space, simulates each trajectory, and then calculates a score for each trajectory based on an evaluation function. The velocity combination corresponding to the trajectory with the highest score is selected. , As the motion control command for the current moment, it is sent to the motion execution module to drive the robot to move towards... Move the point.

[0106] Step S107: Landmark arrival judgment and update. Real-time monitoring of the robot's current position, calculating the robot's current position and... The Euclidean distance between the points. When the distance is less than 0.3m, the determination is made... Point reached. Next, move on to the next landmark. (20.0m, 15.0m) is used as a new temporary target point. Return to step S106 to continue executing DWA local path planning.

[0107] Repeat the above process to reach the destinations in sequence. Point and Point. When arrived After clicking, The point serves as a temporary target point, driving the robot to move towards the entrance of the general manager's office.

[0108] Step S108, navigation terminates when the robot arrives. When all landmarks are reached within 0.3m, the robot is considered to have been reached, and the navigation task is complete. The robot stops moving and notifies the user via voice and screen that navigation is complete.

[0109] Step S109: Dynamic replanning. During navigation, if the robot detects that the path leading to the current temporary target point is blocked by an obstacle (for example, someone has placed a box in the corridor), the replanning mechanism is triggered.

[0110] Assuming the robot is heading to During the process, a large box was detected completely blocking the T-junction between the main corridor and the east corridor. At this time, the replanning trigger module first detected a static obstacle measuring 1.2m × 0.8m at the T-junction using a 16-line LiDAR, with global coordinates ranging from (20.0m, 14.5m) to (20.0m, 15.5m), completely blocking the passage to the east corridor. Then, the obstacle was projected onto the global semantic map with the semantic label "completely blocked," generating an updated local semantic map image centered at the robot's current position (15.0m, 15.0m) within a radius of 15m. Finally, the robot's current pose (15.0m, 15.0m, 0.0rad), the remaining waypoint description "go to the east corridor, then to the general manager's office," and the updated local semantic map image were input into the VLM.

[0111] The system inputs the following thought chain prompts into the VLM: The robot is currently located at coordinates (15.0, 15.0), with an orientation angle of 0.0 rad. The original plan was to proceed to the east corridor and then to the general manager's office, but an obstacle has been detected blocking the T-junction ahead. Please analyze the current scene image and consider the following questions: What could be the cause of the blockage? Are there any other accessible passageways nearby? Please provide an alternative path from the current location to the general manager's office, and output the waypoint descriptions and corresponding coordinates in a structured format.

[0112] After VLM analysis, a new signpost sequence is output: Go straight to the end of the main corridor; turn right into the west corridor; go straight along the west corridor to the end; turn right into the south corridor; go straight along the south corridor to the back door of the general manager's office; enter the general manager's office.

[0113] Simultaneously, the corresponding global coordinate sequence is output. The landmark parsing and maintenance module updates the landmark sequence, and the robot continues navigation according to the new landmark sequence, eventually reaching the target point.

[0114] Example 2

[0115] Based on Example 1, this embodiment improves the evaluation function of the DWA algorithm by adding a path following guidance term, which further enhances the robot's ability to follow macroscopic paths.

[0116] The improved evaluation function is:

[0117] G′ ;

[0118] in, The formula for calculating the path-following guide item is as follows: - / ;

[0119] In the formula, To simulate the perpendicular distance from the end point of the trajectory to the line connecting the current target landmark and the next landmark, This is the preset maximum allowable deviation distance (usually 1.0m-2.0m). When hour, .

[0120] The weighting coefficients are set as follows: , , , , satisfying α+β+γ+δ=1.

[0121] By introducing path-following guidance, the robot focuses not only on the direction of the current target landmark during local planning but also on following the global path. This allows the robot to better stay on the macroscopic path, reducing the risk of deviating from the path, especially in open areas and long corridors, significantly improving navigation stability and efficiency.

[0122] Example 3

[0123] This embodiment expands upon Embodiment 1 by extending the input of the VLM model to support user input of natural language commands that include macroscopic walking preferences.

[0124] For example, a user inputs a command to go from the elevator to the general manager's office, prioritizing routes along the walls and avoiding densely populated areas. VLM takes these user preferences into account when generating waypoint sequences. Specifically, VLM identifies wall-side passages and potentially crowded areas on the map (such as the entrance to the break room or meeting room), and then plans a route that stays as close to the walls as possible while avoiding crowded areas.

[0125] To support this functionality, when building the fine-tuning dataset, instruction samples containing various walking preferences were added, such as walking along routes near windows, avoiding elevator entrances, and trying to walk through wide passageways. By fine-tuning on these samples, VLM can understand and execute various macroscopic walking preferences of the user.

[0126] Test case

[0127] like Figures 3-4 As shown, to verify the effectiveness of the method of this invention, the performance of the traditional DWA algorithm, the A*+DWA fusion algorithm, and the method of this invention were tested in a test environment containing complex scenarios such as U-shaped obstacles, long corridors, and dynamic pedestrians. The test environment was 50m×50m in size and contained multiple rooms, corridors, and obstacles. Each algorithm was tested 10 times independently, and the navigation success rate, average time, and number of times trapped in local optima were statistically analyzed. The experimental results are shown in the table below:

[0128] The experimental results show that:

[0129] The method of this invention has the highest navigation success rate, reaching 94.5%, which is 27.2 percentage points higher than the traditional DWA algorithm and 10.3 percentage points higher than the A*+DWA fusion algorithm.

[0130] The method of this invention has the shortest average processing time, only 36 seconds, which is 37.9% shorter than the traditional DWA algorithm and 14.3% shorter than the A*+DWA fusion algorithm;

[0131] The method of this invention gets trapped in local optima the fewest times, only once per 10 tests, which is 91.7% less than the traditional DWA algorithm and 75% less than the A*+DWA fusion algorithm.

[0132] The method of this invention is an algorithm that supports natural language commands, resulting in a better human-computer interaction experience.

[0133] Furthermore, in dynamic environment testing, when multiple pedestrians are present, the navigation success rate of the method described in this invention remains above 90%, while the navigation success rate of the A*+DWA fusion algorithm drops to around 75%. This indicates that the method described in this invention has stronger adaptability to dynamic environments.

[0134] The present invention also provides a robot path planning system, comprising:

[0135] The map building module is used to build a 2D navigation map of the environment using laser SLAM or visual SLAM technology, and to semantically annotate the map and store the annotated semantic map.

[0136] The human-computer interaction module includes a text input unit and a voice input unit, which are used to receive the starting point and target point descriptions input by the user in text or voice form, and convert voice commands into text commands;

[0137] The VLM road sign generation module deploys a pre-trained and fine-tuned visual language model, which takes 2D navigation map images and user text commands as input and outputs macroscopic road sign information.

[0138] The road sign parsing and maintenance module is used to parse the macroscopic road sign information output by VLM into a road sign coordinate sequence in the global coordinate system, maintain the current target road sign status, and update the road sign sequence during replanning;

[0139] The DWA local planning module takes the current robot pose and the current target landmark as input and uses the dynamic window method to generate real-time motion control commands.

[0140] The replanning trigger module monitors the local planning status and environmental perception information in real time. When it detects that the current target landmark is unreachable, it triggers the VLM to regenerate the landmark sequence.

[0141] The environmental perception module, including lidar, vision sensors, etc., is used to perceive the environmental information around the robot in real time and provide data support for local obstacle avoidance and replanning.

[0142] The motion execution module receives motion control commands generated by the DWA local planning module and drives the robot's motors and actuators.

[0143] Furthermore, the visual language model in the VLM road sign generation module adopts a lightweight architecture and is trained on the navigation instruction dataset using efficient parameter fine-tuning methods such as LoRA, enabling it to run in real time on edge computing devices.

[0144] Furthermore, the conditions under which the replanning trigger module determines that the current target landmark is unreachable include: the robot cannot approach the target landmark within a continuous preset time, all sampled trajectories have a collision risk, and the sensor detects that the channel is completely blocked.

[0145] This application also provides an embodiment of an electronic device. The electronic device is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors or processing units, memory, and buses connecting different components (including memory and processing units).

[0146] A bus refers to one or more of several bus architectures, including memory buses or memory controllers, peripheral buses, graphics acceleration ports, processors, or local buses using any of the various bus architectures. Examples of these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0147] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0148] The memory may include computer-readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic devices may further include other removable / non-removable, volatile / non-volatile computer device storage media. By way of example only, the storage system may be used to read and write non-removable, non-volatile magnetic media.

[0149] The electronic device can also communicate with one or more external devices (e.g., keyboard, pointing device, camera, etc.), may include a display, and may communicate with one or more devices that enable a user to interact with the electronic device, and / or with any device that enables the electronic device to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via an input / output (I / O) interface. Furthermore, the electronic device can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. The processor executes various functional applications and data processing by running programs stored in memory, such as implementing a robot path planning method provided in the above embodiments of the present invention.

[0150] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot path planning method, characterized by, Includes the following steps: Retrieve the starting point text description, the target point text description, and the corresponding 2D navigation map image; The starting point text, target point text, and 2D navigation map image are input into a visual language model to generate macroscopic road sign information, which includes one or more intermediate road sign points. The macroscopic road sign information is parsed into a road sign coordinate sequence in the global coordinate system; Starting from the current robot pose, and taking the next unreached landmark in the landmark coordinate sequence as the temporary target point, the dynamic window method is used to generate local motion control commands. When the robot reaches the current temporary target point, it updates the temporary target point to the next unreached landmark, until all landmarks are reached; When the current temporary target point is detected to be unreachable, a replanning mechanism is triggered, and information is re-inputted into the visual language model to generate new macroscopic road sign information; The visual language model adopts a multimodal large model architecture, including a visual encoder, modal connectors and a language model backbone, and is fine-tuned by a navigation instruction dataset to enable it to identify key waypoints from map images and generate road sign sequences.

2. The method of claim 1, wherein, The macro-road sign information includes structured text descriptions and their corresponding global coordinates. The structured text descriptions contain semantic information, including area names, landmark objects, or turn instructions.

3. The method of claim 1, wherein, The evaluation function of the dynamic window method includes a landmark orientation guidance term, which encourages the robot to choose a motion trajectory toward the current target landmark.

4. The method of claim 1, wherein, The replanning mechanism includes: acquiring obstacle detection results output by sensors in real time; projecting the obstacle detection results onto a global semantic map in the form of obstacle semantic labels to generate an updated local semantic map image; inputting the current robot pose, remaining landmark descriptions, and the updated local semantic map image into a visual language model; and generating an alternative landmark sequence from the current position to the target point based on the thought chain prompting method.

5. The method of claim 1, wherein, The visual encoder uses the SigLIP, CLIP, or ViT model; the modal connector uses the Q-Former, linear projection layer, or cross-attention layer; and the language model backbone uses the Phi-2, Llama-2-7B, or Mistral-7B model.

6. The method of claim 1, wherein, The navigation instruction dataset is constructed as follows: starting point-target point pairs are randomly sampled in a simulation environment, a reference path is generated using a global path planning algorithm, and the reference path is converted into natural language instructions and road sign sequence descriptions using a large language model, forming a triplet of map image, text instructions, and road sign sequence.

7. The method of claim 3, wherein, The evaluation function of the dynamic window method also includes a path following guidance term, which measures the vertical distance between the end position of the simulated trajectory and the line connecting the current target landmark and the next landmark, encouraging the robot to follow the macroscopic path.

8. The method of claim 4, wherein, The thought chain prompting method includes guiding the visual language model to analyze the reasons for the obstruction, finding alternative channels, and generating a structured sequence of road signs.

9. The method of claim 1, wherein, Conditions under which a temporary target point is unreachable include: the robot being unable to approach the target landmark within a preset time period, all sampling trajectories posing a collision risk, and the sensor detecting that the channel is completely blocked.

10. A system utilizing the robot path planning method of claim 1, wherein, include: The map building module is used to build and store 2D navigation maps of the environment; The human-computer interaction module is used to receive the starting point and target point descriptions input by the user in text form; The VLM road sign generation module deploys a pre-trained visual language model, which takes map images and text commands as input and outputs macroscopic road sign information. The landmark parsing and maintenance module is used to parse the VLM output into a global coordinate sequence and maintain the current target landmark status; The DWA local planning module generates motion control commands in real time, guided by the current target landmark. The replanning trigger module monitors the local planning status. When it detects that the current target landmark is unreachable, it generates an updated local semantic map image and triggers the VLM to regenerate the landmark. The motion execution module receives control commands and drives the robot to move.

Images