Task planning method for two-wheel differential mobile robot based on three-dimensional semantic map

Abstract

The invention relates to the technical field of mobile robots, in particular to a two-wheel differential mobile robot task planning method based on a three-dimensional semantic map, and the method comprises the steps: firstly building a motion model based on a two-wheel differential mobile robot, and then building a three-dimensional semantic map suitable for the mobile robot; and then discussing a mechanism for converting a natural language into a machine language based on a semantic reasoning mechanism, specifically establishing a scene understanding framework based on ROS, integrating with a large language model LLM, further researching a method for optimizing robot path planning according to semantic information, and formulating the method from various use scenes such as path planning, obstacle stopping and avoiding, and narrow area passing. And realizing path planning and target navigation. Experimental results show that semantic information is deeply fused into the task planning process, the intelligence and execution efficiency of task planning are enhanced, and meanwhile, the navigation capacity of the two-wheeled moving differential robot in a complex environment is improved by combining marching area information in a three-dimensional semantic map.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robots, and particularly to a task planning method for a two-wheel differential mobile robot based on a three-dimensional semantic map. Background Art

[0002] Two-wheel differential mobile robots are widely used in fields such as warehousing logistics and service robots due to their advantages of simple structure, strong mobility, and low cost. As the core technology of robot autonomous navigation, Simultaneous Localization and Mapping (SLAM) mainly obtains three-dimensional point cloud data of the environment through sensors such as lidar and depth cameras, and then constructs a geometric map reflecting the spatial structure of the scene, providing support for the mobile navigation and task planning of the robot in the scene.

[0003] However, in unstructured dynamic environments such as home and office scenarios, it is difficult to achieve precise navigation and intelligent decision-making of two-wheel differential mobile robots relying solely on traditional geometric maps. The reasons are twofold: First, the kinematic model of two-wheel differential robots has nonholonomic constraints, and the turning ability is limited in narrow spaces, and traditional path planning methods are prone to cause motion jitter or collisions; Second, the positions and shapes of obstacles in dynamic environments change continuously, and the lack of semantic information in traditional static geometric maps results in the robot's inability to recognize key objects in the scene (such as tables, chairs, doors, and windows), restricting its autonomy in complex tasks.

[0004] In recent years, the development of semantic SLAM technology that obtains environmental semantic information while constructing a geometric map has provided a new solution to the above problems. However, when this technology is applied to two-wheel differential robots, it still faces many challenges: For example, the vibration generated during the movement of two-wheel differential robots will affect the stability of sensor data, resulting in a decrease in the registration accuracy of semantic information and geometric maps; The real-time processing of semantic information and map updating require high computing resources, while the computing platform usually carried by two-wheel differential robots has limited performance. Summary of the Invention

[0005] The purpose of the present invention is to provide a task planning method for a two-wheel differential mobile robot based on a three-dimensional semantic map, aiming to optimize path planning under the motion constraints of two-wheel differential robots, and deeply combine task planning with semantic information to solve the technical problems of low task planning efficiency and insufficient execution ability caused by the lack of semantic information in traditional static geometric maps.

[0006] To achieve the above purpose, the present invention provides a task planning method for a two-wheel differential mobile robot based on a three-dimensional semantic map, including the following steps:

[0007] Step 1: Construct a motion model of a two-wheel differential mobile robot;

[0008] Step 2: Construct a 3D semantic map, perform map preprocessing and scene understanding;

[0009] Step 3: Robot path planning based on the 3D semantic map.

[0010] Optionally, Step 1.1: Coordinate system model analysis to complete the coordinate transformation between the local coordinate system and the global coordinate system of the differential drive mobile robot;

[0011] Step 1.2: Simplify the pose of the differential drive mobile robot into 2 degrees of freedom information and construct the corresponding motion model.

[0012] Optionally, the execution process of Step 2 includes the following steps:

[0013] Step 2.1: Perform preprocessing on the 3D semantic map and the 2D grid map;

[0014] Step 2.2: Construct a scene understanding framework based on the ROS robot operating system;

[0015] Step 2.3: Open vocabulary semantic mapping and robot perception;

[0016] Step 2.4: Task execution based on 3D semantic objects.

[0017] Optionally, in Step 2, the preprocessed 3D semantic map and 2D grid map are stored as an octree map, and the RViz tool is used to visualize the map, and the communication between the coordinates of the scene objects and the robot is realized through the ROS robot operating system.

[0018] Optionally, after the grid map and multi-view semantic information are fused in Step 2, each object in the map is embedded with a semantic vector, and the semantic vector in the map is mapped into coordinate information that can be read by the robot, and the category and spatial information of each object are saved as an executable file of the robot as the semantic knowledge base of the robot.

[0019] Optionally, during the process of task execution based on 3D semantic objects in Step 2.4, the scene understanding framework based on the ROS robot operating system is integrated with the large language model LLM to realize task planning through natural language instructions.

[0020] Optionally, the execution process of Step 3 includes the following steps:

[0021] Step 3.1: Formulate a path planning method;

[0022] Step 3.2: Formulate a method for stopping and avoiding obstacles;

[0023] Step 3.3: Formulate a method for passing through narrow areas.

[0024] Optionally, in step 3.1, the A* algorithm is used as the global path planner, and the TEB algorithm and the DWA algorithm are used as local path planners to complete the navigation task with the help of the move_base framework.

[0025] Optionally, step 3.2 realizes obstacle avoidance based on the dynamic window and realizes obstacle avoidance based on the move base framework. When other paths cannot be planned in a narrow area, the parking strategy of the robot is determined by the relationship between the speed limit and the distance from the obstacle. The sliding window method is used to sample the speed of the robot at each moment, and combined with the obstacle distance information provided by the lidar, a set of obstacle distance evaluations are generated; the obstacle avoidance process is based on the move base framework, using LiDAR point cloud data and inertial measurement unit IMU data, combined with the constructed grid map, through the global path planning and local obstacle avoidance strategy.

[0026] Optionally, in step 3.3, by calculating the center point of the narrow area and generating optimized auxiliary path points around it, the robot is guided to pass through the complex and narrow environment with a smooth and safe trajectory.

[0027] The present invention provides a task planning method for a two-wheeled differential mobile robot based on a three-dimensional semantic map. First, a motion model is established based on the two-wheeled differential mobile robot, then a three-dimensional semantic map suitable for the mobile robot is constructed, and then the mechanism of converting natural language into machine language is explored based on the semantic reasoning mechanism. Specifically, a scene understanding framework based on ROS is established and integrated with the large language model LLM. Further research on the method of optimizing the robot path planning according to semantic information, and formulating methods from multiple usage scenarios such as path planning, obstacle avoidance and narrow area passage to achieve path planning and target navigation. The experimental results show that the present invention deeply integrates semantic information into the task planning process, enhances the intelligence and execution efficiency of the task planning, and at the same time improves the navigation ability of the two-wheeled differential mobile robot in a complex environment by combining the passable area information in the three-dimensional semantic map. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0029] Figure 1 It is a specific flowchart of a task planning method for a two-wheeled differential mobile robot based on a three-dimensional semantic map of the present invention.

[0030] Figure 2It is a schematic diagram of the coordinate system model of the two-wheel differential robot of the present invention.

[0031] Figure 3 It is a schematic diagram of the motion model of the two-wheel differential mobile robot of the present invention.

[0032] Figure 4 It is a schematic diagram of the scene understanding framework based on the ROS robot operating system of the present invention.

[0033] Figure 5 It is a schematic diagram of the entity search process of the entity semantic knowledge base for semantic navigation of the present invention.

[0034] Figure 6 It is a schematic diagram of the process of searching for keywords of the target point and indicating the movement of the robot in the specific embodiment of the present invention.

[0035] Figure 7 It is a schematic diagram of the process of formulating the path planning method in the present invention.

[0036] Figure 8 It is a schematic diagram of the process of formulating the obstacle stopping method in the present invention.

[0037] Figure 9 It is a schematic diagram of the process of formulating the obstacle avoidance method in the present invention.

[0038] Figure 10 It is a schematic diagram of the process of the narrow area passing method in the present invention.

[0039] Figure 11 It is a schematic diagram of the obstacle stopping strategy of the mobile robot in a narrow area in the specific embodiment of the present invention.

[0040] Figure 12 It is a schematic diagram of the obstacle avoidance strategy of the mobile robot in the specific embodiment of the present invention.

[0041] Figure 13 It is a schematic diagram of the test result of the repeated point accuracy of the robot in the specific embodiment of the present invention. Detailed implementation manners

[0042] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0043] The present invention provides a task planning method for a two-wheel differential mobile robot based on a three-dimensional semantic map, including the following steps:

[0044] Step 1: Construct a motion model of the two-wheel differential mobile robot;

[0045] Step 2: Construct a three-dimensional semantic map, perform map preprocessing and scene understanding;

[0046] Step 3: Robot path planning based on the three-dimensional semantic map.

[0047] The specific method execution flow chart is as Figure 1 shown, and the following is a further description in combination with the execution steps:

[0048] Step 1: Kinematic model of a two-wheeled differential mobile robot.

[0049] Step 1.1: Analysis of the coordinate system model of the two-wheeled differential robot. As Figure 2 shown, the navigation algorithm implementation of the mobile robot needs to complete the real-time control of the mobile robot to ensure that the mobile robot can reach the target point in a timely and accurate manner. In this process, the mobile robot completes the transformation of its pose, and the coordinate system in which its pose is located also changes.

[0050] In the process of positioning and path planning of the mobile robot, the coordinates in the global coordinate system represent the absolute position of the mobile robot in the experimental scene, and the global coordinate system does not change with the change of the position of the mobile robot. The global coordinate system is determined by the global map, which ensures the stability of the global map.

[0051] The advantage of the local coordinate system is that it can ensure strong real-time performance and can monitor the environmental information around the mobile robot in real time. However, if it is desired to implement the positioning and path planning of the mobile robot through the pose in the local coordinate system, the pose in this coordinate system needs to be transformed into the pose in the global coordinate system through appropriate transformation.

[0052] The coordinate system of the mobile robot body takes the center of the wheels of the mobile robot as the origin, and the coordinate system is established according to the right-hand rule. The forward direction of the mobile robot can be determined as the positive x-axis direction of the coordinate system of the mobile robot body. In the present invention, the coordinate system of the mobile robot body takes the steering center of the driving wheel as the origin and the forward direction as the positive x-axis direction to establish a real-time coordinate system, which changes with the change of the position of the mobile robot. Since the mobile robot moves on a plane, and the two-wheeled differential mobile robot does not involve the elevation information of the z-axis, as well as the changes in the roll angle and pitch, the pose of the mobile robot at time t relative to the global map coordinate system is:

[0053]

[0054] Convert the pose point p in the coordinate system of the mobile robot body to the map coordinate system, and the conversion relationship formula is as follows:

[0055]

[0056] Among them, C represents the coordinate system of the mobile robot body, and M represents the global map coordinate system. is the transformation matrix of the coordinate system of the mobile robot body relative to the global coordinate system, and the transformation matrix can be expressed as:

[0057]

[0058] Then, after determining information such as the maximum steering angle and minimum turning radius with non-holonomic constraint characteristics, coordinate transformation between different coordinate systems can be achieved through a certain conversion relationship.

[0059] Step 1.2: Analysis of the kinematic model of the two-wheel differential robot. Usually, the pose of a robot is composed of information with 6 degrees of freedom: {x, y, z, roll, pitch, yaw}. However, in the actual test scenario of the present invention, since the translation amount y, elevation information z, rotation amounts roll and pitch of the two-wheel differential mobile robot on the plane do not change, the pose of the mobile differential robot is simplified to be composed of information with 2 degrees of freedom: {x, yaw}, where the rotation amount yaw is perpendicular to the normal vector of the ground plane. The differential wheeled outdoor mobile robot used in the present invention has four wheels, among which the two front wheels are driving wheels for providing power to the mobile robot, and the two rear wheels are driven wheels.

[0060] When using the two-wheel differential kinematic model for the trajectory deduction of the differential wheeled outdoor mobile robot, mainly relying on two parameters, the left wheel speed and the right wheel speed, the actual forward speed V of the differential wheeled outdoor mobile robot can be obtained through calculation x and the angular velocity W x .

[0061] As Figure 3 shown is the motion model of the two-wheel differential mobile robot. Let the linear velocities of the left and right wheels be V L , V R , which can be obtained from the wheel encoder:

[0062]

[0063] Among them, M is the count value of the encoder within the sampling period; P is the number of lines of the encoder, that is, the total number of pulses triggered when the encoding disk rotates one circle; N is the reduction ratio of the motor, r is the radius of the wheel, and Δt is the sampling period of the encoder.

[0064] Suppose that within time t, the left and right wheels have traveled S L , S R , and advanced by an angle φ. The radius of the left wheel's travel is ρ, and the distance between the left and right wheels is d. Then there are:

[0065]

[0066] The value of the traveling angle φ can be obtained as follows:

[0067]

[0068] Center point displacement:

[0069]

[0070] When t is very small, the linear velocity V of the center point x and the angular velocity W x can be expressed as:

[0071]

[0072] Represent the two in matrix form:

[0073]

[0074] So far, the kinematic model of the two-wheel differential mobile robot has been constructed. By the wheel speeds sent by the encoders of the two-wheel differential mobile robot and accurately measuring the wheelbase d between the left driving wheel and the right driving wheel of the differential robot, the linear velocity and angular velocity of the center point can be obtained.

[0075] Step 2: Robot scene understanding based on the 3D semantic map.

[0076] Step 2.1: Preprocessing of the 3D semantic map and the 2D grid map. For the semantic map part, this patent uses the server side as the main tool for constructing the semantic map. The robot side only needs to read the map. In a limited computing platform, obtain the semantic information of the object in the form of reading the object coordinates. Effectively utilize the computing resources to provide high-quality map data support for subsequent robot scene understanding, path planning, and task execution. First, import the 3D semantic map (including geometric information and semantic labels) and the 2D grid map (usually an occupancy grid map) into the ROS (Robot Operating System) environment, and use the RViz tool to visualize the map. Through the visualization interface of RViz, the object categories and semantic labels in the 3D semantic map and the obstacle information in the 2D grid map are displayed in real time to ensure the integrity and accuracy of the map data.

[0077] For the data synchronization and update part, perform data format conversion on the 3D semantic map and the 2D grid map to ensure their compatibility in the ROS environment. Align the 3D semantic map and the 2D grid map through coordinate transformation (such as TF transformation) to ensure their spatial consistency, providing a basis for subsequent multi-map fusion and path planning. At the same time, filter and optimize the semantic information in the 3D semantic map, removing redundant or incorrect semantic labels to ensure the accuracy and reliability of the semantic information; preprocess the 2D grid map, including adjusting the map resolution, removing noise, and supplementing obstacle information to ensure the clarity and usability of the map.

[0078] Store the preprocessed 3D semantic map and 2D grid map as octree maps for subsequent rapid loading and use. Load the stored map data in RViz to verify the integrity and usability of the map, ensuring the smooth progress of subsequent task planning and execution. Through the above steps, complete the preprocessing of the 3D semantic map and the 2D grid map, providing a high-precision and multi-modal environmental representation for the robot and laying a foundation for subsequent scene understanding, path planning, and task execution.

[0079] Step 2.2: ROS-based scene understanding framework. The ROS-based scene understanding framework as a whole consists of multiple modules. The core is to achieve communication between the coordinates of scene objects and the robot through the ROS robot operating system. Through the subscription and publishing communication mechanism of ROS, achieve the task execution planning of the robot, and display the semantic map, octree map, and grid map through Rviz to complete the semantic understanding and dynamic interaction of the scene.

[0080] Such as Figure 4As shown, the ROS-based scene understanding framework consists of user input, topic name, node name, and output name. The user inputs natural language instructions on the server side. The system will extract the corresponding keywords through natural language processing algorithms and locate the coordinates of the target object in the preset 3D environment based on these keywords. These coordinates are managed and tracked by ROS Master, which is responsible for coordinating the communication between various nodes to ensure the correct subscription and publishing mechanisms. When the keyword parsing is completed and the target object is determined, the system publishes the coordinate points of the target object to the mobile robot side in the form of a ROS topic. The mobile robot performs map relocalization by subscribing to the LiDAR point cloud data node. The LiDAR point cloud data is transmitted in real time between the LiDAR and the robot-side node to ensure that the robot can accurately perceive and understand the environment. The robot camera data stream is also published in the form of a node to provide visual feedback for judging whether the robot has approached the target object. In addition, the speed topic of the mobile robot is closely related to path planning. The control algorithm determines the path that the robot should travel through path planning and realizes path tracking by adjusting the speeds of the left and right drive wheels of the robot. The output of the speed topic provides precise motion instructions for the robot to ensure that the robot can move smoothly and efficiently to the specified position. The design of this framework enables the robot to perform efficient and accurate positioning, navigation, and task execution in a complex environment. All topics and visualizations are integrated and displayed in Rviz, including semantic maps, grid maps containing only geometric information, and octree maps that fuse grid maps and multi-view semantic information.

[0081] Throughout the framework, the ROS system is responsible for processing node operations and topic communication of various types of data. Through these different types of map data, including semantic maps, LiDAR point cloud maps, and grid maps, etc., the robot can achieve multi-dimensional scene perception and understanding based on multiple sensor information. These processing results can not only be displayed in real time but also be used for subsequent path planning, target recognition, and dynamic scene interaction. By obtaining and processing data from different sensors in real time, the robot can continuously adjust its actions, which are displayed in the form of the / tf topic, to cope with environmental changes and task requirements. At the same time, the efficient cooperation mechanism of the framework ensures that each module can cooperate smoothly. Based on the multi-sensor information fusion, a dynamic and highly adaptable scene understanding and decision-making system is built, enabling the robot to perform tasks and interact efficiently with the environment in a complex and dynamic environment.

[0082] Step 2.3: Open Vocabulary Semantic Mapping and Robot Perception. After the grid map and multi-view semantic information constructed above are fused, each object in the map is embedded with a semantic vector. Mapping the semantic vector in the map to the coordinate information that can be read by the robot requires conversion into a corresponding semantic knowledge base. Based on the knowledge base constructed above, open vocabulary semantic mapping and robot perception need to read the coordinates of the semantic knowledge base.

[0083] Keyword retrieval can be queried according to the scene category, object category, and object ownership relationship. According to the keyword retrieval, the semantic knowledge to be queried is obtained, the line l where the keyword is located is read, and then according to the character segmentation during knowledge storage, the semantic information related to the robot's semantic navigation is parsed. In the present invention, the specific process of retrieving the semantic knowledge base through keywords is as follows: receiving the message published by the text keyword, this message contains two sets of data, the noun word n = f oj f oj is the semantic label constructed above. These nouns contain the categories of objects. When word n queries the corresponding semantic label, the corresponding object in O T (object set) is read, and the coordinate value of the object in the semantic knowledge base is returned. When traversing the semantic knowledge base and not finding the content in the knowledge base that matches word n , the result is returned, and the queried noun is empty.

[0084] The entity semantic knowledge base for semantic navigation provides the function of quickly querying and managing relevant semantic information for the robot during semantic navigation. When performing an entity search task, after the semantic navigation inference mechanism obtains a natural language command, it reads the content in the knowledge base and analyzes it, extracts the relevant information of the entity in the environment, and infers the possible scenarios that may occur when the object moves, so as to complete the entity dynamic search task, such as Figure 5 shown.

[0085] Step 2.4: Task Execution Based on 3D Semantic Objects. The task planning module in the framework can convert the user's instructions into tasks that the robot can execute. Usually, the user inputs instructions through natural language (such as "move to the chair"), and the task planning module parses the instructions and converts them into actions that the robot can understand. In order to enable the robot to recognize the target object in the environment, the framework converts the scene information into structured 3D graph data and performs reasoning in combination with the semantic information of the object. The robot calculates the optimal path from the current position to the target object through the path planning module, taking into account the obstacles and semantic restrictions in the environment to ensure the smooth progress of task execution.

[0086] To achieve task planning through natural language, the framework is integrated with a large language model (LLM). The LLM can parse the user's natural language query and map the instructions to the robot's environmental map. By processing the text query, the LLM identifies the position and semantic information of the target object and matches it with the robot's current scene graph. In this way, the user does not need to know the specific location of the object or environmental details, but only needs to provide natural language instructions, and the robot can automatically identify the target object and execute the task.

[0087] As Figure 6 shown, for example, when the input is "Move to the door", the robot should first extract the keywords "Move" and "door" from the input. These two keywords correspond to the movement instruction and the target point respectively. Then, it searches for "door" in the 3D semantic object library and generates a target point with the center coordinates of the 3D Bounding Box at the chair's position as the movement end point. This target point is sent to the robot side through ROS. Finally, the computer sends a movement instruction to the robot, and the robot moves to the vicinity of the target position.

[0088] Use the robot semantic knowledge base containing scene and object semantic information to achieve target scene reasoning. In this case, first, the information of the target and the scene needs to be extracted from it, and then the navigation task is planned according to the scene information. The process of robot semantic navigation entity search is shown below. Taking the search for a chair as an example, in this algorithm, first, the ROS system listens for messages from the / speech command topic. These text commands are issued by the user and processed through the speech recognition callback function. The text recognition callback function extracts the keywords in the command by receiving the message msg. If the recognized text command contains an instruction like "Move to the chair", the system will enter the next step of processing.

[0089] Next, the system searches for the target object by extracting the keywords in the text command. The process of extracting keywords will identify the specific action instruction from the text command. After executing extractkeywords, the keywords will be sent to the query process of the 3D object library, and the system will search for 3D objects associated with these keywords in the library. If the keyword is "chair", the system will identify the target object and further obtain the 3D coordinate information of the object.

[0090] When the target is "chair", the system calls the category information and spatial information of "chair" in the semantic knowledge base. In this example, the coordinates of the chair are assumed to be [1.0, 2.0, 0.0], and the category is "chair". The system then publishes this information to the robot side via ROS. After the robot reads the corresponding coordinate information and category information, it moves to the target position. After receiving the target coordinates, the robot enters the path planning and movement stage. The robot performs path planning based on the received target pose and calculates the best path to the target position. Subsequently, the robot generates a path and executes the corresponding motion instructions. Finally, the move base module controls the robot to move to the target position along the planned path to complete the task.

[0091] Step 3: Robot path planning based on the 3D semantic map.

[0092] Step 3.1: Path planning method. As Figure 7As shown in the figure, the A* algorithm is integrated into the robot ROS system as the global path planner, and two local algorithms are used as local path planners respectively. The move_base system function package is utilized to provide a communication mechanism to connect the global and local planning tasks to complete the navigation task. A fusion scheme for a two-wheel differential mobile robot and the requirements of a complex indoor working environment is selected to improve the robustness of the mobile robot autonomous navigation system in a real scenario. The move_base adopted by the fusion algorithm is a framework specifically applied to navigation. It controls the movement of the unmanned vehicle chassis to a given target point position and continuously obtains and compares the information of its own pose and the target point state during the movement. It provides a variety of data interfaces externally, such as: sensor data interface (sensor source), map data interface (map_server), global planner interface (global_planner), local planner interface (local_planner), cost map interface (costmap), etc. In the move_base navigation framework designed by the present invention, first, the coordinates laser_link of the 3D lidar sensor in the mapping part and the coordinates base_link of the unmanned vehicle in the navigation part are converted through the tf tree data structure to obtain the relative position relationship between the unmanned vehicle and the sensor in the map. The position information of the target point is given in advance, and the A* algorithm is transplanted into the global planner, and the local TEB algorithm and the local DWA algorithm are transplanted into the local planner respectively. The global planner plans a global path without obstacles based on the global cost map global_costmap generated from the received global map and sensor data; the local planner optimizes the local path in real time according to the local cost map local_costmap obtained from the sensor data. Finally, the move_base module outputs control instructions to the underlying control drive module through cmd_vel. Considering the real-time and stability of navigation, the delay function provided by the ROS system is used to design the mapping module slightly earlier than the navigation module, so that the mapping part provides the required environmental information for navigation, which is beneficial to the planning decision of the unmanned vehicle in the initial stage.

[0093] Step 3.2: Obstacle stopping and avoidance methods. Aiming at the problems of obstacle stopping and dynamic obstacle avoidance of mobile robots in complex environments, two methods are proposed: an obstacle stopping method based on a dynamic window and an obstacle avoidance method based on the move base framework. These methods aim to improve the safety and navigation performance of the robot in a dynamic environment.

[0094] As Figure 8As shown, the obstacle stopping method is implemented based on the dynamic window method. Especially when other paths cannot be planned in a narrow area, the parking strategy of the robot is mainly determined by the relationship between the speed limit and the distance to the obstacle. Specifically, first, the basic framework of the movement is determined by the starting position and the target position of the robot. To effectively plan the movement trajectory, the sliding window method is used to sample the speed of the robot at each moment, and combined with the obstacle distance information provided by the lidar, a set of obstacle distance evaluations are generated. These evaluations are based on the maximum speed limit of the robot and the shortest safe distance to the obstacle, and trigger the obstacle stopping strategy when the robot approaches the obstacle. Once the distance between the robot and the obstacle is lower than the set safety distance threshold, the robot will automatically decelerate and finally set the speed to zero, thus achieving parking. This method can effectively avoid collisions during the robot's driving process through real-time speed adjustment and obstacle distance perception, ensuring the safe docking of the robot in a dynamic environment.

[0095] As Figure 9 As shown, the obstacle avoidance method is based on the move base framework in ROS, uses LiDAR point cloud data and inertial measurement unit (IMU) data, combines with the constructed grid map, and realizes dynamic obstacle avoidance through global path planning and local obstacle avoidance strategies. During the obstacle avoidance process, first, the grid map is loaded as a prior map, and real-time positioning is performed using LiDAR point cloud and IMU data to obtain the current position and orientation of the robot. Based on the current position, through the global path planning algorithm, a preliminary set of path points is searched and generated in the prior map. The goal of global path planning is to find the shortest path from the starting point to the target point, and the path is optimized through the path smoothing algorithm to eliminate redundant points, thus obtaining an efficient global path.

[0096] During the local obstacle avoidance process, the TEB (Timed Elastic Band) method is used to dynamically adjust the robot's path. The TEB method can automatically adjust the movement trajectory of the robot when facing dynamic obstacles by considering constraints such as time, distance, speed, and acceleration. Specifically, when encountering a dynamic obstacle, the TEB method will perform real-time correction on the global path, and the path will shrink according to the distance to the obstacle, the relative speed, and the current state of the robot. This process not only ensures the timeliness of obstacle avoidance but also ensures that the robot can smoothly adjust its direction and avoid deviating too much from the original path.

[0097] During the TEB planning process, considering the robot's movement speed and the distance to obstacles, the obstacle avoidance strategy optimizes the path according to the constraint relationship between distance and speed. By introducing time information, the path adjustment not only takes into account the spatial position of obstacles but also incorporates motion timeliness, ensuring that the robot travels the same distance within the same time period. Finally, the system generates discrete poses with time information and outputs two rounds of speed commands, enabling the robot to avoid obstacles in a smooth and flexible manner, prevent collisions, and ensure path tracking accuracy.

[0098] Step 3.3: Narrow area passage method. In complex indoor environments, there are usually narrow areas in the passable paths, such as door frames, corridor corners, and narrow passages between furniture. Due to limited space in these areas, robots often have difficulty passing through smoothly. Some existing methods usually rely on the active recognition of specific features (such as clearly marked door frames or corridors) to formulate passage strategies, but these methods have poor generalization ability and are difficult to adapt to complex and changing home environments.

[0099] To solve this problem, an online local path adjustment method is used. Based on the success of global path planning, this method starts from the local path. When the local path planning fails, it automatically detects areas that are too narrow and difficult to pass through and generates new path points to guide the robot through.

[0100] As Figure 10 shown, the core idea is to calculate the center point of the narrow area and generate optimized auxiliary path points around it to guide the robot through complex and narrow environments along a smooth and safe trajectory. During the generation of the center point of the narrow area, a local window area is defined based on the cost map and the robot's current pose. Since the points on the map are known, by extracting the obstacle boundaries within this area, the nearest pair of boundary points can be found, and the midpoint of them is calculated as the center point of the narrow area:

[0101]

[0102] where p 1 =(x 1 ,y 1 ) and p 2 =(x 2 ,y 2 ) are the coordinates of the pair of boundary points at this time, and this pair of points satisfies the condition of the minimum Euclidean distance:

[0103] L min =min||p 1 -p 2 ||

[0104] Based on the center point, extend along the direction perpendicular to the line connecting the nearest boundary point pair to generate the initial auxiliary path points. Assume the direction vector of the boundary point pair is v o =(v x , v y ), and its perpendicular vector is v p =(-v y , v x ). By extending a distance d along the perpendicular vector direction on both sides of the center point, the initial auxiliary points are obtained:

[0105]

[0106] To further optimize the position of the auxiliary points, search for the minimum value of the cost function f(x, y) within a small window around the initial auxiliary points to obtain the finally adjusted auxiliary path points. The cost function f(x, y) is usually based on factors such as obstacle distance and path safety. The optimized points can be expressed as:

[0107] (x a , y a ) = argmin f(x, y)

[0108] where (x, y) belongs to the search window area. Through the above method, the robot can quickly generate a local path adapted to the environment without relying on explicit feature markers, greatly improving its passing ability in complex and difficult areas such as narrow channels, door frames, and turns.

[0109] Furthermore, please refer to Figures 11 to 13 , and the present invention is also comparatively illustrated by specific embodiments: Table 1 shows the comparative experiments on open vocabulary and fixed vocabulary. The results show that the accuracy rate of the system in processing natural language instructions (such as "I want to sit down") is between 80% - 85%, which indicates that the system has strong semantic understanding ability. However, due to the diversity and complexity of natural language, the accuracy rate is slightly lower than that in the fixed vocabulary scenario. In the fixed vocabulary scenario, the system can accurately identify and return the corresponding object for a clear object name (such as "Box"), and the accuracy rate is relatively high, usually between 95% - 98%. This is mainly because the semantics of fixed vocabulary is clear and does not require complex semantic parsing. In addition, the center point coordinates of all returned results are consistent with the actual position of the object corresponding to the semantic instruction, indicating that the system has high accuracy in object positioning.

[0110] Table 1 Comparative experiment data of open vocabulary and fixed vocabulary

[0111]

[0112] Figure 11Shows the obstacle stopping strategy of a mobile robot in a narrow area. The red curve represents the travel distance of the robot, the blue curve represents the linear velocity of the robot, and the green curve represents the angular velocity of the robot. As can be seen from the figure, when at a certain distance from the obstacle, the travel distance is horizontal (i.e., the robot stops), thus achieving the purpose of obstacle stopping. And the linear velocity and angular velocity of the robot are relatively stable, and the robot travels smoothly.

[0113] Figure 12 Shows the obstacle avoidance strategy of a mobile robot. When a moving obstacle appears, the local path makes the mobile robot no longer travel according to the global path planning, but dynamically adjusts the travel route according to the real-time perception information. This strategy effectively avoids the collision risk that may be caused by the fixed global path planning. In the experiment, the local path planning can quickly react when the robot approaches the obstacle, bypass the obstacle by recalculating the local path, and gradually return to the predetermined trajectory of the global path after the obstacle is avoided. This strategy that combines global planning and local path planning not only ensures efficient navigation in a large-scale environment but also enhances the adaptability and flexibility of the robot in a dynamic environment. The experimental results show that when the local path planning is combined with the global planning, the robot can more accurately avoid obstacles and successfully complete the target task, while significantly improving the stability and robustness of the path planning.

[0114] Figure 13 Shows the test results of the robot's repeatability to a point. To verify the positioning accuracy of the 3D semantic map in this paper, relying on the navigation framework that fuses A* and TEB, 4 real indoor environment coordinates (x i , y i ) i＝1,2,3,4 were preset. The experimental results show that the average accuracy of the robot to a point is within ±5 cm, which indicates that this fusion navigation framework can effectively achieve high-precision positioning and navigation. In practical applications, this high-precision ability to reach a point is crucial for ensuring the reliability and stability of the robot in a complex environment, verifying the effectiveness of the 3D semantic map and the high-precision performance of the path planning framework that combines global path planning and local path planning in practical indoor scenarios.

[0115] Table 2 shows the errors of the robot in the x and y directions. The experimental results show that in multiple repeated positioning tests, the errors of the robot in the x and y directions are both kept within a small range. Specifically, the average errors of the robot in the x and y directions are within ±3 cm. These error values reflect the accuracy of the robot when performing tasks in different directions and indicate that the navigation performance in a complex indoor environment is relatively stable. By analyzing these errors, it can be seen that the navigation framework of integrated planning can effectively reduce the positioning deviation. Especially when making real-time path adjustments to a dynamic environment, the error control has been effectively optimized, which further verifies the high efficiency and accuracy of the robot in the actual environment.

[0116] Table 2 Comparison of the Errors of the Robot in the x and y Directions

[0117]

[0118] In summary, the present invention has the following beneficial effects:

[0119] 1. Enhance the intelligence and execution efficiency of task planning: Deeply integrate semantic information into the task planning process, solve the problems of low planning efficiency and insufficient task execution ability caused by the lack of semantic information in traditional methods. Utilize the object categories, regional attributes, and semantic relationships in the 3D semantic map to achieve more efficient task decomposition and execution strategies.

[0120] 2. Improve the navigation ability of the two-wheel differential robot in complex environments: By combining the information of the passable regions in the 3D semantic map, optimize the path planning method of the two-wheel differential robot under motion constraints. For narrow regions and dynamic environments, achieve real-time perception and environmental information update, and be able to accurately stop and avoid obstacles in narrow regions, and dynamically avoid obstacles and adjust the path in passable regions.

[0121] The above-disclosed are only one or more preferred embodiments of the present invention. Of course, it cannot be used to limit the scope of the rights of the present invention. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. A two-wheel differential mobile robot task planning method based on a three-dimensional semantic map, characterized in that: The following steps are involved: Step 1: Construct a motion model of a two-wheel differential mobile robot; Step 2: Build a 3D semantic map, perform map preprocessing and scene understanding; Step 3: Robot path planning based on 3D semantic map.

2. The two-wheel differential mobile robot task planning method based on three-dimensional semantic map according to claim 1 is characterized in that: The execution process of step 1 includes the following steps: Step 1.1: Analyze the coordinate system model and complete the coordinate transformation between the local coordinate system and the global coordinate system of the two-wheel differential mobile robot; Step 1.2: Simplify the position and posture of the two-wheel differential mobile robot into 2 degrees of freedom information and construct the corresponding motion model.

3. The two-wheel differential mobile robot task planning method based on three-dimensional semantic map as claimed in claim 2 is characterized in that: The execution process of step 2 includes the following steps: Step 2.1: Preprocess the 3D semantic map and 2D raster map; Step 2.2: Build a scene understanding framework based on the ROS robot operating system; Step 2.3: Open vocabulary semantic mapping and robot perception; Step 2.4: Task execution based on 3D semantic objects.

4. The two-wheel differential mobile robot task planning method based on three-dimensional semantic map as claimed in claim 3 is characterized in that: In step 2, the preprocessed 3D semantic map and 2D grid map are stored as octree maps, the maps are visualized using the RViz tool, and the communication between the scene object coordinates and the robot is achieved through the ROS robot operating system.

5. The two-wheel differential mobile robot task planning method based on three-dimensional semantic map as claimed in claim 4 is characterized in that: After the raster map is fused with the multi-view semantic information in step 2, each object in the map is embedded with a semantic vector, which is mapped into coordinate information readable by the robot. The category and spatial information of each object are saved as the robot's executable file as the robot's semantic knowledge base.

6. The two-wheel differential mobile robot task planning method based on three-dimensional semantic map as claimed in claim 5, characterized in that: In step 2.4, during the execution of tasks based on three-dimensional semantic objects, the scene understanding framework based on the ROS robot operating system is integrated with the large language model LLM to realize task planning through natural language instructions.

7. The two-wheel differential mobile robot task planning method based on three-dimensional semantic map as claimed in claim 6, characterized in that: The execution process of step 3 includes the following steps: Step 3.1: Path planning method development; Step 3.2: Formulate obstacle stopping and obstacle avoidance methods; Step 3.3: Development of narrow area traffic method.

8. The two-wheel differential mobile robot task planning method based on three-dimensional semantic map as claimed in claim 7 is characterized in that: In step 3.1, the A* algorithm is used as the global path planner, the TEB algorithm and the DWA algorithm are used as local path planners, and the move_base framework is used to complete the navigation task.

9. The two-wheel differential mobile robot task planning method based on three-dimensional semantic map as claimed in claim 7, characterized in that: Step 3.2 implements obstacle parking based on dynamic windows and obstacle avoidance based on the move base framework. When other paths cannot be planned in a narrow area, the robot's parking strategy is determined by the relationship between the speed limit and the obstacle distance. The sliding window method is used to sample the robot's speed at each moment, and combined with the obstacle distance information provided by the lidar, a set of obstacle distance evaluations is generated. The obstacle avoidance process is based on the move base framework, using LiDAR point cloud data and inertial measurement unit IMU data, combined with the constructed grid map, through global path planning and local obstacle avoidance strategies.

10. The two-wheel differential mobile robot task planning method based on three-dimensional semantic map according to claim 7, characterized in that: In step 3.3, the center point of the narrow area is calculated and optimized auxiliary path points are generated around it to guide the robot through the complex and narrow environment with a smooth and safe trajectory.

Images