Real-time non-graphical autonomous navigation system for robot

Abstract

The invention relates to a real-time non-graphical robot autonomous navigation system, and belongs to the field of robots. The system comprises a main control module, an environment sensing module and a hybrid navigation module, the main control module is responsible for man-machine interaction, task flow control and cooperation of other modules, and storing and managing a visual semantic database; the environment sensing module is responsible for controlling the robot to collect image data and depth data under the condition of no prior map, identifying objects in a scene, extracting semantic information of the objects, and integrating identification results into a visual semantic database; and the hybrid navigation module analyzes the instruction sent by the main control module by using a large language model, matches visual memory, determines a navigation target, and completes autonomous navigation of the robot through a hybrid navigation strategy. According to the method, the problem of low navigation task accuracy caused by the sparsity of global information in map-free navigation is solved.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a real-time, graph-free autonomous navigation system for robots. Background Technology

[0002] Currently, autonomous mobile robots are widely used in service, security, warehousing, and home scenarios. To achieve environmental perception and path planning, mainstream technologies generally fall into two categories: one is geometric mapping methods based on Simultaneous Localization and Mapping (SLAM); the other is end-to-end visual control methods relying on deep learning models. The first type of method constructs a global map and performs localization and path planning using sensors such as LiDAR and RGB-D cameras, enabling high-precision navigation in structured environments. However, this type of method is susceptible to failure in dynamic scenes due to occlusion, changes in lighting, and interference from moving objects. Furthermore, the mapping process is time-consuming, generates massive amounts of map data, and places high demands on computing and storage resources, resulting in significant maintenance costs. In addition, when the environment changes or the robot operates in large spaces, the map needs to be frequently rebuilt, leading to insufficient system deployment flexibility. The second type of method generates control commands directly from perceived images using deep neural networks, eliminating the reliance on geometric maps and offering advantages such as rapid deployment and low computational latency. However, this type of method typically lacks global spatial consistency and long-term memory capabilities, resulting in weak generalization performance outside the training scenario and difficulty in ensuring stable operation in complex or dynamic environments. More importantly, the decision-making process of this type of method is difficult to interpret, lacks traceability and explicit semantic reasoning capabilities, and is difficult to engage in high-level semantic interaction with humans.

[0003] In existing technologies, the emergence of Visual-Language Models (VLMs) and Large Language Models (LLMs) has enabled robots to perform semantic recognition, target description, and language reasoning from multimodal information. However, most existing technologies still rely on pre-built semantic maps or can only complete target search and navigation within a local visual range, lacking a unified spatial reference system and dynamic update mechanism, and thus failing to achieve long-term stable map-free navigation. Therefore, existing technologies still face many challenges in achieving natural language-driven autonomous navigation: on the one hand, there is a lack of fast, stable, and low-resource-consumption localization and navigation methods in dynamic environments; on the other hand, there is a lack of a perceptual memory system that can maintain global consistency and interpretability without global map constraints. Summary of the Invention

[0004] This application provides a real-time mapless robot autonomous navigation system that can solve the problem of low navigation accuracy caused by the sparsity of global information in mapless navigation.

[0005] To achieve the above objectives, according to a first aspect of this application, a real-time map-free autonomous robot navigation system is provided. The system includes a main control module, an environment perception module, and a hybrid navigation module. The main control module is responsible for human-computer interaction, task flow control, and coordination with the environment perception module and the hybrid navigation module, storing and managing a visual semantic database. The environment perception module is responsible for controlling the robot to collect image and depth data without a prior map, identifying objects in the scene and extracting their semantic information, and integrating the identification results into a visual semantic database. The hybrid navigation module uses a large language model to parse the instructions issued by the main control module, matches them with visual memory, determines the navigation target, and completes autonomous robot navigation through a hybrid navigation strategy.

[0006] To achieve the above objectives, according to a second aspect of this application, a real-time graph-free autonomous navigation method for robots is also provided, comprising the following steps: The main control module determines the storage status of the visual memory. If the visual memory is empty, the environment perception module performs a panoramic scan and visual language model recognition, constructs the visual memory, stores it in a structured manner, and then returns it to the main control module. If visual memory exists, the main control module waits for the user to input natural language commands, and then parses and semantically matches the commands using a large language model; The hybrid navigation module checks the target coordinates to confirm the storage status of the three-dimensional coordinates. If the three-dimensional target is missing, it initiates an on-demand three-dimensional coordinate learning process, which learns the missing three-dimensional coordinates by returning to the origin and re-identifying them. If the three-dimensional coordinates exist or the on-demand three-dimensional coordinate learning process is completed, the three-dimensional coordinates are input into the hybrid navigation module and a preset mode is selected for the navigation task. Once the hybrid navigation module completes navigation, it updates the visual memory and transmits it to the main control module, awaiting the next instruction.

[0007] Furthermore, the environment perception module completes environment perception and constructs a visual semantic database through an environment perception and memory construction method based on a visual language model, including the following steps: Preset a set of scanning angles The robot starts from its initial position and rotates to each angle in sequence. And acquire RGB images and depth images Simultaneously record the robot's current pose in the ODOM coordinate system. ,in This is the global orientation angle; After the robot comes to a stable stop at each angle, it simultaneously acquires the current RGB color image and depth image, and records the robot's coordinates and precise orientation angle in the odom odometry coordinate system at this time; The acquired RGB image is encoded to obtain an encoded RGB image. The encoded RGB image and the scan command text input by the user are used as input, and after processing by the visual language model, JSON data is output. Each detected object is post-processed to generate a visual memory entry, and a unique identifier is assigned to each newly discovered object. By combining the bounding box (bbox) and the synchronously acquired depth image, the effective average depth value within the object region is calculated. Based on the center pixel coordinates of the object in the image, camera intrinsic parameters, average depth value, and global orientation angle when the robot acquires the image, the absolute angle of the object relative to the robot's initial scanning origin is calculated. A close-up image of the object is obtained by cropping it from the original RGB image based on the bounding box (bbox) and then saved. All object entries detected from all scanning angles are aggregated to form a dictionary or hash table structure with the target ID as the primary key, and a visual semantic database is constructed. The visual semantic database additionally stores the robot's origin pose during this scan.

[0008] Furthermore, upon receiving the scanning instruction text input by the user, the system dynamically constructs a Prompt instruction that includes a world knowledge instruction and a user task instruction. The world knowledge instruction serializes the current complete visual memory JSON object into a string, which is then provided as a world model or context to the large language model. The user task instruction takes the user's original natural language instruction as the task to be executed.

[0009] Furthermore, the system's Prompt instruction guides the visual language model to complete the following steps: Identify all relevant objects in the image based on the Prompt command; The visual language model returns a list of multiple objects in JSON format. The list includes the object name, object category, pixel bounding box of the object in the 2D image, the model's confidence in the detection result, and a natural language description. The natural language description includes the object's color, material, state, and positional relationship with the surrounding environment. For each object and its bounding box identified by the visual language model Calculate the coordinates of the center pixel of the image. :

[0010]

[0011] Combined with camera intrinsic parameters, the camera intrinsic parameters are focal length. He Guangxin Calculate the pixel offset between the center point of the object and the optical center of the camera:

[0012] The camera's horizontal field of view (HFOV) is:

[0013] in, This refers to the horizontal resolution of the image sensor. Focal length A general term; Calculate the camera's angular resolution:

[0014] Calculate the horizontal deflection angle of the object relative to the camera's optical center:

[0015] Finally, the global absolute angle of the object relative to the scan origin is:

[0016] Among them, the global absolute angle The key index for the robot to reorient to the target in the on-demand 3D coordinate learning method is stored in the visual memory entry.

[0017] Furthermore, the specific steps of the on-demand 3D coordinate learning method are as follows: When the navigation task begins, the system checks whether the identified navigation target has a valid position_3d field, i.e., three-dimensional coordinates. If the field or the three-dimensional coordinates do not exist, the coordinate learning process is activated. The coordinate learning process is only triggered when the semantic target entry is used for navigation for the first time and its three-dimensional coordinates are missing, and coordinate completion is performed once or multiple times. The robot returns to the scanning origin based on its robot origin pose from the visual semantic database, and based on the absolute angle from the visual semantic database... Orienting itself back towards the target, the system relocates the target within the field of view using a visual language model and obtains its bounding box (bbox). Pixel coordinates are then calculated using a center point method. ), complete the redirection; The robot initiates a slow forward exploration process with visual confirmation as the ultimate goal. During the exploration process, real-time visual input is continuously fed into the visual language model for matching. When the confidence level of the confirmed target is higher than a preset threshold, the exploration process is terminated, and the newly obtained coordinate data is stored in the corresponding target entry in the visual semantic database.

[0018] Furthermore, the visual language model successfully relocalizes the target in the field of view and returns its bounding box. The system atomically captures the complete state at the current moment through a state snapshot mechanism. The complete state includes the RGB image, depth image, robot pose, and orientation. Specifically, the following steps are executed: The effective depth values ​​within the bbox_new region are extracted from the synchronized depth image, and a robust filtering strategy is used to suppress noise interference. Solve the camera coordinate system based on camera intrinsic parameters. The target's coordinates in the camera coordinate system are calculated using the following formula. :

[0019]

[0020]

[0021] Set the fixed transformation as a rotation matrix. With translation vector Transform the target point coordinates to the chassis coordinate system :

[0022] In a typical forward-mount configuration, the fixed transformation simplifies to:

[0023] in This represents the camera's mounting offset relative to the chassis. The system combines the robot's current global pose. The global coordinates of the target in the ODOM coordinate system are calculated using the following formula ( ):

[0024]

[0025] Calculated ( The three-dimensional coordinates of the target object in the global map are then updated to the visual semantic database.

[0026] Furthermore, the hybrid navigation module includes the following specific processes: After the navigation task begins and the robot faces the target, the system first estimates the straight-line distance to the target. If the target has global 3D coordinates, the Euclidean distance is calculated directly; if the target does not have global 3D coordinates but is within the field of view, depth information is used; if the current Euclidean distance between the robot and the target is... If the distance between the robot and the target exceeds a preset threshold, the system enters the local / global path planning mode in step S302; if the current Euclidean distance between the robot and the target is greater than a preset threshold, the system enters the local / global path planning mode in step S302. If the value is less than a preset threshold, the system enters the visual servo mode of S303. Enter the local-to-global path planning mode, and use the local-to-global path planner to achieve medium-to-long-distance movement and obstacle avoidance; Entering visual servo mode, control laws are designed under the state snapshot atomic capture mechanism to achieve close-range movement and obstacle avoidance; The hybrid navigation module also has a robustness guarantee mechanism to ensure that the robot completes the autonomous navigation task. The robustness guarantee mechanism includes a dynamic obstacle avoidance mechanism in visual servo mode, a path recovery mechanism in local and global navigation mode, and a multi-level recovery process for visual tracking loss.

[0027] Furthermore, the local-to-global path planning mode includes the following specific steps: The system uses the A* algorithm with a view-keeping constraint to ensure that the target object remains within the robot's field of vision during navigation; The target pose update unit is activated. The unit locates the target within the current field of view using a visual language model and calculates the angle that the robot needs to face the target based on the pixel offset of the target in the image. The module calculates the desired stopping point in front of the object and uses real-time LiDAR data to check the safety of the desired stopping point and its surrounding area. If there are obstacles near the initial point, the module will search for an unobstructed and safest alternative point within a preset angle and distance range, centered on the object, as the final navigation target. The system calculates an unobstructed parking spot in front of the target object using a LiDAR-based safe target point search algorithm. ; Obtain the robot's current pose in real time from the robot chassis odometer or TF transformation. The planner receives the robot's current pose. and accessible parking spots Then, an A* search method with view constraints is initiated on the local cost map, such that the current node... equal to parking spot If the search is successful, the planned path is completed. The search proceeds back from the endpoint to the starting point, establishing a map coordinate sequence. Each map coordinate point in the map coordinate sequence is then converted back to world coordinates to obtain a map coordinate sequence in the world coordinate system. The system adopts a look-ahead point tracking strategy, which finds a point on the path that is a preset look-ahead distance away from the robot's current position as the tracking target point, calculates the angular error toward the tracking target point, and generates angular velocity commands through proportional control. The A* search method with vision constraints includes the following steps: The new A* algorithm cost function is obtained by improving the cost function of the standard A* algorithm:

[0028] in, From the starting point to the node The actual cost; For the node The estimated cost to reach the target; To maintain vision, a penalty item is imposed. This is a penalty weighting coefficient used to balance the priority of path length and view preservation; For each candidate node on the path Calculate from node Direction vector pointing to the target point and from node The direction vector pointing to the robot's current orientation The angle between the line of sight and the target is obtained. ; Penalty for maintaining field of vision A piecewise function design is adopted:

[0029] in, To maintain a threshold angle for the field of view, This is the penalty coefficient, used to control the intensity of the penalty.

[0030] Furthermore, the visual servoing mode includes the following specific steps: The target is detected in the current field of view by using a visual language model, and its initial bounding box (bbox) is obtained. A high-performance real-time visual tracker (CSRT) is initialized using the bounding box (bbox). Under the atomic capture mechanism of state snapshot, angular velocity control laws and linear velocity control laws are designed respectively; the angular velocity control law can ensure that the robot continuously faces the target by controlling the horizontal pixel error. and angular velocity A closed-loop negative feedback system is formed; the linear velocity control law, based on the distance to the target, adopts a segmented control strategy to safely drive the robot forward or stop; The real-time vision tracker CSRT returns the current bounding box. Calculate the pixel coordinates of its center point ( ; and the center point of the image The horizontal pixel error is obtained through comparison:

[0031] Angular velocity control quantity Generated by proportional control law:

[0032] in, The proportional gain is used to normalize the error to the [-1,1] interval, thereby achieving smooth steering control; The linear velocity control amount is set segment by segment according to the target distance. :

[0033] in, The average depth value of the bounding box provided for the real-time vision tracker CSRT.

[0034] This application presents a robot autonomous navigation method that does not rely on pre-built maps, including visual semantic memory construction, natural language parsing, on-demand 3D coordinate learning, and a hybrid navigation control process. The hybrid navigation system, integrating multimodal perception and language reasoning, can dynamically switch between local and global path planning and visual servoing modes. An A* path planning method with view-keeping constraints introduces a "view axis-target angle" penalty term into the standard A* cost function, actively biasing the search process to always keep the target object in the center of the camera's field of view, achieving a "planning while seeing" navigation effect. It can directly output a highly visible path without post-processing or secondary smoothing. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0036] Figure 1 This is a diagram illustrating the overall architecture and workflow of a mapless navigation system provided in accordance with embodiments of this application. Figure 2 This is a flowchart of the on-demand three-dimensional coordinate learning method provided according to an embodiment of this application; Figure 3 This is a flowchart of a hybrid navigation control strategy provided according to an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] like Figure 1 The overall system architecture of this application is divided into three modules: Main control module: As the system's central dispatch center, it is responsible for human-computer interaction, task flow control, and coordination between modules. This module decides to initiate environmental perception or navigation tasks based on system status (such as the presence of visual memory) and is responsible for the persistent storage and management of visual memory data.

[0039] Environmental Perception Module: Responsible for the robot's environmental exploration and semantic understanding. Without a prior map, this module controls the robot to perform a 360-degree panoramic scan in place, using an onboard RGB-D camera to acquire images and depth data at preset key angles. By calling a visual language model to analyze the images, it identifies relevant objects in the scene based on user commands (such as "scan all objects in the room") and extracts their semantic information. Finally, the recognition results (including object names, descriptions, bounding boxes, confidence scores, 3D coordinates, relative angles, etc.) are integrated into a structured sparse visual semantic database.

[0040] Hybrid Navigation Module: Responsible for executing specific navigation tasks. This module receives natural language navigation instructions from the main control module, parses the instructions using a large language model (LLM) and matches them with visual memory to determine the navigation target, and then employs a hybrid navigation strategy. The hybrid navigation strategy has three variations: (1) If the target object has known global coordinates, A* is used first to perform local and global path planning to complete point-to-point navigation. The visual language model plays a role in assisting navigation re-identification.

[0041] (2) If the target object is within a close range, the visual servo tracking strategy is activated. At this time, the recognition of the visual language model dominates the control. The target is continuously tracked by the CSRT tracker and periodic visual language model re-detection. Real-time obstacle avoidance is performed in combination with lidar data to control the robot to approach the target.

[0042] (3) When the target object lacks global coordinates, this module triggers the 3D position learning process, controls the robot to return to the initial scanning origin, and searches again in the general direction of the target. It also calculates the global 3D coordinates of the target through the camera and depth information to update the visual memory.

[0043] The hybrid navigation module also includes an asynchronous visual language model verification unit. As an auxiliary unit of the hybrid navigation module, it runs on an independent thread to achieve non-blocking target verification and coordinate correction, effectively solving the navigation lag problem that may be caused by traditional synchronous verification. Its core lies in atomically capturing an initial state snapshot (image, pose, depth), calling the visual language model to verify whether the target is within the field of view, recording the reference angle and pose, and performing operations such as "periodic checks" and "coordinate deviation correction".

[0044] Figure 1 The specific process is as follows: The main control module determines the storage status of the visual memory. If the visual memory is empty, the environment perception module performs a panoramic scan and visual language model recognition, constructs the visual memory, stores it in a structured manner, and returns it to the main control module. If the visual memory exists, the main control module waits for the user to input natural language commands and parses and semantically matches the commands using a large language model. The hybrid navigation module checks whether the target coordinates exist in three-dimensional coordinates. If the three-dimensional target is missing, it starts an on-demand three-dimensional coordinate learning process, learning the missing three-dimensional coordinates by returning to the origin and re-identifying. If the three-dimensional coordinates exist or the on-demand three-dimensional coordinate learning process is completed, the three-dimensional coordinates are input to the hybrid navigation module, and a preset mode is selected for the navigation task. When the hybrid navigation module completes navigation, it updates the visual memory and transmits it to the main control module, waiting for the next command.

[0045] The environment perception module completes environment perception and builds a visual semantic database through an environment perception and memory construction method based on a visual language model. The specific method is as follows: To fully perceive the surrounding environment, the environmental perception module presets a set of scanning angles. The robot starts from its initial position and rotates to each angle in sequence. And acquire RGB images and depth images Simultaneously record the robot's current pose in the ODOM coordinate system. ,in This is the global orientation angle.

[0046] After the robot comes to a stable stop at each angle, it simultaneously acquires the current RGB color image and depth image, and records the robot's coordinates and precise orientation angle in the odom odometry coordinate system.

[0047] The acquired RGB image is encoded and used as input along with the user-inputted scanning command text to invoke a large visual language model. The designed Prompt command guides the visual language model to complete the following tasks: Object recognition: Based on the intent of the Prompt command, identify all relevant objects in the image.

[0048] Structured output: The visual language model returns a list of multiple objects in JSON format. Each object contains the following key information: object_name: the specific, detailed name of the object; category: the general category of the object; bbox: the pixel bounding box of the object in the 2D image. Confidence: The model's confidence in the detection results; Description: A detailed natural language description, including the object's color, material, state, and its positional relationship with the surrounding environment, providing rich context for subsequent disambiguation and human-computer interaction. For each object and its bounding box identified by the visual language model... Calculate the coordinates of the center pixel of the image. :

[0049]

[0050] Combined with camera intrinsic parameters, the camera intrinsic parameters are focal length. He Guangxin Calculate the pixel offset between the center point of the object and the optical center of the camera:

[0051] The camera's horizontal field of view (HFOV) is:

[0052] in, This represents the horizontal resolution (number of pixels) of the image sensor. This is the camera's focal length, expressed in pixels. A general term; Calculate the camera's angular resolution:

[0053] Calculate the horizontal deflection angle of the object relative to the camera's optical center:

[0054] Finally, the global absolute angle of the object relative to the scan origin is:

[0055] in, The key index for the robot to reorient to the target in the on-demand 3D coordinate learning method is stored in the visual memory entry.

[0056] After receiving the JSON data returned by the visual language model, the module post-processes each detected object to generate a visual memory entry. A unique identifier is assigned to each newly discovered object; combining the bounding box (bbox) and the synchronously acquired depth image, the effective average depth value within the object's region is calculated. Based on the object's center pixel coordinates in the image, camera intrinsics, the average depth value, and the global orientation angle when the robot acquired the image, the absolute angle of the object relative to the robot's initial scanning origin is calculated. This absolute angle information is a key index for subsequent "on-demand 3D coordinate learning."

[0057] A close-up image of the object is cropped from the original RGB image based on the bounding box (bbox) and saved for subsequent visual tracking and re-identification. When re-detecting the target, the system inputs the current image and the stored reference image into the visual language model for comparison and recognition.

[0058] The object entries detected from all scanning angles are aggregated into a dictionary or hash table structure with the target ID as the primary key, i.e., visual semantics. This visual semantic database also additionally stores the robot's "origin" pose during the scan, providing a benchmark for subsequent position learning.

[0059] To achieve smooth human-computer interaction, the hybrid navigation module first needs to accurately understand the user's navigation intent. This application utilizes the reasoning capabilities of a large language model (LLM) to accurately map fuzzy natural language instructions to specific targets in a visual semantic database.

[0060] The natural language instruction understanding and target determination method based on a large-scale language model in this application is as follows: When a user command is received, the system dynamically constructs a Prompt command containing two parts: a world knowledge command and a user task command. The world knowledge command serializes the current complete "visual memory" JSON object into a string, which is provided as a "world model" or context to the large language model. The user task command takes the user's original natural language command as the task to be executed.

[0061] The system sends the constructed Prompt instruction to a large language model with powerful language understanding and reasoning capabilities. The Prompt instruction explicitly instructs the large language model to perform the following task: "Analyze the user's instruction and find the ID of the target that best matches the user's instruction from the provided visual memory list. Return the result in the specified JSON format ({"reasoning": "...", "target_id": "..."}), without any additional explanation." The system receives the JSON response from the LLM, extracts the ID field of target_id, which is the target that the large language model deems to best match the user's intent after comprehensively matching information based on semantics, description, spatial relationships, and other dimensions. The navigation module then retrieves the complete information corresponding to this ID from the visual semantic database as the final target for subsequent navigation tasks.

[0062] like Figure 2 This application proposes a closed-loop method for on-demand 3D coordinate learning to address the inherent contradiction between "global information sparsity" and "navigation task accuracy" in mapless navigation. This method abandons the high-cost and highly redundant traditional approach of performing full 3D modeling of all objects in the initial perception stage, instead employing a "lazy evaluation" resource optimization strategy. Only when a semantic target is first invoked by the navigation task, and its spatial coordinate data in the visual semantic database is missing, will the system trigger an active coordinate learning and memory enhancement process with global consistency guarantees. The specific steps of the on-demand 3D coordinate learning method are as follows: At the start of the navigation task, the system checks whether the identified navigation target has a valid position_3d field, i.e., three-dimensional coordinates. If the field or the three-dimensional coordinates do not exist, the coordinate learning process is activated. The coordinate learning process is only triggered when the semantic target entry is used for navigation for the first time and its three-dimensional coordinates are missing. It can perform coordinate completion in one or multiple iterations.

[0063] The coordinate learning process is as follows: Ensuring the robot can rediscover and lock onto the target is crucial to the success of this method, given the sparse information of only the absolute angle of the target object. After the robot returns to the scanning origin based on the robot's origin pose stored in the visual semantic database, it then uses the memorized absolute angle... After reorienting itself to the target, the target is relocated within the field of view using a visual language model, and its bounding box (bbox) is obtained. Pixel coordinates are then calculated using the same center point calculation method. This angle represents the object's global absolute orientation relative to the scanning origin during the initial scan, when the object is identified by the visual language model. The robot only needs one precise local rotation to adjust its orientation to this recorded angle. Because this angle is measured at a global reference point, it provides a deterministic, high-probability target location prediction, ensuring that the target can re-enter the vehicle-mounted camera's field of view with minimal search cost, reducing the complexity and uncertainty of subsequent visual searches.

[0064] After retargeting, the robot initiates a slow forward exploration process with visual confirmation as the ultimate goal. This process involves continuously feeding real-time visual input into the visual language model for matching. When the confidence level of the confirmed target exceeds a preset threshold, the exploration process immediately terminates. This newly acquired coordinate data is immediately written back to the corresponding target entry in the visual semantic database, thus completing a one-time, permanent enhancement of the memory.

[0065] Once the visual language model successfully relocalizes the target in the field of view and returns its bounding box, the system atomically captures the complete state at the current moment (including RGB image, depth image, robot pose, and orientation) through a state snapshot mechanism, ensuring multimodal data time synchronization, and then performs the following steps: 1. Extract the effective depth values ​​within the bbox_new region from the synchronized depth image. To suppress noise interference, the following robust filtering strategy is adopted: (1) Extract the effective depth values ​​within the bounding box and ignore invalid values ​​(such as 0, NaN, infinity); (2) Extract the effective depth values ​​of the central region (40%×40%) of the bounding box, wherein the length and width of the central region of the bounding box account for 40% of the length and width of the bounding box, and the center of the central region of the bounding box coincides with the center of the bounding box; (3) Divide the effective depth values ​​into multiple intervals and count the number of depth values ​​in each interval; (4) Select the interval containing the most depth values ​​and calculate the median of all effective depth values ​​in the interval as the final depth value used for calculation. .

[0066] 2. Solve the camera coordinate system based on camera intrinsic parameters. The target's coordinates in the camera coordinate system are calculated using the following formula. :

[0067]

[0068]

[0069] Since there is a fixed transformation between the robot chassis coordinate system and the camera coordinate system defined by the URDF model, let this transformation be a rotation matrix. With translation vector Transform the target point coordinates to the chassis coordinate system. :

[0070] In a typical forward-installation configuration, this transformation can be simplified to:

[0071] in This represents the camera's mounting offset relative to the chassis.

[0072] The learned 3D coordinates are continuously monitored and corrected by an asynchronous visual language model verifier during subsequent navigation. This verifier periodically checks the accuracy of the target coordinates in an independent thread. When a depth error exceeds a preset threshold, it automatically triggers a coordinate correction process, sending a coordinate update request to the main thread via a queue mechanism to ensure the long-term accuracy of the coordinate information.

[0073] 3. The system combines the robot's current global pose. The global coordinates of the target in the ODOM coordinate system are calculated using the following formula ( ):

[0074]

[0075] Calculated ( This refers to the three-dimensional coordinates of the target object on the global map. Since the map in the simulation environment is a three-dimensional map, but the Z-axis is set to 0 by default, meaning the robot is assumed to move in a plane, only the coordinates of the target object need to be calculated. and The three-dimensional coordinates are updated in the visual semantic database for use in subsequent navigation tasks.

[0076] like Figure 3 To balance efficiency in large-scale movement with accuracy in close-range engagement, the hybrid navigation module of this application employs a dynamically switching navigation strategy. This strategy intelligently switches between a "local-to-global path planning mode" and a "visual servoing mode" based on the distance between the robot and the target, and integrates multi-sensor-based safety measures. The specific steps of the hybrid navigation control method of this application are as follows: 1. Navigation Mode Decision and Switching: After the navigation task begins and the robot faces the target, the system first estimates the straight-line distance to the target. If the target has global 3D coordinates, the Euclidean distance is calculated directly; otherwise, but the target is within the field of view, depth information is used. The system then calculates the current Euclidean distance between the robot and the target. If the value exceeds a preset threshold, the system enters the Local Global Path Planning (LOCAL_GLOBAL_NAV) mode; if... If the value is below this threshold, the system will directly enter visual servo (VISUAL_SERVO) mode.

[0077] Let the mode switching distance threshold be Navigation mode decision function The definition is as follows:

[0078] This mechanism ensures efficient navigation using a mature global path planner at long distances, while switching to visual servo mode for fine control at short distances or when the global planner is unavailable.

[0079] During global navigation, the system continuously monitors the distance to the target. Once the distance falls below a switching threshold, it automatically cancels the global navigation target and smoothly transitions to visual servoing mode. The system uses the Euclidean distance between the robot and the target. Dynamically select navigation mode. Distance threshold. Set to 1.5 meters, when If the path is in the local global path planning (LOCAL_GLOBAL_NAV) mode of S402, it will enter the visual servoing (VISUAL_SERVO) mode.

[0080] 2. Local-Global-Navi (LOCAL_GLOBAL_NAV) Mode: This mode aims to utilize a local-global path planner to handle medium- to long-distance movement and obstacle avoidance. This planner is based on the A* algorithm with a cost map and introduces a view-keeping constraint. It calculates the angle between the direction vector from the path point to the target object and the direction vector from the path point to the stopping point; the larger the angle, the higher the penalty value, ensuring that the target object remains within the robot's field of vision during navigation. In other words, the A* algorithm with a view-keeping constraint ensures that the target object remains within the robot's field of vision during navigation.

[0081] Before sending the navigation target, a target pose update unit of the system is activated. This unit first attempts to locate the target in the current field of view using a visual language model. If successful, it then accurately calculates the angle at which the robot needs to face the target based on the pixel offset of the target in the image.

[0082] This unit calculates a "desired stopping point" in front of the object. It then uses real-time LiDAR data to check the safety of this desired stopping point and its surrounding area. If there are obstacles near the initial point, the module searches for an unobstructed and safest alternative point within a certain angle and distance range, centered on the object, as the final navigation target.

[0083] The system calculates an unobstructed parking spot in front of the target object using a LiDAR-based safe target point search algorithm. ; Obtain the robot's current pose in real time from the robot chassis odometer or TF transformation. The planner receives the robot's current pose. and accessible parking spots Then, an A* search with vision constraints is initiated on the local cost map, such that the current node... equal to parking spot If the search is successful, the planned path is completed. The search proceeds back from the endpoint to the starting point, establishing a map coordinate sequence. Each map coordinate point in the map coordinate sequence is then converted back to world coordinates to obtain a map coordinate sequence in the world coordinate system.

[0084] After the path is planned, the system adopts a look-ahead point tracking strategy. A point on the path that is a certain look-ahead distance (usually 0.5 meters) from the robot's current position is found as the tracking target point. The angular error toward that point is calculated, and angular velocity commands are generated through proportional control to achieve smooth path tracking.

[0085] In global path planning based on a local cost map, an independent asynchronous visual language model validator thread continuously monitors the accuracy of the target coordinates. When a coordinate error is detected, a coordinate update request is sent to the main thread via a thread-safe queue, triggering path replanning and ensuring navigation accuracy.

[0086] 3. The specific method for Visual Servo (VISUAL_SERVO) mode is as follows: The system first detects the target in the current field of view using a visual language model, obtains its initial bounding box (bbox), and uses the bounding box (bbox) to initialize a high-performance real-time visual tracker (CSRT).

[0087] Under the atomic capture mechanism of state snapshot, angular velocity control laws and linear velocity control laws are designed respectively; the angular velocity control law can ensure that the robot continuously faces the target by controlling the horizontal pixel error. and angular velocity This forms a closed-loop negative feedback system. The linear velocity control law, based on the distance to the target, employs a segmented control strategy to safely drive the robot forward or to stop, conforming to the intuitive driving logic of "fast in straight lines, slow in turns," preventing collisions with the target due to excessive speed or stalling at a distance, ultimately resulting in smoother movement. The atomic state snapshot capture mechanism, at critical moments when calling the VLM for target verification or tracker initialization, atomically captures and locks the following data at the same instant, forming a "state snapshot": - RGB image - Depth image - Robot pose and orientation angle in the ODOM coordinate system. .

[0088] The tracker returns the current bounding box. Calculate the pixel coordinates of its center point ( ; and the center point of the image The horizontal pixel error is obtained through comparison:

[0089] Angular velocity control quantity Generated by proportional control law:

[0090] in, The proportional gain is used to normalize the error to the [-1,1] interval, thereby achieving smooth steering control.

[0091] Linear velocity control quantity Based on segmentation of target distance:

[0092] in, The average depth value of the bounding box provided for the tracker.

[0093] To address the inconsistency in data timestamps caused by visual recognition and robot motion in dynamic environments, this application designs an atomic state snapshot capture mechanism in the asynchronous visual language model validator. This mechanism atomically and synchronously captures and locks multiple modal data at the same moment during the critical instant of calling the visual language model for target verification. These data include RGB images, depth images, and the robot's pose and orientation angle in the global coordinate system. All subsequent target coordinate calculations and corrections are based on this state snapshot, fundamentally avoiding coordinate calculation errors caused by data acquisition time differences and ensuring the temporal consistency of multimodal perception data.

[0094] To ensure the robot can complete tasks stably and safely in complex and ever-changing real-world environments, this application designs a multi-layered robustness guarantee mechanism, covering various abnormal situations ranging from visual tracking failure to path encounters with obstacles. The robustness guarantee mechanism includes a dynamic obstacle avoidance mechanism in visual servo mode, a path recovery mechanism in local-global navigation mode, and a multi-level recovery process for lost visual tracking.

[0095] Dynamic obstacle avoidance mechanism in visual servo mode: During visual servo tracking, the system continuously monitors the minimum forward distance reading from the LiDAR. When the distance to an obstacle ahead is detected to be less than the preset obstacle avoidance trigger threshold, and the visually estimated distance between the robot and the target is far, the system immediately triggers obstacle avoidance procedures such as "emergency stop and reverse", "state reset and stabilization wait", and "tracker CSRT reinitialization".

[0096] Path recovery mechanism in local-to-global navigation mode: In the A*-based local-to-global path planning mode, the system handles path execution exceptions through a multi-layer state machine. (1) Obstacle detection recovery: When an obstacle is detected ahead during path tracking, the system immediately clears the current path and enters the recovery turn (RECOVERY_TURN) state.

[0097] (2) Replanning: When the steering state is restored, the system attempts to replan a feasible path based on the current pose and obstacle distribution.

[0098] (3) Gradual recovery: If replanning fails, the system will not immediately declare the task a failure, but will try to readjust the orientation based on the target location to create conditions for subsequent recovery.

[0099] Multi-level recovery process for visual tracking loss: When the CSRT tracker continuously loses targets for more than a set frame threshold (usually 15 frames), the system initiates a tiered recovery strategy: Level 1 recovery (VLM re-detection): First, the target is re-identified in the current field of view using a visual language model. If successful, the tracker is re-initialized.

[0100] Secondary recovery (position regression): If VLM re-detection fails, the system readjusts the robot's orientation based on the target's latest estimated position (or the memorized 3D position).

[0101] Level 3 Recovery (Panoramic Search): When the above recovery methods fail, a 360-degree panoramic search is performed to relocate the target, forming a complete anomaly recovery loop.

[0102] In the Local-Global-Navi (LOCAL_GLOBAL_NAV) path planning mode, this application proposes an A* path planning method with view-preserving constraints. This method introduces a "view axis-target angle" penalty term into the cost function of the standard A* algorithm, specifically implemented as follows: 1. The cost function for the standard A* algorithm This application improves upon it to obtain:

[0103] in, From the starting point to the node The actual cost; For the node The estimated cost to reach the target is usually the Euclidean distance; To maintain vision, a penalty item is imposed. This is a penalty weighting coefficient used to balance the priority of path length and view preservation.

[0104] 2. For each candidate node on the path Calculate from node Direction vector pointing to the target point and from node The direction vector pointing to the robot's current orientation The angle between the line of sight and the target is obtained. .

[0105] 3. The penalty term is designed using a piecewise function:

[0106] in, To maintain a threshold angle for the field of view, This is the penalty coefficient, used to control the intensity of the penalty.

[0107] The semantic navigation provided in this application can achieve natural language-driven operation in unknown or dynamic environments. Its core consists of three major functional modules: an environment perception module, a task understanding and decision-making module, and a hybrid navigation control module. The modules interact with each other through a unified semantic memory interface.

[0108] In the initial stage of system operation, the robot is positioned at the initial scanning location, and the environment perception module first initiates the panoramic perception program. By rotating the robot body or gimbal, multi-angle images and depth data of the environment are simultaneously acquired using an RGB-D camera and LiDAR. After processing by a Visual Language Model (VLM), the acquired images can identify the main object categories and semantic labels in the scene, such as "table," "chair," and "water dispenser," and extract their spatial location features and confidence information. The system writes the recognition results into a structured visual semantic database, with each entry containing fields such as object name, category, recognition confidence, image coordinates, depth statistics, robot pose during scanning, and the target's global absolute angle. Through this step, the system can form a sparse semantic memory of the environment without constructing a global geometric map, providing semantic priors for subsequent task execution.

[0109] When a user issues a task command in natural language, such as "go to the red chair" or "move to the printer," the main control module calls a Large Language Model (LLM) to perform semantic parsing of the statement, extracting the target description and behavioral intent. Subsequently, the system searches the visual semantic database for the entry most similar to the target semantically and determines the target object's identifier based on the matching result. If the matched target lacks valid 3D spatial coordinate information, the system triggers an "on-demand 3D coordinate learning" mechanism. At this time, the robot automatically returns to the initial scanning origin to eliminate coordinate deviations caused by odometry drift and adjusts its orientation according to the target's absolute angle stored in memory, entering the forward exploration phase. During exploration, the VLM continuously detects the camera feed. When a high-confidence bounding box of a target is detected, the system simultaneously reads depth data, calculates the 3D coordinates of the target's center point, and fuses this information with the current pose to obtain the target's precise position in the global coordinate system. After calculation, the system writes the coordinate results into the visual semantic database, achieving real-time completion of the target's spatial information.

[0110] After completing coordinate learning, the robot enters the navigation execution phase. The system dynamically selects the control mode based on the Euclidean distance between the target and the current pose. When the target is far away, the system enters a local-to-global path planning mode, using A* to plan a feasible path and execute travel control. In this mode, the system corrects the planned endpoint using a LiDAR-based safe target point search algorithm, ensuring that there are no obstacles around the target point and that the safe radius constraint is met, thus avoiding navigation failure due to obstruction at the path termination point. When the robot approaches the target and the distance is less than a preset threshold (e.g., 1.2 meters), the system automatically switches to visual servo mode. In this mode, the robot initializes the CSRT tracker based on the VLM recognition results, adjusts the angular velocity by calculating the pixel deviation between the target center and the image center, and adjusts the linear velocity based on the target depth value to achieve smooth and continuous approach control. To avoid cumulative errors caused by long-term tracking, the system periodically triggers VLM re-detection at fixed time intervals or distances to calibrate the tracking results and maintain navigation accuracy.

[0111] To enhance stability in complex and dynamic environments, this application also designs a multi-layered robust recovery mechanism. When path planning is interfered with by temporary obstacles, the system triggers a local detour strategy, dynamically generating a temporary obstacle avoidance path through a local cost graph, bypassing the obstacle, and then resuming the main task trajectory. This multi-layered mechanism effectively ensures the continuity and safety of task execution.

[0112] In practical applications, the system of this invention can achieve plug-and-play deployment without the need for pre-built maps. The following describes a complete process from environmental perception to navigation, performed using the method of this application, in an office environment that the robot has never entered before.

[0113] Step 1: Environmental Perception and Visual Memory Construction. Upon powering on the robot, the main control module detects the absence of a local "visual memory" file, enters standby mode, and prompts the user via voice or interface: "Environment unknown, please issue a scan command to perform environmental perception." The user inputs a natural language command: "Scan the office, find all chairs and tables." The main control module receives the command and activates the environmental perception module. The environmental perception module records the robot's current odometry pose (e.g., ...). The robot's chassis, centered at 0°, 60°, 120°, 180°, 240°, and 300°, serves as the "scanning origin" for this task and is stored in the visual semantic database to be created. The perception module controls the robot chassis to rotate sequentially from its original position to 0°, 60°, 120°, 180°, 240°, and 300°. When rotating to 60°, the robot acquires a frame of RGB-D image. The image contains a red office chair and a wooden desk. This RGB image is sent to the VLM (Virtual Machine Library), which, according to the instruction "scan the office, find all chairs and desks," returns structured JSON data. The other object is processed similarly. The perception module assigns IDs (target_001, target_002) to these two objects, crops and saves reference images based on their bounding boxes, and calculates their initial absolute angles by combining the robot's 60° orientation and the pixel positions of the objects in the image. At this point, the 3D coordinate fields in the memory entries for these two targets are empty. This process is repeated at all scanning angles. Finally, the perception module integrates all discovered targets, generates a complete visual semantic database file, and saves it. The environmental perception phase is complete.

[0114] Step 2: Navigation Task and On-Demand 3D Coordinate Learning. The robot prompts: "Environmental perception complete, please issue navigation commands." The user inputs: "Go to that red office chair." The main control module activates the hybrid navigation module, which calls the LLM (Local Language Management) to analyze the command "go to the wooden door" along with the content of the visual semantic database. The LLM accurately returns the ID of target_001. The navigation module checks the memory entry for target_001. If its 3D coordinates are empty, it triggers the on-demand 3D coordinate learning process. The robot first moves back to the "scan origin" pose it has memorized. The robot then precisely aligns its front end with the target based on the initial absolute angle recorded by target_001. The robot moves in a straight line at a speed of 0.1 m / s, while continuously sending camera images to the VLM for rapid matching. After moving 1.5 meters, the VLM successfully locates the red office chair in the current field of view. The robot immediately stops and, based on the current depth image, calculates the distance between the chair and the robot to be 2.8 meters. This is combined with the robot's current global pose (e.g., ...). The distance between the chair and the 2.8-meter distance was used to calculate the global coordinates of the chair using the 3D coordinate solving unit. The coordinates are updated in the memory entry for target_001, and the entire visual memory file is saved. Coordinate learning is complete.

[0115] Step 3: Hybrid navigation execution. The navigation module calculates the robot's current position ( ) and target location ( The distance to the chair is approximately 2.4 meters, which exceeds the preset switching threshold of 1.2 meters, thus triggering a local-to-global planning mode. The intelligent target setting module calculates the desired stopping point 1.0 meter in front of the chair. The safe location search module scans this point using LiDAR, finds no obstacles, and issues this safe point as the navigation target for A*. The robot begins moving towards this safe point. When the straight-line distance between the robot and the chair decreases to 1.1 meters, the system automatically cancels the A* target and switches to visual servo mode. The system calls VLM to obtain the precise bounding box of the chair in the current frame and uses this box to initialize the CSRT visual tracker. The robot enters visual servo closed-loop control, adjusting its angular velocity in real time based on the bounding box position fed back by the tracker to keep the chair in the center of the field of view, while dynamically adjusting its linear velocity based on depth information to smoothly approach the chair.

[0116] When the robot reaches its final position, the system calls the re-identification function to perform near-range target verification. The specific process is as follows: Scene adaptation: Considering that only local features of the target (such as the surface and edge of the object) can be observed from a close-up perspective, the requirement for complete shape recognition is relaxed, and only more than 70% visual features (color, texture) need to be matched for confirmation.

[0117] Multimodal verification: If a reference image exists, compare the features of the reference image with the current local image to verify the color ratio (greater than 70%) and the local shape matching degree (greater than 80%); if there is no reference image, analyze whether there are visual features matching the target description (such as "red cloth" or "wood texture") in the central region of the current image.

[0118] Result judgment: If the verification confidence is greater than or equal to 0.5, the target is confirmed and the robot stops moving; if the confidence is less than 0.5, VLM re-identification is triggered, and if it still fails, the task is judged as failed.

[0119] Table 1 shows the environmental perception and memory construction test results after 20 repeated trials in the simulation environment of the above embodiments, and Table 2 shows the comparative test results of the hybrid navigation strategy of this application. Experiments show that the system can accurately identify semantic targets and complete natural language navigation tasks in home, office, and warehouse environments. Compared with traditional SLAM-based navigation systems, this application shows significant advantages in deployment time, task success rate, and energy consumption. For example, under the same conditions, the average navigation success rate of the system of this invention exceeds 95%, the average path planning time is reduced by about 40%, and it can effectively avoid positioning failures caused by map errors. In summary, the embodiments of this invention not only have good engineering feasibility and scalability, but also provide an efficient, stable, and interpretable technical approach for service robots and intelligent agents to achieve semantic understanding and autonomous navigation in map-free conditions.

[0120] Table 1. Results of Environmental Perception and Memory Construction Tests

[0121] Table 2 Comparison Test Results of Hybrid Navigation Strategies

[0122] This application proposes a real-time autonomous navigation system and method for robots that does not rely on a global map and integrates multimodal perception and hybrid decision-making. Compared with existing technologies, it has the following significant advantages and beneficial effects: 1. Significantly reduces reliance on pre-built maps and enhances environmental adaptability: Compared to traditional navigation technologies that rely on SLAM mapping, this application eliminates the need for pre-building or real-time maintenance of high-precision global geometric maps. This fundamentally solves the problems of map failure and high update costs in dynamic, complex, or large-scale environments. The robot can immediately perform tasks in unknown environments after being unpacked, exhibiting extremely high adaptability to environmental changes.

[0123] 2. Achieving truly natural language interaction and enhancing intelligence: This application integrates a large-scale language model (LLM) and a visual language model (VLM) to enable robots to understand and execute complex, richly descriptive natural language commands. Compared to traditional interaction methods that require users to specify precise coordinates or click on targets on a map, this is an improvement in usability, lowering the barrier to entry and making human-computer interaction as natural and intuitive as communication between people.

[0124] 3. Efficient "on-demand" information acquisition, reducing resource consumption: This application uses "visual memory" instead of dense geometric maps, recording only the semantic, visual, and sparse location information of key targets. Its core "on-demand 3D coordinate learning" mechanism actively acquires the precise spatial coordinates of targets only when the navigation task actually requires it, avoiding the enormous computational and storage overhead of 3D reconstruction of the entire environment. This strategy improves the system's operating efficiency and its friendliness to hardware resources.

[0125] 4. Multi-layered robust design ensures navigation success rate and safety: This application integrates multi-layered navigation and recovery strategies from global to local perspectives. The hybrid navigation mode balances the efficiency of long-distance movement with the accuracy of close-range approach; the LiDAR-based safe target point selection mechanism effectively avoids collisions near the target point; the periodic visual language model calibration and multi-level recovery process after tracking loss in visual servoing ensure that the robot still has a high probability of completing the task even when vision is interfered with.

[0126] 5. Strong interpretability of system decisions: Compared to some end-to-end, "black box" non-graphical navigation methods, the decision-making process of this application is clear and interpretable. The content of the visual semantic database is intuitive and easy to understand, and the state of the robot at each stage from perception, understanding, learning to execution is clear, which facilitates developers to debug, optimize and expand functions.

[0127] 6. Achieving navigation effect with simultaneous planning and visibility: The A* path planning method with view-preserving constraints proposed in this application introduces a penalty term for the view axis-target angle in the cost function, enabling the path search process to actively select nodes that can keep the target within the field of view. This method achieves a "planning-while-visible" navigation effect, directly outputting a highly visible path without post-processing or path smoothing, significantly improving the stability and reliability of target tracking during navigation.

[0128] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the aforementioned real-time imageless robot autonomous navigation system embodiments. The storage medium may be, for example, ROM / RAM, magnetic disk, optical disk, etc.

[0129] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various steps of the aforementioned application embodiments and can achieve the same beneficial effects as the aforementioned real-time mapless robot autonomous navigation system embodiments. To avoid repetition, they will not be described again here.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0132] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A real-time, map-free autonomous navigation system for robots, characterized in that, The system includes a main control module, an environment perception module, and a hybrid navigation module. The main control module is responsible for human-computer interaction, task flow control, and coordination with the environment perception module and the hybrid navigation module, and stores and manages the visual semantic database. The environment perception module is responsible for controlling the robot to collect image and depth data without prior maps, identifying objects in the scene and extracting their semantic information, and integrating the recognition results into the visual semantic database. The hybrid navigation module uses a large language model to parse the instructions issued by the main control module, matches them with visual memory, determines the navigation target, and completes the robot's autonomous navigation through a hybrid navigation strategy.

2. A real-time, map-free autonomous navigation method for robots, characterized in that, Includes the following steps: The main control module determines the storage status of the visual memory. If the visual memory is empty, the environment perception module performs a panoramic scan and visual language model recognition, constructs the visual memory, stores it in a structured manner, and then returns it to the main control module. If visual memory exists, the main control module waits for the user to input natural language commands, and then parses and semantically matches the commands using a large language model; The hybrid navigation module checks the target coordinates to confirm the storage status of the three-dimensional coordinates. If the three-dimensional target is missing, it initiates an on-demand three-dimensional coordinate learning process, which learns the missing three-dimensional coordinates by returning to the origin and re-identifying the target. If the three-dimensional coordinates exist or the on-demand three-dimensional coordinate learning process is completed, the three-dimensional coordinates are input into the hybrid navigation module and a preset mode is selected for the navigation task. Once the hybrid navigation module completes navigation, it updates the visual memory and transmits it to the main control module, awaiting the next instruction.

3. The real-time mapless robot autonomous navigation method according to claim 2, characterized in that, The environment perception module completes environment perception and constructs a visual semantic database through an environment perception and memory construction method based on a visual language model, including the following steps: Preset a set of scanning angles The robot starts from its initial position and rotates to each angle in sequence. And acquire RGB images and depth images Simultaneously record the robot's current pose in the ODOM coordinate system. ,in This is the global orientation angle; After the robot comes to a stable stop at each angle, it simultaneously acquires the current RGB color image and depth image, and records the robot's coordinates and precise orientation angle in the odom odometry coordinate system at this time; The acquired RGB image is encoded to obtain an encoded RGB image. The encoded RGB image and the scan command text input by the user are used as input, and after processing by the visual language model, JSON data is output. Each detected object is post-processed to generate a visual memory entry, and a unique identifier is assigned to each newly discovered object. By combining the bounding box (bbox) and the synchronously acquired depth image, the effective average depth value within the object region is calculated. Based on the center pixel coordinates of the object in the image, camera intrinsic parameters, average depth value, and global orientation angle when the robot acquires the image, the absolute angle of the object relative to the robot's initial scanning origin is calculated. A close-up image of the object is obtained by cropping it from the original RGB image based on the bounding box (bbox) and then saved. All object entries detected from all scanning angles are aggregated to form a dictionary or hash table structure with the target ID as the primary key, and a visual semantic database is constructed. The visual semantic database additionally stores the robot's origin pose during this scan.

4. The real-time mapless robot autonomous navigation method according to claim 3, characterized in that, When the system receives the scanning instruction text input by the user, it dynamically constructs a Prompt instruction that includes a world knowledge instruction and a user task instruction. The world knowledge instruction serializes the current complete visual memory JSON object into a string, which is provided as a world model or context to the large language model. The user task instruction takes the user's original natural language instruction as the task to be executed.

5. The real-time mapless robot autonomous navigation method according to claim 4, characterized in that, The system's Prompt instruction guides the visual language model to complete the following steps: Identify all relevant objects in the image based on the Prompt command; The visual language model returns a list of multiple objects in JSON format. The list includes the object name, object category, pixel bounding box of the object in the 2D image, the model's confidence in the detection result, and a natural language description. The natural language description includes the object's color, material, state, and positional relationship with the surrounding environment. For each object and its bounding box identified by the visual language model Calculate the coordinates of the center pixel of the image. : Combined with camera intrinsic parameters, the camera intrinsic parameters are focal length. He Guangxin Calculate the pixel offset between the center point of the object and the optical center of the camera: The camera's horizontal field of view (HFOV) is: in, This refers to the horizontal resolution of the image sensor. Focal length A general term; Calculate the camera's angular resolution: Calculate the horizontal deflection angle of the object relative to the camera's optical center: Finally, the global absolute angle of the object relative to the scan origin is: Among them, the global absolute angle The key index for the robot to reorient to the target in the on-demand 3D coordinate learning method is stored in the visual memory entry.

6. The real-time mapless robot autonomous navigation method according to claim 5, characterized in that, The specific steps of the on-demand 3D coordinate learning method are as follows: When the navigation task begins, the system checks whether the identified navigation target has a valid position_3d field, i.e., three-dimensional coordinates. If the field or the three-dimensional coordinates do not exist, the coordinate learning process is activated. The coordinate learning process is only triggered when the semantic target entry is used for navigation for the first time and its three-dimensional coordinates are missing, and coordinate completion is performed once or multiple times. The robot returns to the scanning origin based on its robot origin pose from the visual semantic database, and based on the absolute angle from the visual semantic database... Orienting itself back towards the target, the system relocates the target within the field of view using a visual language model and obtains its bounding box (bbox). Pixel coordinates are then calculated using a center point method. ), complete the redirection; The robot initiates a slow forward exploration process with visual confirmation as the ultimate goal. During the exploration process, real-time visual input is continuously fed into the visual language model for matching. When the confidence level of the confirmed target is higher than a preset threshold, the exploration process is terminated, and the newly obtained coordinate data is stored in the corresponding target entry in the visual semantic database.

7. The real-time mapless robot autonomous navigation method according to claim 6, characterized in that, The visual language model successfully relocalizes the target in the field of view and returns its bounding box. The system atomically captures the complete state at the current moment through a state snapshot mechanism. The complete state includes the RGB image, depth image, robot pose, and orientation. The following steps are specifically executed: The effective depth values ​​within the bbox_new region are extracted from the synchronized depth image, and a robust filtering strategy is used to suppress noise interference. Solve the camera coordinate system based on camera intrinsic parameters. The target's coordinates in the camera coordinate system are calculated using the following formula. : Set the fixed transformation as a rotation matrix. With translation vector Transform the target point coordinates to the chassis coordinate system : In a typical forward-mount configuration, the fixed transformation simplifies to: in This represents the camera's mounting offset relative to the chassis. The system combines the robot's current global pose. The global coordinates of the target in the ODOM coordinate system are calculated using the following formula ( ): Calculated ( The three-dimensional coordinates of the target object in the global map are then updated to the visual semantic database.

8. The real-time mapless robot autonomous navigation method according to claim 7, characterized in that, The hybrid navigation module includes the following specific processes: After the navigation task begins and the robot faces the target, the system first estimates the straight-line distance to the target. If the target has global 3D coordinates, the Euclidean distance is calculated directly; if the target does not have global 3D coordinates but is within the field of view, depth information is used; if the current Euclidean distance between the robot and the target is... If the distance between the robot and the target exceeds a preset threshold, the system enters the local / global path planning mode in step S302; if the current Euclidean distance between the robot and the target is greater than a preset threshold, the system enters the local / global path planning mode in step S302. If the value is less than a preset threshold, the system enters the visual servo mode of S303. Enter the local-to-global path planning mode, and use the local-to-global path planner to achieve medium-to-long-distance movement and obstacle avoidance; Entering visual servo mode, control laws are designed under the state snapshot atomic capture mechanism to achieve close-range movement and obstacle avoidance; The hybrid navigation module also has a robustness guarantee mechanism to ensure that the robot completes the autonomous navigation task. The robustness guarantee mechanism includes a dynamic obstacle avoidance mechanism in visual servo mode, a path recovery mechanism in local and global navigation mode, and a multi-level recovery process for visual tracking loss.

9. The real-time mapless robot autonomous navigation method according to claim 8, characterized in that, The local-to-global path planning mode includes the following specific steps: The system uses the A* algorithm with a view-keeping constraint to ensure that the target object remains within the robot's field of vision during navigation; The target pose update unit is activated. The unit locates the target within the current field of view using a visual language model and calculates the angle that the robot needs to face the target based on the pixel offset of the target in the image. The module calculates the desired stopping point in front of the object and uses real-time LiDAR data to check the safety of the desired stopping point and its surrounding area. If there are obstacles near the initial point, the module will search for an unobstructed and safest alternative point within a preset angle and distance range, centered on the object, as the final navigation target. The system calculates an unobstructed parking spot in front of the target object using a LiDAR-based safe target point search algorithm. ; The robot's current pose can be obtained in real time from the robot chassis odometry or TF transformation. The planner receives the robot's current pose. and accessible parking spots Then, an A* search method with view constraints is initiated on the local cost map, such that the current node... equal to parking spot If the search is successful, the planned path is complete. By tracing back from the endpoint to the starting point, a map coordinate sequence is established. Each map coordinate point in the map coordinate sequence is then converted back to world coordinates to obtain a map coordinate sequence in the world coordinate system. The system adopts a look-ahead point tracking strategy, which finds a point on the path that is a preset look-ahead distance away from the robot's current position as the tracking target point, calculates the angular error toward the tracking target point, and generates angular velocity commands through proportional control. The A* search method with vision constraints includes the following steps: The new A* algorithm cost function is obtained by improving the cost function of the standard A* algorithm: in, From the starting point to the node The actual cost; For the node The estimated cost to reach the target; To maintain vision, a penalty item is imposed. This is a penalty weighting coefficient used to balance the priority of path length and view preservation; For each candidate node on the path Calculate from node Direction vector pointing to the target point and from node The direction vector pointing to the robot's current orientation The angle between the line of sight and the target is obtained. ; Penalty for maintaining field of vision A piecewise function design is adopted: in, To maintain a threshold angle for the field of view, This is the penalty coefficient, used to control the intensity of the penalty.

10. The real-time mapless robot autonomous navigation method according to any one of claims 2-9, characterized in that, The visual servoing mode includes the following specific steps: The target is detected in the current field of view by using a visual language model, and its initial bounding box (bbox) is obtained. A high-performance real-time visual tracker (CSRT) is initialized using the bounding box (bbox). Under the atomic capture mechanism of state snapshot, angular velocity control laws and linear velocity control laws are designed respectively; the angular velocity control law can ensure that the robot continuously faces the target by controlling the horizontal pixel error. and angular velocity A closed-loop negative feedback system is formed; the linear velocity control law, based on the distance to the target, adopts a segmented control strategy to safely drive the robot forward or stop; The real-time vision tracker CSRT returns the current bounding box. Calculate the pixel coordinates of its center point ( ; and the center point of the image The horizontal pixel error is obtained through comparison: Angular velocity control quantity Generated by proportional control law: in, The proportional gain is used to normalize the error to the [-1,1] interval, thereby achieving smooth steering control; The linear velocity control amount is set segment by segment according to the target distance. : in, The average depth value of the bounding box provided for the real-time vision tracker CSRT.

Images