Intelligent body target navigation system based on two-dimensional semantic map clicking strategy

Abstract

The invention discloses an agent target navigation system based on a two-dimensional semantic map click strategy, and the system comprises a map construction and management module which employs a laser radar SLAM to construct a two-dimensional map, and achieves the real-time camera pose estimation and map updating through a CUDA acceleration library cuVSLAM; the point selection module extracts space-semantic joint features in the environment map through an image text cross attention mechanism; the path planning module performs global path planning based on the two-dimensional grid map; and the coordinate calculation module fuses the internal reference matrix and the external reference attitude of the camera to carry out depth data and target world coordinate calculation. According to the method, efficient and real-time intelligent agent navigation can be realized, environmental perception and semantic understanding are enhanced through the semantic map, the path search efficiency is optimized, the adaptability and accuracy of target detection are improved, and the precision of coordinate calculation is ensured, so that the reliability, flexibility and overall performance of a navigation system are comprehensively improved.

Description

Technical Field

[0001] This invention relates to the field of robot autonomous navigation technology, and more specifically to an intelligent agent target navigation system based on a two-dimensional semantic map point selection strategy. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, quadruped robots have become a research hotspot in the field of autonomous exploration due to their adaptability and maneuverability in complex environments. However, traditional robot navigation systems have exposed a series of significant technical bottlenecks when dealing with dynamic or unknown scenarios. These systems often rely on single sensor data or outdated algorithm frameworks, leading to error accumulation and difficulties in real-time updates during environmental map construction, thus affecting the reliability of navigation. In terms of path planning, due to the lack of adaptive optimization mechanisms, traditional methods bear a heavy computational burden in complex terrain, easily producing inefficient or suboptimal routes, prolonging exploration time and increasing energy consumption. In addition, target recognition and localization functions are limited by the insufficient robustness of the perception module, with accuracy significantly decreasing under changes in lighting, occlusion, or interference from multiple objects, making it difficult to achieve accurate task execution. These shortcomings collectively highlight the limitations of current technologies in highly dynamic environments. Therefore, there is an urgent need for an autonomous exploration system that integrates advanced algorithms, can efficiently construct accurate environmental maps, achieve precise target recognition and localization, and support multimodal sensor data fusion, in order to improve the overall performance and adaptability of robots in unknown scenarios. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide an intelligent agent target navigation system based on a two-dimensional semantic map point selection strategy. By fusion of LiDAR and visual sensor data through multi-module collaboration, and by employing advanced algorithms to optimize map construction, path planning, and target localization, the system solves the problems of insufficient accuracy, poor real-time performance, and low robustness of traditional navigation systems in dynamic environments.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a map construction and management module, which uses LiDAR SLAM to construct a two-dimensional map and uses the CUDA acceleration library cuVSLAM to achieve real-time camera pose estimation and map updating; a point selection module, which extracts spatial-semantic joint features from the environmental map through an image-text cross-attention mechanism to construct a high-level semantic map representation; a path planning module, which performs global path planning based on a two-dimensional grid map, using the current point and the target point as input, and employing the A* algorithm as the basic path search tool to find the optimal path from the starting point to the destination in the discretized grid space; a target detection and recognition module, which combines visual input with text prompts to achieve real-time target detection and open vocabulary recognition; and a coordinate calculation module, which fuses the camera intrinsic parameter matrix and extrinsic pose parameter to calculate depth data and target world coordinates.

[0005] As a further improvement of the present invention, the specific steps of the map building and management module in building a two-dimensional map are as follows: Step 1, the LiDAR collects environmental point clouds at a fixed frequency, removes noise points through a filtering algorithm, and retains obstacle and free space features; the IMU provides angular velocity and acceleration data of the robot's motion for preliminary estimation of the motion state; Step 2, the two-dimensional pose is updated according to the robot's motion model; Step 3, the LiDAR point cloud is projected onto a two-dimensional plane, and the map is updated through the grid occupancy probability model: for each grid cell, based on the distance information of the current LiDAR point, its occupancy probability is updated using the Bayesian criterion, and finally a binary grid map is generated.

[0006] As a further improvement of the present invention, the map building and management module implements real-time camera pose estimation and map updating in the following specific way: First, the RGB-D camera acquires image sequences, extracts image feature points through ORB or SIFT algorithms, combines a FLANN matcher to achieve preliminary matching of feature points in adjacent frames, and filters out mismatches. Then, based on the matched feature points and the initial pose, the objective function is constructed as follows: in, Represents the projection function. For spatial points, For the first The point at the th The image coordinates in the frame are used to iteratively optimize the spatial points and camera pose using the objective function, minimizing projection error and eliminating cumulative drift. Finally, the pose estimation of LiDAR SLAM and the visual optimization results of cuVSLAM are fused through a Kalman filter to output the final map and pose, achieving real-time updates.

[0007] As a further improvement of the present invention, the point selection module constructs a high-level semantic map representation in the following specific way: First, using a pre-trained multimodal model, image features are aligned with task-related text prompts to generate candidate points with semantic guidance. Then, based on the feature map output by cross-attention, the system further extracts high-response regions as candidate points. After that, a graph neural network is used to model the candidate points, representing the exploration space as a graph structure. Finally, neighborhood information is aggregated through GNN to learn the embedding representation of each candidate point, and the exploration priority of each point is dynamically evaluated based on the attention mechanism.

[0008] As a further improvement of the present invention, the A* algorithm in the global path planning module is an improved A* algorithm based on visual Transformer. This algorithm extracts semantic features from the grid map to guide the path search process, thereby generating multiple candidate path points with different orientation features. Then, the TEB algorithm is used to optimize the selected path in real time, and dynamic obstacle avoidance is achieved by combining sensor feedback data, ensuring that the robot has good safety and smooth motion in actual operation.

[0009] As a further improvement of the present invention, the specific method by which the coordinate calculation module calculates the depth data and the target world coordinates is as follows: fusing the camera intrinsic parameter matrix. With external parameters attitude The system has a depth map Perform coordinate transformation on each pixel: Simultaneously, a Bayesian filter is introduced to compensate for errors in coordinate estimation: in, For the observed values, This is an estimate of the true coordinates.

[0010] The beneficial effects of this invention are as follows: navigation accuracy reaches over 95% in complex indoor environments, path planning latency is less than 100ms, ensuring smooth and safe robot movements. Target detection and tracking accuracy is improved under varying lighting and occlusion conditions, and dynamic tracking continuity is significantly enhanced. Depth camera synchronous calibration technology improves 3D target positioning accuracy to the centimeter level while effectively reducing synchronization errors and drift. Real-world applications demonstrate that the system has strong adaptability to complex environments and can operate stably for extended periods in multi-task, multi-target scenarios, greatly improving the intelligence level of indoor autonomous navigation and service robots. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to the given embodiments.

[0012] This embodiment of the intelligent agent target navigation system based on a two-dimensional semantic map point selection strategy includes a map building and management module, a point selection module, a path planning module, a target detection and recognition module, and a coordinate calculation module. The system constructs an environmental map by fusing LiDAR SLAM and visual sensor data, uses an improved A* algorithm for path planning, and combines multimodal target recognition to achieve accurate navigation. Traditional navigation systems rely on a single sensor, leading to large map building errors. This system uses a Kalman filter to fuse LiDAR and cuVSLAM visual data, controlling the pose estimation error within ±2cm. To address the inefficiency of path planning, visual Transformer semantic features are introduced to guide the search direction, improving path calculation efficiency by 3 times in complex scenes. For target recognition, an open vocabulary model is used, overcoming the limitation of traditional closed-set recognition in handling unknown objects.

[0013] Furthermore, the specific steps for the map building and management module to build a two-dimensional map are as follows: The lidar collects environmental point clouds at a fixed frequency, removes noise points through filtering algorithms, and retains obstacle and free space features; the IMU provides angular velocity and acceleration data of the robot's motion for preliminary estimation of motion state; The robot's state is updated based on its motion model in the SLAM process, updating the 2D pose. It is continuously updated in two-dimensional space, satisfying the following motion model: in, It is the input control quantity. If it is system noise, then the two-dimensional pose can be effectively updated through this motion model; The LiDAR point cloud is projected onto a two-dimensional plane, and the map is updated using an occupancy probability model: for each grid cell, based on the distance information of the current LiDAR point, its occupancy probability is updated using a Bayesian criterion, ultimately generating a binary grid map. This step, through step-by-step filtering and probability update mechanisms, improves the signal-to-noise ratio of the map construction and reduces false obstacle misjudgments compared to the traditional direct projection method.

[0014] Furthermore, the map building and management module implements real-time camera pose estimation and map updating in the following specific way: First, the RGB-D camera acquires image sequences, extracts image feature points using ORB or SIFT algorithms, combines a FLANN matcher to achieve preliminary matching of feature points in adjacent frames, and filters out mismatches. Then, based on the matched feature points and the initial pose, the objective function is constructed as follows: in, Represents the projection function. For spatial points, For the first The point at the th The image coordinates in the frame are used to iteratively optimize the spatial points and camera pose using the objective function, minimizing projection error and eliminating cumulative drift. Finally, the pose estimation of LiDAR SLAM and the visual optimization results of cuVSLAM are fused through a Kalman filter to output the final map and pose, achieving real-time updates.

[0015] Furthermore, the point selection module constructs a high-level semantic map representation as follows: First, using a pre-trained multimodal model, image features are aligned with task-related text prompts to generate semantically guided candidate points. Then, based on the feature map output by cross-attention, the system further extracts high-response regions as candidate points. These points not only contain spatial location information but also carry semantic context (such as "target area," "obstacle avoidance area," etc.). Finally, a graph neural network (GNN) is used to model the candidate points, representing the exploration space as a graph structure. By aggregating neighborhood information through GNN, the embedding representation of each candidate point is learned. And dynamically evaluate the exploration priority of each point based on the attention mechanism. This module innovatively combines language understanding with spatial reasoning, enabling target point selection accuracy to remain high even in occluded scenarios, surpassing traditional geometric feature-based methods.

[0016] Furthermore, during the global path planning process, the A* algorithm is an improved version of the visual Transformer. This algorithm extracts semantic features from the raster map to guide the path search process, thereby generating multiple candidate path points with different orientation features. in It is an input two-dimensional raster map image. These are semantic feature vectors or feature maps extracted from maps. ViT-A* incorporates semantic information to modify or enhance traditional heuristic functions. .

[0017] This can be learned; for example, the output of ViT can be a predicted "semantic cost map" representing the "preference" or "penalty" for navigation in certain regions. An improved A* algorithm based on the visual Transformer is employed to further enhance path diversity and adaptability to complex environments. Subsequently, the TEB algorithm is used for real-time trajectory optimization of the selected path, combined with sensor feedback data to achieve dynamic obstacle avoidance, ensuring good safety and smooth motion during actual operation. The TEB optimization objective is a least-squares problem aimed at minimizing the sum of multiple cost terms.

[0018] in It is the weight of each cost item. These are different cost functions. The cost functions are as follows: Path tracking cost: Penalty for trajectory deviation from the reference path.

[0019] in It is a trajectory point To reference path The minimum distance.

[0020] Obstacle avoidance cost: Penalty for the trajectory point being too close to the obstacle.

[0021] in It is the first An obstacle, It is a trajectory point Obstacles The distance. It is a penalty function that increases sharply when the distance is less than the safety threshold.

[0022] Furthermore, the camera intrinsic parameter matrix is ​​fused. With external parameters attitude The system has a depth map Perform coordinate transformation on each pixel: Simultaneously, a Bayesian filter is introduced to compensate for errors in coordinate estimation: in For the observed values, For accurate coordinate estimation, a closed-loop feedback mechanism is used to correct cumulative drift, ultimately achieving millimeter-level 3D target positioning accuracy. This computational framework, through camera parameter calibration compensation and temporal filtering, controls the target positioning error to within ±3mm, providing a precise coordinate reference for subsequent robotic arm operations.

[0023] In addition, the system integrates LiDAR, vision, and IMU data to improve point cloud stability and environmental matching, providing high-precision world coordinate support for path planning.

[0024] The system in this embodiment achieves complete closed-loop control from environmental perception to target recognition, path planning, and autonomous navigation. The intelligent agent, based on the understanding capabilities of a vision-language model, can comprehend high-level semantic instructions (such as "find the red water glass in the kitchen") in complex and unknown environments, and autonomously complete map construction, key waypoint selection, path planning, and precise target localization.

[0025] This embodiment provides the following examples: Example 1: Facility Inspection A factory needs to conduct regular equipment inspections to ensure normal operation. In this scenario, an intelligent agent autonomously moves to construct a map of equipment distribution and plans inspection routes covering all critical equipment based on the map. Upon reaching each equipment point, the agent rotates 360° to collect environmental data and uses Qwen or InternVL models combined with preset prompts (such as "detect temperature sensors" and "identify leak points") to monitor equipment status in real time. The system accurately locates anomalies using a 3D coordinate calculation module and generates an inspection report. Ultimately, the system can automatically complete inspection tasks, identify potential problems, and improve inspection efficiency and accuracy.

[0026] Example 2: An industrial manufacturing factory needs to automate the handling of goods to reduce human intervention. An intelligent agent builds a map within the factory, identifies work areas and equipment locations, and plans the optimal path based on received task instructions. After navigating to the designated location, the agent uses a Qwen or InternVL model combined with preset prompts (such as "find the box numbered A123" or "count the number of items on the shelf") to identify and locate the target object. Simultaneously, the system supports collaborative work with other robots or workers to complete production tasks. This solution significantly improves handling efficiency and enhances the intelligence and safety of factory production.

[0027] These embodiments demonstrate the effectiveness and broad adaptability of the proposed intelligent agent target navigation system and method based on a two-dimensional semantic map point selection strategy in various practical application scenarios. By integrating LiDAR SLAM, a visual language model (VLM), and advanced path planning and target detection technologies, the system can achieve high-precision map construction, semantic enhancement, target recognition, and 3D localization in unknown or complex environments, and complete efficient navigation by combining a robust motion control strategy.

[0028] In summary, this invention provides a two-dimensional semantic map navigation system integrating LiDAR and vision fusion. Through multimodal sensor data fusion, Transformer semantic-guided path planning, and Bayesian filtering coordinate optimization, it solves the problems of insufficient accuracy, poor real-time performance, and low robustness of traditional navigation systems in dynamic environments. In practical applications, the system achieves centimeter-level positioning accuracy, second-level path planning response, and high target recognition accuracy. It can be widely used in complex scenarios such as service robots and industrial inspection, significantly improving the autonomous navigation capabilities of intelligent agents.

[0029] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An agent target navigation system based on two-dimensional semantic map point selection strategy, characterized in that: Comprise: A map construction and management module that constructs a two-dimensional map using laser radar SLAM and realizes real-time camera pose estimation and map updating through the CUDA acceleration library cuVSLAM; A point selection module that extracts spatial-semantic joint features in the environment map through an image-text cross-attention mechanism, and constructs a high-level semantic map representation; A path planning module that performs global path planning based on a two-dimensional grid map, takes the current point and the target point as input, and uses the A* algorithm as a basic path search tool to find the optimal path from the starting point to the ending point in the discretized grid space; A target detection and recognition module that combines visual input with text prompts to realize real-time target detection and open-vocabulary recognition; A coordinate calculation module that fuses the camera intrinsic matrix and the external parameter pose to calculate depth data and target world coordinates. 2.The agent goal navigation system based on two-dimensional semantic map point selection strategy according to claim 1, characterized in that: The specific steps for the map construction and management module to construct a two-dimensional map are as follows: Step one: The laser radar collects environmental point cloud at a fixed frequency, removes noise points through filtering algorithm, and retains obstacle and free space features; IMU provides angular velocity and acceleration data of robot motion for preliminary estimation of motion state; Step two: Update the two-dimensional pose according to the motion model of the robot; Step three: Project the laser radar point cloud to the two-dimensional plane, and update the map through the occupancy grid probability model: for each grid cell, update its occupancy probability based on the distance information of the current laser point using the Bayesian criterion, and finally generate a binary grid map. 3.The agent goal navigation system based on two-dimensional semantic map point selection strategy according to claim 1, characterized in that: The specific way for the map construction and management module to realize real-time camera pose estimation and map updating is as follows: First, the RGB-D camera collects image sequences, extracts image feature points through ORB or SIFT algorithm, realizes preliminary matching of adjacent frame feature points combined with FLANN matcher, filters out mismatched points, and then based on the matched feature points and the initial pose, constructs the target function as follows: in, Represents the projection function. For spatial points, For the first The point at the th The image coordinates in the frame are used to iteratively optimize the spatial points and camera pose using the objective function, minimizing projection error and eliminating cumulative drift. Finally, the pose estimation of LiDAR SLAM and the visual optimization results of cuVSLAM are fused through a Kalman filter to output the final map and pose, achieving real-time updates.

4. The agent goal navigation system based on two-dimensional semantic map point selection strategy according to any one of claims 1 to 3, characterized in that: The specific way for the point selection module to construct a high-level semantic map representation is as follows: First, use the pre-trained multi-modal model to align the image features with the task-related text prompts, thereby generating candidate points with semantic guidance, then based on the feature map output by the cross-attention, the system further extracts high-response regions as candidate points, then uses graph neural network to model the candidate points, represents the exploration space as a graph structure, finally aggregates neighborhood information through GNN, learns the embedding representation of each candidate point, and dynamically evaluates the exploration priority of each point based on the attention mechanism.

5. The agent goal navigation system based on two-dimensional semantic map point selection strategy according to any one of claims 1 to 3, characterized in that: The A* algorithm in the process of global path planning of the path planning module is an improved A* algorithm based on visual Transformer, which extracts semantic features in the grid map to guide the path search process, thereby generating multiple candidate path points with different characteristics; then uses TEB algorithm to optimize the selected path in real time, combines sensor feedback data to realize dynamic obstacle avoidance, and ensures the safety and motion smoothness of the robot in actual operation.

6. The agent goal navigation system based on two-dimensional semantic map point selection strategy according to any one of claims 1 to 3, characterized in that: The specific manner in which the coordinate calculation module calculates the depth data and the target world coordinates is as follows: fusing the camera intrinsic matrix and the extrinsic pose , the system performs coordinate transformation on each pixel point in the depth map : At the same time, the Bayesian filter is introduced to compensate the error of coordinate estimation: wherein, is the observed value, is the true coordinate estimate.

Images