Robot guidance and control integrated system for adaptation to unfamiliar environment

Abstract

The present disclosure provides a robot sensing and control integrated system for unfamiliar environment adaptation, belonging to the field of intelligent unmanned systems. Each robot in the system includes a sensing device, a perception module, a navigation module and a control module. The perception module is a VLM-based semantic mapper, which performs ground object segmentation and semantic recognition on environmental information, generates a global semantic map and a ground image embedding semantic descriptors of ground object image; the navigation module is a LLM-based semantic planner, which uses LLM to understand the task description of natural language, and infers the task sequence of the current environment; the control module is a VFM-based terrain adaptive controller, which uses VFM to identify terrain features from the ground image, and maps the terrain features and the real-time state of the robot into a driving matrix to realize robot motion control. The present invention realizes the deep collaboration of active perception, autonomous exploration and efficient control of the robot, and enhances the task planning and real-time adjustment capability of the robot in unfamiliar environment.

Description

TECHNICAL FIELD

[0001] The present application relates to mobile robot advanced perception, autonomous task planning, navigation and terrain adaptation control technology, belongs to the field of intelligent unmanned system, and specifically relates to a robot perception, guidance and control integrated system for adapting to unfamiliar environment, which can be used for indoor service, warehouse logistics, search and rescue inspection and the like. BACKGROUND

[0002] With the development of mobile robot technology, more and more robots are deployed in large conference venues, or in outdoor search and rescue inspection and the like. A traditional autonomous mobile robot system uses relatively separated perception, navigation and control modules to achieve the established task requirements. In various application examples, it is found that the autonomous mobile robot system limited by the traditional framework usually needs to design a customized solution for a specific task, and cannot accept diversified and abstract task requirements; meanwhile, when facing an unstructured, terrain-changing, unknown or partially known environment, the perception and control system of the robot is greatly affected, and the task completion effect is poor.

[0003] In the traditional autonomous mobile robot system framework, the perception module provides an environment map and robot positioning for the navigation module. Although a semantic map can provide high-level information of surrounding objects for the robot, the utilization of the navigation module for the information is usually limited to avoiding dynamic and static obstacles, and cannot have a more favorable upper influence on real-time path planning in the task of autonomous exploration in an unfamiliar environment.

[0004] With the rapid development of the field of machine learning and the continuous improvement of hardware computing power, the application of large language models (LLMs) in robot navigation systems has gradually become a new research direction. Natural language has become an effective means of assigning tasks, and LLM-based planners are widely used in multiple fields, including mobile operations, service robots, autonomous driving, navigation, and fault detection. The current mainstream method configures LLMs through contextual cues, enabling them to apply common sense to specific domains without fine-tuning the model. At the task level, existing research has mostly focused on explicit task descriptions, with some teams working on developing LLM planners that can convert task descriptions into formal languages such as linear temporal logic (LTL) or planning domain definition language (PDDL). While formal languages can clearly state subtasks and semantic relationships, they are quite complex. Other research has considered natural language task instructions that are not explicitly described (e.g., "get me something to drink"), but still assumes that the work environment is pre-mapped or has clear structured features. At the environmental level, some researchers have tried to reduce the dependence on pre-constructed semantic maps by introducing feedback from perception systems or specifying semantics at runtime. However, such perception methods are often limited to object detection and are typically only suitable for small rooms or closed environments that can utilize explicit hierarchical structures and natural environmental boundaries.

[0005] Overall, existing research mostly assumes the existence of pre-constructed maps or structured indoor environments, or focuses on specific tasks such as object search and target navigation. These assumptions often cannot be effectively applied in many real-world application scenarios, especially in outdoor environments or tasks where users only provide high-level demand descriptions.

[0006] In recent years, there have been successful examples of combining the perception and control modules of autonomous mobile robots. In terms of perception, using raw images as input, the Visual Foundation Model (VFM) obtained through self-supervised learning has the potential to learn general visual features when the amount of pre-training data is large enough. Through adaptation, fine-tuning, or further model building, it can help robots achieve specific tasks. This method has been applied in multiple tasks in the robotics field, including image semantic segmentation, traversability estimation, and manipulation tasks. Its key advantages lie in its robustness to changes in lighting and occlusion, as well as its good generalization ability across different images of the same scene. In terms of control, Vision-based Reinforcement Learning (VRL) has demonstrated the ability to control agents in simulated environments and has also achieved success in robot control in real-world environments. However, the strategies generated by VRL often lack interpretability and do not guarantee safety and robustness. To address this issue, some researchers have used Inverse Reinforcement Learning (IRL) to interpret terrain accessibility as a reward map, thereby enhancing the robot's understanding of the environment. Despite some progress in combining vision and reinforcement learning, how to incorporate appropriate terrain models into control strategies while ensuring theoretical feasibility and safety remains an urgent problem to be solved. Summary of the Invention

[0007] In view of this, the present invention provides an integrated robot sensing, guidance and control system that is adaptive to unfamiliar environments, which realizes deep collaboration between the robot's active perception, autonomous exploration and efficient control, and enhances the robot's task planning and real-time adjustment capabilities in unfamiliar environments.

[0008] To solve the above-mentioned technical problems, the present invention is implemented as follows.

[0009] A robot sensing, guidance, and control integrated system for adapting to unfamiliar environments, wherein each robot in the system includes a sensing device, a perception module, a navigation module, and a control module;

[0010] The perception module is a semantic mapper based on the Visual Language Model (VLM), used to segment environmental information and perform semantic recognition based on VLM to generate a global semantic map with embedded semantic descriptors of the ground features; the global semantic map is provided to the navigation module, and the segmented ground images are provided to the control module.

[0011] The navigation module is a semantic planner based on a large language model (LLM). It uses LLM to understand the input natural language task description and combines it with the global semantic map provided by the perception module to infer the task sequence in the current environment. The task sequence is continuously optimized as the robot moves and the global semantic map provided by the perception module is updated. The desired trajectory is generated based on the continuously optimized task sequence and sent to the control module.

[0012] The control module is a terrain adaptation controller based on the visual model VFM. VFM is used as a general model for terrain understanding to identify terrain features from the segmented ground image. The terrain features and the robot's real-time state are mapped into a driving matrix through a robot-specific terrain model. Based on the driving matrix and the desired trajectory, the robot's control input is generated to realize the robot's motion control.

[0013] Preferably, the perception module includes: a mapping module, a data association module, and a pose graph optimization module;

[0014] The mapping module acquires image and point cloud information of the environment from the sensing device, as well as the robot's acceleration information. Based on image segmentation algorithms and VLM, it generates local sub-maps that embed semantic descriptors of ground features. Local sub-maps will be continuously generated during the robot's movement.

[0015] The data association module receives local sub-maps sent by other robots through distributed communication and merges them into a global semantic map; it calculates the relative pose transformation matrix of each robot based on the global semantic map and provides it to the pose map optimization module.

[0016] The pose graph optimization module determines the global pose of each robot based on the odometry information obtained using acceleration information and the relative pose transformation matrix between robots.

[0017] Preferably, the mapping module includes an image segmentation module, a visual language module, a visual odometry calculation module, a local sub-map generation module, and a communication module;

[0018] The image segmentation module segments ground objects in the surrounding environment from the image based on the image information of the environment. The ground object image is sent to the visual language module, and the segmented ground image is sent to the control module.

[0019] The visual language module identifies the image of each ground object, generates a corresponding semantic descriptor, and sends it to the local sub-map generation module.

[0020] The visual odometry calculation module generates odometry information based on the image information of the environment and the acceleration information of the robot, and sends it to the local sub-map generation module.

[0021] The local sub-map generation module receives point cloud information of the environment, embeds the semantic descriptor into the point cloud, and combines it with the odometer information to generate a local sub-map.

[0022] The communication module sends the local sub-map of the robot to other robots through distributed communication.

[0023] Preferably, the sensing device includes a depth camera and an inertial measurement unit (IMU); the depth camera acquires image information and point cloud information of the environment; and the IMU acquires the robot's acceleration information.

[0024] Preferably, the global semantic map uses a list to represent the information of three-dimensional features, including the semantic information, centroid location, shape, and size of each feature object in the explored area.

[0025] Preferably, the navigation module includes a decision generation module, a decision verification module, and a path planning module;

[0026] The decision generation module includes an LLM (Limited Language Model) and a behavior library. The LLM analyzes the natural language task description given by the user, combines the global semantic map and the robot's global pose, and calls the preset atomic behaviors in the behavior library to generate a task sequence composed of atomic behaviors. The task sequence is passed to the decision verification module for verification. Atomic actions include three categories: navigation, active perception, and user interaction.

[0027] The decision verification module is used to verify the task sequence; for the verified task sequence, it determines the navigation target point and sends it to the path planning module.

[0028] The path planning module is used to plan the desired trajectory from the current location to the navigation target point and send it to the control module.

[0029] Preferably, the verification of the decision verification module includes semantic verification and spatial verification; the semantic verification checks whether the semantics of the task conform to the predetermined rules, and if there are semantic errors or unreachable targets, it feeds back to the LLM of the decision generation module for adjustment; the spatial verification adopts a frontier exploration approach to ensure that the spatial paths of all tasks are feasible.

[0030] Preferably, the control module includes a three-layer structure: a general terrain understanding layer implemented by VFM, a robot-specific terrain model layer implemented by neural network, and a real-time disturbance adaptation layer.

[0031] The VFM is trained offline based on an image set to achieve basic modeling for terrain recognition;

[0032] The neural network input information combines terrain features and the robot's real-time state; the neural network uses meta-learning technology and is trained based on fragmented driving data of the robot to capture unmodeled dynamics during the robot's interaction with the terrain.

[0033] The real-time perturbation adaptive layer adjusts the weights of the last layer of the deep neural network online. This is to compensate for dynamic perturbations that were not captured during offline training.

[0034] Preferably, the control module includes VFM, a neural network, and a composite adaptive controller;

[0035] The ground image segmented and generated by the perception module is input into the VFM, and the VFM outputs terrain features.

[0036] The terrain features and the current state fed back by the robot are concatenated and input into the neural network, which outputs a driving matrix K.

[0037] The weights of the last layer of the neural network The time-varying linear parameter vector is used for online adaptive control, and the time-varying linear parameter vector is transmitted to the composite adaptive controller together with the driving matrix K.

[0038] During online operation, the composite adaptive controller updates the time-varying linear parameter vector of the neural network in real time according to the set rules, and calculates the robot's control input at the current moment by combining the desired trajectory provided by the navigation module and the driving matrix K.

[0039] Preferably, the neural network is a lightweight deep neural network, including an input layer, two hidden layers and an output layer.

[0040] Beneficial effects:

[0041] (1) The perception module provided by this invention adopts a semantic mapper based on a visual language model (VLM), which realizes multimodal perception and semantic map construction of complex environments. The VLM does not require pre-training for the recognition of specific objects and is less affected by changes in illumination and viewing angle, thus exhibiting strong robustness and versatility. In a preferred embodiment, the semantic map adopts a list format, enabling the robot to efficiently store and transmit local semantic maps, and the low-dimensional data represented by the list facilitates the semantic understanding of the navigation module.

[0042] (2) The navigation module provided by this invention adopts an LLM-based semantic planner. In the prior art, the navigation module mainly performs path planning based on a given location point. However, the input to the navigation module in this invention is the operator's natural language task description. The semantic planner understands the task description with the help of LLM and has the ability to receive high-level tasks described in natural language. Combined with a real-time updated semantic map, it generates and verifies dynamic task sequences online through a recursive method, which significantly enhances the robot's task planning and real-time adjustment capabilities in unfamiliar environments.

[0043] (3) The control module provided by the present invention, through a VFM-based terrain adaptation controller, combined with deep neural networks and adaptive control, realizes real-time disturbance adaptation capability under complex terrain, effectively compensates for dynamic interactions not captured by the offline model, and improves the trajectory tracking performance of autonomous mobile robots under different terrains and dynamic disturbances.

[0044] (5) The control module adopts a three-layer structure: a general terrain understanding layer, a robot-specific terrain model layer, and a real-time disturbance adaptation layer. The general terrain understanding layer is pre-trained on a large-scale image set and can extract features with high semantic relevance from the terrain image. The robot-specific terrain model layer is trained on a small-scale dataset obtained based on robot driving and captures complex dynamics that are not modeled during the interaction between the robot and the terrain. The real-time disturbance adaptation layer adjusts the parameters of the last layer of the deep neural network online through adaptive control to compensate for dynamic disturbances that were not captured during offline training.

[0045] (6) This invention breaks through the limitations of traditional independent modular systems by sharing information and integrating the design of the sensing, navigation and control modules, and realizes deep collaboration of robot active perception, autonomous exploration and efficient control, providing a high-value solution for complex tasks such as disaster relief, on-site exploration and unmanned inspection. Attached Figure Description

[0046] Figure 1 This is a system framework diagram for an embodiment of the present invention.

[0047] Figure 2 This is a framework diagram of the sensing module in an embodiment of the present invention.

[0048] Figure 3 This is a flowchart of the sensing module in an embodiment of the present invention.

[0049] Figure 4 This is a framework diagram of the navigation module according to an embodiment of the present invention.

[0050] Figure 5 This is a flowchart of the navigation module in an embodiment of the present invention.

[0051] Figure 6This is a control module framework diagram according to an embodiment of the present invention.

[0052] Figure 7 This is a flowchart of the control module in an embodiment of the present invention. Detailed Implementation

[0053] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0054] This invention provides an integrated sensing, guidance, and control system for robots that adapt to unfamiliar environments, such as... Figure 1 As shown, the system includes multiple robots that can communicate wirelessly with each other. Each robot includes a sensing device, a perception module, a navigation module, and a control module.

[0055] In this embodiment, the sensing devices configured on the robot include a depth camera and an IMU. The depth camera is used to acquire image information and point cloud information, and the IMU is used to acquire acceleration and angular velocity information. In practice, radar can also be used to acquire point cloud information, and the depth camera can be an Intel RealSense D456 depth camera.

[0056] The perception module is primarily an open-set semantic mapper based on a visual language model (VLM). It uses a depth camera to acquire environmental information, performs feature segmentation and VLM-based semantic recognition, and generates a global semantic map embedded with semantic descriptors of the feature images. This global semantic map is provided to the navigation module, while the segmented ground images are provided to the control module.

[0057] The navigation module is primarily a semantic planner based on a Large Language Model (LLM). In existing technologies, navigation modules mainly plan paths based on given location points. However, this invention inputs the operator's natural language task description into the navigation module. The navigation module uses LLM to understand this task description and, combined with perceptual information provided by the perception module, infers the task sequence in the current environment. As the perceptual information is updated, a real-time, continuously optimized task sequence is generated. Based on the task sequence, a desired trajectory is generated and sent to the control module.

[0058] The control module is primarily a terrain adaptation controller based on a vision-based fundamental model (VFM). The control module identifies terrain features from the ground images segmented by the perception module; it uses a neural network to map the terrain features and the robot's real-time state to a drive matrix; and it generates the robot's control input based on the drive matrix and the desired trajectory provided by the navigation module, thus achieving motion control of the robot. A deep neural network (DNN) is preferably used as the neural network.

[0059] As can be seen, by combining semantic description, LLM and VFM, this invention enables the system to accept high-level tasks described by natural language, analyze and decompose task-related elements, actively perceive and autonomously explore in unfamiliar environments, and realize complex functions such as disaster relief, on-site exploration and unmanned inspection.

[0060] The following section provides a detailed description of the specific implementations of the perception module, navigation module, and control module.

[0061] (1) Perception module

[0062] The perception module is used to construct a global semantic map with inserted semantic descriptors based on high-level features detected by the open set image segmentation algorithm in conjunction with the visual language model (VLM).

[0063] In this embodiment, the input data of the sensing module includes RGB-D depth camera images (containing image information and point cloud information) and acceleration and angular velocity information provided by the inertial measurement unit (IMU).

[0064] See Figure 2 The perception module includes: a mapping module, a data association module, and a pose graph optimization module.

[0065] The mapping module acquires image and point cloud information of the environment from the sensing devices, as well as the robot's acceleration information. Based on image segmentation algorithms and VLM, it generates local sub-maps with embedded semantic descriptors of ground features. Local sub-maps will be continuously generated during the robot's movement.

[0066] The data association module receives local sub-maps sent by other robots through distributed communication; based on the consistency graph data association algorithm, combined with prior information such as gravity, it realizes the alignment of local sub-maps, and the aligned local sub-maps are merged into a global semantic map; based on the global semantic map, it calculates the current pose relative transformation matrix of each robot and provides it to the pose graph optimization module.

[0067] The pose graph optimization module determines the global pose of each robot based on the odometry information obtained using acceleration information and the relative pose transformation matrix between robots.

[0068] Finally, the perception module outputs a global semantic map and pose information for each robot. The global semantic map is preferably a list, including semantic information, centroid location, shape, and size attributes for each ground object within the explored area. This representation allows the robot to efficiently store and transmit local semantic maps, and the low-dimensional data represented by the list facilitates semantic understanding by the navigation module. This perception module does not require pre-training for object-specific recognition, exhibiting strong robustness and versatility.

[0069] In a preferred embodiment, the mapping module includes an image segmentation module, a visual language module, a visual odometry module, a local sub-map generation module, and a communication module. The image segmentation module segments ground features from an image based on environmental image information. Images of these ground features are sent to the visual language module, and the segmented ground images are sent to the control module. The visual language module identifies the image of each ground feature, generates a corresponding semantic descriptor, and sends it to the local sub-map generation module. The visual odometry module generates odometry information based on environmental image information and the robot's acceleration information, and sends it to the local sub-map generation module. The local sub-map generation module receives point cloud information from the environment, embeds the semantic descriptor into the point cloud, and combines it with the odometry information to generate a local sub-map. The communication module (not shown in the figure) sends the local sub-map of the robot to other robots via distributed communication. During robot operation, local sub-maps are continuously generated, and each robot receives local sub-maps sent by other robots through distributed communication.

[0070] Figure 3 The workflow of the sensing module is shown, such as... Figure 3 As shown, the specific steps are as follows:

[0071] Step 11: Receive RGB-D depth camera image input and IMU sensor input, and use the YOLOv7 algorithm to filter out dynamic obstacles in the surrounding environment. Generate odometry information using a visual odometry algorithm.

[0072] Step 12: Use the FastSAM image segmentation algorithm to segment the ground objects in the surrounding environment for semantic map construction, and send the segmented ground images to the control module.

[0073] Step 13: Use the BLIP algorithm as a visual language model to generate semantic descriptors for ground objects. The BLIP algorithm proposes a multimodal mixture architecture—MED (Multimodal mixture of Encoder-Decoder)—which combines the advantages of encoders and decoders, enabling simultaneous handling of understanding and generation tasks.

[0074] Step 14: Project the segmented ground objects into the point cloud, and merge the inter-frame data by calculating and comparing the intersection over union (IOU) of the point clouds. Combine the odometry information provided by the visual odometry method to construct a local semantic map, i.e., a local sub-map.

[0075] Step 15: Transmit local semantic maps between robots using a ROS-based remote topic manager and AP bridge. A single robot receives local sub-maps sent by other robots.

[0076] Step 16: Sub-map alignment and robot loop closure detection are achieved through a data association method based on consistency graphs. At the same time, the global pose of each robot is determined by combining the global semantic map and odometry information. The global semantic map and robot global pose are continuously updated during robot movement and sent to the navigation module.

[0077] (2) Navigation module

[0078] The navigation module is an LLM-based semantic planner that utilizes LLM combined with an action library and a validation module to implement a navigation algorithm for task planning. This algorithm receives the user task and prior information from the semantic mapper recursively, and generates a series of subtasks based on semantic and syntactic rules. The semantic planner can refine these subtasks online to adapt to dynamically changing environments, and the validation module ensures the validity and executability of the task sequence. In each planning iteration, the semantic mapper provides a global semantic map and robot pose updates. The LLM uses these updates to adjust the task sequence and calls atomic actions from the action library to achieve specific tasks. The action library contains atomic actions such as navigation, active perception, and user interaction. The semantic planner generates a task sequence based on the current task and reviews it to ensure the generated task semantics are correct and to verify the task's reachability and spatial validity.

[0079] See Figure 4 The navigation module includes a decision generation module, a decision verification module, and a path planning module.

[0080] The decision generation module includes an LLM (Local Level Management) and a behavior library. The LLM analyzes the high-level language commands given by the user, infers relevant elements, combines the global semantic map and robot pose, and calls pre-defined atomic behaviors from the behavior library to generate a task sequence composed of atomic behaviors (containing parameters for each task). The task sequence is then passed to the decision verification module for validation. For example, if the user's command is: "I sent a robot to collect supplies from an inbound ship. I haven't received a reply yet, what happened?" the LLM, combined with the global semantic map provided by the perception module, should analyze that it is looking for a robot and infer, using the provided context information, that the robot is likely near the dock in the scene. Since there is no direct path to the dock on the map, exploration is necessary to reach the dock and find the task robot. The behavior library pre-sets many atomic behaviors, covering three categories: navigation, active perception, and user interaction. These behaviors can be called through the system-configured API interface, using the behavior library for task inference. In each iteration, the LLM generates a task sequence, which is continuously optimized as the robot moves and the global semantic map and robot pose provided by the perception module are updated. The task sequence generated by the decision generation module will be passed to the decision verification module for verification.

[0081] In a preferred embodiment, LLM can also maintain all input information and task history in the form of context through an API, forming a contextual learning database. This allows LLM to perform more reasonable inferences based on contextual information during the reasoning process.

[0082] The decision verification module is used to verify the task sequence; after the verification is passed, the navigation target point of the task sequence is determined and sent to the path planning module.

[0083] In a preferred embodiment, the decision verification module performs both semantic and spatial verification. To ensure the generated tasks are executable, the decision verification module first checks for semantic correctness and then checks the spatial reachability of the given navigation target. If the task contains unreachable targets or actions that do not conform to predetermined rules, the decision verification module provides feedback to the LLM (Local Management Module) of the decision generation module to adjust the task sequence. For example, if the LLM attempts to invoke the goto action to reach an unreachable location x, the verifier will provide feedback such as: "Location x is unreachable from your current location. Please consider exploring first and update your plan accordingly." Spatial verification employs a forward-looking exploration method to ensure that the spatial paths of all tasks are feasible, preventing the robot from performing dangerous or unachievable tasks. After successful verification, the coordinates of the navigation target point are sent to the path planning module.

[0084] The path planning module plans the desired trajectory from the current position to the navigation target point and sends it to the control module. When the robot performs active perception actions, it will explore unknown areas in the map. At this time, the perception module will send a real-time updated semantic map to the navigation module.

[0085] Figure 5 The workflow of the navigation module is shown, such as... Figure 5 As shown, the specific steps are as follows:

[0086] Step 21: The user issues a natural language task description. The LLM receives the task description, understands the task objectives and environmental constraints, and analyzes relevant elements.

[0087] Step 22: Based on the global semantic map and robot pose provided by the semantic mapper, the semantic planner calls API interfaces such as ChatGPT 4o to infer the task composition in the current environment. During this process, the semantic mapper provides real-time updates to the global semantic map and robot pose to ensure that the task sequence generated by LLM conforms to the current environmental conditions.

[0088] Step 23: Generate task sequences using the available action library through stepwise inference via LLM. All input information and action history are maintained in context via API. Atomic actions include three categories: navigation, active perception, and user interaction. The semantic planner automatically selects and combines appropriate task actions based on current needs.

[0089] Step 24: The generated task sequence is sent to the decision verification module for review. Semantic verification checks the correctness of the tasks, ensuring that the semantics of all tasks conform to predetermined rules. If a task contains semantic errors or an unreachable target, the decision verification module will provide feedback to the LLM for adjustment. Spatial verification employs a forward-looking exploration method to ensure that the spatial paths of all tasks are feasible. During the spatial path verification process, the decision verification module checks whether the robot can safely perform the tasks in the existing environment, avoiding dangerous or unachievable tasks.

[0090] Step 25: After verification, using the coordinates of the navigation target point in the optimized task sequence, plan a desired trajectory from the current position to the navigation target point, and finally send it to the control module. The control module controls the robot to perform corresponding operations according to the instructions of the task sequence.

[0091] (3) Control module

[0092] The control module is a terrain adaptation controller based on the Visual Model (VFM). In this embodiment, it uses a visual model (VFM) combined with a deep neural network (DNN) and adaptive control to implement an algorithm for vehicle motion control.

[0093] The input data for the control module includes ground images segmented by the perception module and robot state information. After processing the ground images using VFM (Voice over Model), general terrain features are extracted, and combined with a robot-specific terrain model to achieve efficient adaptation to complex terrain and dynamic disturbances. The algorithm in the control module comprises a three-layer structure: a general terrain understanding layer, a robot-specific terrain model layer, and a real-time disturbance adaptation layer. The general terrain understanding layer uses VFM, the robot-specific terrain model layer uses a deep neural network (DNN), and the real-time disturbance adaptation layer uses a composite adaptive controller.

[0094] First, terrain features are understood through a Visual Model (VFM). The VFM is pre-trained on an image set of approximately 1.2 million images, enabling it to extract highly semantically relevant features from the terrain images. These features are further combined with robot state information through a deep neural network to generate a terrain-specific driving matrix K. This deep neural network employs meta-learning techniques, trained on a small dataset (e.g., 20 minutes of driving data) to capture complex dynamics (such as track slippage or wheel degradation) that were not modeled during robot interaction with the terrain. During real-time operation, the control module adaptively adjusts the parameters of the last layer of the deep neural network online to compensate for dynamic disturbances not captured during offline training. The control module generates the driving matrix based on the current terrain features and robot state, thereby rapidly adjusting the control strategy to ensure the robot's trajectory tracking performance in complex terrain.

[0095] See Figure 6 The figure illustrates the implementation of the control module, which includes a VFM, a DNN, and a composite adaptive controller. The ground image segmented by the perception module is input into the VFM, and the VFM outputs a terrain feature vector. Topographic feature vector The current state (including current linear velocity and angular velocity) fed back from the robot is concatenated and input into a DNN. This DNN has two hidden layers, and the network structure itself serves as the basis functions required for adaptive control. Its output, after deformation, yields the driving matrix K. The weights of the last layer of the DNN... The time-varying linear parameter vector, which is used for online adaptive control, is passed to the composite adaptive controller along with the driving matrix K. During online operation, the composite adaptive controller can update the time-varying linear parameter vector in real time according to the proposed adaptive control rules. This is combined with the desired trajectory (i.e., the desired linear and angular velocities at the current position) provided by the navigation module, the driving matrix K, and the time-varying linear parameter vector. Calculate the control input at the current moment u (Linear velocity and angular velocity) are used to control the robot's motion platform. Specifically... Update and u The calculation can be implemented by designing adaptive control rules as needed, which will not be detailed here.

[0096] Figure 7 The workflow of the control module is shown, such as... Figure 7 As shown, the specific steps are as follows:

[0097] Step 31: The control module receives ground image input from the perception module and obtains robot status information.

[0098] Step 32: Extract terrain feature vectors from the ground images using a visual base model pre-trained on 1.2M images. .

[0099] Step 33, Terrain Feature Vector The robot's state V is input into a lightweight deep neural network to capture complex, unmodeled dynamics during the robot's interaction with the terrain. The lightweight deep neural network outputs a driving matrix K, which, along with the weights of the last layer of the deep neural network, is passed to a composite adaptive controller.

[0100] Step 34: During real-time robot operation, meta-learning technology is employed to adjust the weights of the last layer of the deep neural network online through a composite adaptive control mechanism. This compensates for dynamic disturbances that were not captured during offline training, ensuring the robot's motion control accuracy in complex environments.

[0101] Step 35: The composite adaptive controller, combining the desired trajectory, the driving matrix K, and the time-varying linear parameter vector provided by the navigation module, calculates the robot's control input at the current moment; based on the current terrain features and robot state, it updates the driving matrix in real time, thereby adjusting the robot's motion state. This ensures that the robot can stably track the target trajectory in complex terrain, overcoming dynamic disturbances and environmental changes.

[0102] The dynamic system model of the control module is represented as follows:

[0103] (1)

[0104] in, Indicates the system status. Indicates control input, t Indicates time, For the nominal dynamic model, Indicates an unknown disturbance;

[0105] Terrain feature vectors generated using VFM Feature mapping through deep neural networks For unknown disturbances The model is constructed, and the calculation method is as follows:

[0106] (2)

[0107] in, For time-varying linear parameter vectors, To indicate the error;

[0108] Feature mapping Obtained through offline training of a deep neural network. It serves as the weight of the last layer in a deep neural network and is continuously and adaptively adjusted during operation.

[0109] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A robot sensing, guidance, and control integrated system for adaptive operation in unfamiliar environments, wherein each robot in the system includes a sensing device, a perception module, a navigation module, and a control module, characterized in that: The perception module is a semantic mapper based on the Visual Language Model (VLM), used to segment environmental information and perform semantic recognition based on VLM to generate a global semantic map with embedded semantic descriptors of ground features. The global semantic map is provided to the navigation module, and the segmented ground image is provided to the control module. The navigation module is a semantic planner based on a Large Language Model (LLM). It uses LLM to understand the input natural language task description and combines it with the global semantic map provided by the perception module to infer a task sequence composed of atomic behaviors in the current environment. The atomic behaviors include three categories: navigation, active perception, and user interaction. The task sequence is continuously optimized as the robot moves and the global semantic map provided by the perception module is updated. The desired trajectory is generated based on the continuously optimized task sequence and sent to the control module. The control module is a terrain adaptation controller based on the visual basic model VFM. VFM is used as a general model for terrain understanding to identify terrain features from the segmented ground image. The terrain features and the robot's real-time state are mapped into a driving matrix through a robot-specific terrain model. The robot's control input is generated based on the driving matrix and the desired trajectory to achieve robot motion control. The operation of the sensing module includes: It receives RGB-D depth camera image input and IMU sensor input, uses the YOLOv7 algorithm to filter out dynamic obstacles in the surrounding environment, and uses a visual odometry algorithm to generate odometry information. The FastSAM image segmentation algorithm is used to segment the ground objects in the surrounding environment for semantic map construction, and the segmented ground images are sent to the control module. The BLIP algorithm is used as a visual language model to generate semantic descriptors for ground objects. The segmented ground objects are projected onto the point cloud, and the overlap of the point clouds is calculated and compared to complete the merging of inter-frame data. The local semantic map, i.e., the local sub-map, is constructed by combining the odometry information provided by the visual odometry method. The transmission of local semantic maps between robots is achieved through a ROS-based remote topic manager and AP bridge; a single robot receives local sub-maps sent by other robots. Sub-map alignment and inter-robot loop detection are achieved through a data association method based on consistency graphs. At the same time, the global pose of each robot is determined by combining global semantic map and odometry information. The global semantic map and robot global pose are continuously updated during robot movement and sent to the navigation module.

2. The robot sensing, guidance, and control integrated system for adaptive unfamiliar environments as described in claim 1, characterized in that, The perception module includes: a mapping module, a data association module, and a pose graph optimization module; The mapping module acquires image and point cloud information of the environment from the sensing device, as well as the robot's acceleration information. Based on image segmentation algorithms and VLM, it generates local sub-maps that embed semantic descriptors of ground features. Local sub-maps will be continuously generated during the robot's movement. The data association module receives local sub-maps sent by other robots through distributed communication and merges them into a global semantic map; it calculates the relative pose transformation matrix of each robot based on the global semantic map and provides it to the pose map optimization module. The pose graph optimization module determines the global pose of each robot based on the odometry information obtained using acceleration information and the relative pose transformation matrix between robots.

3. The robot sensing, guidance, and control integrated system for adaptive unfamiliar environments as described in claim 2, characterized in that, The mapping module includes an image segmentation module, a visual language module, a visual odometry calculation module, a local sub-map generation module, and a communication module. The image segmentation module segments ground objects in the surrounding environment from the image based on the image information of the environment. The ground object image is sent to the visual language module, and the segmented ground image is sent to the control module. The visual language module identifies the image of each ground object, generates a corresponding semantic descriptor, and sends it to the local sub-map generation module. The visual odometry calculation module generates odometry information based on the image information of the environment and the acceleration information of the robot, and sends it to the local sub-map generation module. The local sub-map generation module receives point cloud information of the environment, embeds the semantic descriptor into the point cloud, and combines it with the odometer information to generate a local sub-map. The communication module sends the local sub-map of the robot to other robots through distributed communication.

4. The robot sensing, guidance, and control integrated system for adaptive unfamiliar environments as described in claim 1, characterized in that, The sensing devices include a depth camera and an inertial measurement unit (IMU); the depth camera acquires image information and point cloud information of the environment; the IMU acquires the robot's acceleration information.

5. The robot sensing, guidance, and control integrated system for adaptive unfamiliar environments as described in any one of claims 1-4, characterized in that, The global semantic map uses a list to represent the information of three-dimensional land features, including the semantic information, centroid location, shape, and size of each land feature object in the explored area.

6. The robot sensing, guidance, and control integrated system for adaptive unfamiliar environments as described in claim 1, characterized in that, The navigation module includes a decision generation module, a decision verification module, and a path planning module; The decision generation module includes an LLM (Limited Language Management) and a behavior library. The LLM analyzes the natural language task description given by the user, combines the global semantic map and the robot's global pose, and calls the preset atomic behaviors in the behavior library to generate a task sequence composed of atomic behaviors. The LLM automatically selects and combines the corresponding atomic behaviors according to the current requirements. The task sequence is passed to the decision verification module for verification. The decision verification module is used to verify the task sequence; for the verified task sequence, it determines the navigation target point and sends it to the path planning module. The path planning module is used to plan the desired trajectory from the current position to the navigation target point and send it to the control module; when the robot performs active perception actions, it will explore unknown areas in the map; at this time, the perception module will send a real-time updated semantic map to the navigation module.

7. The robot sensing, guidance, and control integrated system for adaptive unfamiliar environments as described in claim 6, characterized in that, The decision verification module includes semantic verification and spatial verification. The semantic verification checks whether the semantics of the task conform to the predetermined rules. If there are semantic errors or unreachable targets, it feeds back to the LLM of the decision generation module for adjustment. The spatial verification adopts a frontier exploration approach to ensure that the spatial paths of all tasks are feasible.

8. The robot sensing, guidance, and control integrated system for adaptive unfamiliar environments as described in claim 1, characterized in that, The control module comprises a three-layer structure: a general terrain understanding layer implemented by VFM, a robot-specific terrain model layer implemented by neural networks, and a real-time perturbation adaptation layer. The VFM is trained offline based on an image set to achieve basic modeling for terrain recognition; The neural network input information combines terrain features and the robot's real-time state; the neural network uses meta-learning technology and is trained based on fragmented driving data of the robot to capture unmodeled dynamics during the robot's interaction with the terrain. The real-time perturbation adaptive layer adjusts the weights of the last layer of the deep neural network online. This is to compensate for dynamic perturbations that were not captured during offline training.

9. The robot sensing, guidance, and control integrated system for adaptive unfamiliar environments as described in claim 1, characterized in that, The control module includes VFM, neural network, and composite adaptive controller; The ground image segmented and generated by the perception module is input into the VFM, and the VFM outputs terrain features. The terrain features and the current state fed back by the robot are concatenated and input into the neural network, which outputs a driving matrix K. The weights of the last layer of the neural network The time-varying linear parameter vector is used for online adaptive control, and the time-varying linear parameter vector is transmitted to the composite adaptive controller together with the driving matrix K. During online operation, the composite adaptive controller updates the time-varying linear parameter vector of the neural network in real time according to the set rules, and calculates the robot's control input at the current moment by combining the desired trajectory provided by the navigation module and the driving matrix K.

10. The robot sensing, guidance, and control integrated system for adaptive unfamiliar environments as described in claim 8 or 9, characterized in that, The neural network is a lightweight deep neural network, consisting of an input layer, two hidden layers, and an output layer.

Images