Robot adaptive control method and system based on visual radiation perception

Abstract

The present invention relates to a robot adaptive control method and system based on visual radiation perception, which involves the field of intelligent control. Environmental data is acquired by integrating sensors such as laser radar, plantar pressure sensor, joint torque sensor, binocular camera and IMU. After preprocessing, the data is input into an environmental recognition model to obtain initial environment and posture information, and then a local area map is constructed and an initial travel path is planned. At the same time, a flexible capacitor array and plantar pressure sensor are used to evaluate the contact surface type. Finally, the motion mode is matched according to the travel sequence and surface type, and the robot motion is controlled. This solves the technical problems of the robot's limited environmental perception ability, insufficient autonomous decision-making ability and poor motion control flexibility in complex, changeable and unknown environments, and realizes the robot's efficient environmental perception, autonomous path planning and flexible motion control in complex environments.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control, and in particular to a robot adaptive control method and system based on visual radiation perception. Background Art

[0002] When conducting search and rescue operations in complex environments such as caves, ruins, or disaster sites, traditional manual search and rescue methods often face challenges such as high risk, low efficiency, and limited accessibility. With the rapid development of robotics technology, utilizing robots to perform such tasks has become an effective solution. However, existing search and rescue robots still face challenges such as limited environmental perception, insufficient autonomous decision-making capabilities, and poor motion control flexibility when faced with complex and changing unknown environments. Existing robots often rely on preset maps or simple sensor data for navigation, making them difficult to adapt to complex and changing real-world environments. Especially in confined spaces or areas with complex terrain, robots need to possess higher environmental perception accuracy and autonomous decision-making capabilities to adjust their motion strategies in real time to avoid collisions and efficiently complete the mission. Furthermore, different terrains, such as hard ground, soft sand, gravel, and mud, place higher demands on the robot's motion patterns and energy consumption strategies. Robots must be able to dynamically adjust their motion parameters based on real-time environmental information to adapt to more complex application requirements. Summary of the Invention

[0003] The present invention aims to solve the technical problems in the prior art of robots' limited environmental perception ability, insufficient autonomous decision-making ability and poor motion control flexibility in complex, changeable and unknown environments, and provides a robot adaptive control method and system based on visual radiation perception to solve them.

[0004] The technical solution of the present invention to solve the above technical problems is as follows:

[0005] In the first aspect, the present invention provides a robot adaptive control method based on visual radiation perception, which is applied to an intelligent robot, wherein the robot is equipped with a laser radar, a plantar pressure sensor, a joint torque sensor, a binocular camera located on the head, and an IMU built into the torso. The method comprises: obtaining multiple data sets through the laser radar, the plantar pressure sensor, the joint torque sensor, the binocular camera, and the IMU, pre-processing the multiple data sets and inputting them into an environment recognition model to obtain initial environment information and initial posture information of the robot; using a sliding window algorithm to construct a local area map within the minimum data processing range of the initial environment information, and segmenting the local area map based on a region growing algorithm to extract passable areas and obstacle areas. domain; combining the initial posture information of the robot, the traversable area and the obstacle area, with the minimum loss as the constraint, planning the initial travel path on the local area map, generating a travel sequence containing position and posture information, wherein each node in the travel sequence contains the target position coordinates, the expected posture angle and the arrival timestamp; obtaining the pressure distribution, texture characteristics and stiffness information of the contact surface through the flexible capacitance array on the robot surface and the plantar pressure sensor, constructing a three-dimensional surface feature vector, performing spatial surface evaluation based on the surface feature vector, and identifying the surface type; matching the robot's travel sequence and surface type in a preset motion pattern library, obtaining the joint motor control parameter array according to the matching result, and executing the robot's motion control through the parameter array.

[0006] In the second aspect, the present invention provides a robot adaptive control system based on visual radiation perception, wherein the system is applied to an intelligent robot, wherein the robot is equipped with a laser radar, a plantar pressure sensor, a joint torque sensor, a binocular camera located on the head, and an IMU built into the torso. The system comprises: a data acquisition module for obtaining multiple data sets through the laser radar, the plantar pressure sensor, the joint torque sensor, the binocular camera, and the IMU, and inputting the multiple data sets into an environment recognition model after pre-processing to obtain initial environment information and initial posture information of the robot; a region segmentation module for constructing a local region map within the minimum data processing range of the initial environment information using a sliding window algorithm, and segmenting the local region map based on a region growing algorithm to extract passable areas and obstacle areas; a path planning module for extracting passable areas and obstacle areas using a path planning module; a path planning module for extracting passable areas and obstacle areas using a path planning module; a path planning module for extracting passable areas and obstacle areas using a path planning module; a path planning module for extracting passable areas and obstacle areas using a path planning module; a path planning module for extracting passable areas and obstacle areas by means of ... A planning module is used to combine the initial posture information of the robot, the traversable area and the obstacle area, and plan the initial travel path on the local area map with minimum loss as the constraint, to generate a travel sequence containing position and posture information, wherein each node in the travel sequence contains the target position coordinates, the expected posture angle and the arrival timestamp; a surface evaluation module is used to obtain the pressure distribution, texture characteristics and stiffness information of the contact surface through the flexible capacitance array and the plantar pressure sensor of the robot surface, construct a three-dimensional surface feature vector, perform spatial surface evaluation based on the surface feature vector, and identify the surface type; a motion control module is used to match the preset motion pattern library based on the travel sequence and surface type of the robot, obtain the joint motor control parameter array according to the matching result, and perform the motion control of the robot through the parameter array.

[0007] The beneficial effects of the present invention are: environmental data is acquired by integrating sensors such as laser radar, plantar pressure sensor, joint torque sensor, binocular camera and IMU, and after preprocessing, it is input into the environmental recognition model to obtain initial environment and posture information, and then a local area map is constructed and the initial travel path is planned. At the same time, a flexible capacitive array and plantar pressure sensor are used to evaluate the contact surface type, and finally the motion mode is matched and the robot motion is controlled according to the travel sequence and surface type, thereby realizing the robot's efficient environmental perception, autonomous path planning and flexible motion control in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a flow chart of the robot adaptive control method based on visual radiation perception provided by the present invention.

[0009] Figure 2 This is a schematic diagram of the structure of the robot adaptive control system based on visual radiation perception provided by the present invention.

[0010] Description of reference numerals: data acquisition module 11 , region segmentation module 12 , path planning module 13 , surface evaluation module 14 , motion control module 15 . DETAILED DESCRIPTION

[0011] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0012] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the specified features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0013] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or illustration". Any embodiment of the present invention described as "for example" is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed herein.

[0014] Example 1

[0015] like Figure 1 As shown, an embodiment of the present invention provides a robot adaptive control method based on visual radiation perception. The method is applied to an intelligent robot, wherein the robot is equipped with a laser radar, a plantar pressure sensor, a joint torque sensor, a binocular camera located on the head, and an IMU (Inertial Measurement Unit) built into the torso. The method includes:

[0016] S10: Obtain multiple data sets through the laser radar, plantar pressure sensor, joint torque sensor, binocular camera and IMU, pre-process the multiple data sets and input them into the environment recognition model to obtain initial environment information and robot initial posture information.

[0017] For example, the robot first collects multi-source environment and its own status data through its equipped lidar, plantar pressure sensor, joint torque sensor, binocular camera located on the head, and IMU (inertial measurement unit) built into the torso.

[0018] As one of the core sensors, LiDAR is responsible for capturing three-dimensional point cloud data of the surrounding environment. This data accurately depicts the spatial position and shape of objects around the robot, providing a foundation for environmental modeling. Plantar pressure sensors monitor the pressure distribution of the robot's feet in real time when they make contact with the ground. This information is crucial for understanding the material properties of the ground and determining terrain stability. Joint torque sensors record the changes in torque applied to each joint of the robot during motion, helping to analyze the robot's mechanical state during motion and providing a basis for subsequent motion control. Binocular cameras simulate human vision to capture two-dimensional images of the environment. Combined with depth information extraction technology, they can further enrich the dimensions of environmental perception. The IMU continuously monitors the robot's posture changes by measuring its acceleration and angular velocity, ensuring that the robot maintains a stable motion state in complex environments.

[0019] After collecting the aforementioned multi-source data sets, preprocessing steps such as data cleaning, denoising, and time synchronization are performed before they are uniformly input into the environment recognition model. Based on multimodal data fusion technology, the environment recognition model comprehensively analyzes information from different sensors to extract initial features of the environment, including but not limited to terrain type (e.g., flat, rugged, soft sand), obstacle location and size, and the robot's current initial posture information (e.g., position coordinates, heading angle, etc.). For example, in a caving search and rescue scenario, a lidar radar may detect a narrow passage ahead, while a plantar pressure sensor indicates that the ground is relatively soft. Combined with the image information from the binocular camera, the environment recognition model can comprehensively determine that the area is soft sand terrain and confirm that the robot is currently at the entrance of the passage, ready to enter and carry out the search and rescue mission. This series of processing steps provides solid data support for the robot's subsequent development of adaptive motion strategies and path planning.

[0020] S20: constructing a local area map within the minimum data processing range of the initial environmental information using a sliding window algorithm, and segmenting the local area map based on a region growing algorithm to extract passable areas and obstacle areas.

[0021] Furthermore, to effectively analyze and utilize the initial environmental information, a sliding window algorithm is employed to dynamically construct a local region map within the minimum data processing scope defined by the initial environmental information. This algorithm uses a window of a specific size (which can be flexibly adjusted based on the robot's size and motion characteristics) that slides across the three-dimensional environmental data space, gradually capturing and integrating the environmental information within the window. This creates a series of local environment snapshots, which together form the basis of the local region map. This process ensures that the system can efficiently focus on the current region of interest, even in the presence of large amounts of data or complex environments, reducing computational overhead.

[0022] The constructed local region map is then finely segmented using a region growing algorithm. Starting from a seed point (typically a point with significant features or known attributes), the algorithm gradually merges adjacent pixels or voxels that meet certain criteria into the same region based on pre-defined similarity criteria (such as normal vector similarity and spatial continuity) until no further expansion is possible. This process effectively divides the local region map into several regions with clear boundaries, which are then further classified into traversable areas and obstacle areas based on their internal characteristics.

[0023] For example, in a cave search and rescue scenario, a sliding window algorithm might first focus on an area in front of the robot, constructing a local environmental map of that area. A region growing algorithm would then identify flat, obstacle-free portions of the map as traversable areas, while marking raised, irregular, or significantly obstructed areas as obstacles. This segmentation result not only provides clear navigation guidance for the robot but also provides an important basis for subsequent path planning and motion control, ensuring the robot can safely and efficiently traverse complex environments and complete search and rescue missions.

[0024] S30: Combining the initial posture information, traversable area and obstacle area of ​​the robot, with minimum loss as a constraint, planning an initial travel path on the local area map, and generating a travel sequence containing position and posture information, wherein each node in the travel sequence contains the target position coordinates, the expected posture angle and the arrival timestamp.

[0025] Preferably, after obtaining the robot's initial posture information and the distribution of traversable areas and obstacle areas on the local area map, the path planning task is further performed. The path planning task takes minimizing energy loss as the core constraint, aiming to ensure that the robot moves in the most economical way in a complex environment. Specifically, the system first integrates the robot's current position and orientation (i.e., initial posture information) with the local area map data, and uses a path planning algorithm (such as the A* algorithm) to search for the optimal path from the starting point to the target point in the traversable area. The algorithm not only considers the geographical distance, but also combines terrain characteristics (such as slope, surface material, etc.) and the robot dynamics model to estimate the energy consumption of each candidate path and select the path with the lowest total energy loss as the initial travel path.

[0026] The planned initial path is then converted into a series of movement sequence nodes containing position and attitude information. Each node precisely specifies the robot's target position coordinates at that point, the desired attitude angles (such as pitch and yaw), and the estimated arrival timestamp at that point. These timestamps are dynamically calculated based on the robot's velocity, acceleration limits, and terrain changes along the path. This ensures the robot can smoothly transition at the desired speed and avoids energy waste from sudden stops and starts. For example, in a cave rescue scenario, if the robot needs to traverse a terrain consisting of alternating hard rock and soft sand, the system's initial movement path will prioritize the hard rock areas, as these areas provide more stable movement and require less energy. The nodes in the movement sequence detail the robot's target position on each hard rock, the required attitude (such as maintaining a horizontal position for stable movement), and the estimated arrival time, ensuring the robot can complete the rescue mission efficiently and safely.

[0027] S40: Obtain pressure distribution, texture characteristics and stiffness information of the contact surface through the robot surface flexible capacitance array and the plantar pressure sensor, construct a three-dimensional surface feature vector, perform spatial surface evaluation based on the surface feature vector, and identify the surface type.

[0028] To more precisely adapt to varying terrain conditions, the robot utilizes a flexible capacitive array and plantar pressure sensors integrated onto its surface to provide detailed perception of the surface's properties. The flexible capacitive array captures minute surface deformations, revealing surface texture and local stiffness variations. The plantar pressure sensors monitor and record the pressure distribution within the contact area in real time. Together, these data form the foundation for the robot's perception of the external environment.

[0029] The system fuses the data obtained by these two types of sensors and constructs a three-dimensional surface feature vector through algorithm analysis. This vector not only contains the three-dimensional spatial information of pressure distribution, but also integrates the surface texture details and stiffness characteristics, providing comprehensive and accurate data support for subsequent spatial surface evaluation.

[0030] Then, based on the three-dimensional surface feature vectors, the system executes a spatial surface assessment program. Using pattern recognition and machine learning techniques, it compares and analyzes the perceived surface characteristics with a pre-set database of surface types, accurately identifying the type of surface currently in contact, such as hard ground, soft sand, gravel, or mud. For example, during a caving search and rescue mission, when a robot reaches an unknown area, the system can quickly identify the ground as soft sand through the collaborative work of the flexible capacitive array and the plantar pressure sensor. It then adjusts the robot's motion pattern and joint torque, adopting a creeping or slow, steady movement strategy to adapt to the sandy environment and ensure the smooth progress of the search and rescue mission.

[0031] S50: matching is performed in a preset motion pattern library based on the movement sequence and surface type of the robot, obtaining a joint motor control parameter array according to the matching result, and executing motion control of the robot through the parameter array.

[0032] Specifically, after obtaining the robot's movement sequence and identifying the surface type through surface feature vectors, the system further performs motion pattern matching and control parameter generation. The system combines the position and posture information of each node in the movement sequence with the currently identified surface type (such as hard ground, soft sand, gravel, etc.) as input parameters and accurately matches it to a preset motion pattern library. This motion pattern library predefines a variety of motion patterns for different terrains and task requirements, each corresponding to a specific set of joint motor control parameters.

[0033] Through pattern matching, the system can quickly find the motion mode that best matches the current travel requirements and surface conditions, and extract the corresponding joint motor control parameter array. These parameter arrays specify in detail the torque, speed, and position instructions that each joint motor should output during the robot's travel sequence, ensuring that the robot can flexibly adapt to different terrain conditions according to the preset path and posture, achieving efficient and stable motion control. For example, in a cave exploration and rescue mission, if the robot needs to traverse an area of ​​soft sand, the system will automatically select a creeping mode suitable for soft sand terrain through matching and generate the corresponding joint motor control parameter array, guiding the robot to move at low speed and high torque to avoid getting stuck in the sand and ensure the smooth progress of the search and rescue mission.

[0034] In a preferred embodiment, the method further includes: performing time synchronization and spatial alignment on the point cloud data acquired by the lidar and the image data collected by the binocular camera to generate a multimodal observation frame, extracting ORB feature points and depth information in the multimodal observation frame, performing feature matching through a bag-of-words model, and constructing a common view relationship between key frames; constructing an initial pose graph based on the common view relationship, optimizing the pose parameters of the key frames using a bundle adjustment method, and obtaining a pose graph in combination with the pre-integration constraints of the IMU; identifying scene revisits through a loop detection mechanism, triggering global pose graph optimization based on the pose graph, generating a consistent global environment map, and fusing the global environment map with the local area map to generate a hybrid map containing global topological structure and local detail features.

[0035] Optionally, to improve the accuracy and robustness of environmental modeling, the system performs a series of multimodal data fusion and map-building operations. First, the LiDAR continuously scans the surrounding environment, generating high-precision 3D point cloud data, while the binocular camera simultaneously captures color image data of the environment. To ensure temporal and spatial consistency of the multi-source data, the system synchronizes and spatially aligns the point cloud data with the image data, generating a multimodal observation frame containing both geometric and visual information.

[0036] The system then extracts ORB feature points (Oriented FAST and Rotated BRIEF) from these observation frames. These feature points are rotationally and scale-invariant, stably describing salient features in the environment. Simultaneously, combined with the disparity information from the binocular cameras, depth information is extracted from these feature points, providing a basis for subsequent 3D reconstruction.

[0037] Furthermore, the extracted feature points are encoded and matched using a bag-of-words model to construct co-viewing relationships between keyframes, i.e., which keyframes observe the same environmental features. Based on these co-viewing relationships, an initial pose graph is constructed, where nodes represent keyframes and edges represent relative pose transformations between keyframes.

[0038] To further improve the accuracy of pose estimation, bundle adjustment is used to optimize the pose parameters of the keyframes. This method adjusts the pose by minimizing the reprojection error, ensuring better spatial alignment of the 3D point cloud and image features. Furthermore, the system further optimizes the pose graph by incorporating IMU pre-integration constraints. The system uses the acceleration and angular velocity information provided by the IMU to continuously predict and correct pose changes, resulting in a more accurate pose graph.

[0039] Furthermore, to eliminate accumulated errors and improve map consistency, a loop detection mechanism is introduced. This mechanism identifies scene revisits by comparing the similarity between current observations and historical keyframes. Once a loop is detected, the system triggers global pose graph optimization based on the pose graph, adjusting the poses of all relevant keyframes to eliminate accumulated drift errors and ultimately generate a consistent global environmental map. This ensures that the map maintains coherence and accuracy in terms of spatial scale, feature description, and time. The spatial relationships between various map components are correct, and the descriptions of the same environmental features are unified without contradictions or deviations. Furthermore, during the map construction and update process, data collected at different times can be rationally integrated to ensure the overall consistency of the map, providing the robot with a reliable and stable foundation for environmental cognition.

[0040] The global environment map is a digital map constructed based on multi-sensor data fusion that comprehensively describes the robot's surroundings. This map encompasses the environment's topological structure, such as spatial layout and path connectivity, as well as its geometric features, including the location, shape, and size of obstacles. It provides the robot with a comprehensive, accurate, and comprehensive understanding of its environment, enabling autonomous positioning, global path planning, and real-time perception of its position and surroundings during navigation, enabling efficient and safe motion control and task execution.

[0041] After the global environment map is constructed, it is used for the robot's positioning and navigation in the entire environment. The robot's pose information recorded in the pose graph helps to accurately determine the robot's position and posture in the global environment map, providing a key reference for the robot's positioning on the map, so that the robot can know its own position and direction in the global environment map based on the pose graph information. The global environment map also contains rich environmental information, such as obstacle distribution, traversable areas, etc. This information, combined with the pose graph, allows the robot to have a more comprehensive understanding of the relationship between itself and the environment. It not only knows where it is (the pose graph provides the posture), but also the surrounding environment conditions (the global environment map provides environmental information), so as to better perform path planning, obstacle avoidance and other operations.

[0042] Finally, the global environment map is fused with the previously constructed local area map. The local area map provides detailed information about the robot's surroundings, while the global map ensures the overall consistency of the environmental representation. This fusion strategy generates a hybrid map that combines global topology and local details, providing richer and more accurate environmental information for the robot's autonomous navigation and path planning. For example, in a cave search and rescue mission, the hybrid map can clearly display the overall layout of the cave and the distribution of local obstacles, helping the robot complete the search and rescue mission efficiently and safely.

[0043] In a preferred embodiment, the global environment map is fused with the local area map to generate a hybrid map containing global topological structure and local detail features, including: extracting topological feature points in the global environment map and geometric feature points in the local area map, determining the transformation relationship between the maps through feature matching, using a weighted average method to fuse the obstacle information and passable area information in the local area map into the global environment map, updating the confidence distribution of the map, and obtaining a hybrid map.

[0044] Specifically, during the hybrid map construction process, feature extraction is first performed on the global environment map and the local area map. Specifically, topological feature points are extracted from the global environment map. These feature points represent key locations and connections within the map, such as channel intersections, and together they constitute the global topology of the environment. Simultaneously, geometric feature points are extracted from the local area map. These feature points describe the geometry and spatial layout of the robot's surroundings in detail, such as the outlines of obstacles and the undulations of the ground, providing detailed local information about the environment.

[0045] The system then uses a feature matching algorithm to compare the topological feature points in the global environment map with the geometric feature points in the local area map to determine the spatial transformation relationship between the two maps, including parameters such as translation, rotation, and scaling. This step ensures that the global map and the local map are accurately aligned in space.

[0046] After determining the transformation relationship between the maps, a weighted averaging method is used to fuse the obstacle information and traversable area information in the local area map into the global environment map. Specifically, based on factors such as the feature significance of each area in the local area map and the quality of sensor data, different weights are assigned to different areas. The information of these areas is then weighted averaged with the information of the corresponding areas in the global map to obtain the fused map data.

[0047] During the fusion process, the confidence distribution of the map is updated. This means that the reliability assessment of each area in the global map is adjusted based on the new information provided by the local area map. For example, if the local area map shows that an area originally marked as passable actually has an obstacle, the system will lower the confidence of that area in the global map and adjust its passability marking accordingly.

[0048] Through the above steps, a hybrid map containing global topological structure and local detailed features is obtained. This map not only retains the overall layout information of the global environment map, but also incorporates the detailed environmental characteristics of the local area map, providing more comprehensive and accurate environmental perception support for the robot's autonomous navigation and path planning. For example, in cave exploration and rescue missions, the hybrid map can simultaneously display the overall channel structure and local obstacle distribution within the visible distance of the cave, helping the robot to more effectively plan the search and rescue path.

[0049] In a preferred embodiment, the method also includes: matching the observation data of the plantar pressure sensor and joint torque sensor at the current moment with the hybrid map, and predicting the real-time posture of the robot through a particle filtering algorithm; planning a global path on the hybrid map based on the real-time posture and task objectives, and using the RRT* algorithm to generate a collision-free optimal path; decomposing the global path into local sub-goals, combining the travel sequence and motion pattern library, and generating continuous joint control instructions; monitoring the changes in the local area map in real time, and when a new obstacle is detected, triggering local path replanning and executing the joint control instruction update.

[0050] For example, the system continuously matches the current observation data from the plantar pressure sensor and joint torque sensor with a pre-built hybrid map. The plantar pressure sensor provides mechanical information about the robot's contact with the ground, while the joint torque sensor reflects the force state of each joint during movement. Together, these data constitute a direct perception of the robot's current state. Furthermore, through a particle filtering algorithm, these sensor data are probabilistically matched with environmental features in the hybrid map to predict the robot's real-time position in the global environment, including position coordinates and attitude angles, ensuring the robot's accurate estimation of its own position.

[0051] Based on the predicted real-time pose and the pre-set mission objective, the system performs global path planning on the hybrid map. This process utilizes the RRT* algorithm (Rapidly-Exploring Random Trees Star), which uses random sampling and path optimization to search for the optimal collision-free path from the current pose to the mission objective within the hybrid map. The RRT* algorithm considers not only path length but also path safety, ensuring efficient and safe robot movement in complex environments.

[0052] The planned global path is then broken down into a series of local sub-goals, each corresponding to an intermediate position that the robot needs to gradually approach. Combining the previously generated travel sequence with a library of pre-set motion patterns, the system further generates continuous joint control commands, which specify the angles and velocities that each joint should achieve at each time step to achieve precise movement from the current pose to the local sub-goal.

[0053] While the robot is performing its mission, the system monitors changes in the local area map in real time. This is achieved primarily by continuously analyzing data from sensors such as lidar and binocular cameras. Once a new obstacle is detected on the robot's path, the system immediately triggers the local path replanning mechanism. This mechanism uses the latest environmental information to recalculate the safe path from the current pose to the next local sub-target in the hybrid map and updates the joint control instructions accordingly, ensuring that the robot can flexibly respond to environmental changes and continue to perform tasks efficiently. For example, in a cave search and rescue mission, if an obstacle caused by a landslide suddenly appears in front of the robot, the system will quickly adjust the path to avoid the obstacle and replan the subsequent route to ensure the smooth progress of the search and rescue mission.

[0054] In a preferred embodiment, the steps of constructing the environment recognition model include: collecting multimodal training samples of lidar point cloud data, plantar pressure sensor data, joint torque sensor data, binocular camera image data, and IMU posture data, constructing a training data set, and labeling the samples in the training data set, including terrain type labels, posture mode labels, obstacle labels, and surface morphology labels; constructing a multi-branch neural network architecture, the multi-branch neural network architecture including a point cloud processing branch, an image processing branch, and a timing processing branch; using the training data set as input and multiple labels as output, performing model training based on the multi-branch neural network architecture to obtain an environment recognition model.

[0055] Furthermore, when building an environmental recognition model, it is necessary to first collect multimodal training samples, including lidar point cloud data, plantar pressure sensor data, joint torque sensor data, binocular camera image data, and IMU posture data. This data captures the characteristics of the robot's environment from different dimensions. For example, lidar point cloud data provides information about the environment's geometry, plantar pressure sensor data reflects the robot's contact with the ground, binocular camera image data presents the environment's visual appearance, and IMU posture data records the robot's motion. By integrating this multimodal data, a comprehensive training dataset is constructed, providing a rich and diverse sample foundation for subsequent model training.

[0056] Each sample in the training dataset is then meticulously labeled with terrain type labels (e.g., flat ground, slope, stairs), posture mode labels (e.g., standing, walking, crawling), obstacle labels (e.g., rocks, trees, walls), and surface morphology labels (e.g., smooth, rough, soft). These labels provide clear supervision information for the model, enabling it to accurately understand the relationship between different data and corresponding environmental features during the learning process.

[0057] In terms of model architecture design, a multi-branch neural network architecture is constructed, comprising a point cloud processing branch, an image processing branch, and a time series processing branch. The point cloud processing branch is responsible for processing LiDAR point cloud data, extracting geometric features from the point cloud through operations such as 3D convolution. The image processing branch processes binocular camera image data, using convolutional neural networks to extract visual features such as texture and color. The time series processing branch processes data with time series characteristics, such as plantar pressure sensor data, joint torque sensor data, and IMU posture data, capturing temporal dependencies within the data through recurrent neural networks or their variants.

[0058] Finally, the model is trained based on a multi-branch neural network, using the constructed training dataset as input and the multiple annotated labels as output. During the training process, the model continuously adjusts the parameters of each branch network to minimize the error between the predicted label and the true label, thereby learning an effective mapping relationship from multimodal data to environmental features. After sufficient training iterations, an environmental recognition model is finally obtained. This model can accurately identify information such as the terrain type, posture mode, obstacles, and surface morphology of the robot's environment, providing strong support for the robot's autonomous navigation and environmental adaptation. For example, in cave exploration and rescue missions, the environmental recognition model can accurately identify rocky terrain, narrow passages, and slippery surfaces in the cave, helping the robot make reasonable movement decisions.

[0059] Specifically, geometric features (such as the three-dimensional structure and depth information of lidar point clouds) are primarily used for terrain type identification and surface morphology analysis. Differences in terrain type (e.g., flat, sloped, stepped) and surface morphology (e.g., rough, smooth) are essentially differences in the geometric structure of the environment. These differences can be effectively distinguished by analyzing geometric features such as height distribution and curvature variations in point clouds. For example, stepped terrain appears as a layered, abrupt change in height in a point cloud, while the point cloud distribution of a rough surface is more discrete. Visual features (such as texture, color, and semantic information from binocular camera images) focus on obstacle identification and assist in terrain type recognition. Obstacles typically have unique visual semantic features. Using convolutional neural networks to extract edges and contours in images can accurately detect dynamic or static obstacles in the environment. Furthermore, combining visual and geometric features (such as spatiotemporal registration of point clouds and images) can further improve the accuracy of terrain type recognition. For example, image texture can be used to determine terrain material (e.g., grass, concrete), which can be used to assist geometric features in more accurately distinguishing different subtypes of the same terrain. Motion features (such as acceleration and angular velocity from IMUs and joint torque sensor data) primarily address posture pattern recognition. A robot's posture parameters (such as pitch and roll angles) and joint force patterns exhibit specific patterns under different terrains or motion states (e.g., straight-line movement, turning, and climbing). By analyzing time-series motion feature data, the robot can identify the current posture mode (e.g., "climbing posture" or "obstacle avoidance and turning posture") and provide feedback for motion control. Through multi-branch feature extraction and fusion, the model maps geometric, visual, and motion features to the spatial structure, visual semantics, and motion state of the environment, ultimately achieving comprehensive recognition of terrain type, surface morphology, obstacles, and posture patterns.

[0060] In a preferred embodiment, a sliding window algorithm is used to construct a local area map within the minimum data processing range of the initial environmental information, and the local area map is segmented based on a region growing algorithm to extract the traversable area and the obstacle area, including: determining the size parameters of the sliding window, the size parameters including the length, width and height of the window, and dynamically adjusting the size of the sliding window based on the size and motion characteristics of the robot; moving the sliding window in the three-dimensional space of the initial environmental information, extracting the point cloud data in each window, downsampling the point cloud data, and generating a local point cloud subset; constructing an octree structure based on the local point cloud subset, calculating the occupancy probability of each voxel, and generating a probability map, wherein the probability map Each voxel in the map contains an occupancy probability value and an uncertainty measure; a region growing algorithm is applied to the probability map, and voxels are clustered based on the normal vector similarity and spatial continuity between the voxels, and the point cloud is divided into different regions. Feature extraction is performed on each clustered region, and the geometric feature parameters of the region, including flatness, curvature and surface roughness, are calculated. The initial obstacle region and the initial traversable region are identified based on the geometric feature parameters; the boundary of the identified initial obstacle region and the initial traversable region are processed, and the region boundaries are optimized through expansion and erosion operations to generate a local region map that ultimately contains the obstacle region and the traversable region. The local region map contains the position, size and shape information of the obstacle region and the connectivity information of the traversable region.

[0061] Specifically, when constructing the local area map, the sliding window size parameters, including the length, width, and height, must be determined. To ensure the sliding window can adapt to data collection requirements in different environments, these size parameters must be dynamically adjusted based on the robot's size and motion characteristics. For example, if the robot is large or moves quickly, the sliding window size must be increased accordingly to cover a wider range of environmental information.

[0062] After determining the size parameters, the sliding window is moved within the three-dimensional space of the initial environmental information, extracting point cloud data window by window. Because the original point cloud data may contain redundancy and noise, to improve data processing efficiency, the extracted point cloud data needs to be downsampled to generate a local point cloud subset. Next, an octree structure is constructed based on the local point cloud subset. This structure divides the three-dimensional space into multiple voxels and calculates the occupancy probability of each voxel to generate a probability map. Each voxel in the probability map contains an occupancy probability value and an uncertainty measure, which provides an important basis for subsequent region segmentation.

[0063] The probability map is then subjected to a region growing algorithm, which clusters voxels based on normal vector similarity and spatial continuity, dividing the point cloud into distinct regions. Feature extraction is performed on each clustered region, and geometric characteristic parameters such as flatness, curvature, and surface roughness are calculated to further identify initial obstacle regions and initial traversable areas. For example, areas with high flatness may correspond to flat ground or walls, while areas with high curvature may correspond to obstacles or uneven terrain.

[0064] Finally, to optimize region boundaries and improve the accuracy of the local region map, the boundaries of the identified initial obstacle regions and initial traversable regions are processed. Region boundaries are adjusted through dilation and erosion operations to generate the final local region map. This local region map not only contains information about the location, size, and shape of the obstacle regions but also reflects the connectivity of the traversable regions, providing important support for the robot's autonomous navigation and path planning. For example, in cave exploration and rescue missions, the generated local region map can clearly display the distribution of obstacles and traversable pathways within the cave, helping the robot complete the search and rescue mission safely and efficiently.

[0065] In a preferred embodiment, in combination with the initial posture information, traversable area and obstacle area of ​​the robot, with minimum loss as a constraint, an initial travel path is planned on the local area map, and a travel sequence containing position and posture information is generated, including: establishing a path planning cost function, the cost function including distance cost, energy loss cost, posture adjustment cost and safety margin cost, wherein the energy loss cost is calculated based on the robot kinematic model and terrain features, and the posture adjustment cost is calculated based on the change in robot joint angle; defining an initial node and a target node in the traversable area of ​​the local area map, the initial node including the initial position coordinates and posture angle of the robot, and the target node including the position coordinates and expected posture of the task target; using the A* algorithm to search for the optimal path on the local area map, and during the search process, based on the cost The function calculates the total cost of each candidate node and selects the node with the smallest total cost as the extended node; smoothes the initial path obtained by the search, uses B-spline curves to interpolate the path points, generates a continuous and smooth path curve, and presets a safety distance from obstacles based on the path curve; discretizes the smoothed path curve into a travel sequence containing multiple nodes, and assigns target position coordinates, expected posture angle and arrival timestamp to each node, wherein the expected posture angle is calculated based on the path curvature and terrain characteristics, and the arrival timestamp is determined based on the robot's motion speed and acceleration constraints; verifies the feasibility of the travel sequence, checks whether the posture angle of each node is within the motion range of the robot joint, and whether the motion between adjacent nodes meets the robot's dynamic constraints, corrects or replans infeasible nodes, and obtains the final travel sequence.

[0066] Specifically, during the robot's local path planning process, a path planning cost function must be established to generate a movement sequence that includes position and posture information. This cost function comprehensively considers multiple factors, including distance cost, which refers to the impact of path length on time and resource consumption; energy loss cost, calculated based on the robot's kinematic model and terrain characteristics (such as slope and ground friction coefficient). Different terrains require different amounts of energy for robot movement; for example, moving on a slope consumes more energy than on flat ground; posture adjustment cost, calculated based on the change in the robot's joint angles. Frequent or large joint angle adjustments increase energy consumption and mechanical wear; and safety margin cost, which ensures that the robot maintains a sufficient safe distance from obstacles.

[0067] Next, the initial and target nodes are defined within the traversable region of the local region graph. The initial node contains the robot's initial position coordinates and attitude angles, which define the starting state of the robot's planning. The target node contains the position coordinates and desired attitude of the mission goal. For example, in a cave search and rescue mission, the goal might be to reach a specific location within the cave and perform search and rescue operations in the appropriate attitude.

[0068] Then, the A algorithm is used to search for the optimal path on the local area graph. Using a heuristic search strategy, the A algorithm calculates the total cost of each candidate node based on the aforementioned cost function. It then considers factors such as distance, energy, posture adjustment, and safety, selecting the node with the lowest total cost as the extension node and gradually approaches the target node, thereby finding the optimal path under given constraints.

[0069] The initial path obtained by the search is smoothed, and the path points are interpolated using B-spline curves. B-spline curves can generate a continuous and smooth path curve, avoiding sharp turns and ensuring the smooth movement of the robot. Furthermore, a safe distance from obstacles is preset based on the path curve to ensure that the robot does not collide with obstacles during movement.

[0070] The smoothed path curve is then discretized into a progression sequence consisting of multiple nodes, each of which is assigned a target position coordinate, a desired attitude angle, and an arrival timestamp. The desired attitude angle is calculated based on the path curvature and terrain characteristics. For example, around turns, the robot needs to adjust its attitude to maintain stability. The arrival timestamp is determined based on the robot's velocity and acceleration constraints, ensuring that the robot reaches each node within a reasonable timeframe.

[0071] Finally, the feasibility of the movement sequence is verified. The pose angle of each node is checked to ensure it is within the range of motion of the robot's joints. If it is outside this range, the robot will be unable to achieve that pose. The movement between adjacent nodes is also checked to ensure that it meets the robot's dynamic constraints, such as acceleration and speed limits. For infeasible nodes, corrections or replanning are performed, such as adjusting node position, pose, or timestamps. Ultimately, a final movement sequence that satisfies all constraints is obtained, providing a reliable basis for the robot's autonomous navigation.

[0072] The robot adaptive control method based on visual radiation perception provided by the embodiment of the present invention has at least the following technical effects:

[0073] 1. By integrating multi-source data such as lidar, plantar pressure sensors, joint torque sensors, binocular cameras, and IMUs, a comprehensive and accurate environment recognition model is constructed. This model further generates a hybrid map containing global topological structures and local detail features, providing the robot with richer and more accurate environmental information, significantly improving the robot's perception and understanding capabilities in complex environments.

[0074] 2. Based on the robot's initial posture, traversable area, and obstacle area, the initial travel path is planned with minimum loss as the constraint, and a travel sequence containing position and posture information is generated. The joint motor control parameter array is matched in the motion pattern library based on the surface type to realize the intelligent and adaptive robot motion control, enabling the robot to flexibly adjust its motion strategy according to different environments and task requirements, thereby improving motion efficiency and safety.

[0075] 3. Real-time monitoring of changes in local area maps. When new obstacles are detected, it can trigger local path replanning and update joint control instructions to ensure that the robot continues to maintain efficient and safe movement in dynamic environments, enhancing the robot's adaptability and response capabilities to complex and changing environments.

[0076] Example 2

[0077] like Figure 2 As shown, based on the same inventive concept as the robot adaptive control method based on visual radiation perception provided in Example 1, an embodiment of the present invention further provides a robot adaptive control system based on visual radiation perception. The system is applied to an intelligent robot, which is equipped with a laser radar, a plantar pressure sensor, a joint torque sensor, a binocular camera located on the head, and an IMU built into the torso. The system includes:

[0078] The data acquisition module 11 is used to obtain multiple data sets through the laser radar, plantar pressure sensor, joint torque sensor, binocular camera and IMU, pre-process the multiple data sets and input them into the environment recognition model to obtain initial environment information and robot initial posture information.

[0079] The region segmentation module 12 is used to construct a local region map within the minimum data processing range of the initial environmental information using a sliding window algorithm, and to segment the local region map based on a region growing algorithm to extract the passable region and the obstacle region.

[0080] The path planning module 13 is used to combine the initial posture information of the robot, the traversable area and the obstacle area, and plan the initial travel path on the local area map with the minimum loss as the constraint, and generate a travel sequence containing position and posture information, wherein each node in the travel sequence contains the target position coordinates, the expected posture angle and the arrival timestamp.

[0081] The surface evaluation module 14 is used to obtain the pressure distribution, texture characteristics and stiffness information of the contact surface through the robot surface flexible capacitance array and the plantar pressure sensor, construct a three-dimensional surface feature vector, perform spatial surface evaluation based on the surface feature vector, and identify the surface type.

[0082] The motion control module 15 is used to match the robot's travel sequence and surface type in a preset motion pattern library, obtain a joint motor control parameter array according to the matching result, and perform robot motion control through the parameter array.

[0083] Furthermore, the system is further configured to perform the following steps:

[0084] The point cloud data acquired by the lidar is temporally synchronized and spatially aligned with the image data collected by the binocular camera to generate a multimodal observation frame, and the ORB feature points and depth information in the multimodal observation frame are extracted. Feature matching is performed using a bag-of-words model to construct a common-view relationship between key frames. An initial pose graph is constructed based on the common-view relationship, and the pose parameters of the key frames are optimized using a bundle adjustment method. The pose graph is obtained by combining the pre-integration constraints of the IMU. Scene revisits are identified through a loop detection mechanism, and global pose graph optimization is triggered based on the pose graph to generate a consistent global environment map. The global environment map is fused with the local area map to generate a hybrid map containing global topological structure and local detail features.

[0085] Furthermore, the system is further configured to perform the following steps:

[0086] The topological feature points in the global environment map and the geometric feature points in the local area map are extracted, the transformation relationship between the maps is determined through feature matching, the obstacle information and the passable area information in the local area map are integrated into the global environment map using a weighted average method, the confidence distribution of the map is updated, and a hybrid map is obtained.

[0087] Furthermore, the system is further configured to perform the following steps:

[0088] The observation data of the plantar pressure sensor and joint torque sensor at the current moment are matched with the hybrid map, and the real-time posture of the robot is predicted using a particle filter algorithm. Based on the real-time posture and task objectives, a global path is planned on the hybrid map, and an optimal collision-free path is generated using the RRT* algorithm. The global path is decomposed into local sub-goals, and continuous joint control instructions are generated by combining the travel sequence and motion pattern library. Changes in the local area map are monitored in real time. When a new obstacle is detected, local path replanning is triggered and the joint control instruction update is executed.

[0089] Furthermore, the data acquisition module 11 is further configured to perform the following steps:

[0090] Collect multimodal training samples of lidar point cloud data, plantar pressure sensor data, joint torque sensor data, binocular camera image data, and IMU posture data to construct a training data set, and label the samples in the training data set, including terrain type labels, posture mode labels, obstacle labels, and surface morphology labels; construct a multi-branch neural network architecture, which includes a point cloud processing branch, an image processing branch, and a time series processing branch; use the training data set as input and multiple labels as output, perform model training based on the multi-branch neural network architecture, and obtain an environment recognition model.

[0091] Furthermore, the region segmentation module 12 is further configured to perform the following steps:

[0092] Determine the size parameters of the sliding window, which include the length, width and height of the window, and dynamically adjust the size of the sliding window based on the size and motion characteristics of the robot; move the sliding window in the three-dimensional space of the initial environmental information, extract the point cloud data within each window, downsample the point cloud data, and generate a local point cloud subset; construct an octree structure based on the local point cloud subset, calculate the occupancy probability of each voxel, and generate a probability map, wherein each voxel in the probability map contains an occupancy probability value and an uncertainty measure; apply a region growing algorithm to the probability map, based on the voxel spacing, The point cloud is clustered based on the normal vector similarity and spatial continuity, and the point cloud is divided into different regions. Feature extraction is performed on each cluster region, and the geometric feature parameters of the region are calculated, including flatness, curvature and surface roughness. The initial obstacle region and the initial traversable region are identified based on the geometric feature parameters. The boundary of the identified initial obstacle region and the initial traversable region are processed, and the region boundaries are optimized through expansion and erosion operations to generate a local region map containing the obstacle region and the traversable region. The local region map contains the position, size and shape information of the obstacle region and the connectivity information of the traversable region.

[0093] Furthermore, the path planning module 13 is further configured to perform the following steps:

[0094] A path planning cost function is established, wherein the cost function includes distance cost, energy loss cost, posture adjustment cost and safety margin cost, wherein the energy loss cost is calculated based on the robot kinematic model and terrain features, and the posture adjustment cost is calculated based on the change in robot joint angle; an initial node and a target node are defined in the traversable area of ​​the local area graph, wherein the initial node includes the initial position coordinates and posture angle of the robot, and the target node includes the position coordinates and expected posture of the task target; an A* algorithm is used to search for the optimal path on the local area graph, and during the search process, the total cost of each candidate node is calculated based on the cost function, and the node with the smallest total cost is selected as the extension node; the initial path obtained by the search is Smoothing is performed, and the path points are interpolated using B-spline curves to generate a continuous and smooth path curve. The safety distance from obstacles is preset based on the path curve. The smoothed path curve is discretized into a travel sequence containing multiple nodes, and each node is assigned a target position coordinate, an expected posture angle, and an arrival timestamp, wherein the expected posture angle is calculated based on the path curvature and terrain characteristics, and the arrival timestamp is determined based on the robot's motion speed and acceleration constraints. The feasibility of the travel sequence is verified to check whether the posture angle of each node is within the motion range of the robot joint and whether the motion between adjacent nodes meets the robot's dynamic constraints. Infeasible nodes are corrected or replanned to obtain the final travel sequence.

[0095] Through the above-mentioned detailed description of the robot adaptive control method based on visual radiation perception in this specification, those skilled in the art can clearly understand the robot adaptive control system based on visual radiation perception in this embodiment. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0096] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot adaptive control method based on visual radiation perception, characterized in that: The method is applied to an intelligent robot, which is equipped with a laser radar, a plantar pressure sensor, a joint torque sensor, a binocular camera located on the head, and an IMU built into the torso. The method includes: Acquire multiple data sets through the laser radar, plantar pressure sensor, joint torque sensor, binocular camera and IMU, pre-process the multiple data sets and input them into the environment recognition model to obtain initial environment information and initial posture information of the robot; A sliding window algorithm is used to construct a local area map within the minimum data processing range of the initial environmental information, and the local area map is segmented based on a region growing algorithm to extract the passable area and the obstacle area; Combining the robot's initial posture information, the traversable area, and the obstacle area, and taking minimum loss as a constraint, planning an initial travel path on the local area graph, generating a travel sequence containing position and posture information, wherein each node in the travel sequence contains the target position coordinates, the expected posture angle, and the arrival timestamp; The robot obtains pressure distribution, texture characteristics, and stiffness information of the contact surface through a flexible capacitive array on the robot surface and a plantar pressure sensor, constructs a three-dimensional surface feature vector, performs spatial surface evaluation based on the surface feature vector, and identifies the surface type, where the surface type is the type of the current contact surface, including hard ground, soft sand, gravel, or silt. Matching the robot's travel sequence and surface type in a preset motion pattern library, obtaining a joint motor control parameter array according to the matching result, and executing the robot's motion control through the parameter array; A sliding window algorithm is used to construct a local area map within the minimum data processing range of the initial environmental information, and the local area map is segmented based on a region growing algorithm to extract the passable area and the obstacle area, including: Determining size parameters of the sliding window, including the length, width, and height of the window, and dynamically adjusting the size of the sliding window based on the size and motion characteristics of the robot; Moving the sliding window in the three-dimensional space of the initial environment information, extracting point cloud data within each window, downsampling the point cloud data, and generating a local point cloud subset; constructing an octree structure based on the local point cloud subset, calculating the occupancy probability of each voxel, and generating a probability map, wherein each voxel in the probability map includes an occupancy probability value and an uncertainty measure; Applying a region growing algorithm to the probability map, clustering based on normal vector similarity and spatial continuity between voxels, dividing the point cloud into different regions, extracting features from each clustered region, calculating geometric feature parameters of the region, including flatness, curvature, and surface roughness, and identifying an initial obstacle region and an initial traversable region based on the geometric feature parameters; Performing boundary processing on the identified initial obstacle area and initial passable area, optimizing the area boundaries through dilation and erosion operations, and generating a local area map that ultimately includes the obstacle area and the passable area, wherein the local area map includes the location, size, and shape information of the obstacle area and the connectivity information of the passable area; Combining the robot's initial posture information, the traversable area, and the obstacle area, and taking minimum loss as a constraint, an initial travel path is planned on the local area map to generate a travel sequence containing position and posture information, including: Establishing a path planning cost function, the cost function includes a distance cost, an energy loss cost, a posture adjustment cost, and a safety margin cost, wherein the energy loss cost is calculated based on the robot kinematic model and terrain features, and the posture adjustment cost is calculated based on the robot joint angle change; An initial node and a target node are defined in the traversable area of ​​the local area graph, wherein the initial node includes the initial position coordinates and posture angle of the robot, and the target node includes the position coordinates and expected posture of the task target; An A* algorithm is used to search for an optimal path on the local area graph. During the search, a total cost of each candidate node is calculated based on the cost function, and a node with the smallest total cost is selected as an expansion node. The initial path obtained by the search is smoothed, and the path points are interpolated using a B-spline curve to generate a continuous and smooth path curve. The safe distance from obstacles is preset based on the path curve. Discretize the smoothed path curve into a travel sequence containing multiple nodes, and assign a target position coordinate, a desired posture angle, and an arrival timestamp to each node, wherein the desired posture angle is calculated based on the path curvature and terrain characteristics, and the arrival timestamp is determined based on the robot's motion speed and acceleration constraints; The feasibility of the movement sequence is verified to check whether the posture angle of each node is within the motion range of the robot joint and whether the motion between adjacent nodes meets the dynamic constraints of the robot. Infeasible nodes are corrected or replanned to obtain the final movement sequence.

2. The method according to claim 1, characterized in that The method further comprises: The point cloud data acquired by the lidar is temporally synchronized and spatially aligned with the image data collected by the binocular camera to generate a multimodal observation frame, and the ORB feature points and depth information in the multimodal observation frame are extracted. Feature matching is performed using the bag-of-words model to construct a common view relationship between key frames; An initial pose graph is constructed based on the common view relationship, the pose parameters of the key frames are optimized using the bundle adjustment method, and the pose graph is obtained by combining the pre-integration constraints of the IMU; Scene revisits are identified through a loop detection mechanism, and global pose graph optimization is triggered based on the pose graph to generate a consistent global environment map. The global environment map is fused with the local area map to generate a hybrid map containing global topological structure and local detail features.

3. The method according to claim 2, characterized in that The global environment map is merged with the local area map to generate a hybrid map containing global topological structure and local detail features, including: The topological feature points in the global environment map and the geometric feature points in the local area map are extracted, the transformation relationship between the maps is determined through feature matching, the obstacle information and the passable area information in the local area map are integrated into the global environment map using a weighted average method, the confidence distribution of the map is updated, and a hybrid map is obtained.

4. The method according to claim 3, characterized in that The method further comprises: Matching the observation data of the plantar pressure sensor and joint torque sensor at the current moment with the hybrid map, and predicting the real-time posture of the robot through a particle filter algorithm; Based on the real-time pose and mission objectives, a global path is planned on the hybrid map, and an optimal collision-free path is generated using the RRT* algorithm; Decomposing the global path into local sub-goals, and combining the travel sequence and motion pattern library to generate continuous joint control instructions; The changes of the local area map are monitored in real time. When a new obstacle is detected, the local path replanning is triggered and the joint control instruction update is executed.

5. The method according to claim 1, characterized in that The steps of constructing the environment recognition model include: Collect multimodal training samples of lidar point cloud data, plantar pressure sensor data, joint torque sensor data, binocular camera image data, and IMU posture data, construct a training dataset, and annotate the samples in the training dataset, including terrain type labels, posture mode labels, obstacle labels, and surface morphology labels; Constructing a multi-branch neural network architecture, wherein the multi-branch neural network architecture includes a point cloud processing branch, an image processing branch, and a time series processing branch; The training data set is used as input and the multiple labels are used as output. Model training is performed based on a multi-branch neural network architecture to obtain an environment recognition model.

6. A robot adaptive control system based on visual radiation perception, characterized in that: The system is applied to an intelligent robot, which is equipped with a laser radar, a plantar pressure sensor, a joint torque sensor, a binocular camera located on the head, and an IMU built into the torso, and is used to implement the robot adaptive control method based on visual radiation perception according to any one of claims 1 to 5. The system includes: a data acquisition module, configured to obtain multiple data sets through the laser radar, plantar pressure sensor, joint torque sensor, binocular camera, and IMU, pre-process the multiple data sets, and input them into the environment recognition model to obtain initial environment information and initial robot posture information; A region segmentation module is used to construct a local region map within the minimum data processing range of the initial environmental information using a sliding window algorithm, and to segment the local region map based on a region growing algorithm to extract traversable areas and obstacle areas; a path planning module, configured to plan an initial travel path on the local area graph based on the robot's initial posture information, the traversable area, and the obstacle area, with minimum loss as a constraint, and generate a travel sequence containing position and posture information, wherein each node in the travel sequence contains the target position coordinates, the expected posture angle, and the arrival timestamp; a surface assessment module for acquiring pressure distribution, texture characteristics, and stiffness information of the contact surface through the robot's surface flexible capacitance array and the plantar pressure sensor, constructing a three-dimensional surface feature vector, performing spatial surface assessment based on the surface feature vector, and identifying the surface type, where the surface type is the type of the current contact surface, including hard ground, soft sand, gravel, or mud; The motion control module is used to match the robot's travel sequence and surface type in a preset motion pattern library, obtain a joint motor control parameter array according to the matching result, and perform robot motion control through the parameter array.

Images