Robotic interactive local map construction method and system based on depth camera

Abstract

The application provides a kind of robot interactive local map construction method and system based on depth camera, it is related to robot real-time map construction technical field. Including: obtaining depth image and color image by depth camera, obtaining IMU data by IMU;Color image, IMU data are input into global odometer module, and the pose of robot under global coordinate system is estimated in real time;According to global pose, depth image is constructed to the elevation grid map with robot as center, the terrain feature and the passage cost of each grid are calculated, and the local passability map is constructed;The region to be modified selected by operator in color image is obtained, the map grid corresponding to the region to be modified in color image is obtained based on image alignment module, and the passage cost of corresponding grid is modified, to realize the interactive modification of local passability map.The application combines the data of depth camera and the operation intention of human operator, realizes the interactive construction and modification of map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of real-time map construction of robots, in particular to a robot interactive local map construction method and system based on a depth camera. BACKGROUND

[0002] With the development of technology, mobile robots have been widely used in service, agriculture, and medical fields due to their versatility and flexibility, and their potential value in urban security, national defense, and space exploration has become increasingly prominent. Mobile robots can work flexibly in unstructured environments by integrating intelligent navigation, environmental perception, and autonomous decision-making technologies, greatly expanding the application scenarios of robots. In particular, in human-robot collaboration, mobile robots have shown unique advantages by using advanced environmental perception and path planning capabilities to achieve safe and efficient collaboration with humans.

[0003] When mobile robots perform complex tasks in unknown environments, such as autonomous navigation, they need to have the ability to perceive their own state and environmental information through sensors, as well as the ability to accurately locate and construct environmental maps. However, traditional map construction methods have obvious limitations. These methods usually directly process raw point cloud data, and the constructed maps can only express basic geometric and topological information of the environment, and cannot obtain and describe semantic information and other higher-dimensional terrain features in the environment. This results in robots being able to establish geometric cognition of the surrounding environment, but not being able to truly understand the content and meaning of the environment. In other words, the robot knows "what the environment looks like", but does not know "what the environment is" and "what impact the environment has on me", which limits the robot's ability to analyze the surrounding environment in higher dimensions and severely limits the robot's ability to perform advanced tasks in complex environments. To solve this problem, human cognition of the environment needs to be integrated into the map.

[0004] Real-time interactive map construction, as a systematic discipline technology involving computer vision, sensor data processing, map construction, and human-computer interaction, is one of the popular directions in current robot research. The core task of map construction is to obtain three-dimensional point cloud data of the environment in real time through a depth camera, and dynamically update the environmental map combined with the motion information of the robot. During the construction process, the map needs to be interactive, i.e., it can adjust and optimize the map content in real time according to environmental changes or operator needs. For example, when dynamic obstacles or structural changes occur in the environment, the map can quickly update and reflect these changes, thereby providing accurate navigation support for the robot.

[0005] As the core sensor, depth cameras can provide high-resolution three-dimensional point cloud data, capturing detailed information of the environment. However, depth cameras may have data noise or accuracy degradation problems in strong light or long-distance scenes. Therefore, in order to achieve high-precision map construction in complex environments, it is usually necessary to combine other sensor data (such as inertial measurement unit IMU or lidar) for multi-source data fusion to improve the robustness and accuracy of the map. In addition, interactive maps not only need to update environmental information in real time, but also need to support operators to edit, label or plan paths through the interactive interface, so as to meet the diversified application requirements. By integrating the three-dimensional perception capability of depth cameras and real-time data processing technology, the interactive map construction method can achieve high-precision and high-robustness map construction and updating in dynamic environments, providing strong technical support for robot navigation, environmental perception and human-computer interaction. The application of this technology will greatly promote the autonomy and intelligence of intelligent robots in complex environments.

[0006] The Chinese patent with application number 201710244943.2 discloses a method for restricting the movement space of a mobile robot based on interaction mode. This method is based on a visual marker method, which specifies a custom virtual boundary through marker switching, thereby restricting the movement space of the mobile robot. This system can only demarcate a virtual boundary in a map that has already been constructed, and cannot modify the map while constructing it.

[0007] The Chinese patent with application number 202310845919.X discloses a human-computer interaction method applied to service robots. This method understands the operator's instructions through a sequence model, and can achieve point navigation control in home or office scenarios. However, this method lacks interactive capabilities for maps, and still uses traditional methods for map construction, which cannot incorporate the operator's intentions into the map construction process.

[0008] The Chinese patent with application number 202410851954.7 discloses a robot terrain perception and autonomous exploration method in complex environments. This method uses a lidar as a sensor, outputs a point cloud map through the A-LOAM algorithm, and calculates an elevation raster map. It combines various terrain feature data to obtain passable map information, and finally performs autonomous exploration on this map. However, this method lacks interactive capabilities for maps, and still uses traditional methods for map construction, and uses a lidar as a sensor, which is relatively expensive.

[0009] A patent with the application number 202411578428.4 discloses a method and system for marking obstacles in a map by an unmanned vehicle. The system can understand the instructions of an operator through a sequence model and can realize point navigation control in a home or office scene. However, the method lacks interactive ability for the map, and the traditional method is still used in map construction, which cannot combine the operator's intention into the map construction process. SUMMARY

[0010] To solve the problem that the real-time interactive map construction system in the prior art cannot effectively integrate artificial experience, an embodiment of the present application provides a robot interactive local map construction method and system based on a depth camera. The technical solution is as follows:

[0011] In one aspect, a robot interactive local map construction method based on a depth camera is provided. The method is realized by a robot interactive local map construction system based on a depth camera. The system includes an operator, an interactive operation platform, and a mobile robot. The method includes the following steps:

[0012] S1. The depth camera of the mobile robot acquires a depth image and a color image. The IMU of the mobile robot acquires IMU data. The depth image, the color image, and the IMU data are sent to the interactive operation platform.

[0013] S2. The interactive operation platform estimates the pose of the robot in a global coordinate system in real time according to the color image, the IMU data, and a global odometer module.

[0014] S3. The interactive operation platform constructs an elevation grid map centered on the robot according to the pose of the robot in the global coordinate system and the depth image, calculates the terrain features and the passing cost of each grid, further constructs a local passability map centered on the robot, and feeds back the local passability map centered on the robot to the operator through a local passability map display module.

[0015] S4. The operator selects a region to be modified in the color image through the interactive image display window modification module of the interactive operation platform. The interactive operation platform acquires the corresponding map grid of the region to be modified in the local passability map centered on the robot according to the image alignment module of the depth camera, modifies the passing cost of the grid, realizes real-time modification of the map by the operator, and generates a map that can be used for robot navigation.

[0016] Optionally, the step S3 of constructing an elevation grid map centered on the robot according to the pose of the robot in the global coordinate system and the depth image, calculating the terrain features and the passing cost of each grid, and further constructing a local passability map centered on the robot includes the following steps:

[0017] S31, setting a passability weight according to the passability of the robot for different terrain features.

[0018] S32, obtaining an elevation grid map centered on the robot according to the pose of the robot in the global coordinate system and the depth image.

[0019] S33, filtering the elevation grid map using a Savitzky-Golay filter to obtain a filtered elevation grid map.

[0020] S34, calculating parameters for describing the terrain features of each grid on the filtered elevation grid map.

[0021] S35, normalizing the terrain features to obtain the pass cost of the robot for each type of terrain.

[0022] S36, obtaining a local passability map centered on the robot according to the passability weight and the pass cost.

[0023] Optionally, the calculation of the parameters for describing the terrain features of each grid in S33 includes:

[0024] calculating the slope, as shown in the following formula (1):

[0025]

[0026] In the formula, slope represents the slope, Gx represents the height gradient in the x direction, and Gy represents the height gradient in the y direction. Gx is calculated according to the following formula (2):

[0027]

[0028] In the formula, h(i,j) represents the height of any point in the window, i represents the horizontal coordinate of the grid, j represents the vertical coordinate of the grid, and resolution represents the resolution of the map.

[0029] Gy is calculated according to the following formula (3):

[0030]

[0031] The roughness is calculated as shown in the following formula (4):

[0032]

[0033] In the formula, roughness represents the roughness, h(i,j) represents the height of any point in the window, h_center represents the height of the center point, and n represents the number of grids in the window.

[0034] The steepness is calculated as shown in the following equation (5):

[0035]

[0036] In the equation, stepness represents the steepness, h max represents the maximum height value within the window, h min represents the minimum height value within the window, and resolution represents the resolution of the map grid.

[0037] The slope change rate is calculated as shown in the following equation (6):

[0038]

[0039] In the equation, slope_change_rate represents the slope change rate, dsx represents the slope change rate in the x direction, and dsy represents the slope change rate in the y direction. Here, dsx is calculated according to the following equation (7):

[0040]

[0041] dsy is calculated according to the following equation (8):

[0042]

[0043] Optionally, the robot in S34 calculates the traversal cost for each type of terrain as shown in the following equation (9):

[0044] cost = 1 - e (-decay_factor*normalized_feature_value) (9)

[0045] In the equation, cost represents the traversal cost, decay_factor represents an adjustable parameter used to control the rate of cost growth, and normalized_feature_value represents the result of normalizing the feature value to the range [0, 1].

[0046] Optionally, the image alignment module of the depth camera is configured to establish a pixel-level mapping relationship between the color image and the depth image, and align the depth image and the color image according to the pixel-level mapping relationship.

[0047] Here, the two-dimensional coordinates in the depth image are restored to three-dimensional coordinates in the depth camera coordinate system according to the following equation (10):

[0048]

[0049] In the equation, P d represents the three-dimensional space coordinates corresponding to a pixel in the depth image, K d represents the intrinsic matrix of the depth lens in the depth camera, and (ud ,v d ) represents the pixel coordinates of each pixel in the depth image, and d represents the depth value corresponding to each pixel.

[0050] Project the three-dimensional points in the depth image into the pixel coordinate system of the color image by the following formula (11):

[0051]

[0052] In the formula, (u c ,v c ) represents the pixel coordinates of the pixel block in the color image, K c represents the intrinsic matrix of the color lens in the depth camera, R represents the rotation matrix between the color camera and the depth camera, and T trans represents the translation vector between the color camera and the depth camera.

[0053] Optionally, the method further comprises:

[0054] According to the mechanical structure characteristics and kinematic constraints of the robot, the physical threshold range of the terrain feature parameters is determined.

[0055] A weighted evaluation matrix is constructed, and the influence weight coefficients of each terrain feature on the motion energy consumption, safety risk and passing efficiency of the robot are quantified by the analytic hierarchy process.

[0056] On the other hand, a depth camera-based robot interactive local map construction system is provided, which is applied to the depth camera-based robot interactive local map construction method, and the system comprises an operator, an interactive operation platform and a mobile robot, wherein:

[0057] The operator is used to select a to-be-modified region in the color image through the interactive image display window modification module of the interactive operation platform.

[0058] The interactive operation platform is used to estimate the pose of the robot in the global coordinate system in real time according to the color image, the IMU data and the global odometer module; construct an elevation grid map centered on the robot according to the pose of the robot in the global coordinate system and the depth image, calculate the terrain feature and the passing cost of each grid, and further construct a local passability map centered on the robot, and feed back the local passability map centered on the robot to the operator through the local passability map display module; and modify the passing cost of the grid corresponding to the to-be-modified region in the local passability map centered on the robot according to the image alignment module of the depth camera, so as to realize real-time modification of the map by the operator and generate a map that can be used for robot navigation.

[0059] The mobile robot acquires a depth image and a color image through a depth camera of the mobile robot, acquires IMU data through an IMU of the mobile robot, and sends the depth image, the color image and the IMU data to the interactive operation platform.

[0060] Optionally, the interactive operation platform is further used for:

[0061] S31, setting a passability weight according to a passability of the robot to different terrain features.

[0062] S32, obtaining an elevation grid map centered on the robot according to a pose of the robot in a global coordinate system and the depth image.

[0063] S33, performing filtering processing on the elevation grid map using a Savitzky-Golay filter to obtain a filtered elevation grid map.

[0064] S34, calculating a parameter for describing a terrain feature of each grid on the filtered elevation grid map.

[0065] S35, performing normalization processing on the terrain feature to obtain a pass cost of the robot to each kind of terrain.

[0066] S36, obtaining a local passability map centered on the robot according to the passability weight and the pass cost.

[0067] Optionally, the interactive operation platform is further used for:

[0068] calculating a slope, as shown in the following formula (1):

[0069]

[0070] In the formula, slope represents the slope, Gx represents a height gradient in the x direction, and Gy represents a height gradient in the y direction. Gx is calculated according to the following formula (2):

[0071]

[0072] In the formula, h(i,j) represents a height of any point in the window, i represents a horizontal coordinate of the grid, j represents a vertical coordinate of the grid, and resolution represents a resolution of the map.

[0073] Gy is calculated according to the following formula (3):

[0074]

[0075] calculating a roughness, as shown in the following formula (4):

[0076]

[0077] where roughness represents roughness, h(i,j) represents height of any point in the window, h_center represents height of the center point, and n represents number of grids in the window.

[0078] The steepness is calculated as shown in the following equation (5):

[0079]

[0080] where stepness represents steepness, h max represents maximum height value in the window, h min represents minimum height value in the window, and resolution represents resolution of the map grid.

[0081] The slope change rate is calculated as shown in the following equation (6):

[0082]

[0083] where slope_change_rate represents slope change rate, dsx represents slope change rate in x direction, and dsy represents slope change rate in y direction. The dsx is calculated according to the following equation (7):

[0084]

[0085] The dsy is calculated according to the following equation (8):

[0086]

[0087] Optionally, the robot has a passing cost for each type of terrain, which is shown in the following equation (9):

[0088] cost = 1 - w (-decay_factor*normalized_feature_value) (9)

[0089] where cost represents passing cost, decay_factor represents an adjustable parameter for controlling rate of cost growth, and normalized_feature_value represents a result of normalizing the feature value to the range of [0, 1].

[0090] Optionally, the depth camera has an image alignment module for establishing a pixel-level mapping relationship between the color image and the depth image, and aligning the depth image and the color image according to the pixel-level mapping relationship.

[0091] where the two-dimensional coordinates in the depth image are restored to three-dimensional coordinates in the depth camera coordinate system by the following equation (10):

[0092]

[0093] wherein P d represents the three-dimensional space coordinates corresponding to a certain pixel in the depth image, K d represents the intrinsic matrix of the depth lens in the depth camera, (u d ,v d represents the pixel coordinates of each pixel in the depth image, and d represents the depth value corresponding to each pixel.

[0094] The three-dimensional point in the depth image is projected into the pixel coordinate system of the color image by the following formula (11):

[0095]

[0096] wherein (u c ,v c represents the pixel coordinates of the pixel block in the color image, K c represents the intrinsic matrix of the color lens in the depth camera, R represents the rotation matrix between the color camera and the depth camera, and T trans represents the translation vector between the color camera and the depth camera.

[0097] Optionally, the method further comprises:

[0098] According to the mechanical structure characteristics and kinematic constraints of the robot, a physical threshold range of the terrain feature parameters is determined.

[0099] A weighted evaluation matrix is constructed, and the influence weight coefficients of each terrain feature on the motion energy consumption, safety risk and passing efficiency of the robot are quantified by the analytic hierarchy process.

[0100] In another aspect, a robot interactive local map construction device is provided, which comprises a processor and a memory having computer readable instructions stored thereon, wherein the computer readable instructions are executed by the processor to implement any one of the above robot interactive local map construction methods based on a depth camera.

[0101] In another aspect, a computer readable storage medium is provided, which stores at least one instruction, wherein the at least one instruction is loaded and executed by a processor to implement any one of the above robot interactive local map construction methods based on a depth camera.

[0102] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0103] In the present application, a system capable of realizing real-time interactive map construction is provided. The system is equipped with a depth camera, and through the image data returned by the depth camera, combined with the operation intention of the human operator, the interactive construction and modification of the map are realized. The present application supports real-time modification and dynamic update in the map construction process, can adapt to complex dynamic environment, and expands the application range of the robot. Using the depth camera as the main sensor reduces the cost of hardware equipment. At the same time, through the visual interface and the mixed decision mechanism combined with the operator behavior, the semantic understanding ability of the robot to the map is improved, so that the reliability and usability of the map are significantly improved. A solid foundation is provided for the subsequent navigation logic of the robot. BRIEF DESCRIPTION OF DRAWINGS

[0104] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0105] Figure 1 It is a flow chart of a robot interactive local map construction method based on a depth camera provided by the embodiment of the present application;

[0106] Figure 2 It is a flow chart of a robot system for following a specific person autonomously in all-weather provided by the embodiment of the present application;

[0107] Figure 3 It is a schematic diagram of an interactive map construction module provided by the embodiment of the present application;

[0108] Figure 4 It is a schematic diagram of an interactive interface provided by the embodiment of the present application;

[0109] Figure 5 It is a flow chart of map modification of an interactive interface provided by the embodiment of the present application;

[0110] Figure 6 It is a schematic diagram of the effect of aligning the depth image and the color image provided by the embodiment of the present application;

[0111] Figure 7 It is a system block diagram of a robot interactive local map construction system based on a depth camera provided by the embodiment of the present application;

[0112] Figure 8 It is a structural schematic diagram of a robot interactive local map construction device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0113] The technical solutions in the present application will be described below with reference to the drawings.

[0114] In the embodiments of the present application, the words such as "example", "for example" are used to represent an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0115] In the embodiments of the present application, "image" and "picture" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "corresponding" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0116] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0117] In order to make the technical problems, technical solutions and advantages of the present application more clear, the following will be described in detail with reference to the drawings and specific embodiments.

[0118] The embodiments of the present application provide a robot interactive local map construction method based on a depth camera, which can be implemented by a robot interactive local map construction device, which can be a terminal or a server. As shown in the robot interactive local map construction method flowchart based on a depth camera, the processing flow of the method can include the following steps: Figure 1 、 Figure 2 As shown in the robot interactive local map construction method flowchart based on a depth camera, the processing flow of the method can include the following steps:

[0119] S1, the depth camera of the mobile robot acquires a depth image and a color image, the IMU of the mobile robot acquires IMU data, and the depth image, the color image and the IMU data are sent to an interactive operation platform.

[0120] S2, the interactive operation platform estimates the pose of the robot in the global coordinate system in real time according to the color image, the IMU data and a global odometer module.

[0121] In a feasible implementation, in order to ensure that the constructed map can accurately reflect the information of the environment around the robot, the pose of the robot in the global environment must be determined. The global odometer is a basic module of robot positioning and map construction, and its core task is to estimate the pose (position and attitude) of the robot in the global coordinate system in real time through multi-sensor data fusion, to provide a time and space reference for subsequent map construction, path planning and human-computer interaction.

[0122] In the present application, the VINS-MONO algorithm is used as the algorithm of the global odometer module, which fuses monocular vision and IMU (Inertial Measurement Unit) data through tight coupling optimization, and outputs high-precision robot pose in real time, providing a time and space reference for point cloud alignment of the depth camera, dynamic map construction and operator interaction, and ensuring the positioning robustness of the system in complex environments. By fusing monocular camera and IMU data, the pose estimation of the robot is realized, and finally the pose estimation of the robot in the global environment is output. Through the high-precision pose estimation of the VINS-MONO algorithm, the point cloud data of the depth camera can be aligned with the robot pose information, thereby providing an accurate time and space reference for dynamic map construction and operator interaction, and ensuring the positioning robustness of the system in complex environments.

[0123] S3, the interactive operation platform constructs an elevation grid map centered on the robot according to the pose of the robot in the global coordinate system and the depth image, calculates the terrain features and passability cost of each grid, and then constructs a local passability map centered on the robot, and feeds back the local passability map centered on the robot to the operator through a local passability map display module.

[0124] Optionally, the above step S3 can include steps S31-S36 as follows:

[0125] S31, set the passability weight according to the passability of the robot to different terrain features.

[0126] S32, obtain an elevation grid map centered on the robot according to the pose of the robot in the global coordinate system and the depth image.

[0127] In a feasible implementation, the passability map generated by the present application is a local map centered on the robot, and compared with maintaining a global map, the construction method of the local map has higher calculation efficiency and is suitable for rapid perception and response in unknown environments. The elevation grid map is a dynamic local elevation map, which is specially used to create and maintain a local elevation map centered on a small mobile platform. This method effectively avoids the huge calculation and storage overhead brought by the global map, and is more suitable for real-time application scenarios.

[0128] Further, the system accepts the original point cloud collected by the sensor and the odometer information of the platform to generate an elevation raster map centered on the robot. Subsequently, the system processes the elevation data in the elevation raster map. According to the received elevation map data, the calculation of the terrain feature and the corresponding cost, the creation of the original cost map, and the generation of the final map after filtering are realized, as shown in the flowchart of FIG. 3. Figure 3

[0129] S33, filtering the elevation raster map using a Savitzky-Golay filter to obtain a filtered elevation raster map.

[0130] In a feasible implementation, due to the limitations of platform motion and sensor characteristics, the original data often has problems such as noise, holes, and discontinuity, which leads to the generation of discontinuous invalid data in the elevation raster map. The Savitzky-Golay filter is used to filter the elevation raster map. The filter can smooth the noise while retaining the high-order derivative information of the signal, avoiding the over-smoothing caused by the traditional moving average filter, and is particularly suitable for processing terrain elevation data with obvious peaks and valleys:

[0131]

[0132] In the formula, D'(x, y) is the height value of the filtered raster. c i,j is a filter coefficient obtained by least squares. w represents the window size of the two-dimensional filter. represents the original height data in the window.

[0133] S34, calculating parameters for describing the terrain feature of each raster on the filtered elevation raster map.

[0134] In a feasible implementation, in order to realize the calculation of the terrain passability, the terrain feature of each raster is described by calculating the following parameters:

[0135] 1. Calculate the slope (slope):

[0136] The 3x3 Sobel operator is used to calculate the height gradient in the X direction and the Y direction. The gradient in the X direction is calculated according to formula (2):

[0137]

[0138] The gradient in the Y direction is calculated according to formula (3):

[0139]

[0140] The final slope value is the Euclidean norm of the gradients in the two directions: ​

[0141]

[0142] 2. Calculate roughness:

[0143] Calculate the standard deviation of height values in a local window of specified size (default 5x5). For each point in the window, calculate the sum of squares of height difference from the center point. The formula is shown in equation (5):

[0144]

[0145] where roughness represents roughness, h(i,j) represents height of any point in the window, h_center represents height of the center point, and n represents the number of grids in the window.

[0146] 3. Calculate steepness:

[0147] Calculate the ratio of maximum height difference to resolution in a local window of 3x3. The formula is shown in equation (6):

[0148]

[0149] where stepness represents steepness, h_max represents the maximum height value in the window, h_min represents the minimum height value in the window, and resolution represents the resolution of the map grid. max min

[0150] 4. Calculate slope change rate:

[0151] Calculate the rate of change of slope in X and Y directions for each grid. The rate of change in X direction:

[0152]

[0153] The rate of change in Y direction:

[0154]

[0155] The final slope change rate is the modulus of the rate of change in X and Y directions:

[0156]

[0157] where slope_change_rate represents slope change rate, dsx represents the rate of change of slope in x direction, and dsy represents the rate of change of slope in y direction.

[0158] ​​S35, normalize the terrain features to obtain the passing cost of the robot for each kind of terrain.

[0159] In an embodiment, according to the passing ability of the robot for different terrain features, the operator can set the passing ability weight of the robot for different terrain features in the interactive interface. All terrain features are normalized to obtain the passing cost of the robot for each kind of terrain:

[0160] cost=1-e (-decay_factor*normalized_feature_value) (10)

[0161] In the formula, cost represents the passing cost, decay_factor represents an adjustable parameter for controlling the rate of cost growth, and normalized_feature_value represents the result of normalizing the feature value to the range of [0, 1].

[0162] S36, obtain the local passability map centered on the robot according to the passing ability weight and the passing cost.

[0163] In an embodiment, the weighted sum of the terrain passing cost and the terrain passing ability weight of the map is the total passing cost of the map. Thus, the local passability map centered on the robot can be obtained.

[0164] S4, the operator selects the region to be modified in the color image through the interactive image display window of the interactive operation platform, and the interactive operation platform obtains the corresponding map grid of the region to be modified in the local passability map centered on the robot according to the image alignment module of the depth camera, and modifies the passing cost of the grid, so as to realize real-time modification of the map by the operator, and generate a map that can be used for robot navigation.

[0165] In an embodiment, the present application adopts a modular architecture design, and the interactive modification of the map is mainly realized by the interactive interface and the passing cost calculation module. Through the collaborative work of the multi-modal human-computer interaction interface and the intelligent decision algorithm, adaptive map construction and optimization in a dynamic environment are realized. The front end of the system develops a cross-platform visual interactive interface based on the QT framework, integrates a remote communication protocol stack, supports the operator to establish a low-latency bidirectional data link with the mobile robot through the TCP / IP protocol, and effectively guarantees the real-time interaction demand in the remote operation scene.

[0166] In the system initialization phase, the user needs to complete the motion characteristics modeling of the mobile platform through the parameter configuration panel. Specifically, according to the mechanical structure characteristics and kinematic constraints of the target robot, the physical threshold range of the multi-dimensional terrain characteristic parameters is input in the interactive interface, including but not limited to the maximum climbable slope, ground roughness tolerance, vertical obstacle crossing height and other key performance indicators. At the same time, a weighted evaluation matrix needs to be constructed, and the influence weight coefficients of each terrain feature on the robot's motion energy consumption, safety risk and passing efficiency are quantified by the analytic hierarchy process (AHP), providing a mathematical basis for subsequent passability modeling.

[0167] As shown in the interface, the human-computer interaction interface adopts a three-view layout design: the left side is a two-dimensional grid map visualization window constructed based on SLAM technology, which displays the local passability map centered on the robot in real time; the central area integrates an image labeling module, which supports the visualization and interactive labeling of depth camera raw data; the right side control panel includes motion instruction issuing, map correction confirmation and system state monitoring function units, forming a complete map closed-loop correction system. Figure 4

[0168] Interactive modification of the map is mainly realized through the operator's delineation operation in the interactive image display window and the corresponding buttons. In the interactive image display window, the current image can be fixed by clicking with the mouse, and the system will record the global pose information of the robot at the current time, and then the delineation operation can be performed in the image.

[0169] By clicking or dragging with the mouse, an area can be selected in the color image, and then the operator can change the properties of the selected area according to their needs (such as considering the selected area as a puddle). The system will preset road, grass, muddy ground, obstacle and other preset terrain properties. After the area is delineated in the image, the system will find the corresponding map grid of the delineated image area through the image alignment module, and establish the mapping between the image and the grid map. After the operator selects the terrain properties, the system will comprehensively consider the robot's passability and the terrain characteristics of the delineated area, and recalculate the passability cost of the delineated area. The passability cost of the delineated area in the image will be changed and reflected in the constructed passability map, changing the value of the map grid corresponding to the delineated image area. In this way, interactive modification of the passability map is realized.

[0170] ​All the manual correction regions are recorded in the system, when the robot subsequently perceives the same environment region, the system will preferentially adopt the manual correction data to cover the real-time perception result, so as to ensure the effective inheritance and persistent application of human experience. This hybrid decision mechanism not only retains the environment adaptation ability of the autonomous mapping system, but also integrates the domain knowledge of human operators, significantly improving the map reliability in complex scenes. The process of interactive map modification is as shown in Figure 5 .

[0171] Optionally, an image alignment module of the depth camera is configured to establish a pixel-level mapping relationship between the color image and the depth image, and align the depth image and the color image according to the pixel-level mapping relationship.

[0172] In a feasible implementation, in a depth camera system, the core task of image alignment is to establish a pixel-level mapping relationship between a color image (RGB) and a depth image (Depth), so as to ensure that the color information and the depth value of the same scene point are accurately corresponding. This process is the basis for subsequent three-dimensional reconstruction and interactive map construction, and the alignment effect is as shown in Figure 6 .

[0173] The depth camera can collect a pair of depth images and color images in the same frame. For the camera, the intrinsic matrix K can be represented as:

[0174]

[0175] In the formula, dx and dy respectively represent the physical size of each pixel on the image plane, (u0, v0) is the origin coordinate in the image coordinate system, and f is the focal length of the camera. Let the intrinsic matrix of the depth lens in the depth camera be K d , and the intrinsic matrix of the color lens be K c . The extrinsic matrix T between the color camera and the depth camera, which contains the rotation matrix R and the translation vector T trans , can be represented as:

[0176]

[0177] In order to restore the two-dimensional coordinates in the depth image to the three-dimensional coordinates in the camera coordinate system, the following formula can be used:

[0178]

[0179] In the formula, P d represents the three-dimensional space coordinates corresponding to a pixel in the depth image, K d represents the intrinsic matrix of the depth lens in the depth camera, (u d , v drepresents the pixel coordinate of each pixel in the depth image, and d represents the depth value corresponding to each pixel, i.e. the distance between the pixel and the camera.

[0180] In order to map the corresponding three-dimensional point in the depth image to the color image, the following formula needs to be used:

[0181]

[0182] In the formula, P d is the three-dimensional point coordinate in the depth image (converted by the depth value), R and T trans represent the rotation matrix and the translation vector between the color camera and the depth camera, K c represents the intrinsic matrix of the color lens, (u c , v c ) represents the pixel coordinate of the pixel block in the color image.

[0183] After the image alignment operation, the pixels in the color image will establish a corresponding mapping relationship with the pixels in the depth image, and finally the depth value corresponding to each pixel in the color image will be obtained, which provides a basis for subsequent user interactive map labeling. It is a prerequisite for realizing the present application to obtain accurate camera intrinsic matrix K and extrinsic matrix T through camera calibration.

[0184] In the embodiment of the present application, a system capable of realizing real-time interactive map construction is provided. The system is equipped with a depth camera, and through the image data returned by the depth camera, the interactive construction and modification of the map are realized in combination with the operation intention of the human operator. The present application supports real-time modification and dynamic update in the map construction process, can adapt to complex dynamic environments, and expands the application range of robots. Using a depth camera as the main sensor reduces the cost of hardware devices. At the same time, through the visual interface and the mixed decision mechanism combined with the behavior of the operator, the semantic understanding ability of the robot to the map is improved, so that the reliability and ease of use of the map are significantly improved. A solid foundation is provided for the subsequent navigation logic of the robot.

[0185] Figure 7 is a robot interactive local map construction system block diagram based on a depth camera according to an example embodiment, and the system is used for a robot interactive local map construction method based on a depth camera. Referring to Figure 7 , the system includes an operator, an interactive operation platform, and a mobile robot, wherein:

[0186] The operator is used to select a to-be-modified region in the color image through the interactive image display window modification module of the interactive operation platform.

[0187] The interactive operation platform is used for estimating the pose of the robot in a global coordinate system in real time according to color images, IMU data and a global odometer module; constructing an elevation grid map centered on the robot according to the pose of the robot in the global coordinate system and a depth image, calculating the terrain feature and the passing cost of each grid, and then constructing a local passability map centered on the robot, and feeding back the local passability map centered on the robot to the operator through a local passability map display module; obtaining the corresponding grid of the region to be modified in the local passability map centered on the robot according to an image alignment module of the depth camera, and modifying the passing cost of the grid, so as to realize real-time modification of the map by the operator, and generate a map that can be used for robot navigation.

[0188] The mobile robot obtains a depth image and a color image through a depth camera of the mobile robot, and obtains IMU data through an IMU of the mobile robot, and sends the depth image, the color image and the IMU data to the interactive operation platform.

[0189] Optionally, the interactive operation platform is further used for:

[0190] S31, setting a passing ability weight according to the passing ability of the robot for different terrain features.

[0191] S32, obtaining an elevation grid map centered on the robot according to the pose of the robot in the global coordinate system and the depth image.

[0192] S33, filtering the elevation grid map using a Savitzky-Golay filter to obtain a filtered elevation grid map.

[0193] S34, calculating a parameter for describing the terrain feature of each grid for the filtered elevation grid map.

[0194] S35, normalizing the terrain feature to obtain the passing cost of the robot for each kind of terrain.

[0195] S36, obtaining a local passability map centered on the robot according to the passing ability weight and the passing cost.

[0196] Optionally, the interactive operation platform is further used for:

[0197] calculating the slope, as shown in the following formula (1):

[0198]

[0199] In the formula, slope represents the slope, Gx represents the height gradient in the x direction, and Gy represents the height gradient in the y direction. Gx is calculated according to the following formula (2):

[0200]

[0201] where h(i,j) represents the height of any point in the window, i represents the horizontal coordinate of the grid, j represents the vertical coordinate of the grid, and resolution represents the resolution of the map.

[0202] Gy is calculated according to the following formula (3):

[0203]

[0204] The roughness is calculated as shown in the following formula (4):

[0205]

[0206] where roughness represents the roughness, h(i,j) represents the height of any point in the window, h_center represents the height of the center point, and n represents the number of grids in the window.

[0207] The steepness is calculated as shown in the following formula (5):

[0208]

[0209] where stepness represents the steepness, h max represents the maximum height value in the window, h min represents the minimum height value in the window, and resolution represents the resolution of the map grid.

[0210] The slope change rate is calculated as shown in the following formula (6):

[0211]

[0212] where slope_change_rate represents the slope change rate, dsx represents the change rate of the slope in the x direction, and dsy represents the change rate of the slope in the y direction. The dsx is calculated according to the following formula (7):

[0213]

[0214] The dsy is calculated according to the following formula (8):

[0215]

[0216] Optionally, the passing cost of the robot for each type of terrain is shown in the following formula (9):

[0217] cost = 1 - e (-decay_factor*normalized_feature_value) (9)

[0218] In the formula, cost represents the passing cost, decay_factor represents an adjustable parameter for controlling the rate of cost growth, and normalized_feature_value represents the result of normalizing the feature value to the range [0, 1].

[0219] Optionally, the depth camera comprises an image alignment module configured to establish a pixel-level mapping relationship between the color image and the depth image, and align the depth image and the color image according to the pixel-level mapping relationship.

[0220] In the formula, the two-dimensional coordinates in the depth image are restored to three-dimensional coordinates in the depth camera coordinate system by the following formula (10):

[0221]

[0222] In the formula, P d represents the three-dimensional space coordinates corresponding to a pixel in the depth image, K d represents the intrinsic matrix of the depth lens in the depth camera, (u d ,v d ) represents the pixel coordinates of each pixel in the depth image, and d represents the depth value corresponding to each pixel.

[0223] The three-dimensional point in the depth image is projected into the pixel coordinate system of the color image by the following formula (11):

[0224]

[0225] In the formula, (u c ,v c ) represents the pixel coordinates of the pixel block in the color image, K c represents the intrinsic matrix of the color lens in the depth camera, R represents the rotation matrix between the color camera and the depth camera, and T trans represents the translation vector between the color camera and the depth camera.

[0226] Optionally, the method further comprises:

[0227] According to the mechanical structure characteristics and kinematic constraints of the robot, a physical threshold range of the terrain feature parameters is determined.

[0228] A weighted evaluation matrix is constructed, and the influence weight coefficients of each terrain feature on the motion energy consumption, safety risk and passing efficiency of the robot are quantified by the analytic hierarchy process.

[0229] This invention provides a system for real-time interactive map construction. The system is equipped with a depth camera. Through image data transmitted from the depth camera and combined with the operator's intentions, it enables interactive map construction and modification. This invention supports real-time modification and dynamic updates during map construction, adapting to complex dynamic environments and expanding the robot's application scope. Using a depth camera as the main sensor reduces hardware costs. Simultaneously, the use of a visual interface and a hybrid decision-making mechanism incorporating operator behavior enhances the robot's semantic understanding of the map, significantly improving its reliability and usability. This provides a solid foundation for the robot's subsequent navigation logic.

[0230] Figure 8 This is a schematic diagram of the structure of a robot-interactive local map building device provided in an embodiment of the present invention, such as... Figure 8 As shown, the robot-interactive local map building device may include the above-mentioned Figure 7 The illustrated system is a robot-interactive local map building system based on a depth camera. Optionally, the robot-interactive local map building device 410 may include a first processor 2001.

[0231] Optionally, the robot-interactive local map building device 410 may also include a memory 2002 and a transceiver 2003.

[0232] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0233] The following is combined Figure 8 A detailed description of each component of the robot-interactive local map building device 410 is provided below:

[0234] The first processor 2001 is the control center of the robot interactive local map building device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0235] Optionally, the first processor 2001 can execute various functions of the robot interactable local map construction device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0236] In a specific implementation, as an example, the first processor 2001 can include one or more CPUs, such as the CPU0 and the CPU1 shown in FIG. 2. Figure 8

[0237] In a specific implementation, as an example, the robot interactable local map construction device 410 can also include multiple processors, such as the first processor 2001 and the second processor 2004 shown in FIG. 2. Each of these processors can be a single-CPU or a multi-CPU. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Figure 8

[0238] The memory 2002 is configured to store software programs for implementing the solutions of the present application and is controlled by the first processor 2001 to perform the implementation, and the specific implementation can refer to the above method embodiments, which will not be repeated here.

[0239] Optionally, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but not limited to this. The memory 2002 can be integrated with the first processor 2001 or exist independently and be coupled with the first processor 2001 through an interface circuit (not shown in FIG. 2) of the robot interactable local map construction device 410, and the embodiments of the present application do not make specific limitations here. Figure 8

[0240] ​​​The transceiver 2003 is configured to communicate with a network device or a terminal device.

[0241] Optionally, the transceiver 2003 can include a receiver and a transmitter (not shown in the figure separately). Figure 8 The receiver is configured to implement a receiving function, and the transmitter is configured to implement a transmitting function.

[0242] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or can exist independently and be coupled to the first processor 2001 through an interface circuit (not shown in the figure) of the robot interactable local map construction device 410, and the embodiments of the present application do not make a specific limitation thereon. Figure 8

[0243] It should be noted that the structure of the robot interactable local map construction device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements. Figure 8

[0244] In addition, the technical effects of the robot interactable local map construction device 410 can refer to the technical effects of the robot interactable local map construction method based on the depth camera described in the above method embodiments, which will not be described here.

[0245] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0246] ​​It should also be understood that the memory in the embodiments of the present application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be random access memory (RAM) used as external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0247] The above-described embodiments can be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0248] It should be understood that the term "and / or" herein merely describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects, but can also represent an "and / or" relationship, which can be understood in the context before and after.

[0249] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0250] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0251] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0252] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0253] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0254] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0255] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0256] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0257] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for constructing interactive local maps for robots based on depth cameras, characterized in that, The method is implemented by a depth camera-based interactive local map building system for robots, the system including an operator, an interactive operating platform, and a mobile robot, and the method includes: S1. The depth camera of the mobile robot acquires depth images and color images, the IMU of the mobile robot acquires IMU data, and the depth images, color images and IMU data are sent to the interactive operating platform. S2. The interactive operating platform estimates the robot's pose in the global coordinate system in real time based on the color image, IMU data, and global odometry module. S3. The interactive operation platform constructs an elevation grid map centered on the robot based on the robot's pose in the global coordinate system and the depth image, calculates the terrain features and passage cost of each grid, and then constructs a local accessibility map centered on the robot. The local accessibility map display module then feeds the local accessibility map centered on the robot back to the operator. S4. The operator selects the area to be modified in the color image through the interactive image display window modification module of the interactive operation platform. The interactive operation platform obtains the map grid corresponding to the area to be modified in the local accessibility map centered on the robot according to the image alignment module of the depth camera, and modifies the access cost of the grid, so that the operator can modify the map in real time and generate a map that can be used for robot navigation.

2. The method for constructing interactive local maps for robots based on depth cameras according to claim 1, characterized in that, S3 involves constructing an elevation grid map centered on the robot based on the robot's pose in the global coordinate system and the depth image, calculating the terrain features and passage cost of each grid cell, and then constructing a local drivability map centered on the robot, including: S31. Set the accessibility weights according to the robot's ability to navigate different terrain features; S32. Based on the robot's pose in the global coordinate system and the depth image, obtain an elevation grid map centered on the robot. S33. The elevation raster map is filtered using a Savitzky-Golay filter to obtain a filtered elevation raster map. S34. For the filtered elevation raster map, calculate the parameters used to describe the terrain features of each raster. S35. Normalize the terrain features to obtain the robot's passage cost for each type of terrain. S36. Based on the accessibility weights and access costs, a local accessibility map centered on the robot is obtained.

3. The method for constructing interactive local maps for robots based on depth cameras according to claim 2, characterized in that, The parameters calculated in S33 to describe the terrain features of each grid cell include: The slope is calculated as shown in equation (1): In the formula, slope represents the gradient, Gx represents the height gradient in the x-direction, and Gy represents the height gradient in the y-direction; where Gx is calculated according to the following formula (2): In the formula, h(i,j) represents the height of any point within the window, i represents the horizontal coordinate of the grid, j represents the vertical coordinate of the grid, and resolution represents the resolution of the map. Gy is calculated according to the following formula (3): The roughness is calculated as shown in equation (4): In the formula, roughness represents roughness, h(i,j) represents the height of any point within the window, h_center represents the height of the center point, and n represents the number of grid cells within the window; The steepness is calculated as shown in equation (5): In the formula, stepness represents steepness, and h max h represents the maximum height value within the window. min This indicates the minimum height value within the window, and resolution indicates the resolution of the map raster. The slope change rate is calculated as shown in equation (6): In the formula, slope_change_rate represents the rate of change of slope, dsx represents the rate of change of slope in the x-direction, and dsy represents the rate of change of slope in the y-direction. Wherein, dsx is calculated according to the following formula (7): dsy is calculated according to the following formula (8); 4. The method for constructing interactive local maps for robots based on depth cameras according to claim 2, characterized in that, The travel cost of the robot in S34 for each type of terrain is shown in the following equation (9): cost=1-e (-decay_factor*normalized_feature_value) (9) In the formula, cost represents the cost of passage, decay_factor represents an adjustable parameter used to control the rate of cost growth, and normalized_feature_value represents the result of normalizing the feature value to the range of [0,1].

5. The method for constructing interactive local maps for robots based on depth cameras according to claim 1, characterized in that, The image alignment module of the depth camera is used to establish a pixel-level mapping relationship between the color image and the depth image, and to align the depth image and the color image according to the pixel-level mapping relationship; The two-dimensional coordinates in the depth image are restored to the three-dimensional coordinates in the depth camera coordinate system using the following formula (10): In the formula, P d K represents the three-dimensional spatial coordinates corresponding to a pixel in a depth image. d This represents the intrinsic parameter matrix of the depth lens in a depth camera, (u d ,v d ) represents the pixel coordinates of each pixel in the depth image, and d represents the depth value corresponding to each pixel; The three-dimensional points in the depth image are projected into the pixel coordinate system of the color image using the following equation (11): In the formula, (u c ,v c K represents the pixel coordinates of a pixel block in a color image. c R represents the intrinsic parameter matrix of the color lens in the depth camera, R represents the rotation matrix between the color camera and the depth camera, and T represents the intrinsic parameter matrix of the color lens in the depth camera. trans This represents the translation vector between the color camera and the depth camera.

6. The method for constructing interactive local maps for robots based on depth cameras according to claim 1, characterized in that, The method further includes: Based on the robot's mechanical structure characteristics and kinematic constraints, determine the physical threshold range of terrain feature parameters; A weighted evaluation matrix was constructed, and the influence of various terrain features on robot motion energy consumption, safety risks and passage efficiency was quantified by the analytic hierarchy process.

7. A robot interactive local map construction system based on a depth camera, wherein the robot interactive local map construction system based on a depth camera is used to implement the robot interactive local map construction method based on a depth camera as described in any one of claims 1-6, characterized in that, The system includes an operator, an interactive operating platform, and a mobile robot, wherein: The operator is used to select the area to be modified in the color image through the interactive image display window modification module of the interactive operation platform; The interactive operating platform is used to estimate the robot's pose in the global coordinate system in real time based on the color image, IMU data, and global odometry module; construct an elevation grid map centered on the robot based on the robot's pose in the global coordinate system and the depth image, calculate the terrain features and passage cost of each grid, and then construct a local accessibility map centered on the robot. The local accessibility map display module then feeds the local accessibility map centered on the robot back to the operator; the image alignment module of the depth camera obtains the map grid corresponding to the area to be modified in the local accessibility map centered on the robot, modifies the passage cost of the grid, and enables the operator to modify the map in real time, generating a map that can be used for robot navigation; The mobile robot acquires depth and color images through its depth camera and IMU data through its IMU, and then sends the depth images, color images, and IMU data to the interactive operating platform.

8. The robot interactive local map building system based on a depth camera according to claim 7, characterized in that, Based on the robot's pose in the global coordinate system and the depth image, an elevation grid map centered on the robot is constructed. The terrain features and passage cost of each grid cell are calculated, thereby constructing a local drivability map centered on the robot, including: S31. Set the accessibility weights according to the robot's ability to navigate different terrain features; S32. Based on the robot's pose in the global coordinate system and the depth image, obtain an elevation grid map centered on the robot. S33. The elevation raster map is filtered using a Savitzky-Golay filter to obtain a filtered elevation raster map. S34. For the filtered elevation raster map, calculate the parameters used to describe the terrain features of each raster. S35. Normalize the terrain features to obtain the robot's passage cost for each type of terrain. S36. Based on the accessibility weights and access costs, a local accessibility map centered on the robot is obtained.

9. A robot-interactive local map building device, characterized in that, The robot-interactive local map building device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images