Indoor SLAM map construction method and system

Abstract

The invention provides an indoor SLAM map construction method and system, and relates to the technical field of image processing, and the construction method comprises the steps: building a unified global space reference through cross-device space-time calibration, supplementing the visual blind area information of a robot in real time through the global static features extracted by an environment camera, and obtaining the visual blind area information of the robot; and meanwhile, dynamic interference features are accurately eliminated in combination with continuous frame analysis, and finally, the pose of the robot is continuously corrected by taking global features as anchor point constraints in back-end optimization. The complete technology chain enables the constructed indoor three-dimensional map to have higher integrity, precision and consistency, significantly improves the positioning robustness and navigation reliability of the robot in an environment with dense goods shelves and dynamic activities, reduces the performance dependence on a robot body sensor and the scene reconstruction cost, and improves the reliability of the robot. And a stable and efficient environment sensing basis is provided for automatic operation of industries such as storage and archive management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an indoor SLAM map construction method and system. BACKGROUND

[0002] The existing indoor SLAM technology mainly relies on the local sensor carried by the mobile robot to realize positioning and mapping. Typical schemes include: a method based on a laser radar, which obtains high-precision geometric point clouds by scanning, but the cost is high and it is difficult to provide semantic information; a method based on a visual sensor, which restores the three-dimensional structure and texture of the scene through feature matching, but it is sensitive to light changes and weak texture environments; and a collaborative scheme that fuses visual, inertial and other multi-source data, aiming to improve the robustness and accuracy of the system.

[0003] However, in indoor scenes such as warehouses, archives and the like where shelves are dense, the passageway is narrow, and personnel and equipment often move, the above-mentioned schemes that rely on the self-perception of the robot mobile platform face common bottlenecks: first, the field of view of the sensor is limited, and local observation blind areas are easily formed in the stacking of goods and the shielding area of shelves, resulting in missing maps or geometric distortion; second, although existing research has focused on dynamic scene processing, in environments such as warehouses where dynamic objects and static backgrounds have similar appearances and intermittent motion may occur, the accurate identification and removal of dynamic features remains a great challenge, which easily leads to pose estimation drift; more importantly, existing methods essentially rely on the robot's own motion trajectory for recursive map stitching and optimization, and lack a fixed, global and continuous observation reference to anchor the map coordinate system, so the cumulative error cannot be fundamentally suppressed, resulting in a decline in the overall consistency of the map after a long time and a large range of operation. These defects collectively limit the practical application of existing SLAM technology in automated warehouse operations that require high precision and high reliability. SUMMARY

[0004] The present application solves one or more of the above-mentioned related technical problems.

[0005] To solve the above-mentioned problems, the present application provides an indoor SLAM map construction method and system.

[0006] In a first aspect, the present application provides an indoor SLAM map construction method, comprising: observing a static calibration target by a visual sensor carried by a robot and an environmental camera arranged indoors, and establishing a coordinate transformation relationship between a robot local coordinate system in which the visual sensor is located and a global coordinate system in which the environmental camera is located; extracting static features based on a global image obtained by the environmental camera, and constructing and updating a global landmark database according to the static features based on the global coordinate system; Based on the local image and the corresponding depth information acquired by the visual sensor, local features are extracted, and based on the robot local coordinate system, a local feature point cloud is formed according to the local features and the corresponding depth information; based on the coordinate conversion relationship, the static features in the global landmark database are converted to the robot local coordinate system and matched and fused with the local feature point cloud to form a fused SLAM point cloud feature set; Based on the global image, a dynamic target is identified and a dynamic region mask is generated, the dynamic region mask is converted to the global coordinate system, and based on the coordinate conversion relationship, the dynamic region mask is converted to the robot local coordinate system; from the fused SLAM point cloud feature set, feature points located in the dynamic region mask in the robot local coordinate system are removed to obtain an updated SLAM point cloud feature set; Based on the updated SLAM point cloud feature set, a pose graph is constructed, the static features in the global landmark database are added as global anchor points to the pose graph for constraint optimization to obtain an optimized robot pose, and based on the optimized robot pose, the depth information collected by the visual sensor, and the global landmark database, a three-dimensional map of the indoor environment is generated.

[0007] Optionally, the establishment of the coordinate conversion relationship between the robot local coordinate system in which the visual sensor is located and the global coordinate system in which the environment camera is located comprises: A plurality of static calibration targets are determined, and the visual sensor and the environment camera are used to collect visual sensor images and environment camera images corresponding to the static calibration targets, respectively; Feature point pixel coordinates corresponding to the static calibration targets are extracted from the visual sensor images and the environment camera images, respectively; Based on the perspective-n-point algorithm, an initial rotation matrix and a translation vector are obtained according to the feature point pixel coordinates, parameter data, and actual physical dimensions of the static calibration targets; The rotation matrix and the translation vector are optimized by a predetermined optimization algorithm to obtain the coordinate conversion relationship.

[0008] Optionally, the extraction of local features based on the local image acquired by the visual sensor comprises: The local image is processed according to a predetermined semantic segmentation network to obtain a semantic segmentation result, and the semantic segmentation result includes a static target category result; The static features corresponding to the static targets are extracted according to the static target category result; The static features include the position and contour information of the static target in the global image, and the static target categories include shelves, columns, and aisle marker lines.

[0009] Optionally, the construction process of the preset semantic segmentation network includes: Obtain the original YOLOv5s network, and improve the original YOLOv5s network to obtain the improved YOLOv5s network; The original YOLOv5s network includes a backbone network, a Neck network, and an output network; the improvement process includes: The standard Bottleneck residual module in the backbone network is replaced with an enhanced residual module that integrates channel and spatial attention mechanisms to enhance the feature extraction capability of scene rule structures. A cross-scale attention module was added to the Neck network, and the output network was transformed into a detection-segmentation integrated head.

[0010] Optionally, the semantic segmentation result further includes dynamic target category results; the step of identifying dynamic targets and generating dynamic region masks based on the global image acquired by the environmental camera includes: Perform inter-frame difference operation on the global image to obtain dynamic candidate regions, and compare the dynamic candidate regions with the semantic segmentation results; If any of the dynamic candidate regions overlaps with the region of the dynamic target category result, the corresponding dynamic candidate region is determined to correspond to the dynamic target. Based on the dynamic candidate region corresponding to the dynamic target, generate the two-dimensional pixel region coordinates of the dynamic target; Based on the coordinate transformation relationship, the coordinates of the two-dimensional pixel region are converted into a three-dimensional spatial region in the global coordinate system, and a dynamic region mask is generated.

[0011] Optionally, constructing a pose graph based on the updated SLAM point cloud feature set includes: Based on the angular velocity and acceleration data collected by the robot's inertial measurement unit, the robot's attitude angle change data and instantaneous velocity are determined. Based on the robot's attitude angle change data and instantaneous velocity, the robot's initial motion trajectory is calculated according to the mileage data. Keyframes are determined based on the initial motion trajectory and the local image. The trajectory pose at the time corresponding to the keyframe is used as the initial pose of the corresponding keyframe node. The keyframe and the keyframe node correspond one-to-one. Static features in the global landmark database are used as nodes with fixed locations to form fixed nodes; based on the updated SLAM point cloud feature set, when a robot key frame observes a static feature corresponding to the fixed node, an observation constraint edge is established between the corresponding key frame node and the fixed node; between adjacent key frame nodes, based on the initial motion trajectory, a motion constraint edge is established; based on the key frame nodes, the fixed nodes, the observation constraint edges and the motion constraint edges, the pose graph is obtained.

[0012] Optionally, the static features in the global landmark database are added to the pose graph as global anchor points for constraint optimization to obtain an optimized robot pose, including: the pose graph is solved by a nonlinear optimization algorithm to obtain the optimized robot pose; wherein the objective function of the nonlinear optimization algorithm is composed of a weighted sum of a pose prediction error term and a global anchor constraint error term.

[0013] Optionally, based on the optimized robot pose, the depth information collected by the visual sensor, and the global landmark database, a three-dimensional map of the indoor environment is generated, including based on the optimized robot pose and the depth information, a three-dimensional point cloud map of the robot's direct observation area is generated; the static feature data of the corresponding area in the global landmark database is called to supplement the structural information of the area not directly observed by the robot through a three-dimensional reconstruction algorithm to generate a corresponding three-dimensional geometric model; the three-dimensional point cloud map and the three-dimensional geometric model are registered and fused in the global coordinate system to form a three-dimensional map of the indoor environment.

[0014] In a second aspect, the present application provides an indoor SLAM map construction system, comprising: an acquisition unit configured to observe a static calibration target by a visual sensor carried by a robot and an environment camera arranged in an indoor environment, and establish a coordinate conversion relationship between a robot local coordinate system in which the visual sensor is located and a global coordinate system in which the environment camera is located; a processing unit configured to obtain static features based on a global image obtained by the environment camera, and construct and update a global landmark database according to the static features based on the global coordinate system; The processing unit is further configured to extract local features based on the local image and corresponding depth information acquired by the visual sensor, form a local feature point cloud based on the robot local coordinate system according to the local features and corresponding depth information, convert the static features in the global landmark database to the robot local coordinate system based on the coordinate conversion relationship, and match and fuse the static features in the robot local coordinate system with the local feature point cloud to form a fused SLAM point cloud feature set. The processing unit is further configured to identify a dynamic target based on the global image and generate a dynamic region mask, convert the dynamic region mask to the global coordinate system, convert the dynamic region mask to the robot local coordinate system based on the coordinate conversion relationship, and remove feature points in the fused SLAM point cloud feature set that are located in the dynamic region mask in the robot local coordinate system to obtain an updated SLAM point cloud feature set. The construction unit is configured to construct a pose graph based on the updated SLAM point cloud feature set, add static features in the global landmark database as global anchor points to the pose graph for constraint optimization, obtain an optimized robot pose, and generate a three-dimensional map of the indoor environment based on the optimized robot pose, depth information acquired by the visual sensor, and the global landmark database.

[0015] In a third aspect, the present application provides an indoor SLAM map construction device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the indoor SLAM map construction method of the first aspect when executing the computer program.

[0016] The indoor SLAM map construction method and system of the present application have the following advantages: Firstly, a unified global spatial reference is established through cross-device space-time calibration, laying a foundation for multi-source data fusion and solving the problem of lack of global reference in traditional SLAM; secondly, static features are extracted from global images of environment cameras and a landmark database is constructed, effectively supplementing the observation blind area caused by the limited field of view of the robot and enhancing the completeness of the map; thirdly, through matching and fusion of local features and global features, and combining with dynamic recognition and removal mechanism, the details of the map are enriched while the interference of dynamic objects is excluded, improving the robustness of the system; finally, global anchor points are introduced in the back-end optimization to correct the pose, significantly suppressing the cumulative drift, thereby realizing high-precision, high-consistency, and strong-adaptability real-time map construction in complex indoor environments as a whole. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 FIG. 1 is a flowchart of an indoor SLAM map construction method according to an embodiment of the present application; Figure 2 A schematic diagram of a pose graph construction process of an embodiment of the present application; Figure 3 A structural schematic diagram of an indoor SLAM map construction system of an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the above objectives, features and advantages of the present application more apparent, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided so as to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are merely for illustrative purposes, and are not intended to limit the scope of protection of the present application.

[0019] It should be understood that each of the steps recited in the method embodiments of the present application can be executed in different orders, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0020] The term “comprising” and variations thereof as used herein are open-ended, that is, “comprising but not limited to”; the term “based on” is “based at least in part on”; the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments”; the term “optional” means “optional in at least some embodiments.” Related definitions are given throughout the description. It should be noted that the concepts mentioned in the present application are merely used to distinguish different devices, modules or units, and are not intended to limit the functions performed by these devices, modules or units, or the sequence or interdependence of these functions.

[0021] It should be noted that the modification of “one” “multiple” mentioned in the present application is illustrative rather than limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as “one or more”.

[0022] The names of the messages or information exchanged between the devices in the embodiments of the present application are merely for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0023] As shown in FIG. 1, an indoor SLAM map construction method provided by an embodiment of the present application includes the following steps. Figure 1 ​Step S100, a static calibration target is observed by a vision sensor carried by a robot and an environment camera arranged in a room, and a coordinate conversion relationship between a robot local coordinate system in which the vision sensor is located and a global coordinate system in which the environment camera is located is established.

[0024] Specifically, a network of environment cameras is arranged fixedly in a warehouse room (for example, a warehouse, an archive management, etc.), and a vision sensor (for example, an RGB-D camera) carried on a mobile robot can jointly observe the same fixed physical feature (static calibration target), for example, a specific shelf corner, a ground preset mark point or a wall high-contrast texture. By controlling the robot to move to the vicinity of several such preset static calibration targets, the robot vision sensor image and the environment camera image are synchronously collected. By using computer vision technology, feature points corresponding to the same physical target are detected and extracted from the two images respectively. Through a series of mathematical transformations and optimizations, the conversion parameters (including rotation and translation) that can map any point in the local coordinate system in which the robot sensor is located to the global coordinate system defined by the environment camera are finally accurately calculated. At the same time, the clocks of the two types of devices are synchronized to ensure that the timestamps of the data are consistent, thereby completing the unification of the time and space reference across devices.

[0025] For example, the upper left corner of a permanent shelf in a warehouse is selected as a calibration target. The robot drives to the front of the shelf, and its front-facing camera captures an image containing the corner point; at the same time, the environment camera installed on the warehouse ceiling, which has a field of view covering the area, also captures an image containing the same corner point. The corner point in the two images is automatically identified, and a known internal parameter (such as focal length) of the camera and the approximate spatial relationship of the corner point in the real world are combined to solve an accurate conversion relationship. Thereafter, any point seen by the robot camera can be converted into the global map coordinates with the warehouse ceiling camera as the origin through this relationship.

[0026] Through the established accurate space-time alignment relationship, a unified reference framework is provided for subsequent fusion processing, which fundamentally breaks the closed nature of traditional SLAM relying only on the robot's own recursive estimation. By introducing the environment camera as a stable global observation reference, the robot's local, relative and easily drifting pose and map data are anchored in a fixed, absolute and persistent global coordinate system. This lays an indispensable foundation for subsequent use of global images to supplement blind areas, identify and eliminate global dynamic disturbances, and impose global anchor point constraints in the back-end optimization to correct cumulative errors, and is the primary key technology link to improve accuracy, consistency and robustness in complex indoor environments.

[0027] Step S200, based on the global image obtained by the environment camera, extract static features, and based on the global coordinate system, construct and update the global landmark database according to the static features.

[0028] Specifically, the process is the core link of constructing the global environment prior knowledge base, aiming to use the perspective of fixed environment camera to establish a stable, comprehensive and real-time updateable static space reference framework for the whole system.

[0029] The environment camera deployed at the key position of the warehouse continuously collects global images covering the entire work area. These images are processed to automatically identify and extract those position-fixed and structure-stable environmental elements as static features. These features are not ordinary visual corners, but macro landmarks with clear semantics and stable geometric meaning, such as complete shelf contours, fixed wall columns, painted ground passage indication lines, and permanent facility edges, etc. After extracting the positions of these features in the image, the coordinate conversion relationship established in step S100 is used to accurately convert them from image pixel coordinates to three-dimensional space coordinates in the global coordinate system. The three-dimensional positions, semantic types, and source camera numbers of all these features are systematically organized, stored and indexed to form a dynamic global landmark database. This database is not built once, but periodically updated and verified with the continuous observation of the environment camera, ensuring that it can reflect the true state of the environment, such as the new layout after the shelves are moved.

[0030] For example: an environment camera installed in the center of the warehouse ceiling, its picture clearly shows three rows of parallel shelves, four load-bearing columns and the main cross passage. Analyzing the picture, identifying the vertical edge lines on both sides of each row of shelves, the bottom center points of each column, and the identification corner points at the intersection of the passage. Then, according to the known coordinate conversion relationship, the start and end point coordinates of each edge line in the warehouse global coordinate system, the precise three-dimensional position of the bottom of each column, and the spatial coordinates of each identification corner point are calculated, and these information together with its corresponding physical semantic label (such as "shelf edge", "column", "passage point") are recorded into the database to form a "global map skeleton" composed of precise coordinates and semantic information.

[0031] The global landmark database constructed and maintained provides global environmental cognition ability beyond the instantaneous local perception of the robot, which creates a static environment digital twin skeleton based on fixed view continuous observation independent of the robot motion trajectory. This database effectively breaks through the inherent perception blind area of the robot due to physical obstruction and field of view angle limitation, so that the robot can obtain accurate structural information of the area by querying the database even if the shelf back or remote corner cannot be seen by the sensor, thereby ensuring the global integrity of the constructed map. At the same time, these constant features in the database provide stable and absolute spatial reference anchors for subsequent robot positioning, which fundamentally provides key data basis for solving the inherent problem of cumulative error in the SLAM system.

[0032] In step S300, based on the local image and the corresponding depth information obtained by the visual sensor, local features are extracted, a local feature point cloud is formed based on the local coordinate system of the robot according to the local features and the corresponding depth information, the static features in the global landmark database are converted to the local coordinate system of the robot based on the coordinate conversion relationship, and the local feature point cloud is matched and fused to form a fused SLAM point cloud feature set.

[0033] Specifically, step S300 is a key link for deep fusion of global prior knowledge and real-time perception of the robot, aiming to construct an environment feature representation with global consistency and local high precision.

[0034] During the robot's movement, its mounted visual sensor (e.g. RGB-D camera) captures real-time local images and corresponding depth information (i.e. data representing the actual physical distance from the sensor's optical center to the surface point represented by each pixel in the color image captured by the robot's visual sensor) of the environment in front of it. Local features (e.g. corner points of a shelf's layer, texture points of a package) are extracted from these data and combined with the depth information to form a local feature point cloud describing the fine geometric structure in the current field of view in the robot's own local coordinate system. At the same time, according to the robot's current rough position, all static features (e.g. the overall contour of a shelf, the position of a rear column) in the area around the robot's current position (including the area in the current field of view and the area outside the current field of view that is blocked) are retrieved from the global landmark database constructed in S200. Using the coordinate transformation relationship established in S100, these static features stored in global coordinates are accurately transformed into the current robot's local coordinate system. Subsequently, the system matches and fuses these two sets of features in the same coordinate system, i.e. the locally generated feature point cloud and the globally static features converted from the database. The matching process confirms the correspondence between the two (e.g. a corner point extracted from the local image is actually the same point as a shelf corner point labeled in the database), and the fusion process combines them and optimizes the position to ultimately generate a unified, more informative SLAM point cloud feature set.

[0035] Example: The robot is driving in a passageway between two shelves, with its camera facing forward, obtaining a local feature point cloud of the front and lower parts of the two shelves. Due to the limited field of view, it cannot see the structure behind and on the upper layers of the shelves. At this time, the system calls all static features of the passageway and adjacent areas from the global database (including the endpoint of the passageway behind the robot, the top edges of the two shelves, and the entrance coordinates of the adjacent passageway blocked by the current shelf), and transforms these features into the robot's current head coordinate system. Thus, the robot's "perception" is instantly enhanced: it not only "sees" the details in front of it, but also "knows" the structure behind and above it, and even "perceives" the situation across the wall. All this information is fused into a unified feature set for subsequent positioning and mapping.

[0036] The dynamic feature fusion realized by step S300 combines the instantaneous local observation of the robot with the global prior knowledge of the environment, which completely solves the map breaking and detail missing problem caused by the dependence of continuous observation in traditional SLAM. By "injecting" the features in the global landmark database into the local coordinate system of the robot in real time, the key environmental structure information that cannot be directly observed by the robot due to the occlusion of the field of view (such as the back of the shelf and the back of the body) can be continuously and seamlessly supplemented, ensuring the spatiotemporal continuity and structural integrity of the constructed feature set. This not only makes the real-time generated map avoid the "blind area" caused by the void and distortion, providing a more reliable environmental model for navigation and planning, but also provides more abundant and stable feature constraints for subsequent positioning optimization, significantly improving the perception robustness and accuracy of the entire system in complex and narrow environments.

[0037] In step S400, a dynamic target is identified based on the global image, and a dynamic region mask is generated. The dynamic region mask is converted to the global coordinate system, and based on the coordinate conversion relationship, the dynamic region mask is converted to the local coordinate system of the robot. The feature points in the fused SLAM point cloud feature set that are located in the dynamic region mask in the local coordinate system of the robot are removed, and an updated SLAM point cloud feature set is obtained.

[0038] Specifically, this process is the core link of dynamic interference perception, labeling and exclusion, aiming to ensure the static consistency of the constructed map and avoid pollution of mobile objects to positioning and map.

[0039] The fixedly deployed environmental camera continuously captures continuous frame global images covering the entire scene with its stationary and wide field of view. Using this time sequence image stream, by analyzing the pixel changes between consecutive frames and combining semantic understanding, dynamic targets in the picture are identified, such as moving forklifts, passing staff or temporarily transported goods. For each dynamic target identified, a dynamic region mask is generated on the image area where the target is located, which accurately indicates the pixel range occupied by the target in the image. First, this two-dimensional mask based on image pixels is converted to a three-dimensional space region description in the global coordinate system using the camera model. Then, with the help of the coordinate conversion relationship established in S100, the dynamic three-dimensional region in the global coordinate system is accurately mapped to the current local coordinate system of the robot. Finally, the fused SLAM point cloud feature set generated by S300 is checked, and all feature points falling within the dynamic region mask in the local coordinate system are identified as unreliable features contaminated by dynamic objects, and are removed, thereby obtaining a pure updated SLAM point cloud feature set containing only static environmental information.

[0040] For example, an environmental camera captures a picture of a forklift crossing the main aisle. By analyzing several consecutive frames, the system locks onto the forklift and outlines its precise contours in the image, generating a mask corresponding to the forklift's shape. This mask is converted into a piece of three-dimensional space in the global coordinate system of the warehouse (i.e., the volume of space actually occupied by the forklift). At this point, the robot working nearby may have some feature points extracted from the forklift's body in its SLAM point cloud feature set. Through coordinate transformation, the system maps the global dynamic mask representing the "forklift-occupied space" into the robot's local field of view and finds that several points in the local feature point cloud are located within this space, determining that these points belong to dynamic interference and deleting them.

[0041] Through the established dynamic interference processing mechanism, the stable, continuous, and global observation angle provided by the fixed environmental camera is innovatively utilized to achieve high-robustness identification and spatial calibration of dynamic objects in the scene. By accurately transferring and mapping the identified dynamic regions between the global coordinate system and the robot's local coordinate system, the system can directly and actively eliminate feature points associated with dynamic objects at the front-end data level, effectively preventing dynamic features from participating in subsequent pose estimation and map construction. This mechanism fundamentally avoids the trajectory drift and map distortion caused by the traditional SLAM method mistaking dynamic objects as static, ensuring the purity and long-term consistency of the constructed map, greatly improving the practicality and reliability of the system in dynamic indoor scenes such as warehouses and logistics that have frequent human-robot interactions.

[0042] Step S500, based on the updated SLAM point cloud feature set, construct a pose graph, add static features in the global landmark database as global anchor points to the pose graph for constraint optimization, obtain an optimized robot pose, and generate a three-dimensional map of the indoor environment based on the optimized robot pose, depth information collected by the vision sensor, and the global landmark database.

[0043] Specifically, the final link of fusion optimization and map generation aims to utilize global anchor point constraints to globally correct the robot trajectory and ultimately output a complete and accurate indoor three-dimensional map.

[0044] Based on the updated SLAM point cloud feature set obtained in the previous step, combined with the robot inertial measurement unit (IMU) and odometry data of the wheel speed meter, a pose graph is constructed with the robot motion trajectory as the core. The graph takes the position and attitude (pose) of the robot at different times as the node, and the motion estimation and feature observation relationship between adjacent times as the edge. The core innovation of this process is to add the static features (such as fixed corners and column centers) in the global landmark database constructed by S200 as global anchor nodes with fixed position to the pose graph. When a certain trajectory node (pose) of the robot observes a global anchor through its feature set, a strong constraint edge is established between them. Subsequently, the whole graph is solved by a pre-set nonlinear optimization algorithm, which considers both the motion continuity constraint and the absolute position constraint of the global anchor, thereby significantly correcting the cumulative drift generated by the robot's own sensor calculation, and obtaining the optimized robot pose. Finally, based on this series of accurate poses, depth information collected by the visual sensor, and the global landmark database used to supplement the blind area, a three-dimensional map covering the entire indoor environment is generated through three-dimensional reconstruction and fusion technology, which is accurate in scale and rich in details.

[0045] For example: the robot drives around the warehouse along the circular channel and returns to the starting point, but due to the errors of the odometry and visual estimation, the calculated end position deviates significantly from the actual starting point (cumulative drift). At this time, the pose graph contains the various pose nodes of the robot's one-way journey and the constraints between them. At the same time, multiple global anchors (such as the columns at the four corners of the warehouse) are also added to the graph. The optimization algorithm finds that the feature observed by a certain node in the robot's trajectory deviates significantly from the theoretical observation position of the southeast corner column anchor. In order to satisfy both the motion continuity and the absolute position constraint, the algorithm will adjust the node poses of the entire circular trajectory cooperatively, and finally "pull back" the trajectory to the correct position, making the end point coincide with the starting point, thereby eliminating the closed-loop error. Based on this corrected accurate trajectory, all observed depth point clouds are fused to generate a seamless and globally consistent three-dimensional map of the real warehouse.

[0046] The core beneficial effect of introducing global anchor constraints in the back-end optimization and multi-source data fusion reconstruction is that it creatively introduces the inherent static structure of the environment as an absolute reference into the SLAM optimization process, achieving fundamental suppression and real-time correction of the robot's pose cumulative drift, thereby ensuring the global consistency of the map under long-time and large-range motion. On this basis, the three-dimensional map generated by fusing the optimized accurate pose, local depth observation, and global prior structure not only has a high local detail resolution, but also has global scale accuracy and spatial integrity, providing a truly reliable spatial digital foundation for the robot's precise navigation, task planning, and environmental interaction.

[0047] In this embodiment, through the deep fusion and collaborative optimization of the local sensor of the robot and the global information of the environment camera, the core problems such as the limited field of view, dynamic interference and cumulative error of the traditional SLAM technology in the indoor dense scene are effectively solved. The specific beneficial effects are that a unified global spatial reference is established through cross-device space-time calibration, and the global static features extracted by the environment camera are used to supplement the visual blind area information of the robot in real time, and the dynamic interference features are accurately removed by combining continuous frame analysis, and finally the global features are used as anchor points to constrain the continuous correction of the robot pose in the back-end optimization. This complete technical chain makes the indoor three-dimensional map constructed have higher integrity, precision and consistency, significantly improves the positioning robustness and navigation reliability of the robot in the dense shelf and dynamic activity environment, reduces the performance dependence of the robot on the sensor and the scene transformation cost, and provides a stable and efficient environment perception basis for the automation of the warehousing, archive management and other industries.

[0048] Optionally, the establishing of the coordinate conversion relationship between the robot local coordinate system in which the visual sensor is located and the global coordinate system in which the environment camera is located comprises: determining a plurality of static calibration targets, and respectively collecting visual sensor images and environment camera images corresponding to the static calibration targets through the visual sensor and the environment camera; respectively extracting feature point pixel coordinates corresponding to the static calibration targets from the visual sensor images and the environment camera images; obtaining an initial rotation matrix and a translation vector based on the feature point pixel coordinates, parameter data and the actual physical size of the static calibration target according to the perspective-n-point algorithm; wherein the parameter data includes internal parameters of the visual sensor and the environment camera; optimizing the rotation matrix and the translation vector through a preset optimization algorithm to obtain the coordinate conversion relationship.

[0049] Specifically, this process is a core technical means for realizing the spatial alignment of multi-source data, which aims to establish a unified spatial reference between the robot mobile platform and the fixed environment infrastructure through accurate mathematical calculation and optimization.

[0050] Static calibration target determination and image acquisition: a plurality of (usually ≥4) objects with clear geometric features and fixed positions are preselected in the deployment environment as static calibration targets. These targets should ensure that they can be seen in the robot working path and the field of view of the environment camera. For example, they include special identification patterns (such as AprilTag, checkerboard) pasted on the wall or floor, inherent vertical edges and corners on the shelf, and specific corner points of structural columns.

[0051] Among them, the static calibration target can be a special marker set artificially (high precision, stable features), or it can directly use the inherent and easily detected permanent structural features in the environment (without modifying the environment, but the feature extraction difficulty may increase). The robot moves to each target in turn, ensuring that its vision sensor and environmental camera can simultaneously and clearly capture images containing the target.

[0052] Feature point pixel coordinate extraction: For each set of images collected (robot image and environmental camera image), use feature detection algorithms (such as Harris corner detection, SIFT, or specialized detectors for calibration patterns) to automatically identify and extract the precise feature point pixel coordinates of the calibration target on the respective image plane. For example, if a checkerboard is used, the pixel coordinates of all its internal corner points are extracted; if a shelf corner is used, sub-pixel precision coordinates of the corner point are extracted. It should be noted that this process ensures that the feature points extracted from two different angle images physically correspond to the same three-dimensional space point.

[0053] Initial solution based on perspective-n-point (PnP) algorithm: The PnP algorithm is the core of solving this problem. Its input includes: 1, a set of feature point pixel coordinates in the environmental camera image; 2, the three-dimensional coordinates of these feature points in the world coordinate system (usually established based on the calibration target itself, with known actual physical dimensions such as checkerboard grid spacing); 3, the internal parameters of the environmental camera (obtained by pre-calibration, including focal length, principal point, distortion coefficient). The algorithm calculates the rotation matrix R and translation vector t that project three-dimensional points in the world coordinate system to the environmental camera image plane through mathematical solution (direct linear transformation DLT or more stable iterative method). Similarly, the pose of the robot vision sensor relative to the same set of world coordinates can be calculated. Finally, through relative pose transformation, the initial rotation matrix R and translation vector t from the robot local coordinate system to the environmental camera global coordinate system are obtained.

[0054] It should be noted that in addition to the classic PnP, more robust EPnP (efficient PnP) or UPnP (unbiased PnP) algorithms can also be used as initial solvers to improve calculation speed and numerical stability. For multiple sets of calibration target data, they can be calculated separately and then averaged or used as optimization initial values.

[0055] Refinement by optimization algorithm: the initial solution obtained by PnP is affected by image noise, feature extraction error, etc., and is not optimal. Therefore, a nonlinear optimization algorithm (preset optimization algorithm) is introduced for refinement. For example, the preset optimization algorithm can use an iterative optimization algorithm such as the Levenberg-Marquardt (LM) algorithm or the Gauss-Newton method to automatically adjust R and t to minimize the total re-projection error. This process can fuse observation data from multiple different positions and different calibration targets to optimize a set of optimal and globally consistent coordinate transformation parameters. To enhance robustness, the RANSAC algorithm can be used to remove possible mismatched point pairs before optimization.

[0056] The coordinate transformation relationship establishment process realizes high precision and high robustness of cross-device coordinate system by fusing multiple sets of high-precision static calibration target observation data and using two-stage refinement calculation from PnP analytical solution to nonlinear iterative optimization. This process is not only theoretically rigorous, but also can reliably solve three-dimensional space transformation from two-dimensional image observation. Moreover, by introducing nonlinear optimization for global optimal fusion of multi-source observation data, single-point observation noise and error are effectively suppressed, thereby obtaining highly accurate and stable coordinate transformation relationship. This high-precision space alignment is a fundamental prerequisite for subsequent effective projection of global landmarks, accurate mapping of dynamic interference, and role of global anchor constraints, providing essential basic guarantees for positioning accuracy and global consistency of the entire system, and avoiding systematic deviation caused by inaccurate coordinate system. Figure 1

[0057] Optionally, the local image obtained based on the visual sensor is processed to extract local features, including: processing the local image according to a preset semantic segmentation network to obtain a semantic segmentation result, the semantic segmentation result including a static target category result; extracting the static feature corresponding to the static target according to the static target category result; wherein the static feature includes position and contour information of the static target in the global image, and the static target category includes shelves, pillars and passage marking lines.

[0058] Optionally, the construction process of the preset semantic segmentation network includes: obtaining an original YOLOv5s network, and improving the original YOLOv5s network to obtain the improved YOLOv5s network; wherein the original YOLOv5s network includes a backbone network, a Neck network and an output network; and the improvement process includes: ​The standard bottleneck residual module in the backbone network is replaced by an enhanced residual module integrating channel and spatial attention mechanisms to enhance the feature extraction capability of the scene regular structure. A cross-scale attention module is added to the Neck network, and the output network is modified into a detection-segmentation integrated head.

[0059] Optionally, the output channel dimension of the detection-segmentation integrated head is adapted.

[0060] Specifically, the local feature extraction process based on the improved semantic segmentation network is the core of realizing accurate semantic perception of the environment, aiming to enable the robot not only to "see" the environment, but also to "understand" the class and accurate shape of the key structure in the environment, so as to extract high-value static features with clear semantic meaning.

[0061] Semantic segmentation processing and feature extraction: the local image collected by the robot vision sensor is input into a pre-trained semantic segmentation network. The network classifies the image at the pixel level, labels each pixel, and outputs the semantic segmentation result. This result is not a single output, and it at least includes: Pixel-level class label map: a matrix with the same size as the input image, and the value at each position represents that the pixel belongs to the "shelf", "pillar", "passage identification line" or "background" class.

[0062] For example: for different individuals of the same class (such as two independent shelves), the network can distinguish and identify them.

[0063] Static feature generation: according to the area marked as a static target class (such as a shelf or a pillar) in the segmentation result, the corresponding static feature is extracted. This is not just extracting the corner points or spots used in traditional SLAM, but extracting high-level features bound to semantic entities. For example: position information: the pixel coordinates of the center of the shelf pillar bottom in the image, or the main vertical edge line of the shelf can be extracted. Contour information: the accurate pixel mask of the "shelf" class area is directly obtained, so that the complete two-dimensional contour is known.

[0064] Among them, the selection of features can be flexibly adjusted according to subsequent tasks. For example, for positioning, it may be preferred to extract corner points and center points; for reconstruction, more attention is paid to contour lines. The extraction method can be the corner points of the minimum bounding rectangle of the mask, or the center line is extracted using the skeletonization algorithm.

[0065] Improved YOLOv5s semantic segmentation network construction: to adapt to the warehouse scene, the general target detection network YOLOv5s is improved in a targeted manner, so that it has high-performance real-time semantic segmentation capability.

[0066] Backbone network (CSPDarknet53) enhancement: Replace the original standard bottleneck residual module with an enhanced residual module that integrates channel and spatial attention mechanisms. Purpose: Enhance the network's ability to extract regular structural features (such as the straight edges of shelves, the vertical contours of columns, and the parallel lines of aisles) from warehouse images and suppress irrelevant texture interference.

[0067] For example: Integrated attention mechanism: A typical and efficient optimization method is to replace the 3x3 convolution layer in the standard bottleneck residual module with an enhanced convolution module that integrates channel attention and spatial attention.

[0068] Channel attention (such as SE module) example: This mechanism first compresses each feature channel into a scalar through global pooling, and then generates the weight of each channel through a small fully connected network. In the warehouse scenario, feature channels representing "edge directionality" and "corner response" are crucial for identifying shelves and columns, and this mechanism can automatically enhance the weights of these key channels.

[0069] Spatial attention example: At the same time or additionally introduce a spatial attention submodule that generates a weight mask based on the spatial position of the feature map, making the network pay more attention to areas with structural features (such as the area where the shelves are located) rather than homogeneous areas such as the ground or ceiling.

[0070] Structural reparameterization optimization: Another optional method is to use the reparameterization idea to design a multi-branch residual module during training (such as the RepVGG block), which enriches feature representation through multi-branch structure during training, and combines into a single efficient structure during inference, thereby improving feature extraction capability without increasing inference time.

[0071] Regardless of the specific method, the ultimate goal of optimization is to make the backbone network more sensitive to structural features with directionality, regularity, and high contrast in the warehouse scenario.

[0072] Neck network (Neck layer (FPN+PAN)) optimization: Add a cross-scale attention module. Purpose: To solve the problem of multiple target sizes in the warehouse (such as the details of nearby shelves and the overall outline of distant objects), achieve adaptive and intelligent fusion of multi-scale features, and improve the segmentation consistency of different size targets (from thin columns to long aisles).

[0073] For example: Inject attention on the feature pyramid path: On the feature fusion path of FPN (semantics propagated from top to bottom) and PAN (positioning enhanced from bottom to top), instead of simply adding or splicing, a lightweight cross-scale attention module is inserted. This module receives feature maps from different levels (e.g., high semantic features from deep layers and high-resolution features from shallow layers). It calculates the correlation or importance weight between different scale feature maps. For example, when fusing information to predict the precise boundaries of a large shelf, the module may give higher weight to shallow high-resolution features to obtain details; while when judging whether a region belongs to the "shelf" category, it will rely more on deep semantic features.

[0074] The adaptive weight learning mechanism in BiFPN (weighted bidirectional feature pyramid) can also be used, or a miniature Transformer-style cross-attention unit can be designed to calculate and fuse multi-scale context information.

[0075] Detection head modification and output dimension adaptation: The goal is to upgrade a single target detection head to an integrated head that outputs both detection boxes and segmentation masks, and to accurately adapt its output to the specific needs of the warehouse scenario.

[0076] For detection-segmentation integrated head design: Retain the branches in the original YOLO detection head for predicting bounding boxes (BBox), objectness, and original class probabilities. On this basis, a new segmentation branch is added in parallel. This segmentation branch is usually composed of several convolutional layers, and finally outputs a feature map associated with the spatial resolution of the input image, with a channel number equal to the number of classes (including background). Each channel corresponds to a class-wise probability map.

[0077] Output channel dimension adaptation: This is a key step in "specializing" a general-purpose model. The classifier dimension of the original YOLOv5s output layer is designed for general-purpose datasets (such as the 80 classes of COCO). This improvement process requires: 1. Redesign the output channel number of the classifier (the last convolution layer). The new channel number calculation formula needs to adapt to the warehouse categories, for example: (5+N)×A, where 5 represents the bounding box coordinates and confidence, N represents the number of warehouse-specific target categories (at least including shelves, pillars, and aisle marker lines, and possibly including "background" or "other" categories), and A is the number of anchor boxes. 2. The output channel number of the segmentation branch is directly set to N+1 (including background).

[0078] Based on the improved semantic segmentation network local feature extraction process, the feature focusing ability of the warehouse regular structure is enhanced by introducing the attention mechanism, the multi-scale information fusion is optimized by using the cross-scale attention, and the detection-segmentation integrated head is designed to realize the efficient and accurate output, which realizes the technical breakthrough of extracting high-quality semantic features from the local visual data of the robot. This process enables the robot to accurately and reliably identify and strip out key static structure entities (such as shelves, pillars) from complex environments, and extract their features with clear physical meaning (such as center, edge, contour), rather than just abstract feature points. These features rich in semantic information not only have stronger distinguishability and matching robustness, but also provide crucial high-quality input for subsequent feature fusion, dynamic rejection (because static entities can be clearly distinguished), and semantic-based map understanding and application, fundamentally improving the perception intelligence and task adaptability of the entire SLAM system in structured indoor environments.

[0079] Optionally, the semantic segmentation result further includes a dynamic target category result; and the identifying a dynamic target and generating a dynamic area mask based on the global image collected by the environment camera comprises: performing inter-frame difference operation on the global image to obtain a dynamic candidate area, and comparing the dynamic candidate area with the semantic segmentation result; if any of the dynamic candidate areas overlaps with the area of the dynamic target category result, determining that the corresponding dynamic candidate area corresponds to the dynamic target; generating a two-dimensional pixel area coordinate of the dynamic target based on the dynamic candidate area corresponding to the dynamic target; converting the two-dimensional pixel area coordinate into a three-dimensional space area in the global coordinate system based on the coordinate conversion relationship, and generating a dynamic area mask.

[0080] Specifically, this process is the core technical link to realize dynamic interference robustness exclusion, aiming to accurately distinguish static background from dynamic objects in the environment, and generate a spatial exclusion area that can be directly used by subsequent steps.

[0081] Dynamic candidate area acquisition based on inter-frame difference: using continuous frame global images collected by a fixed environment camera, the gray value or feature change of corresponding pixels between adjacent frames is calculated. Usually, inter-frame difference method is used, for example, the absolute difference D between the current frame I t and the previous frame I t-1 is calculated. The pixels in the difference D that exceed the preset threshold are marked as "changed pixels".

[0082] Region generation: morphological operations (such as dilation to connect adjacent points) and connected component analysis are performed on these discrete "changed pixels" to form a series of connected dynamic candidate regions. These regions can correspond to real moving objects, or be false changes caused by light and shadow changes, camera noise, etc.

[0083] wherein, to improve sensitivity and reduce ghosting, a three-frame difference method can be used. A more robust but slightly more computationally intensive method is to use a background modeling algorithm (such as ViBe or Gaussian Mixture Model, GMM) to establish and maintain a dynamically updated background model, and directly compare the current frame with the background model to obtain the foreground region.

[0084] Dynamic target determination combined with semantic segmentation results: the dynamic candidate regions obtained in the previous step are compared with the semantic segmentation results output by the same semantic segmentation network used in step S300 for the current frame image. The semantic segmentation results not only include static target categories (shelves, pillars), but also include dynamic target category results (such as pixel-level labels for "person", "forklift", and "trolley").

[0085] Determination logic: for a dynamic candidate region, calculate the spatial overlap (such as the intersection over union, IoU) between it and all pixel regions labeled as dynamic target categories in the semantic segmentation results of the frame. If the overlap exceeds a determination threshold (for example, IoU>0.3), it is considered that the dynamic candidate region is caused by the movement of a real dynamic target (such as a forklift), and thus the candidate region is determined to correspond to a dynamic target. If there is no significant overlap with any dynamic category region, it can be determined to be a light and shadow disturbance and excluded.

[0086] This process combines "motion information" and "semantic information". Pure inter-frame difference can misjudge the moving curtain shadow or light change caused by turning on or off the light as dynamic, while pure single-frame semantic segmentation can miss the temporarily stationary dynamic objects. The combination of the two achieves high-accuracy dynamic object recognition.

[0087] Dynamic target two-dimensional pixel region coordinate generation: for each region determined to be a real dynamic target, determine its bounding rectangle boundary on the entire image plane. Usually defined by the minimum column coordinate u_min, maximum column coordinate u_max, minimum row coordinate v_min, and maximum row coordinate v_max of all pixel points in the region, i.e., generate two-dimensional pixel region coordinates (u_min, v_min, u_max, v_max). The coordinates define the precise two-dimensional range of the dynamic target in the environment camera image, which is the input for mapping it from the image space to the real three-dimensional space.

[0088] Three-dimensional space generation of dynamic region mask: using the intrinsic parameters (intrinsic matrix) of the environmental camera, the key points on the two-dimensional pixel boundary of the dynamic target (such as the four corner points of the rectangular frame) are back-projected to the three-dimensional rays in the environmental camera coordinate system. Combined with certain known geometric assumptions (for example, assuming that the bottom of the target contacts the ground plane), or using the sparse depth information of the region (if the environmental camera is a depth camera), the three-dimensional space points corresponding to these corner points can be estimated, thus outlining the three-dimensional space region (such as an approximate cube or prism) occupied by the dynamic target in the global coordinate system. The definition of this three-dimensional space region is the dynamic region mask.

[0089] Finally, using the coordinate conversion relationship established by S100, the three-dimensional dynamic region mask defined in the global coordinate system is converted to the robot local coordinate system at the current time. The final result is a mask that defines "which three-dimensional space positions are temporarily occupied by dynamic objects" in the current self-coordinate system of the robot.

[0090] The dynamic target recognition and mask generation process realizes high-precision and high-robustness perception of scene dynamic interference by innovatively cooperating inter-frame difference motion detection with semantic segmentation category recognition. The core beneficial effect is that this method effectively overcomes the defects of traditional single motion detection being easily disturbed by environmental factors such as light and shadow, and makes up for the deficiency of pure semantic segmentation possibly missing detection when dynamic objects are static. By generating a three-dimensional space mask bound with the semantics of specific dynamic targets, the system can actively and accurately exclude data associated with dynamic objects at the feature level in the front end of the robot's SLAM calculation, thereby fundamentally preventing dynamic features from being incorrectly included in the map or used for pose estimation, ensuring the pure static consistency of the constructed map, and greatly improving the practicality and reliability of the entire system in real and dynamically changing indoor working environments.

[0091] Optionally, as shown in Figure 2 constructing a pose graph based on the updated SLAM point cloud feature set, comprising: determining robot attitude angle change data and instantaneous speed according to the angular velocity and acceleration data collected by the inertial measurement unit of the robot, and calculating the initial motion trajectory of the robot according to the driving mileage data based on the robot attitude angle change data and the instantaneous speed; determining key frames based on the initial motion trajectory and the local image, taking the trajectory pose at the corresponding time of the key frame as the initial pose of the corresponding key frame node, the key frame and the key frame node corresponding one by one; forming a fixed node by taking the static features in the global landmark database as a node with fixed position; Based on the updated SLAM point cloud feature set, when the robot key frame observes the static feature corresponding to the fixed node, an observation constraint edge is established between the corresponding key frame node and the fixed node; Between adjacent key frame nodes, motion constraint edges are established based on the initial motion trajectory; Based on the key frame nodes, the fixed nodes, the observation constraint edges and the motion constraint edges, the pose graph is obtained.

[0092] Optionally, the static features in the global landmark database are added as global anchor points to the pose graph for constraint optimization to obtain an optimized robot pose, including: The pose graph is solved by a nonlinear optimization algorithm to obtain an optimized robot pose; The objective function of the nonlinear optimization algorithm is based on the weighted sum of the pose prediction error term and the global anchor constraint error term.

[0093] Specifically, pose graph construction: initial motion trajectory calculation: fuse inertial measurement unit and wheel speed meter data. Integrate the angular velocity collected by the IMU to obtain the attitude angle change (heading, pitch, roll); integrate the acceleration and combine with the gravity compensation to obtain the instantaneous speed. At the same time, the wheel speed meter provides relative travel distance (displacement) data. Through a sensor fusion algorithm (such as complementary filtering or Kalman filtering), the two are combined to calculate the initial motion trajectory of the robot over time (i.e. a preliminary estimate of a series of poses). Pre-integration technology can also be used to efficiently and accurately process IMU data, avoiding repeated integration during optimization. For the wheel speed meter, its system error (such as scale factor, wheel radius) can be modeled and estimated in optimization.

[0094] Key frame node initialization: not all image frames participate in optimization. Key frames are selected according to the initial motion trajectory and the content change of the local image (such as displacement exceeding a threshold or sufficient parallax). Each key frame represents an important state in the robot's motion. In the constructed pose graph, a key frame node is created for each key frame, and its initial pose is directly taken from the initial motion trajectory value at the time corresponding to the key frame.

[0095] For example: the robot may select a frame every 2 meters when driving in a straight line; when turning, a frame may be selected every 0.5 meters or every 10 degrees of rotation due to the rapid change in viewing angle.

[0096] Fixed node introduction: static features (such as shelf corner F1, column center P1) stored in the global landmark database, whose coordinates are known and fixed in the global coordinate system, are added to the pose graph as fixed nodes (also known as anchor points) with constant positions. The three-dimensional coordinates of these nodes remain unchanged during optimization and serve as global references.

[0097] Observation constraint edge: When a certain keyframe (through its associated updated SLAM point cloud feature set) observes and matches to a certain physical feature corresponding to a fixed node, an observation constraint edge is established between the keyframe node and the fixed node. The constraint implied by this edge is: according to the current estimated keyframe pose, the global coordinates of the fixed node are converted to the camera coordinate system of the keyframe, and the projected position should be consistent with the observed feature pixel position in the actual image. Its error is usually defined as the reprojection error.

[0098] Motion constraint edge: A motion constraint edge is established between adjacent keyframe nodes. The constraint implied by this edge is: the relative pose transformation between two keyframes should be as consistent as possible with the relative transformation calculated from the initial motion trajectory (or calculated by visual feature matching). Its error is usually defined as the relative pose error.

[0099] Pose graph formation: The pose graph is composed of two types of nodes (keyframe nodes, fixed nodes) and two types of edges (observation constraint edges, motion constraint edges), forming a sparse graph model representing the relationship between robot motion, observation and global environment.

[0100] Global anchor constraint optimization: Nonlinear optimization solver: input the above constructed pose graph into a nonlinear optimization solver (such as Ceres Solver, g2o). The optimization variable is the pose (position and attitude) of all keyframe nodes. The coordinates of the fixed nodes are used as known constants in the calculation.

[0101] Objective function: The objective function F is the weighted sum of two types of error terms: the pose prediction error term, which is the sum of the squared errors of all motion constraint edges. The global anchor constraint error term, which is the sum of the squared errors of all observation constraint edges. This is the key to the process, which introduces global absolute coordinate information as a strong constraint.

[0102] The solver adjusts the poses of all keyframe nodes through an iterative algorithm (such as Levenberg-Marquardt) to minimize the total objective function F, thereby obtaining the optimized robot pose (i.e. the optimal pose estimate of all keyframe nodes in the graph). In addition to weighted least squares, robust kernel functions (such as Huber kernel) can be used to handle possible outliers (incorrect matching edges), enhancing the robustness of the optimization. After optimization, edge pruning can be performed to maintain the size of the graph, achieving real-time performance.

[0103] The pose graph construction and global anchor constraint optimization process realizes the fundamental suppression of the cumulative error of the SLAM system by incorporating the global static features representing the inherent structure of the environment as fixed anchors into the pose graph and forming a unified optimization model with the robot motion trajectory and local observations. This method breaks through the limitation of traditional pose graph optimization which only relies on relative motion constraints and local loop constraints. Through the absolute spatial reference provided by the global anchor, the global drift of the trajectory can be directly and effectively corrected during the optimization process, especially in long-distance operation or scenes lacking loops. The cooperative optimization mechanism of "global anchor constraint" and "local motion constraint" ensures that the final robot pose is not only locally smooth but also highly consistent with the real-world coordinate system, thereby laying the most critical trajectory foundation for generating an accurate, seamless, and globally consistent three-dimensional environment map, greatly improving the practical precision and reliability of the entire system in large and complex indoor scenes.

[0104] Optionally, generating the three-dimensional map of the indoor environment based on the optimized robot pose, the depth information collected by the visual sensor, and the global landmark database includes generating a three-dimensional point cloud map of the direct observation area of the robot based on the optimized robot pose and the depth information; calling the static feature data of the corresponding area in the global landmark database to supplement the structural information of the area not directly observed by the robot through a three-dimensional reconstruction algorithm to generate a corresponding three-dimensional geometric model; registering and fusing the three-dimensional point cloud map and the three-dimensional geometric model in the global coordinate system to form the three-dimensional map of the indoor environment.

[0105] Optionally, the three-dimensional reconstruction algorithm uses a Poisson reconstruction algorithm and maps the color texture collected by the visual sensor carried by the robot to the three-dimensional point cloud to generate a textured dense map.

[0106] Specifically, the three-dimensional point cloud map generation of the direct observation area is the main part of the map. For each optimized robot pose, each valid pixel point in the depth map collected by the visual sensor corresponding to the pose is back-projected to the camera coordinate system of the robot itself using the camera intrinsic parameters to obtain a three-dimensional point. Then, the local three-dimensional point is converted to the global coordinate system using the optimized pose (R, t). Repeat this operation for all depth pixels of all key frames and collect all points converted to the global coordinate to form a three-dimensional point cloud map covering the area directly "seen" by the robot during its entire walking path. The generated original point cloud usually contains noise and redundancy. Voxel grid downsampling can be performed to uniformize the point density and reduce the data volume, and a statistical outlier removal filter can be applied to remove isolated noise points, thereby obtaining a clean, uniform, and dense three-dimensional point cloud map.

[0107] Unobserved region 3D geometry model completion: This is a direct manifestation of solving the "blind spot" problem. Identify the regions that the robot does not directly observe (e.g. the back of the shelf, the far end of the aisle that the robot does not enter). For these regions, call the corresponding static feature data in the global landmark database (e.g. the 3D coordinates of the shelf edge line, the column center point located in this region extracted from the environment camera image). These features are sparse but absolutely accurate in position.

[0108] 3D reconstruction algorithm (take Poisson reconstruction as an example): Take the above sparse but accurate static feature points as input. The core idea of the Poisson reconstruction algorithm is to regard these points as samples of the gradient of a 3D indicator function (the function is 1 inside the object surface and 0 outside). The algorithm reconstructs this indicator function by solving a Poisson equation, and then extracts its isosurface (usually the zero isosurface), thereby generating a smooth, closed and volumetric 3D triangular mesh model. This process can "infer" the complete surface shape from sparse points.

[0109] Alternative algorithms: For example: Delaunay triangulation / Alpha Shapes: suitable for generating surface shells from point clouds, and works well for regular structures such as planes and cylinders. Or a completion network based on deep learning: train a neural network to learn to predict complete 3D geometric structures from sparse semantic feature points (such as shelf corner points).

[0110] Global model fusion and texture mapping: Since the directly observed point cloud and the completed geometric model are both placed in a unified global coordinate system, they are naturally spatially aligned. The main task of fusion is to solve the smooth transition and possible overlap / gap problems at the interface of the two.

[0111] Fusion operations can include: simple splicing: directly add the completed mesh model to the point cloud map as an independent model component.

[0112] Seamless fusion: at the interface of the point cloud and the mesh, perform Poisson reconstruction on the point cloud to convert it into a mesh, then perform Boolean operation (union) and mesh smoothing with the completed mesh to form a single, seamless mesh model. Alternatively, use the truncated signed distance function (TSDF) to fuse the information of the two into a unified volumetric representation, and then extract the isosurface.

[0113] Color texture mapping: In order to generate a more intuitive and information-rich textured dense map, it is necessary to map the color information in the color images captured by the robot's vision sensors onto the final 3D model.

[0114] For the part of direct observation point cloud: each three-dimensional point has its source pixel color, which can directly give the point color attribute to form a color point cloud.

[0115] For the part of completed mesh model: texture baking: project multiple color images from different perspectives onto the mesh surface, solve the color inconsistency and occlusion problem between different perspectives through multi-view texture fusion algorithm (such as MVS), and generate a texture map covering the entire mesh surface.

[0116] For example: calculate the visibility and projected color of each vertex or patch of the mesh model in multiple color images, assign a final color through weighted average or optimal view selection, and store these color information as a texture map.

[0117] The three-dimensional map generation and fusion process realizes complete, accurate and usable digital reproduction of indoor environment by deeply integrating and optimizing local high-precision observation with global prior structure reconstruction and giving realistic visual texture. The core benefit is that the final output of this method is not only a simple geometric point cloud map, but also a global scale accurate three-dimensional semantic environment model with millimeter level detail accuracy in the "direct observation area" and complete structure inference in the "unobserved area". This completely solves the problem of "unknown holes" in the traditional mobile robot mapping due to the limitation of its own field of view. The generated map can be directly used for precise navigation planning, semantic level positioning of shelves and goods, digital inspection and simulation, providing a solid and reliable spatial data infrastructure for the overall intelligent management of warehouse, archive and other scenes, greatly improving the automation level and system decision-making ability.

[0118] Optionally, the indoor SLAM map construction method is suitable for robot autonomous navigation and operation scheduling scenarios in intelligent warehouse, smart archive and smart library.

[0119] Specifically, the core advantage of the indoor SLAM map construction method proposed by the technical solution is to realize high-precision, high-completeness and strong-robustness three-dimensional map construction and real-time positioning of complex indoor environment by fusing global observation of fixed environment cameras. This feature makes it have outstanding application value in the following indoor scenes with core needs for automation and fine operation: Intelligent warehouse: scene characteristics: dense shelves, narrow aisles, presence of forklifts, AGVs, frequent movement of personnel (dynamic interference), and need for precise goods storage and retrieval, inventory checking and path planning.

[0120] The method can effectively solve the problems of positioning drift and map blind area caused by shelf shielding of traditional warehouse robots, supplement blind area information through a global map, has strong dynamic interference elimination capability, and can ensure that the map remains stable in a dynamic environment mixed with people and vehicles, and the generated high-precision three-dimensional map with semantics can be directly used for accurate positioning at the storage location and multi-machine cooperative scheduling, thereby greatly improving the automation level and efficiency of warehouse operation.

[0121] Intelligent archives library: scene features: the layout of the archives shelf / bookshelf may be adjusted, the environment requires quietness, constant temperature and humidity, the storage and retrieval operation needs to be accurate and error-free, and regular inventory and literature positioning need to be supported.

[0122] The method relies on the inherent structure of the environment (shelf body, column) as a landmark, does not need to paste easily damaged two-dimensional codes and other marks, meets the environmental requirements of archives / library protection. When the shelf layout is adjusted, only the global database needs to be updated through the environment camera, without the need for reprogramming or large-scale re-running of mapping, thereby greatly reducing the system maintenance cost. The constructed centimeter-level precision map can support the robot to accurately store and retrieve specific archives boxes or books, and realize efficient and unmanned inventory operation.

[0123] In summary, the method (the indoor SLAM map construction method) is not a general SLAM scheme, but a solution optimized for indoor automation scenarios with dense and regular structures, dynamic interference, and strict requirements for operation accuracy and map integrity. It converts the environmental infrastructure (fixed camera) into an intelligent component that enhances the perception ability of the robot, provides an indispensable high-quality spatial perception basis for reliable autonomous navigation, accurate operation execution and efficient system scheduling of the robot in the above-mentioned scenarios, and is a key enabling technology for upgrading these fields to “intelligent”.

[0124] As shown in Figure 3 The indoor SLAM map construction system provided by the embodiment of the application comprises: An acquisition unit is configured to observe a static calibration target through a vision sensor carried by a robot and an environment camera arranged indoors, and establish a coordinate conversion relationship between a robot local coordinate system in which the vision sensor is located and a global coordinate system in which the environment camera is located; A processing unit is configured to obtain a static feature based on a global image acquired by the environment camera, and construct and update a global landmark database according to the static feature based on the global coordinate system; The processing unit is further configured to extract local features based on the local image and corresponding depth information acquired by the visual sensor, form a local feature point cloud based on the local coordinate system of the robot according to the local features and the corresponding depth information, convert the static features in the global landmark database to the local coordinate system of the robot based on the coordinate conversion relationship, and match and fuse the static features with the local feature point cloud to form a fused SLAM point cloud feature set. The processing unit is further configured to identify a dynamic target based on the global image and generate a dynamic region mask, convert the dynamic region mask to the global coordinate system, convert the dynamic region mask to the local coordinate system of the robot based on the coordinate conversion relationship, and remove feature points in the fused SLAM point cloud feature set that are located in the dynamic region mask in the local coordinate system of the robot to obtain an updated SLAM point cloud feature set. The constructing unit is configured to construct a pose graph based on the updated SLAM point cloud feature set, add static features in the global landmark database as global anchor points to the pose graph for constraint optimization, obtain an optimized robot pose, and generate a three-dimensional map of the indoor space based on the optimized robot pose, depth information acquired by the visual sensor, and the global landmark database.

[0125] The embodiment of the present application provides an indoor SLAM map construction device, which comprises a memory and a processor.

[0126] The embodiment of the present application provides a computer readable storage medium, wherein the storage medium stores a computer program.

[0127] Although the present application is disclosed as above, the protection scope of the present application is not limited to this. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and these changes and modifications shall fall within the protection scope of the present application.

Claims

1. An indoor SLAM map construction method, characterized in that, include: The robot uses its onboard visual sensors and indoor environmental cameras to observe static calibration targets and establish coordinate transformation relationships between the robot's local coordinate system (where the visual sensors are located) and the global coordinate system (where the environmental cameras are located). Based on the global image acquired by the environmental camera, static features are extracted, and a global landmark database is constructed and updated based on the global coordinate system and the static features. Based on the local image and corresponding depth information acquired by the visual sensor, local features are extracted. Based on the robot's local coordinate system, a local feature point cloud is formed according to the local features and the corresponding depth information. Based on the coordinate transformation relationship, the static features in the global landmark database are transformed to the robot's local coordinate system and matched and fused with the local feature point cloud to form a fused SLAM point cloud feature set. Based on the global image, dynamic targets are identified and dynamic region masks are generated. The dynamic region masks are transformed to the global coordinate system and then transformed to the robot local coordinate system based on the coordinate transformation relationship. Feature points located within the dynamic region mask in the robot local coordinate system are removed from the fused SLAM point cloud feature set to obtain an updated SLAM point cloud feature set. A pose graph is constructed based on the updated SLAM point cloud feature set. Static features from the global landmark database are added to the pose graph as global anchor points for constraint optimization to obtain an optimized robot pose. Based on the optimized robot pose, the depth information collected by the visual sensor, and the global landmark database, a 3D map of the indoor space is generated.

2. The indoor SLAM map construction method according to claim 1, characterized in that, Establishing the coordinate transformation relationship between the robot's local coordinate system where the vision sensor is located and the global coordinate system where the environmental camera is located includes: Multiple static calibration targets are identified, and visual sensor images and environmental camera images corresponding to the static calibration targets are acquired respectively through the visual sensor and the environmental camera; The feature point pixel coordinates corresponding to the static calibration target are extracted from the visual sensor image and the environmental camera image, respectively. Based on the perspective-n-point algorithm, the initial rotation matrix and translation vector are obtained according to the feature point pixel coordinates, parameter data, and the actual physical size of the static calibration target. The rotation matrix and translation vector are optimized using a preset optimization algorithm to obtain the coordinate transformation relationship.

3. The indoor SLAM map construction method according to claim 1, characterized in that, The extraction of static features from the global image acquired by the environmental camera includes: The local image is processed according to a preset semantic segmentation network to obtain a semantic segmentation result, which includes a static target category result. Extract the static features corresponding to the static target based on the static target category results; The preset semantic segmentation network is constructed based on an improved YOLOv5s network; the static features include the position and contour information of the static target in the global image, and the static target categories include shelves, columns, and aisle marker lines.

4. The indoor SLAM map construction method according to claim 3, characterized in that, The construction process of the preset semantic segmentation network includes: Obtain the original YOLOv5s network, and improve the original YOLOv5s network to obtain the improved YOLOv5s network; The original YOLOv5s network includes a backbone network, a Neck network, and an output network; the improvement process includes: The standard Bottleneck residual module in the backbone network is replaced with an enhanced residual module that integrates channel and spatial attention mechanisms to enhance the feature extraction capability of scene rule structures. A cross-scale attention module was added to the Neck network, and the output network was transformed into a detection-segmentation integrated head.

5. The indoor SLAM map construction method according to claim 3, characterized in that, The semantic segmentation result also includes dynamic target category results; the process of identifying dynamic targets and generating dynamic region masks based on the global image captured by the environmental camera includes: Perform inter-frame difference operation on the global image to obtain dynamic candidate regions, and compare the dynamic candidate regions with the semantic segmentation results; If any of the dynamic candidate regions overlaps with the region of the dynamic target category result, the corresponding dynamic candidate region is determined to correspond to the dynamic target. Based on the dynamic candidate region corresponding to the dynamic target, generate the two-dimensional pixel region coordinates of the dynamic target; Based on the coordinate transformation relationship, the coordinates of the two-dimensional pixel region are converted into a three-dimensional spatial region in the global coordinate system, and a dynamic region mask is generated.

6. The indoor SLAM map construction method according to claim 1, characterized in that, The construction of the pose graph based on the updated SLAM point cloud feature set includes: Based on the angular velocity and acceleration data collected by the robot's inertial measurement unit, the robot's attitude angle change data and instantaneous velocity are determined. Based on the robot's attitude angle change data and instantaneous velocity, the robot's initial motion trajectory is calculated according to the mileage data. Keyframes are determined based on the initial motion trajectory and the local image. The trajectory pose at the time corresponding to the keyframe is used as the initial pose of the corresponding keyframe node. The keyframe and the keyframe node correspond one-to-one. Static features in the global landmark database are used as nodes with fixed locations to form fixed nodes; Based on the updated SLAM point cloud feature set, when the robot observes the static features corresponding to the fixed node in a keyframe, an observation constraint edge is established between the corresponding keyframe node and the fixed node. Between adjacent keyframe nodes, motion constraint edges are established based on the initial motion trajectory; The pose graph is obtained based on the keyframe node, the fixed node, the observation constraint edge, and the motion constraint edge.

7. The indoor SLAM map construction method according to claim 6, characterized in that, The step of adding static features from the global landmark database as global anchor points to the pose graph for constraint optimization to obtain the optimized robot pose includes: The pose graph is solved by a nonlinear optimization algorithm to obtain the optimized robot pose. The objective function of the nonlinear optimization algorithm is based on the weighted sum of the pose prediction error term and the global anchor point constraint error term.

8. The indoor SLAM map construction method according to claim 1, characterized in that, The process involves generating a 3D map of the indoor space based on the optimized robot pose, the depth information collected by the visual sensor, and the global landmark database. Based on the optimized robot pose and the depth information, a 3D point cloud map of the robot's directly observed area is generated. The static feature data of the corresponding region in the global landmark database is called, and the structural information of the region that the robot has not directly observed is supplemented by a 3D reconstruction algorithm to generate the corresponding 3D geometric model. The three-dimensional point cloud map and the three-dimensional geometric model are registered and fused in the global coordinate system to form the three-dimensional map of the interior.

9. An indoor SLAM map building system, characterized in that, include: The acquisition unit is used to observe the static calibration target through the vision sensor mounted on the robot and the environmental camera deployed indoors, and to establish the coordinate transformation relationship between the robot's local coordinate system where the vision sensor is located and the global coordinate system where the environmental camera is located. The processing unit is used to obtain static features based on the global image acquired by the environmental camera, and to construct and update a global landmark database based on the static features according to the global coordinate system. The processing unit is further configured to extract local features based on the local image and corresponding depth information acquired by the visual sensor, form a local feature point cloud based on the robot's local coordinate system and the local features and corresponding depth information, transform the static features in the global landmark database to the robot's local coordinate system based on the coordinate transformation relationship, and match and fuse them with the local feature point cloud to form a fused SLAM point cloud feature set. The processing unit is further configured to identify dynamic targets and generate dynamic region masks based on the global image, transform the dynamic region masks to the global coordinate system, and transform the dynamic region masks to the robot local coordinate system based on the coordinate transformation relationship, and remove feature points located within the dynamic region masks in the robot local coordinate system from the fused SLAM point cloud feature set to obtain an updated SLAM point cloud feature set. The construction unit is used to construct a pose graph based on the updated SLAM point cloud feature set, add static features from the global landmark database as global anchor points to the pose graph for constraint optimization, obtain the optimized robot pose, and generate the indoor 3D map based on the optimized robot pose, the depth information collected by the visual sensor, and the global landmark database.

10. An indoor SLAM map building device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the indoor SLAM map construction method as described in any one of claims 1 to 8.

Images