Large-space real-time positioning method and system based on SLAM (Simultaneous Localization and Mapping) and visual graphics

Abstract

The invention provides a large-space real-time positioning method and system based on SLAM and visual graphics, and relates to the technical field of space positioning and intelligent navigation, and the method comprises the steps: obtaining a space scene image and an environment point cloud, and carrying out the preprocessing; detecting the scene image based on a deep learning target detection model, and identifying a dynamic target and a corresponding category and a bounding box; static space feature points of the environment point cloud are reserved, and an initial space point cloud map is constructed; oRB feature extraction is carried out on the static spatial feature points to obtain feature descriptors, and a spatial visual feature map is constructed; performing joint optimization on the initial spatial point cloud map and the spatial visual feature map to generate a multi-modal global spatial map; the dynamic target is subjected to pose estimation in combination with space inertial measurement information to obtain an initial pose, and the initial pose is optimized to obtain a final pose, so that the positioning precision and robustness in a large-space dynamic environment can be improved, the consistency of a multi-modal map can be optimized, and the positioning stability in the dynamic environment can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of space positioning and intelligent navigation technology, and in particular to a large-space real-time positioning method and system based on SLAM and visual graphics. Background Art

[0002] In large space environments, real-time and accurate positioning capabilities are the core foundation supporting technologies such as robot autonomous movement, intelligent monitoring and scheduling, and AR / VR immersive interaction.

[0003] In traditional positioning solutions, GPS technology is susceptible to interference in indoor or complex building occlusion scenarios, making it difficult to meet accuracy requirements; although laser SLAM technology can construct a dense geometric map of the environment, it lacks visual texture information and is prone to positioning deviations due to dynamic interference in environments with a large number of moving objects.

[0004] Existing visual SLAM technology relies on matching and tracking environmental feature points. During long-distance movement across large spaces, positioning errors can easily increase due to the accumulation of feature point drift. Single-sensor solutions have significant limitations. Visual sensors are prone to matching failures in areas with drastic lighting changes or sparse features, while inertial measurement units suffer from short-term drift, making it difficult to maintain long-term stable positioning on their own. Furthermore, dynamic features such as pedestrians and moving objects in dynamic environments can interfere with the effective matching of feature points and map construction. Insufficient temporal and spatial calibration accuracy during multi-sensor data fusion, as well as poor global map consistency, further restrict the reliability and stability of large-scale positioning.

[0005] Therefore, it is necessary to provide a large-space real-time positioning method and system based on SLAM and visual graphics to solve the above technical problems. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides a large space real-time positioning method and system based on SLAM and visual graphics, which is used to solve the problems of insufficient positioning accuracy and robustness of existing technologies in large space dynamic environments, and the positioning accuracy of multi-sensor fusion constructed Figure 1 The consistency is poor, and the interference of dynamic features can easily lead to deviations in pose estimation.

[0007] The present invention provides a large-space real-time positioning method based on SLAM and visual graphics, the method comprising: Acquire spatial scene image data through a visual sensor, acquire spatial environment point cloud data through a laser radar, perform distortion correction and grayscale processing on the spatial scene image data, and perform denoising and downsampling processing on the spatial environment point cloud data; Performing dynamic detection on the spatial scene image data based on a deep learning target detection model to identify dynamic targets and corresponding target categories and target bounding boxes; Based on the target category and the target bounding box, dynamic spatial feature points in the spatial environment point cloud data are eliminated by a motion analysis method, static spatial feature points are retained, and an initial spatial point cloud map is constructed; Performing ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and constructing a spatial visual feature map based on the spatial feature descriptor; Jointly optimizing the initial spatial point cloud map and the spatial visual feature map through a graph optimization algorithm to generate a multimodal global spatial map; A visual inertial odometry framework is used, combined with the spatial inertial measurement information of the dynamic target collected by an inertial measurement unit, to perform real-time pose estimation on the dynamic target to obtain an initial pose. Based on the multimodal global spatial map, the initial pose is optimized through the bundle adjustment method to obtain a final pose.

[0008] Preferably, the performing dynamic detection on the spatial scene image data based on the deep learning target detection model to identify dynamic targets and corresponding target categories and target bounding boxes specifically includes: The deep learning target detection model adopts an improved Faster R-CNN model; Inputting the distortion-corrected and grayscale-processed spatial scene image data into the backbone network of the improved FasterR-CNN model, extracting multi-scale features of the spatial scene image data step by step through convolutional layers and pooling layers to generate a spatial scene feature map F, wherein the spatial scene feature map F includes feature maps of the dynamic target and the static background; Input the spatial scene feature map F into the region proposal network of the improved Faster R-CNN model, generate spatial scene anchor boxes of preset sizes through a sliding window, predict the probability that each spatial scene anchor box belongs to the dynamic target and the parameters of the candidate bounding box, and set the corresponding prediction loss function; By minimizing the prediction loss function, the prediction process is optimized and a set of candidate bounding boxes is output; The candidate bounding boxes in the candidate bounding box set are mapped to the spatial scene feature map F, and the candidate bounding box features corresponding to the candidate bounding boxes are extracted through ROIAlign and input into the classification regression head for judgment: Obtain a preset category library, output the category matching probability between the dynamic target and each preset category in the preset category library based on the classifier of the classification regression head, and select the preset category corresponding to the maximum category matching probability as the target category; The regressor of the classification regression head outputs a bounding box correction coefficient and updates the coordinates of the candidate bounding box to obtain the coordinates of the target bounding box.

[0009] Preferably, the prediction loss function is expressed as follows: Where, represents the prediction loss function; Indicates the total number of spatial scene anchor boxes; Indicates the number of spatial scene anchor boxes in dynamic objects; represents the cross entropy loss function, which is used to optimize the classification of dynamic targets and static backgrounds; represents the smooth L1 loss function, which is used to optimize the parameters of the candidate bounding box; Indicates the probability that the i-th spatial scene anchor box belongs to a dynamic target; Represents the i-th true label. When the i-th spatial scene anchor box belongs to a dynamic target, , when the i-th spatial scene anchor box belongs to the static background, ; represents the balance coefficient, ; represents the parameters of the i-th candidate bounding box, , Respectively represent the offset of the center coordinates of the i-th candidate bounding box in the x and y directions, Represent the width and height scaling factors of the i-th candidate bounding box respectively; represents the relative parameters of the i-th candidate bounding box and the spatial scene anchor box, , Respectively represent the relative offset of the center coordinates of the i-th candidate bounding box in the x and y directions, Represent the relative scaling factors of the width and height of the i-th candidate bounding box respectively.

[0010] Preferably, the step of removing dynamic spatial feature points from the spatial environment point cloud data based on the target category and the target bounding box by a motion analysis method, retaining static spatial feature points, and constructing an initial spatial point cloud map specifically includes: A bounding box expansion coefficient is set according to the target category, and the product of the bounding box area corresponding to the target bounding box and the bounding box expansion coefficient is the bounding box expansion area, and the spatial environment point cloud data located in the bounding box expansion area is screened to generate a dynamic candidate point cloud set; Performing inter-frame motion analysis on the dynamic candidate point clouds in the dynamic candidate point cloud set by the motion analysis method, and calculating the spatial position change of the dynamic candidate point clouds in two consecutive frames; Setting a motion change threshold, when the spatial position change exceeds the motion change threshold, determining the dynamic candidate point cloud as the dynamic spatial feature point, otherwise determining it as the static spatial feature point; Eliminating the dynamic spatial feature points and retaining the static spatial feature points; The static spatial feature points are spliced ​​in time series, and the static spatial feature points are downsampled using a voxel filtering algorithm to retain key geometric features in the static spatial feature points to generate the initial spatial point cloud map.

[0011] Preferably, performing ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and constructing a spatial visual feature map based on the spatial feature descriptor, specifically includes: Selecting the static spatial feature points whose grayscale change degree is greater than the significant grayscale change threshold as valid static feature points; Calculate the direction angle of the effective static feature point by using the grayscale centroid method, and rotate the 31×31 pixel neighborhood of the effective static feature point to the positive direction corresponding to the direction angle; Performing grayscale comparison on 256 pairs of pixels within a 31×31 pixel neighborhood of the rotated valid static feature point to generate the spatial feature descriptor with a length of 256 bits; The spatial feature descriptors are screened and matched by calculating the Hamming distance, a corresponding relationship between the effective static feature points is established, and the spatial visual feature map is constructed.

[0012] Preferably, the calculation formula of the direction angle is: Where, Indicates the direction angle; The first-order grayscale moment in the x-direction of the 31×31 pixel neighborhood of the valid static feature point is used to reflect the centroid shift of the grayscale distribution of the 31×31 pixel neighborhood of the valid static feature point in the x-direction; The first-order grayscale moment in the y direction of the 31×31 pixel neighborhood of the valid static feature point is used to reflect the grayscale distribution center offset in the y direction of the 31×31 pixel neighborhood of the valid static feature point; Represents the grayscale value of the pixel with coordinates (x, y) in the 31×31 pixel neighborhood of the valid static feature point. ; arctan() represents the inverse tangent function; The binary form of the spatial feature descriptor is: Where D represents the binary form of the spatial feature descriptor; The ath pixel pair in the 31×31 pixel neighborhood representing the valid static feature point after rotation; Represents the a-th pixel pair Gray value.

[0013] Preferably, the joint optimization of the initial spatial point cloud map and the spatial visual feature map by a graph optimization algorithm to generate a multimodal global spatial map specifically includes: Get the point cloud map node set of the initial spatial point cloud map , , M represents the total number of point cloud map nodes in the point cloud map node set, and the visual feature map node set of the spatial visual feature map , , R represents the total number of visual feature map nodes in the visual feature map node set, and constructs the map node set , ; Set pose node collection , , K represents the total number of pose nodes in the pose node set, and sets the point cloud matching edge and visual reprojection edges , combined with the map node set , construct a spatial factor graph, where the point cloud matches the edge Used to connect the pose node and the point cloud map node, the visual reprojection edge Used to connect pose nodes and visual feature map nodes; Based on the spatial factor graph, set the point cloud matching error and visual reprojection error , where the point cloud matching error Used to represent pose nodes Point cloud map node under and map nodes The deviation, the visual reprojection error Used to represent pose nodes Visual feature map node under The reprojection bias of Construct the overall objective function, use the Gauss-Newton method to iteratively optimize the overall objective function, and solve the incremental equation , U represents the Hessian matrix of the total objective function, Represents the parameter update amount, b represents the gradient vector, and updates the pose node set and the map node set , by minimizing the point cloud matching error through weighted summation and the visual reprojection error The sum of squares, output optimized pose node set And the optimized map node set , and the optimized map node set Includes an optimized point cloud map node set and the optimized visual feature map node set ; If the optimized point cloud map node and optimized visual feature map nodes The spatial distance is less than , is the spatial fusion threshold, the weighted average method is used to optimize the point cloud map nodes and optimized visual feature map nodes Perform spatial fusion and generate spatial fusion map nodes ; Associate the spatial fusion map node and the corresponding spatial feature descriptors to generate the multimodal global spatial map.

[0014] Preferably, the point cloud matching error The calculation formula is as follows: Where, Represents pose node The corresponding transformation matrix; Represents pose node The observed point cloud map node; The visual reprojection error The calculation formula is as follows: Where, represents the perspective projection function; represents the internal parameter matrix; Represents pose node The visual feature map node observed below.

[0015] Preferably, the point cloud matching error is minimized by weighted summation. and the visual reprojection error The corresponding calculation formula is as follows: Where, The information matrices representing the point cloud matching error and visual reprojection error respectively; The spatial fusion map node The calculation formula is as follows: Where, Represents the optimized point cloud map nodes respectively , optimized visual feature map node The corresponding spatial fusion weight.

[0016] A large-space real-time positioning system based on SLAM and visual graphics, comprising: A data acquisition module is used to acquire spatial scene image data through a visual sensor, acquire spatial environment point cloud data through a lidar, perform distortion correction and grayscale processing on the spatial scene image data, and perform denoising and downsampling processing on the spatial environment point cloud data; A target recognition module is used to perform dynamic detection on the spatial scene image data based on a deep learning target detection model, and identify dynamic targets and corresponding target categories and target bounding boxes; a point cloud map construction module, configured to remove dynamic spatial feature points from the spatial environment point cloud data based on the target category and the target bounding box by a motion analysis method, retain static spatial feature points, and construct an initial spatial point cloud map; A visual map construction module is used to perform ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and to construct a spatial visual feature map based on the spatial feature descriptor; A joint optimization module, configured to jointly optimize the initial spatial point cloud map and the spatial visual feature map using a graph optimization algorithm to generate a multimodal global spatial map; The pose determination module is used to use the visual inertial odometry framework, combined with the spatial inertial measurement information of the dynamic target collected by the inertial measurement unit, to perform real-time pose estimation on the dynamic target to obtain an initial pose, and based on the multimodal global spatial map, optimize the initial pose through the bundle adjustment method to obtain a final pose.

[0017] Compared with related technologies, the large-space real-time positioning method and system based on SLAM and visual graphics provided by the present invention have the following beneficial effects: The present invention obtains spatial scene image data through visual sensors, obtains spatial environment point cloud data through laser radar, performs distortion correction and grayscale processing on the spatial scene image data, and performs denoising and downsampling processing on the spatial environment point cloud data; performs dynamic detection on the spatial scene image data based on a deep learning target detection model, identifies dynamic targets and the corresponding target categories and target bounding boxes; based on the target categories and target bounding boxes, removes dynamic spatial feature points in the spatial environment point cloud data through a motion analysis method, retains static spatial feature points, and constructs an initial spatial point cloud map; performs ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and The feature descriptor constructs a spatial visual feature map; the initial spatial point cloud map and the spatial visual feature map are jointly optimized through a graph optimization algorithm to generate a multimodal global spatial map; the visual inertial odometry framework is adopted, combined with the spatial inertial measurement information of the dynamic target collected by the inertial measurement unit, to perform real-time pose estimation of the dynamic target to obtain the initial pose, and based on the multimodal global spatial map, the initial pose is optimized through the bundle adjustment method to obtain the final pose, thereby improving the positioning accuracy and robustness in large-space dynamic environments. Through multi-sensor fusion and dynamic feature processing, the consistency of the multimodal map is optimized, the positioning deviation is suppressed, and the positioning stability in dynamic environments is enhanced.

[0018] Through preprocessing of multi-sensor data, the present invention achieves precise spatiotemporal alignment of visual and lidar data, effectively eliminating noise and distortion in the raw data. This provides a high-quality input foundation for subsequent fusion calculations and ensures the reliability of data fusion. By combining a dynamic feature rejection mechanism with deep learning target detection and motion feature analysis, the present invention accurately identifies and filters dynamic interference factors in the environment, reducing the impact of invalid features on static feature extraction. This allows the extracted static features to better align with the real-world structure, providing stable feature support for positioning. When constructing a multimodal map, the present invention jointly optimizes the initial point cloud map and the visual feature map using a graph optimization algorithm. This addresses the lack of global consistency in traditional single maps, enhances the map's ability to comprehensively represent the geometric and texture features of large-scale environments, and provides a more reliable spatial reference for positioning. Through a collaborative optimization strategy combining visual inertial odometry and bundle adjustment, the present invention effectively suppresses the cumulative deviation of pose estimation during long-distance movement, ensuring the continuity and accuracy of positioning results. Furthermore, the present invention's dynamic environment adaptive adjustment mechanism can sense environmental changes in real time and update static areas of the map, enabling the positioning system to maintain stable operation in complex dynamic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Flowchart of the large space real-time positioning method based on SLAM and visual graphics of the present invention; Figure 2 1 is a system block diagram of a large-space real-time positioning system based on SLAM and visual graphics of the present invention; Figure 3 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] like Figure 1 FIG. 1 is a flow chart of a large-space real-time positioning method based on SLAM and visual graphics provided by an embodiment of the present invention. Figure 1 The execution subject of the method shown may be a software and / or hardware device. The execution subject of the present application may include but is not limited to at least one of the following: user equipment, network equipment, etc. Among them, the user equipment may include but is not limited to computers, smart phones, personal digital assistants (PDAs) and the electronic devices mentioned above. Network equipment may include but is not limited to a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. It includes steps S1 to S6, as follows: S1, acquiring spatial scene image data through a visual sensor, acquiring spatial environment point cloud data through a lidar, performing distortion correction and grayscale processing on the spatial scene image data, and performing denoising and downsampling processing on the spatial environment point cloud data; Among them, visual sensors are devices used to capture the visual texture and structure of the environment and collect images of spatial scenes. LiDAR is a device that acquires three-dimensional point cloud data of the spatial environment by emitting laser signals.

[0022] Visual sensors capture image information of spatial scenes, while lidar uses the environment's three-dimensional point cloud data to form a multi-dimensional perception of the spatial environment. The captured image data requires distortion correction to eliminate geometric deviations introduced by the optical system and restore the scene's true spatial structure. Grayscaling is then used to convert color information into a grayscale format containing only brightness features, simplifying the data dimension while retaining key visual textures. For the point cloud data acquired by the lidar, denoising is performed to filter out invalid points caused by equipment noise and environmental interference, improving data reliability. Downsampling is then performed to reduce the total amount of data while retaining core geometric features, improving efficiency for subsequent processing.

[0023] S2, performing dynamic detection on the spatial scene image data based on a deep learning target detection model to identify dynamic targets and corresponding target categories and target bounding boxes; It can be understood that a deep learning object detection model refers to a model built using deep learning technology for automatically identifying and locating dynamic objects in spatial scene images. Dynamic objects are objects whose position or shape changes over time in spatial scene image data. The object category is the type of dynamic object, such as pedestrians or vehicles. The object bounding box is a rectangular area that accurately marks the spatial extent of a dynamic object in a spatial scene image.

[0024] Based on deep learning object detection models, preprocessed spatial scene image data can be analyzed and pixel distribution patterns within the image can be learned and matched to automatically detect dynamic objects. This process not only identifies moving objects but also determines their category and precisely marks their spatial extent within the image using bounding boxes. This stage provides a critical basis for subsequently distinguishing dynamic and static features in space, ensuring that subsequent map construction retains only stable structures within the environment.

[0025] S3, based on the target category and the target bounding box, removing dynamic spatial feature points in the spatial environment point cloud data through a motion analysis method, retaining static spatial feature points, and constructing an initial spatial point cloud map; It should be noted that motion analysis methods refer to methods that determine whether spatial feature points change with motion by analyzing the position changes of spatial environment point clouds in continuous time series. Dynamic spatial feature points are feature points in the spatial environment point cloud whose positions change with the motion of dynamic targets. Static spatial feature points are feature points in the spatial environment point cloud whose positions do not change with the motion of dynamic targets. They are used to reflect the fixed structure of the spatial environment. The initial spatial point cloud map is a preliminary three-dimensional point cloud map formed by stitching together static spatial feature points, which is used to reflect the static geometric structure of the spatial environment.

[0026] Based on target category and bounding box information, the spatial environment point cloud data is filtered using motion analysis methods. Specifically, the corresponding feature points in the point cloud are associated with the dynamic target's category attributes and its bounding range in the image. The position changes of these points in the time series are analyzed to determine whether they move with the dynamic target. By eliminating these dynamic spatial feature points and retaining static spatial feature points that do not change over time, these static spatial feature points are then spliced ​​and integrated to form an initial spatial point cloud map that reflects the fixed geometry of the environment.

[0027] S4, performing ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and constructing a spatial visual feature map based on the spatial feature descriptor; The spatial feature descriptor is information used to quantify the local visual attributes of static spatial feature points. The spatial visual feature map is a map constructed based on the spatial feature descriptor, which contains the visual attributes of static feature points and their spatial correspondences.

[0028] It should be noted that ORB features are rotationally invariant and scale-adaptive, effectively describing the local visual attributes of feature points. This process yields spatial feature descriptors that quantitatively represent these attributes. Based on these descriptors, the visual information of static spatial feature points is associated with their spatial positions to construct a spatial visual feature map that includes visual texture features.

[0029] S5, jointly optimizing the initial spatial point cloud map and the spatial visual feature map through a graph optimization algorithm to generate a multimodal global spatial map; Graph optimization algorithms build factor graphs that combine poses and map nodes, optimizing node parameters to minimize overall error. Multimodal global spatial maps are global maps that fuse point cloud geometry with visual feature information to comprehensively reflect the geometric structure and visual texture of the environment.

[0030] Specifically, a graph optimization algorithm can be used to coordinately adjust the initial spatial point cloud map and the spatial visual feature map, constructing a factor graph containing pose nodes and map nodes, and optimizing the parameters of each node by minimizing the overall error. This process can eliminate the spatial deviation that may occur when the two maps are constructed independently, so that the geometric information of the point cloud and the visual feature information are consistent in spatial position. Ultimately, a multimodal global spatial map that integrates geometric structure and visual texture is generated, providing a unified and reliable reference for subsequent pose estimation.

[0031] S6, using the visual inertial odometry framework, combined with the spatial inertial measurement information of the dynamic target collected by the inertial measurement unit, performs real-time pose estimation on the dynamic target to obtain an initial pose, and optimizes the initial pose through the bundle adjustment method based on the multimodal global spatial map to obtain a final pose.

[0032] The visual-inertial odometry framework refers to a technical framework used to fuse visual sensor data with inertial measurement data to estimate motion trajectory and posture in real time. An inertial measurement unit refers to a sensor device used to collect inertial information such as an object's angular velocity and acceleration. Spatial inertial measurement information refers to data such as angular velocity and acceleration used to reflect the motion state of a dynamic target. The initial pose refers to the preliminary position and posture results of a dynamic target obtained through real-time pose estimation. The bundle adjustment method uses a multimodal global spatial map as a reference. Based on the initial pose estimation results, it achieves fine optimization of the posture and position parameters of the dynamic target by minimizing the deviation between the observed position and the theoretical projected position of the visual feature points, and ultimately outputs an accurate pose result that is consistent with the global map space. The final pose refers to the precise position and posture results of the dynamic target obtained after optimization by the bundle adjustment method.

[0033] It is understandable that the visual inertial odometry framework can be first used to fuse the image information obtained by the visual sensor with the inertial measurement information of the dynamic target space collected by the inertial measurement unit, and the position and attitude of the dynamic target can be calculated in real time to obtain the initial pose. Subsequently, the initial pose is optimized through the bundle adjustment method with the multimodal global spatial map as a reference: by minimizing the deviation between the observed position of the static feature points on the image and the theoretical projection position calculated based on the initial pose, the pose parameters are finely adjusted to finally obtain an accurate final pose consistent with the global map space, realizing the precise tracking of the motion state of the dynamic target in a complex environment.

[0034] In a specific implementation process, the deep learning target detection model is used to dynamically detect the spatial scene image data, identify dynamic targets and corresponding target categories and target bounding boxes, and specifically include: The deep learning target detection model adopts an improved Faster R-CNN model; Inputting the distortion-corrected and grayscale-processed spatial scene image data into the backbone network of the improved FasterR-CNN model, extracting multi-scale features of the spatial scene image data step by step through convolutional layers and pooling layers to generate a spatial scene feature map F, wherein the spatial scene feature map F includes feature maps of the dynamic target and the static background; Input the spatial scene feature map F into the region proposal network of the improved Faster R-CNN model, generate spatial scene anchor boxes of preset sizes through a sliding window, predict the probability that each spatial scene anchor box belongs to the dynamic target and the parameters of the candidate bounding box, and set the corresponding prediction loss function; By minimizing the prediction loss function, the prediction process is optimized and a set of candidate bounding boxes is output; The candidate bounding boxes in the candidate bounding box set are mapped to the spatial scene feature map F, and the candidate bounding box features corresponding to the candidate bounding boxes are extracted through ROIAlign and input into the classification regression head for judgment: Obtain a preset category library, output the category matching probability between the dynamic target and each preset category in the preset category library based on the classifier of the classification regression head, and select the preset category corresponding to the maximum category matching probability as the target category; The regressor of the classification regression head outputs a bounding box correction coefficient and updates the coordinates of the candidate bounding box to obtain the coordinates of the target bounding box.

[0035] The expression of the prediction loss function is as follows: Where, represents the prediction loss function; Indicates the total number of spatial scene anchor boxes; Indicates the number of spatial scene anchor boxes in dynamic objects; represents the cross entropy loss function, which is used to optimize the classification of dynamic targets and static backgrounds; represents the smooth L1 loss function, which is used to optimize the parameters of the candidate bounding box; Indicates the probability that the i-th spatial scene anchor box belongs to a dynamic target; Represents the i-th true label. When the i-th spatial scene anchor box belongs to a dynamic target, , when the i-th spatial scene anchor box belongs to the static background, ; represents the balance coefficient, ; represents the parameters of the i-th candidate bounding box, , Respectively represent the offset of the center coordinates of the i-th candidate bounding box in the x and y directions, Represent the width and height scaling factors of the i-th candidate bounding box respectively; represents the relative parameters of the i-th candidate bounding box and the spatial scene anchor box, , Respectively represent the relative offset of the center coordinates of the i-th candidate bounding box in the x and y directions, Represent the relative scaling factors of the width and height of the i-th candidate bounding box respectively.

[0036] First, the distortion-corrected and grayscaled spatial scene image data can be fed into the backbone network of the improved Faster R-CNN model. The backbone network extracts features through the collaborative operation of multiple convolutional and pooling layers. The convolutional layers use sliding convolution kernels to capture features of local image regions, gradually extracting basic features such as edges and textures, as well as higher-level semantic features. The pooling layers reduce the dimensionality of the feature map through downsampling, improving computational efficiency while retaining key information. This step-by-step extraction method ultimately generates a spatial scene feature map that integrates multi-scale information. This feature map encompasses both the salient features of dynamic targets and the structural features of static backgrounds, comprehensively characterizing the spatial distribution and attribute characteristics of targets of different scales within the image.

[0037] Next, the spatial scene feature map is fed into the region proposal network of the Faster R-CNN model. The region proposal network generates fixed-size spatial scene anchor boxes on the feature map using a sliding window. Based on these anchor boxes, the region proposal network simultaneously performs two core prediction tasks: first, determining the probability that each anchor box belongs to a dynamic target to distinguish the target from the background; and second, predicting the parameters of the candidate bounding box, including the center coordinate offset and width and height scaling factors, to preliminarily locate the spatial extent of the target. To optimize prediction accuracy, the region proposal network sets a corresponding prediction loss function, which constrains the prediction process by comprehensively considering the classification error and the bounding box regression error.

[0038] The prediction loss function consists of two parts: the cross-entropy loss function is used to optimize the classification task of dynamic objects and static backgrounds. It improves the accuracy of anchor box classification by quantifying the deviation between the predicted probability and the true label. The smoothed L1 loss function focuses on optimizing the parameters of the candidate bounding boxes, enhancing the positioning accuracy of the bounding boxes by reducing the prediction errors caused by the bounding box coordinate offset and scale scaling. At the same time, this loss function introduces a balance coefficient to adjust the weights of the two loss types, ensuring the coordinated optimization of the classification and regression tasks. By minimizing this loss function, the Faster R-CNN model continuously iteratively updates its parameters, ultimately outputting a set of filtered candidate bounding boxes.

[0039] After the candidate bounding box set is generated, the candidate bounding boxes can be mapped to the spatial scene feature map, and ROIAlign is used to extract the feature area corresponding to each candidate box: ROIAlign avoids quantization errors in the feature alignment process through precise coordinate mapping and interpolation processing, ensuring that the extracted candidate bounding box features can truly reflect the local properties of the target. The extracted features are input into the classification and regression head, where the classifier calculates the matching probability between the dynamic target and each category based on a preset category library and selects the category with the highest probability as the target category; the regressor outputs the bounding box correction coefficient, which finely updates the bounding box coordinates by adjusting the center coordinate offset and width and height scaling factors of the candidate bounding box, ultimately obtaining a target bounding box that accurately marks the spatial range of the dynamic target.

[0040] The method of removing dynamic spatial feature points from the spatial environment point cloud data based on the target category and the target bounding box by a motion analysis method, retaining static spatial feature points, and constructing an initial spatial point cloud map specifically includes: A bounding box expansion coefficient is set according to the target category, and the product of the bounding box area corresponding to the target bounding box and the bounding box expansion coefficient is the bounding box expansion area, and the spatial environment point cloud data located in the bounding box expansion area is screened to generate a dynamic candidate point cloud set; Performing inter-frame motion analysis on the dynamic candidate point clouds in the dynamic candidate point cloud set by the motion analysis method, and calculating the spatial position change of the dynamic candidate point clouds in two consecutive frames; Setting a motion change threshold, when the spatial position change exceeds the motion change threshold, determining the dynamic candidate point cloud as the dynamic spatial feature point, otherwise determining it as the static spatial feature point; Eliminating the dynamic spatial feature points and retaining the static spatial feature points; The static spatial feature points are spliced ​​in time series, and the static spatial feature points are downsampled using a voxel filtering algorithm to retain key geometric features in the static spatial feature points to generate the initial spatial point cloud map.

[0041] First, the target bounding box is expanded by setting a corresponding bounding box expansion coefficient based on the attribute characteristics of the target category. The expanded bounding box area is obtained by multiplying the original area of ​​the target bounding box by the expansion coefficient. This process can compensate for possible deviations in bounding box positioning, more comprehensively covering the point cloud range associated with dynamic targets in space, and avoiding the omission of dynamic point clouds due to insufficient bounding box accuracy. Based on this expanded area, all point cloud data within this area is filtered from the spatial environment point cloud data to form a dynamic candidate point cloud set, which focuses on potential dynamic feature points for subsequent motion analysis.

[0042] Subsequently, a motion analysis method is used to analyze the inter-frame motion characteristics of the dynamic candidate point cloud set. By extracting the spatial coordinate information of the same dynamic candidate point cloud from two consecutive frames of data, the change in its spatial position over time is accurately calculated to quantify the point cloud's motion state. This process effectively captures the motion trajectory characteristics of the point cloud associated with the dynamic target, providing a quantitative basis for distinguishing true dynamic points from static points.

[0043] Next, a motion change threshold is set as the criterion for feature point classification. When the spatial position change of a dynamic candidate point cloud exceeds this threshold, it indicates that it has significantly moved with the dynamic target and is therefore classified as a dynamic spatial feature point. If the change does not exceed the threshold, it is classified as a static spatial feature point unaffected by the dynamic target's motion. This threshold setting effectively filters out false motion signals caused by measurement noise or minor environmental disturbances, ensuring the accuracy of the classification results.

[0044] Finally, the identified dynamic spatial feature points are removed, retaining only the static spatial feature points. These retained static spatial feature points are spliced ​​together according to the time sequence of acquisition, integrating the static structural information of the environment acquired at different moments. Simultaneously, a voxel filtering algorithm is used to downsample the spliced ​​static point cloud data, reducing data redundancy and optimizing data size while preserving key geometric features such as corners and columns. This process ultimately generates an initial spatial point cloud map that accurately reflects the fixed geometry of the environment.

[0045] The step of performing ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and constructing a spatial visual feature map based on the spatial feature descriptor, specifically includes: Selecting the static spatial feature points whose grayscale change degree is greater than the significant grayscale change threshold as valid static feature points; Calculate the direction angle of the effective static feature point by using the grayscale centroid method, and rotate the 31×31 pixel neighborhood of the effective static feature point to the positive direction corresponding to the direction angle; Performing grayscale comparison on 256 pairs of pixels within a 31×31 pixel neighborhood of the rotated valid static feature point to generate the spatial feature descriptor with a length of 256 bits; The spatial feature descriptors are screened and matched by calculating the Hamming distance, a corresponding relationship between the effective static feature points is established, and the spatial visual feature map is constructed.

[0046] The calculation formula of the direction angle is: Where, Indicates the direction angle; The first-order grayscale moment in the x-direction of the 31×31 pixel neighborhood of the valid static feature point is used to reflect the centroid shift of the grayscale distribution of the 31×31 pixel neighborhood of the valid static feature point in the x-direction; The first-order grayscale moment in the y direction of the 31×31 pixel neighborhood of the valid static feature point is used to reflect the grayscale distribution center offset in the y direction of the 31×31 pixel neighborhood of the valid static feature point; Represents the grayscale value of the pixel with coordinates (x, y) in the 31×31 pixel neighborhood of the valid static feature point. ; arctan() represents the inverse tangent function; The binary form of the spatial feature descriptor is: Where D represents the binary form of the spatial feature descriptor; The ath pixel pair in the 31×31 pixel neighborhood representing the valid static feature point after rotation; Represents the a-th pixel pair Gray value.

[0047] First, by evaluating the grayscale change of each static spatial feature point, points with grayscale changes greater than the significant grayscale change threshold are selected as valid static feature points. These points are more conducive to subsequent feature extraction and matching due to their obvious grayscale distribution differences.

[0048] Next, the grayscale centroid method is used to analyze the grayscale distribution of a 31×31 pixel neighborhood of a valid static feature point to determine the orientation angle. Specifically, the first-order grayscale moments in the x- and y-directions are calculated within the neighborhood. The x-direction grayscale moment reflects the offset of the neighborhood's grayscale distribution center along the x-axis, and the y-direction grayscale moment reflects the offset of the center along the y-axis. Based on these two grayscale moments, the inverse tangent function is used to calculate the orientation angle, which represents the main direction of the grayscale distribution within the neighborhood. Subsequently, the 31×31 pixel neighborhood of the valid static feature point is rotated to the positive direction corresponding to this orientation angle to ensure that the subsequent feature description is rotationally invariant.

[0049] Then, for the rotated 31×31 pixel neighborhood, 256 pairs of pixels are selected and the grayscale values ​​of each pair are compared: if the grayscale value of the previous pixel is smaller than that of the next pixel, the corresponding bit is recorded as 1; otherwise, it is recorded as 0, ultimately generating a 256-bit binary spatial feature descriptor. This binary encoding method can efficiently quantify the grayscale distribution characteristics of the neighborhood, with low computational and storage costs, making it suitable for large-scale feature processing.

[0050] Finally, the Hamming distance between the spatial feature descriptors corresponding to different valid static feature points is calculated to measure the similarity of the descriptors. Based on the Hamming distance, descriptors with high matching scores are selected to establish spatial correspondences between valid static feature points. By integrating these correspondences, the spatial positions of valid static feature points are associated with their visual feature descriptors. Ultimately, a spatial visual feature map is constructed that contains the spatial distribution and visual attributes of the feature points, providing an accurate visual feature reference for subsequent multimodal map joint optimization.

[0051] The method of jointly optimizing the initial spatial point cloud map and the spatial visual feature map by a graph optimization algorithm to generate a multimodal global spatial map specifically includes: Get the point cloud map node set of the initial spatial point cloud map , , M represents the total number of point cloud map nodes in the point cloud map node set, and the visual feature map node set of the spatial visual feature map , , R represents the total number of visual feature map nodes in the visual feature map node set, and constructs the map node set , ; Set pose node collection , , K represents the total number of pose nodes in the pose node set, and sets the point cloud matching edge and visual reprojection edges , combined with the map node set , construct a spatial factor graph, where the point cloud matches the edge Used to connect the pose node and the point cloud map node, the visual reprojection edge Used to connect pose nodes and visual feature map nodes; Based on the spatial factor graph, set the point cloud matching error and visual reprojection error , where the point cloud matching error Used to represent pose nodes Point cloud map node under and map nodes The deviation, the visual reprojection error Used to represent pose nodes Visual feature map node under The reprojection bias of Construct the overall objective function, use the Gauss-Newton method to iteratively optimize the overall objective function, and solve the incremental equation , U represents the Hessian matrix of the total objective function, Represents the parameter update amount, b represents the gradient vector, and updates the pose node set and the map node set , by minimizing the point cloud matching error through weighted summation and the visual reprojection error The sum of squares, output optimized pose node set And the optimized map node set , and the optimized map node set Includes an optimized point cloud map node set and the optimized visual feature map node set ; If the optimized point cloud map node and optimized visual feature map nodes The spatial distance is less than , is the spatial fusion threshold, the weighted average method is used to optimize the point cloud map nodes and optimized visual feature map nodes Perform spatial fusion and generate spatial fusion map nodes ; Associate the spatial fusion map node and the corresponding spatial feature descriptors to generate the multimodal global spatial map.

[0052] The point cloud matching error The calculation formula is as follows: Where, Represents pose node The corresponding transformation matrix; Represents pose node The observed point cloud map node; The visual reprojection error The calculation formula is as follows: Where, represents the perspective projection function; represents the internal parameter matrix; Represents pose node The visual feature map node observed below.

[0053] The point cloud matching error is minimized by weighted summation and the visual reprojection error The corresponding calculation formula is as follows: Where, The information matrices representing the point cloud matching error and visual reprojection error respectively; The spatial fusion map node The calculation formula is as follows: Where, Represents the optimized point cloud map nodes respectively , optimized visual feature map node The corresponding spatial fusion weight.

[0054] First, the point cloud map node set contained in the initial spatial point cloud map and the visual feature map node set contained in the spatial visual feature map are obtained, and the two are merged into a unified map node set, thereby realizing the node-level integration of point cloud geometric information and visual feature information.

[0055] Next, we set a pose node set consisting of multiple pose nodes and define point cloud matching edges and visual reprojection edges: point cloud matching edges connect pose nodes to point cloud map nodes; visual reprojection edges connect pose nodes to visual feature map nodes. Combining this integrated map node set, we construct a spatial factor graph consisting of pose nodes, map nodes, and two types of constraint edges. This graph structure quantifies the geometric and projection constraints between nodes.

[0056] Subsequently, point cloud matching error and visual reprojection error are defined: point cloud matching error is used to measure the spatial deviation between the observed point cloud map node and the map node under the transformation matrix corresponding to the pose node; visual reprojection error is used to characterize the reprojection deviation of the visual feature map node observed under the pose node after transformation by the perspective projection function and the intrinsic parameter matrix. Based on these two types of errors, an overall objective function is constructed and iterative optimization is performed using the Gauss-Newton method: the incremental equation consisting of the Hessian matrix, the parameter update amount, and the gradient vector is solved, and the sum of the squares of the point cloud matching error and the visual reprojection error is minimized in a weighted summation manner. The pose node set and the map node set are continuously updated, and the optimized pose node and the map node set containing the optimized point cloud node and the optimized visual node are finally output.

[0057] If the spatial distance between the optimized point cloud map node and the visual feature map node is less than the spatial fusion threshold, the two represent the same spatial position and are fused using the weighted average method: the corresponding spatial fusion weights are assigned according to the confidence of the two types of nodes, and the spatial fusion map node is generated through weighted calculation to achieve information fusion of geometric and visual features at the same spatial position.

[0058] Finally, the spatial fusion map nodes are associated with their corresponding spatial feature descriptors, and the geometric-visual composite features of the fusion map nodes are integrated with the optimized pose information to form a multimodal global spatial map that contains both precise spatial geometric structure and rich visual texture information, providing a unified and high-precision spatial reference benchmark for subsequent dynamic target pose estimation.

[0059] like Figure 2 FIG. 1 is a system block diagram of a large-space real-time positioning system based on SLAM and visual graphics provided by an embodiment of the present invention, wherein the system includes: A data acquisition module is used to acquire spatial scene image data through a visual sensor, acquire spatial environment point cloud data through a lidar, perform distortion correction and grayscale processing on the spatial scene image data, and perform denoising and downsampling processing on the spatial environment point cloud data; A target recognition module is used to perform dynamic detection on the spatial scene image data based on a deep learning target detection model, and identify dynamic targets and corresponding target categories and target bounding boxes; a point cloud map construction module, configured to remove dynamic spatial feature points from the spatial environment point cloud data based on the target category and the target bounding box by a motion analysis method, retain static spatial feature points, and construct an initial spatial point cloud map; A visual map construction module is used to perform ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and to construct a spatial visual feature map based on the spatial feature descriptor; A joint optimization module, configured to jointly optimize the initial spatial point cloud map and the spatial visual feature map using a graph optimization algorithm to generate a multimodal global spatial map; The pose determination module is used to use the visual inertial odometry framework, combined with the spatial inertial measurement information of the dynamic target collected by the inertial measurement unit, to perform real-time pose estimation on the dynamic target to obtain an initial pose, and based on the multimodal global spatial map, optimize the initial pose through the bundle adjustment method to obtain a final pose.

[0060] Figure 2 The apparatus of the embodiment shown can be used to perform Figure 1 The implementation principles and technical effects of the steps in the method embodiment shown are similar and will not be repeated here.

[0061] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the steps of the large-space real-time positioning method based on SLAM and visual graphics as described above.

[0062] like Figure 3 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. The electronic device 30 includes: a processor 31, a memory 32 and a computer program; The memory 32 is used to store the computer program, which may also be a flash memory. The computer program is, for example, an application program or a functional module for implementing the above method.

[0063] The processor 31 is configured to execute the computer program stored in the memory to implement the various steps performed by the device in the above method. For details, please refer to the relevant description in the above method embodiment.

[0064] Optionally, the memory 32 may be independent or integrated with the processor 31 .

[0065] When the memory 32 is a device independent of the processor 31, the device may further include: The bus 33 is used to connect the memory 32 and the processor 31 .

[0066] A readable storage medium stores a computer program, which, when executed by a processor, is used to implement the steps of the large-space real-time positioning method based on SLAM and visual graphics as described above.

[0067] The readable storage medium may be a computer storage medium or a communication medium. Communication media include any medium that facilitates the transfer of computer programs from one location to another. Computer storage media may be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium may also be an integral part of the processor. The processor and the readable storage medium may be located in an application-specific integrated circuit (ASIC). In addition, the ASIC may be located in a user device. Of course, the processor and the readable storage medium may also exist as discrete components in a communication device. The readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0068] The present invention also provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and at least one processor executes the execution instructions so that the device implements the methods provided in the various embodiments described above.

[0069] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large-space real-time positioning method based on SLAM and visual graphics, characterized by: The method comprises: Acquire spatial scene image data through a visual sensor, acquire spatial environment point cloud data through a laser radar, perform distortion correction and grayscale processing on the spatial scene image data, and perform denoising and downsampling processing on the spatial environment point cloud data; Performing dynamic detection on the spatial scene image data based on a deep learning target detection model to identify dynamic targets and corresponding target categories and target bounding boxes; Based on the target category and the target bounding box, dynamic spatial feature points in the spatial environment point cloud data are eliminated by a motion analysis method, static spatial feature points are retained, and an initial spatial point cloud map is constructed; Performing ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and constructing a spatial visual feature map based on the spatial feature descriptor; Jointly optimizing the initial spatial point cloud map and the spatial visual feature map through a graph optimization algorithm to generate a multimodal global spatial map; A visual inertial odometry framework is used, combined with the spatial inertial measurement information of the dynamic target collected by an inertial measurement unit, to perform real-time pose estimation on the dynamic target to obtain an initial pose. Based on the multimodal global spatial map, the initial pose is optimized through the bundle adjustment method to obtain a final pose.

2. The large space real-time positioning method based on SLAM and visual graphics according to claim 1 is characterized in that, The performing dynamic detection on the spatial scene image data based on the deep learning target detection model to identify dynamic targets and corresponding target categories and target bounding boxes specifically includes: The deep learning target detection model adopts an improved Faster R-CNN model; Inputting the distortion-corrected and grayscale-processed spatial scene image data into the backbone network of the improved Faster R-CNN model, extracting multi-scale features of the spatial scene image data step by step through convolutional layers and pooling layers to generate a spatial scene feature map F, wherein the spatial scene feature map F includes feature maps of the dynamic target and the static background; Input the spatial scene feature map F into the region proposal network of the improved Faster R-CNN model, generate spatial scene anchor boxes of preset sizes through a sliding window, predict the probability that each spatial scene anchor box belongs to the dynamic target and the parameters of the candidate bounding box, and set the corresponding prediction loss function; By minimizing the prediction loss function, the prediction process is optimized and a set of candidate bounding boxes is output; The candidate bounding boxes in the candidate bounding box set are mapped to the spatial scene feature map F, and the candidate bounding box features corresponding to the candidate bounding boxes are extracted through ROIAlign and input into the classification regression head for judgment: Obtain a preset category library, output the category matching probability between the dynamic target and each preset category in the preset category library based on the classifier of the classification regression head, and select the preset category corresponding to the maximum category matching probability as the target category; The regressor of the classification regression head outputs a bounding box correction coefficient and updates the coordinates of the candidate bounding box to obtain the coordinates of the target bounding box.

3. The large space real-time positioning method based on SLAM and visual graphics according to claim 2 is characterized in that, The expression of the prediction loss function is as follows: Where, represents the prediction loss function; Indicates the total number of spatial scene anchor boxes; Indicates the number of spatial scene anchor boxes in dynamic objects; represents the cross entropy loss function, which is used to optimize the classification of dynamic targets and static backgrounds; represents the smooth L1 loss function, which is used to optimize the parameters of the candidate bounding box; Indicates the probability that the i-th spatial scene anchor box belongs to a dynamic target; Represents the i-th true label. When the i-th spatial scene anchor box belongs to a dynamic target, , when the i-th spatial scene anchor box belongs to the static background, ; represents the balance coefficient, ; represents the parameters of the i-th candidate bounding box, , Respectively represent the offset of the center coordinates of the i-th candidate bounding box in the x and y directions, Represent the width and height scaling factors of the i-th candidate bounding box respectively; represents the relative parameters of the i-th candidate bounding box and the spatial scene anchor box, , Respectively represent the relative offset of the center coordinates of the i-th candidate bounding box in the x and y directions, Represent the relative scaling factors of the width and height of the i-th candidate bounding box respectively.

4. The large space real-time positioning method based on SLAM and visual graphics according to claim 1 is characterized in that, The method of removing dynamic spatial feature points from the spatial environment point cloud data based on the target category and the target bounding box by a motion analysis method, retaining static spatial feature points, and constructing an initial spatial point cloud map specifically includes: A bounding box expansion coefficient is set according to the target category, and the product of the bounding box area corresponding to the target bounding box and the bounding box expansion coefficient is the bounding box expansion area, and the spatial environment point cloud data located in the bounding box expansion area is screened to generate a dynamic candidate point cloud set; Performing inter-frame motion analysis on the dynamic candidate point clouds in the dynamic candidate point cloud set by the motion analysis method, and calculating the spatial position change of the dynamic candidate point clouds in two consecutive frames; Setting a motion change threshold, when the spatial position change exceeds the motion change threshold, determining the dynamic candidate point cloud as the dynamic spatial feature point, otherwise determining it as the static spatial feature point; Eliminating the dynamic spatial feature points and retaining the static spatial feature points; The static spatial feature points are spliced ​​in time series, and the static spatial feature points are downsampled using a voxel filtering algorithm to retain key geometric features in the static spatial feature points to generate the initial spatial point cloud map.

5. The large space real-time positioning method based on SLAM and visual graphics according to claim 1 is characterized in that, The step of performing ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and constructing a spatial visual feature map based on the spatial feature descriptor, specifically includes: Selecting the static spatial feature points whose grayscale change degree is greater than the significant grayscale change threshold as valid static feature points; Calculate the direction angle of the effective static feature point by using the grayscale centroid method, and rotate the 31×31 pixel neighborhood of the effective static feature point to the positive direction corresponding to the direction angle; Performing grayscale comparison on 256 pairs of pixels within a 31×31 pixel neighborhood of the rotated valid static feature point to generate the spatial feature descriptor with a length of 256 bits; The spatial feature descriptors are screened and matched by calculating the Hamming distance, a corresponding relationship between the effective static feature points is established, and the spatial visual feature map is constructed.

6. The large space real-time positioning method based on SLAM and visual graphics according to claim 5 is characterized in that, The calculation formula of the direction angle is: Where, Indicates the direction angle; The first-order grayscale moment in the x-direction of the 31×31 pixel neighborhood of the valid static feature point is used to reflect the centroid shift of the grayscale distribution of the 31×31 pixel neighborhood of the valid static feature point in the x-direction; The first-order grayscale moment in the y direction of the 31×31 pixel neighborhood of the valid static feature point is used to reflect the grayscale distribution center offset in the y direction of the 31×31 pixel neighborhood of the valid static feature point; Represents the grayscale value of the pixel with coordinates (x, y) in the 31×31 pixel neighborhood of the valid static feature point. ; arctan() represents the inverse tangent function; The binary form of the spatial feature descriptor is: Where D represents the binary form of the spatial feature descriptor; The ath pixel pair in the 31×31 pixel neighborhood representing the valid static feature point after rotation; Indicates the a-th pixel pair Gray value.

7. The large space real-time positioning method based on SLAM and visual graphics according to claim 1 is characterized in that, The method of jointly optimizing the initial spatial point cloud map and the spatial visual feature map by a graph optimization algorithm to generate a multimodal global spatial map specifically includes: Get the point cloud map node set of the initial spatial point cloud map , , M represents the total number of point cloud map nodes in the point cloud map node set, and the visual feature map node set of the spatial visual feature map , , R represents the total number of visual feature map nodes in the visual feature map node set, and constructs the map node set , ; Set pose node collection , , K represents the total number of pose nodes in the pose node set, and sets the point cloud matching edge and visually reprojected edges , combined with the map node set , construct a spatial factor graph, where the point cloud matches the edge Used to connect the pose node and the point cloud map node, the visual reprojection edge Used to connect pose nodes and visual feature map nodes; Based on the spatial factor graph, the point cloud matching error is set and visual reprojection error , where the point cloud matching error Used to represent pose nodes Point cloud map node under and map nodes The deviation, the visual reprojection error Used to represent pose nodes Visual feature map node under The reprojection bias of Construct the overall objective function, use the Gauss-Newton method to iteratively optimize the overall objective function, and solve the incremental equation , U represents the Hessian matrix of the total objective function, represents the parameter update amount, b represents the gradient vector, and updates the pose node set and the map node set , by minimizing the point cloud matching error through weighted summation and the visual reprojection error The sum of squares, output optimized pose node set And the optimized map node set , and the optimized map node set Includes an optimized point cloud map node set and the optimized visual feature map node set ; If the optimized point cloud map node and optimized visual feature map nodes The spatial distance is less than , is the spatial fusion threshold, the weighted average method is used to optimize the point cloud map nodes and optimized visual feature map nodes Perform spatial fusion and generate spatial fusion map nodes ; Associate the spatial fusion map node and the corresponding spatial feature descriptors to generate the multimodal global spatial map.

8. The large space real-time positioning method based on SLAM and visual graphics according to claim 7 is characterized in that, The point cloud matching error The calculation formula is as follows: Where, Represents pose node The corresponding transformation matrix; Represents pose node The observed point cloud map node; The visual reprojection error The calculation formula is as follows: Where, represents the perspective projection function; represents the internal parameter matrix; Represents pose node The visual feature map node observed below.

9. The large space real-time positioning method based on SLAM and visual graphics according to claim 7 is characterized in that, The point cloud matching error is minimized by weighted summation and the visual reprojection error The corresponding calculation formula is as follows: Where, The information matrices representing the point cloud matching error and visual reprojection error respectively; The spatial fusion map node The calculation formula is as follows: Where, Represents the optimized point cloud map nodes respectively , optimized visual feature map node The corresponding spatial fusion weight.

10. A large space real-time positioning system based on SLAM and visual graphics, applied to a large space real-time positioning method based on SLAM and visual graphics as claimed in any one of claims 1 to 9, characterized in that: The system comprises: A data acquisition module is used to acquire spatial scene image data through a visual sensor, acquire spatial environment point cloud data through a lidar, perform distortion correction and grayscale processing on the spatial scene image data, and perform denoising and downsampling processing on the spatial environment point cloud data; A target recognition module is used to perform dynamic detection on the spatial scene image data based on a deep learning target detection model, and identify dynamic targets and corresponding target categories and target bounding boxes; a point cloud map construction module, configured to remove dynamic spatial feature points from the spatial environment point cloud data based on the target category and the target bounding box by a motion analysis method, retain static spatial feature points, and construct an initial spatial point cloud map; A visual map construction module is used to perform ORB feature extraction on the static spatial feature points to obtain a spatial feature descriptor, and to construct a spatial visual feature map based on the spatial feature descriptor; A joint optimization module, configured to jointly optimize the initial spatial point cloud map and the spatial visual feature map using a graph optimization algorithm to generate a multimodal global spatial map; The pose determination module is used to use the visual inertial odometry framework, combined with the spatial inertial measurement information of the dynamic target collected by the inertial measurement unit, to perform real-time pose estimation on the dynamic target to obtain an initial pose, and based on the multimodal global spatial map, optimize the initial pose through the bundle adjustment method to obtain a final pose.

Images