Large space real-time positioning method and system based on slam and visual graph

Abstract

The application provides a large space real-time positioning method and system based on SLAM and visual graphics, relates to the technical field of space positioning and intelligent navigation, and the method comprises the following steps: acquiring a space scene image and an environment point cloud and performing pretreatment; detecting the scene image based on a deep learning target detection model, identifying a dynamic target and a corresponding category and a boundary box; reserving static space feature points of the environment point cloud and constructing an initial space point cloud map; performing ORB feature extraction on the static space feature points to obtain feature descriptors and constructing a space visual feature map; jointly optimizing the initial space point cloud map and the space visual feature map to generate a multi-modal global space map; combining space inertial measurement information to estimate the pose of the dynamic target to obtain an initial pose, and obtaining a final pose after optimization, so that the positioning accuracy and robustness in a large space dynamic environment can be improved, the consistency of the multi-modal map is optimized, and the positioning stability in a dynamic environment is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of spatial positioning and intelligent navigation, and particularly relates to a large space real-time positioning method and system based on SLAM and visual graphics. BACKGROUND

[0002] In a large space environment, real-time accurate positioning capability is the core basis for supporting robot autonomous movement, intelligent monitoring and scheduling, AR / VR immersive interaction and other technologies.

[0003] In a traditional positioning scheme, GPS technology is susceptible to interference in indoor or complex building shielding scenes, and is difficult to meet the accuracy requirements; although laser SLAM technology can construct a dense geometric map of the environment, it lacks visual texture information, and is prone to positioning deviation due to dynamic interference in an environment with a large number of moving objects.

[0004] Existing visual SLAM technology relies on matching and tracking of environmental feature points, and in a large space long-distance movement process, positioning error is prone to increase due to feature point drift accumulation; a single sensor scheme has obvious limitations, and a visual sensor is prone to matching failure in a region with severe light changes or sparse features, and an inertial measurement unit has a short-term drift problem, and is difficult to maintain long-term stable positioning alone. In addition, dynamic features such as pedestrians and moving objects in a dynamic environment can interfere with effective matching of feature points and map construction, and problems such as insufficient time and space calibration accuracy and poor global consistency of the map during multi-sensor data fusion further restrict the reliability and stability of large space positioning.

[0005] Therefore, it is necessary to provide a large space real-time positioning method and system based on SLAM and visual graphics to solve the above technical problems. SUMMARY

[0006] To solve the above technical problems, the present application provides a large space real-time positioning method and system based on SLAM and visual graphics, which is used to solve the problems of insufficient positioning accuracy and robustness in a dynamic environment in the prior art, poor consistency of a map constructed by multi-sensor fusion, and easy deviation of pose estimation caused by dynamic features. Figure One

[0007] The large space real-time positioning method based on SLAM and visual graphics provided by the present application comprises:

[0008] acquiring space scene image data through a visual sensor, acquiring space environment point cloud data through a laser radar, and performing distortion correction and grayscale processing on the space scene image data, and performing denoising and down-sampling processing on the space environment point cloud data;

[0009] ​dynamically detecting the spatial scene image data based on a deep learning target detection model, identifying dynamic targets and corresponding target categories and target bounding boxes;

[0010] based on the target categories and the target bounding boxes, removing dynamic spatial feature points in the spatial environment point cloud data and retaining static spatial feature points by a motion analysis method, and constructing an initial spatial point cloud map;

[0011] extracting spatial feature descriptors from the static spatial feature points by ORB feature extraction, and constructing a spatial visual feature map based on the spatial feature descriptors;

[0012] jointly optimizing the initial spatial point cloud map and the spatial visual feature map by a graph optimization algorithm to generate a multi-modal global spatial map;

[0013] adopting a visual-inertial odometer framework, combining spatial inertial measurement information of the dynamic targets collected by an inertial measurement unit, performing real-time pose estimation on the dynamic targets to obtain an initial pose, and optimizing the initial pose by a bundle adjustment method based on the multi-modal global spatial map to obtain a final pose.

[0014] Preferably, the deep learning target detection model dynamically detects the spatial scene image data, identifies dynamic targets and corresponding target categories and target bounding boxes, and specifically includes:

[0015] the deep learning target detection model adopts an improved Faster R-CNN model;

[0016] inputting the spatial scene image data after distortion correction and grayscale processing to a backbone network of the improved Faster R-CNN model, extracting multi-scale features of the spatial scene image data by convolution layers and pooling layers step by step, generating a spatial scene feature map F, and the spatial scene feature map F includes feature maps of the dynamic targets and static backgrounds;

[0017] inputting the spatial scene feature map F to a region proposal network of the improved Faster R-CNN model, generating spatial scene anchor point boxes of a preset size by a sliding window, predicting the probability of each spatial scene anchor point box belonging to the dynamic targets and the parameters of the candidate bounding boxes, and setting a corresponding prediction loss function;

[0018] optimizing the prediction process by minimizing the prediction loss function and outputting a candidate bounding box set;

[0019] mapping a candidate bounding box in the candidate bounding box set to the spatial scene feature map F, extracting a candidate bounding box feature corresponding to the candidate bounding box through ROIAlign and inputting the candidate bounding box feature to a classification and regression head for judgment:

[0020] obtaining a preset category library, outputting a category matching probability of the dynamic target and each preset category in the preset category library based on a classifier of the classification and regression head, and selecting a preset category corresponding to a maximum category matching probability as the target category;

[0021] outputting a bounding box correction coefficient based on a regressor of the classification and regression head, and updating coordinates of the candidate bounding box to obtain coordinates of the target bounding box.

[0022] Preferably, the expression of the prediction loss function is as follows:

[0023] In the formula, represents a prediction loss function; represents a total number of spatial scene anchor boxes; represents a number of spatial scene anchor boxes in a dynamic target; represents a cross-entropy loss function, which is used for optimizing classification of a dynamic target and a static background; represents a smooth L1 loss function, which is used for optimizing parameters of a candidate bounding box; represents a probability that an i-th spatial scene anchor box belongs to a dynamic target; represents an i-th real label, when the i-th spatial scene anchor box belongs to a dynamic target, when the i-th spatial scene anchor box belongs to a static background, ; represents a balance coefficient, ; represents parameters of an i-th candidate bounding box, , respectively represent offset amounts of a center coordinate of the i-th candidate bounding box in x and y directions, respectively represent width and height scaling factors of the i-th candidate bounding box; represents relative parameters of the i-th candidate bounding box and a spatial scene anchor box, , respectively represent relative offset amounts of a center coordinate of the i-th candidate bounding box in x and y directions, respectively represent width and height relative scaling factors of the i-th candidate bounding box.

[0024] Preferably, based on the target category and the target bounding box, dynamic spatial feature points in the spatial environment point cloud data are removed by a motion analysis method, static spatial feature points are retained, and an initial spatial point cloud map is constructed, specifically including:

[0025] According to the target category, a bounding box expansion coefficient is set, and the product of the target bounding box corresponding bounding box region and the bounding box expansion coefficient is a bounding box expansion region. The spatial environment point cloud data located in the bounding box expansion region is screened to generate a dynamic candidate point cloud set;

[0026] The spatial position change amount of the dynamic candidate point cloud in the dynamic candidate point cloud set is calculated by inter-frame motion analysis of the motion analysis method;

[0027] A motion change amount threshold is set. When the spatial position change amount exceeds the motion change amount threshold, the dynamic candidate point cloud is determined to be the dynamic spatial feature point, otherwise it is determined to be the static spatial feature point;

[0028] The dynamic spatial feature points are removed, and the static spatial feature points are retained;

[0029] The static spatial feature points are spliced according to the time sequence, the static spatial feature points are down-sampled by a voxel filtering algorithm, the key geometric features in the static spatial feature points are retained, and the initial spatial point cloud map is generated.

[0030] Preferably, the ORB feature extraction is performed on the static spatial feature points to obtain spatial feature descriptors, and a spatial visual feature map is constructed based on the spatial feature descriptors, specifically including:

[0031] The static spatial feature points with a gray change degree greater than a significant gray change threshold are selected as effective static feature points;

[0032] The direction angle of the effective static feature points is calculated by a gray centroid method, and the 31x31 pixel neighborhood of the effective static feature points is rotated to the positive direction corresponding to the direction angle;

[0033] The 256 pairs of pixel points in the 31x31 pixel neighborhood of the rotated effective static feature points are compared in gray to generate a spatial feature descriptor with a length of 256 bits;

[0034] The spatial feature descriptors are screened and matched by calculating the Hamming distance to establish the corresponding relationship between the effective static feature points and construct the spatial visual feature map.

[0035] Preferably, the calculation formula of the direction angle is:

[0036] wherein, denotes the direction angle; denotes the first order gray moment of the 31x31 pixel neighborhood of the valid static feature point in x direction, for reflecting the gray distribution barycenter offset of the 31x31 pixel neighborhood of the valid static feature point in x direction; denotes the first order gray moment of the 31x31 pixel neighborhood of the valid static feature point in y direction, for reflecting the gray distribution barycenter offset of the 31x31 pixel neighborhood of the valid static feature point in y direction; denotes the gray value of the pixel point with coordinate (x, y) in the 31x31 pixel neighborhood of the valid static feature point, ; arctan() denotes the inverse tangent function;

[0037] The binary form of the spatial feature descriptor is:

[0038] wherein, D denotes the binary form of the spatial feature descriptor; denotes the a-th pixel point pair in the 31x31 pixel neighborhood of the valid static feature point after rotation; denotes the gray value of the a-th pixel point pair .

[0039] Preferably, the initial spatial point cloud map and the spatial visual feature map are jointly optimized by the graph optimization algorithm to generate a multi-modal global spatial map, specifically comprising:

[0040] obtaining a point cloud map node set of the initial spatial point cloud map , , M denotes the total number of point cloud map nodes in the point cloud map node set, and a visual feature map node set of the spatial visual feature map , , R denotes the total number of visual feature map nodes in the visual feature map node set, and a map node set is constructed , ;

[0041] a pose node set is set , , K denotes the total number of pose nodes in the pose node set, and point cloud matching edges and visual re-projection edges are set, in combination with the map node set , a spatial factor graph is constructed, wherein the point cloud matching edges are used to connect the pose nodes and the point cloud map nodes, and the visual re-projection edges a pose node and a visual feature map node are connected;

[0042] a point cloud matching error and a visual re-projection error are set based on the spatial factor graph wherein the point cloud matching error is used to represent a point cloud map node under a pose node and the visual re-projection error is used to represent a re-projection deviation of a visual feature map node under a pose node ;

[0043] a total target function is constructed, and the total target function is iteratively optimized by using a Gauss-Newton method to solve an incremental equation , U represents a Hessian matrix of the total target function, represents a parameter update amount, b represents a gradient vector, the pose node set and the map node set are updated, and the point cloud matching error and the visual re-projection error are minimized by weighted summation of squares, and an optimized pose node set and an optimized map node set are outputted, and the optimized map node set includes an optimized point cloud map node set and an optimized visual feature map node set ;

[0044] if a spatial distance between the optimized point cloud map node and the optimized visual feature map node is less than , a spatial fusion threshold, then the optimized point cloud map node and the optimized visual feature map node are spatially fused by using a weighted average method to generate a spatial fusion map node ;

[0045] the spatial fusion map node and a corresponding spatial feature descriptor are associated to generate the multi-modal global spatial map.

[0046] Preferably, a calculation formula of the point cloud matching error is as follows:

[0047] in the formula, pose node corresponding transformation matrix; pose node observed point cloud map node;

[0048] the visual re-projection error The calculation formula is as follows:

[0049] In the formula, indicates a perspective projection function; indicates an intrinsic matrix; pose node observed visual feature map node.

[0050] Preferably, the sum of squares of the point cloud matching error and the visual re-projection error is minimized by weighted summation, and the corresponding calculation formula is as follows:

[0051] In the formula, respectively indicate the information matrixes of the point cloud matching error and the visual re-projection error;

[0052] The calculation formula of the spatial fusion map node is as follows:

[0053] In the formula, respectively indicate the optimized point cloud map node , the optimized visual feature map node and the corresponding spatial fusion weight.

[0054] The large space real-time positioning system based on SLAM and visual graphics, the system comprises:

[0055] A data acquisition module is configured to acquire spatial scene image data through a visual sensor, acquire spatial environment point cloud data through a laser radar, and perform distortion correction and grayscale processing on the spatial scene image data, and perform denoising and down-sampling processing on the spatial environment point cloud data.

[0056] A target recognition module is configured to perform dynamic detection on the spatial scene image data based on a deep learning target detection model, recognize dynamic targets and corresponding target categories and target bounding boxes.

[0057] A point cloud map construction module is configured to eliminate dynamic spatial feature points in the spatial environment point cloud data and retain static spatial feature points based on the target categories and the target bounding boxes through a motion analysis method, and construct an initial spatial point cloud map.

[0058] a visual map construction module configured to perform ORB feature extraction on the static spatial feature points to obtain spatial feature descriptors, and construct a spatial visual feature map based on the spatial feature descriptors;

[0059] a joint optimization module configured to perform joint optimization on the initial spatial point cloud map and the spatial visual feature map by a graph optimization algorithm to generate a multi-modal global spatial map;

[0060] a pose determination module configured to perform real-time pose estimation on the dynamic target by using a visual-inertial odometer framework in combination with spatial inertial measurement information of the dynamic target collected by an inertial measurement unit to obtain an initial pose, and perform optimization on the initial pose by a bundle adjustment method based on the multi-modal global spatial map to obtain a final pose.

[0061] Compared with the related art, the method and system for real-time positioning in a large space based on SLAM and visual graphics provided by the application have the following beneficial effects:

[0062] The application obtains spatial scene image data by a visual sensor, obtains spatial environment point cloud data by a laser radar, and performs distortion correction and grayscale processing on the spatial scene image data, and performs denoising and down-sampling processing on the spatial environment point cloud data; performs dynamic detection on the spatial scene image data based on a deep learning target detection model, identifies a dynamic target and corresponding target categories and target bounding boxes; based on the target categories and the target bounding boxes, removes dynamic spatial feature points in the spatial environment point cloud data by a motion analysis method, retains static spatial feature points, and constructs an initial spatial point cloud map; performs ORB feature extraction on the static spatial feature points to obtain spatial feature descriptors, and constructs a spatial visual feature map based on the spatial feature descriptors; performs joint optimization on the initial spatial point cloud map and the spatial visual feature map by a graph optimization algorithm to generate a multi-modal global spatial map; by using a visual-inertial odometer framework in combination with spatial inertial measurement information of the dynamic target collected by an inertial measurement unit, performs real-time pose estimation on the dynamic target to obtain an initial pose, and performs optimization on the initial pose by a bundle adjustment method based on the multi-modal global spatial map to obtain a final pose, so that the positioning accuracy and robustness in a large space dynamic environment can be improved, the consistency of the multi-modal map can be optimized by multi-sensor fusion and dynamic feature processing, the positioning deviation can be suppressed, and the positioning stability in a dynamic environment can be enhanced.

[0063] The application realizes accurate space-time alignment of visual and laser radar data by preprocessing of multi-sensor data, effectively eliminates noise and distortion interference in original data, provides high-quality input basis for subsequent fusion calculation, and guarantees the reliability of data fusion. The application can accurately identify and filter dynamic interference factors in the environment by combining deep learning target detection and motion feature analysis through a dynamic feature elimination mechanism, reduce the influence of invalid features on static feature extraction, make the extracted static features more consistent with the real environment structure, and provide stable feature support for positioning. When constructing a multi-modal map, the application solves the problem of insufficient global consistency of traditional single map by jointly optimizing the initial point cloud map and the visual feature map through a graph optimization algorithm, enhances the comprehensive representation ability of the map for the geometric and texture features of large space environment, and provides a more reliable spatial reference for positioning. Through the cooperative optimization strategy of visual inertial odometer and beam adjustment method, the cumulative deviation of pose estimation is effectively suppressed during long-distance movement, guaranteeing the continuity and accuracy of the positioning result. At the same time, the dynamic environment adaptive adjustment mechanism of the application can sense environmental changes in real time and update the static area of the map, so that the positioning system can still operate stably in complex dynamic scenes. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A flowchart of a large space real-time positioning method based on SLAM and visual graphics according to an embodiment of the application is shown in

[0065] Figure 2 A system block diagram of a large space real-time positioning system based on SLAM and visual graphics according to an embodiment of the application is shown in

[0066] Figure 3 A hardware structure schematic diagram of an electronic device according to an embodiment of the application is shown in DETAILED DESCRIPTION

[0067] To make the objectives, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described below in detail with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0068] As shown in Figure 1 , a flowchart of a large space real-time positioning method based on SLAM and visual graphics according to an embodiment of the application is shown in Figure 1The execution subject of the method can be a software and / or a hardware device. The execution subject of the present application can include, but is not limited to, at least one of the following: a user equipment, a network equipment, and the like. Among them, the user equipment can include, but is not limited to, a computer, a smart phone, a personal digital assistant (PDA), and the above-mentioned electronic devices, and the like. The network equipment can include, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of computers or network servers based on cloud computing, wherein the cloud computing is a kind of distributed computing, and a super virtual computer composed of a group of loosely coupled computers. The present embodiment does not make any limitation. It includes steps S1 to S6, and the details are as follows:

[0069] S1, acquiring spatial scene image data through a visual sensor, acquiring spatial environment point cloud data through a laser radar, and performing distortion correction and grayscale processing on the spatial scene image data, and performing denoising and down-sampling processing on the spatial environment point cloud data;

[0070] Among them, the visual sensor refers to a device for capturing the visual texture and structure of the environment, and collecting spatial scene image data. The laser radar refers to a device for acquiring three-dimensional point cloud data of the spatial environment by emitting laser signals.

[0071] The image information of the spatial scene is captured by the visual sensor, and the three-dimensional point cloud data of the environment is acquired by the laser radar to form multi-dimensional perception of the spatial environment. For the collected image data, distortion correction is needed to eliminate the geometric deviation introduced by the optical system and restore the true spatial structure of the scene; then grayscale processing is performed to convert the color information into grayscale form containing only brightness features, which simplifies the data dimension while retaining the key visual texture. For the point cloud data acquired by the laser radar, denoising processing is needed to filter out invalid points caused by device noise and environmental interference, and to improve data reliability; then down-sampling processing is performed to reduce the total amount of data while retaining the core geometric features, thereby improving the efficiency of subsequent processing.

[0072] S2, performing dynamic detection on the spatial scene image data based on a deep learning target detection model to identify dynamic targets and corresponding target categories and target bounding boxes;

[0073] It can be understood that the deep learning target detection model refers to a model constructed based on deep learning technology for automatically identifying and locating dynamic targets from spatial scene image data. Dynamic targets refer to objects in spatial scene image data whose position or shape changes over time. The target category is the type to which the dynamic target belongs, such as pedestrians, vehicles, etc. The target bounding box refers to a rectangular area used to accurately mark the spatial range of the dynamic target in the spatial scene image.

[0074] Based on the deep learning target detection model, the pre-processed spatial scene image data can be analyzed, and the pixel distribution pattern in the image can be learned and matched to realize automatic detection of dynamic targets. In this process, not only the target in motion state is identified, but also its category is determined, and its spatial range in the image is accurately marked by the target bounding box. This stage provides a key basis for distinguishing dynamic and static features in space for subsequent map construction, ensuring that only stable structures in the environment are retained in subsequent map construction.

[0075] S3, based on the target category and the target bounding box, removing dynamic spatial feature points in the spatial environment point cloud data by a motion analysis method, retaining static spatial feature points, and constructing an initial spatial point cloud map;

[0076] It should be noted that the motion analysis method refers to a method of analyzing the position change of the spatial environment point cloud in a continuous time sequence to determine whether the spatial feature point changes with the motion state. The dynamic spatial feature point is a feature point in the spatial environment point cloud whose position changes with the motion of the dynamic target. The static spatial feature point is a feature point in the spatial environment point cloud whose position does not change with the motion of the dynamic target, and is used to reflect the fixed structure of the spatial environment. The initial spatial point cloud map is a preliminary three-dimensional point cloud map formed by splicing static spatial feature points, and is used to reflect the static geometric structure of the spatial environment.

[0077] Based on the target category and the target bounding box information, the spatial environment point cloud data is filtered by a motion analysis method. Specifically, according to the category attribute of the dynamic target and its boundary range in the image, the corresponding feature points in the point cloud are associated, the position change rule of these points in the time sequence is analyzed, and it is determined whether it moves with the dynamic target. By removing such dynamic spatial feature points, static spatial feature points that do not change with time are retained, and then these static spatial feature points are spliced and integrated to form an initial spatial point cloud map that can reflect the fixed geometric form of the environment.

[0078] S4, extracting ORB features from the static spatial feature points to obtain spatial feature descriptors, and constructing a spatial visual feature map based on the spatial feature descriptors;

[0079] Wherein, the spatial feature descriptor refers to information for quantitatively describing the local visual attributes of the static spatial feature points. The spatial visual feature map refers to a map constructed based on the spatial feature descriptors, which contains the visual attributes of the static feature points and their spatial correspondence.

[0080] It should be noted that the ORB feature has rotation invariance and scale adaptability, can effectively describe the local visual attributes of the feature points, and the spatial feature descriptor quantifying these attributes is obtained through this process. Based on these descriptors, the visual information of the static spatial feature points is associated with their spatial positions to construct a spatial visual feature map containing visual texture features.

[0081] S5, jointly optimizing the initial spatial point cloud map and the spatial visual feature map through a graph optimization algorithm to generate a multi-modal global spatial map;

[0082] The graph optimization algorithm refers to an algorithm that optimizes node parameters to minimize the overall error by constructing a factor graph containing pose and map nodes. The multi-modal global spatial map refers to a global map that fuses point cloud geometric information and visual feature information, which is used to comprehensively reflect the geometric structure and visual texture of the environment.

[0083] Specifically, the initial spatial point cloud map and the spatial visual feature map can be collaboratively adjusted using a graph optimization algorithm to construct a factor graph containing pose nodes and map nodes, and the node parameters are optimized by minimizing the overall error. This process can eliminate the spatial deviations that may occur during the independent construction of the two maps, so that the geometric information and visual feature information of the point cloud remain consistent in spatial position, and finally a multi-modal global spatial map that fuses geometric structure and visual texture is generated, providing a unified and reliable reference for subsequent pose estimation.

[0084] S6, using a visual-inertial odometry framework, combining the spatial inertial measurement information of the dynamic target collected by the inertial measurement unit, performing real-time pose estimation on the dynamic target to obtain an initial pose, and based on the multi-modal global spatial map, optimizing the initial pose through bundle adjustment to obtain a final pose.

[0085] The visual-inertial odometry framework refers to a technical framework for fusing visual sensor data and inertial measurement data to estimate motion trajectory and attitude in real time. The inertial measurement unit refers to a sensor device for collecting inertial information such as angular velocity and acceleration of an object. The spatial inertial measurement information refers to data such as angular velocity and acceleration that reflect the motion state of the dynamic target. The initial pose refers to the preliminary position and attitude result of the dynamic target obtained through real-time pose estimation. The bundle adjustment method takes the multi-modal global spatial map as a reference, based on the initial pose estimation result, and through minimizing the deviation between the observed position and the theoretical projection position of the visual feature points, it realizes the fine optimization of the attitude and position parameters of the dynamic target, and finally outputs the accurate pose result consistent with the global map space. The final pose refers to the accurate position and attitude result of the dynamic target obtained after optimization by the bundle adjustment method.

[0086] It can be understood that firstly, a visual inertial odometer framework can be adopted to fuse image information acquired by a visual sensor and dynamic target space inertial measurement information collected by an inertial measurement unit, to calculate a position and an attitude of the dynamic target in real time, and obtain an initial pose. Subsequently, the initial pose is optimized by a bundle adjustment method with reference to a multi-modal global space map: by minimizing the deviation between the observed position of a static feature point on an image and the theoretical projection position calculated based on the initial pose, the pose parameters are finely adjusted, and finally an accurate final pose consistent with the global map space is obtained, so as to realize accurate tracking of the motion state of the dynamic target in a complex environment.

[0087] In the specific implementation process, the deep learning target detection model performs dynamic detection on the spatial scene image data, identifies the dynamic target and the corresponding target category and target bounding box, and specifically includes the following steps:

[0088] The deep learning target detection model adopts an improved Faster R-CNN model;

[0089] The spatial scene image data after distortion correction and grayscale processing is input into the backbone network of the improved Faster R-CNN model, and multi-scale features of the spatial scene image data are extracted step by step through convolution layers and pooling layers to generate a spatial scene feature map F, and the spatial scene feature map F includes feature maps of the dynamic target and the static background;

[0090] The spatial scene feature map F is input into the region proposal network of the improved Faster R-CNN model, a spatial scene anchor box of a preset size is generated through a sliding window, the probability that each spatial scene anchor box belongs to the dynamic target and the parameters of the candidate bounding box are predicted, and a corresponding prediction loss function is set;

[0091] By minimizing the prediction loss function, the prediction process is optimized and a candidate bounding box set is output;

[0092] The candidate bounding box in the candidate bounding box set is mapped to the spatial scene feature map F, and the candidate bounding box feature corresponding to the candidate bounding box is extracted through ROIAlign and input into the classification and regression head for judgment:

[0093] A preset category library is acquired, the classifier of the classification and regression head is used to output the class matching probability of the dynamic target and each preset category in the preset category library, and the preset category corresponding to the maximum class matching probability is selected as the target category;

[0094] The regressor of the classification and regression head outputs a bounding box correction coefficient, and the coordinates of the candidate bounding box are updated to obtain the coordinates of the target bounding box.

[0095] The expression of the prediction loss function is as follows:

[0096] In the formula, represents the prediction loss function; represents the total number of spatial scene anchor boxes; represents the number of spatial scene anchor boxes in the dynamic target; represents the cross-entropy loss function for optimizing the classification of dynamic targets and static backgrounds; represents the smooth L1 loss function for optimizing the parameters of the candidate bounding box; represents the probability that the i-th spatial scene anchor box belongs to the dynamic target; represents the i-th real label, when the i-th spatial scene anchor box belongs to the dynamic target, when the i-th spatial scene anchor box belongs to the static background, ; represents the balance coefficient, ; represents the parameters of the i-th candidate bounding box, , respectively represent the offset of the center coordinates of the i-th candidate bounding box in the x and y directions, respectively represent the width and height scaling factors of the i-th candidate bounding box; represents the relative parameters of the i-th candidate bounding box and the spatial scene anchor box, , respectively represent the relative offset of the center coordinates of the i-th candidate bounding box in the x and y directions, respectively represent the relative width and height scaling factors of the i-th candidate bounding box.

[0097] First, the spatial scene image data after distortion correction and grayscale processing can be input into the improved Faster R-CNN model backbone network. The backbone network completes feature extraction through the cooperative operation of multiple convolutional layers and pooling layers: the convolutional layer captures features in local image regions with sliding convolutional kernels, gradually extracting basic features such as edges and textures, and higher-level semantic features; the pooling layer reduces the dimension of the feature map through downsampling, preserving key information while improving computational efficiency. Through this step-by-step extraction method, a spatial scene feature map that integrates multi-scale information is finally generated. This feature map contains not only the prominent features of dynamic targets, but also the structural features of static backgrounds, and can fully represent the spatial distribution and attribute features of different scale targets in the image.

[0098] Then, the spatial scene feature map is sent into the region proposal network of the Faster R-CNN model. The region proposal network generates fixed-size spatial scene anchor boxes on the feature map through sliding windows. Based on these anchor boxes, the region proposal network simultaneously performs two core prediction tasks: one is to judge the probability that each anchor box belongs to a dynamic target, so as to distinguish the target from the background; the other is to predict the parameters of the candidate bounding box, including the center coordinate offset and the width-height scaling factor, for preliminarily locating the spatial range of the target. To optimize the prediction accuracy, the region proposal network sets a corresponding prediction loss function, which realizes the constraint on the prediction process by comprehensively considering the classification error and the bounding box regression error.

[0099] The prediction loss function consists of two parts: the cross-entropy loss function is used to optimize the classification task of dynamic targets and static backgrounds, and by quantifying the deviation of the predicted probability and the true label, the accuracy of the anchor box classification is improved; the smooth L1 loss function focuses on the optimization of the candidate bounding box parameters, and by reducing the prediction error of the bounding box coordinate offset and the scale, the positioning accuracy of the bounding box is enhanced. At the same time, the loss function introduces a balance coefficient to adjust the weights of the two types of losses, ensuring the collaborative optimization of the classification and regression tasks. By minimizing the loss function, the Faster R-CNN model iteratively updates the parameters, and finally outputs a set of filtered candidate bounding boxes.

[0100] After the set of candidate bounding boxes is generated, the candidate bounding boxes can be mapped to the spatial scene feature map, and ROIAlign is used to extract the feature region corresponding to each candidate box: ROIAlign avoids quantization errors in the feature alignment process through precise coordinate mapping and interpolation processing, ensuring that the extracted candidate bounding box features can truly reflect the local attributes of the target. The extracted features are input into the classification and regression head, wherein the classifier calculates the matching probability of the dynamic target and each category based on the preset category library, and selects the category with the highest probability as the target category; the regressor outputs the bounding box correction coefficient, which adjusts the center coordinate offset and width-height scaling factor of the candidate bounding box to finely update the bounding box coordinates, and finally obtains the target bounding box that accurately marks the spatial range of the dynamic target.

[0101] Based on the target category and the target bounding box, dynamic spatial feature points in the spatial environment point cloud data are removed through a motion analysis method, and static spatial feature points are retained to construct an initial spatial point cloud map, specifically including:

[0102] According to the target category, a bounding box expansion coefficient is set, and the product of the bounding box region corresponding to the target bounding box and the bounding box expansion coefficient is a bounding box expansion region, and the spatial environment point cloud data located in the bounding box expansion region is filtered to generate a dynamic candidate point cloud set;

[0103] The motion analysis method is used to analyze inter-frame motion of the dynamic candidate point cloud in the dynamic candidate point cloud set, and a spatial position change amount of the dynamic candidate point cloud in two continuous frames is calculated.

[0104] A motion change amount threshold is set, and when the spatial position change amount exceeds the motion change amount threshold, the dynamic candidate point cloud is determined as the dynamic spatial feature point, otherwise, the dynamic candidate point cloud is determined as the static spatial feature point.

[0105] The dynamic spatial feature points are removed, and the static spatial feature points are retained.

[0106] The static spatial feature points are spliced according to a time sequence, a voxel filtering algorithm is used to down-sample the static spatial feature points, key geometric features in the static spatial feature points are retained, and the initial spatial point cloud map is generated.

[0107] First, a corresponding bounding box expansion coefficient is set according to the attribute features of the target category, and a region expansion process is performed on the target bounding box. The expansion region of the bounding box is obtained by multiplying the original region of the target bounding box by the expansion coefficient. The above process can compensate for the possible deviation of the bounding box positioning, more comprehensively cover the point cloud range associated with the dynamic target in space, and avoid the omission of dynamic point cloud due to insufficient accuracy of the bounding box. Based on the expansion region, all point cloud data in the region are selected from the spatial environment point cloud data to form a dynamic candidate point cloud set, which focuses on potential dynamic feature points for subsequent motion analysis.

[0108] Subsequently, a motion analysis method is used to analyze the inter-frame motion characteristics of the dynamic candidate point cloud set. By extracting the spatial coordinate information of the same dynamic candidate point cloud in two continuous frames of data, the spatial position change amount in the time dimension is accurately calculated, and the motion state of the point cloud is quantified. The above process can effectively capture the motion trajectory features of the dynamic target associated point cloud, and provide a quantitative basis for distinguishing real dynamic points from static points.

[0109] Then, a motion change amount threshold is set as a judgment standard for feature point classification. When the spatial position change amount of the dynamic candidate point cloud exceeds the threshold, it indicates that the dynamic candidate point cloud has a significant position change with the dynamic target, and is determined as a dynamic spatial feature point. If the change amount does not exceed the threshold, it is determined as a static spatial feature point that is not affected by the motion of the dynamic target. The setting of the threshold can effectively filter false motion signals caused by measurement noise or small environmental disturbances, and ensure the accuracy of the classification result.

[0110] Finally, the dynamic spatial feature points are removed, and only the static spatial feature points are reserved. The reserved static spatial feature points are spliced according to the time acquisition sequence, and the static structure information of the environment obtained at different times is integrated. At the same time, the voxel filtering algorithm is used to reduce the sampling of the spliced static point cloud data, and under the premise of retaining key geometric features such as corners and columns, the data redundancy is reduced, and the data size is optimized. Through the above operation, the initial spatial point cloud map which can accurately reflect the fixed geometric form of the environment is finally generated.

[0111] The ORB feature extraction is performed on the static spatial feature points to obtain a spatial feature descriptor, and a spatial visual feature map is constructed based on the spatial feature descriptor, and specifically includes:

[0112] The static spatial feature points with a gray change degree greater than a significant gray change threshold are selected as effective static feature points;

[0113] The direction angle of the effective static feature points is calculated by the gray centroid method, and the 31*31 pixel neighborhood of the effective static feature points is rotated to the positive direction corresponding to the direction angle;

[0114] The 256 pairs of pixel points in the 31*31 pixel neighborhood of the rotated effective static feature points are compared in gray, and the spatial feature descriptor with a length of 256 bits is generated;

[0115] The spatial feature descriptors are screened and matched by calculating the Hamming distance, the corresponding relationship between the effective static feature points is established, and the spatial visual feature map is constructed.

[0116] The calculation formula of the direction angle is:

[0117] In the formula, represents the direction angle; represents the first-order gray moment of the 31*31 pixel neighborhood of the effective static feature points in the x direction, which is used to reflect the gray distribution gravity offset of the 31*31 pixel neighborhood of the effective static feature points in the x direction; represents the first-order gray moment of the 31*31 pixel neighborhood of the effective static feature points in the y direction, which is used to reflect the gray distribution gravity offset of the 31*31 pixel neighborhood of the effective static feature points in the y direction; represents the gray value of the pixel point with coordinates (x, y) in the 31*31 pixel neighborhood of the effective static feature points, ; arctan() represents the inverse tangent function;

[0118] The binary form of the spatial feature descriptor is:

[0119] D represents a binary form of a spatial feature descriptor; represents the a-th pixel pair in the 31x31 pixel neighborhood of the rotated effective static feature point; represents the a-th pixel pair gray value.

[0120] First, by evaluating the gray level change degree of each static spatial feature point, the points with a gray level change greater than a significant gray level change threshold are selected as effective static feature points. Such points have a significant difference in gray level distribution, which is more conducive to subsequent feature extraction and matching.

[0121] Next, the gray centroid method is used to analyze the gray level distribution of the 31x31 pixel neighborhood of the effective static feature point to determine the direction angle. Specifically, the first-order gray moments in the x direction and y direction within the neighborhood are calculated, where the x direction gray moment reflects the gray level distribution center of gravity offset of the neighborhood in the x axis direction, and the y direction gray moment reflects the center of gravity offset in the y axis direction; based on the two gray moments, the direction angle is calculated by the arctangent function, which can represent the main direction of the gray level distribution in the neighborhood. Subsequently, the 31x31 pixel neighborhood of the effective static feature point is rotated to the positive direction corresponding to the direction angle to ensure that the subsequent feature description has rotation invariance.

[0122] Then, for the rotated 31x31 pixel neighborhood, 256 pairs of pixel points are selected, and the gray level values of each pair of pixel points are compared: if the gray level value of the former is less than that of the latter, the corresponding bit is recorded as 1, otherwise as 0, and finally a binary spatial feature descriptor with a length of 256 bits is generated. This binary encoding method can efficiently quantify the gray level distribution features of the neighborhood, and has low calculation and storage cost, which is suitable for large-scale feature processing.

[0123] Finally, the Hamming distance between the spatial feature descriptors corresponding to different effective static feature points is calculated to measure the similarity of the descriptors. Based on the Hamming distance, the descriptors with high matching degree are selected to establish the spatial correspondence between the effective static feature points. By integrating these correspondence relationships, the spatial position of the effective static feature points is associated with their visual feature descriptors, and finally a spatial visual feature map containing the spatial distribution and visual attributes of the feature points is constructed, providing accurate visual feature reference for subsequent multi-modal map joint optimization.

[0124] The initial spatial point cloud map and the spatial visual feature map are jointly optimized by the graph optimization algorithm to generate a multi-modal global spatial map, specifically including:

[0125] obtaining a point cloud map node set of the initial spatial point cloud map , , M represents the total number of point cloud map nodes in the point cloud map node set, and the spatial visual feature map node set of the spatial visual feature map , , R represents the total number of visual feature map nodes in the visual feature map node set, the map node set is constructed , ;

[0126] The pose node set is set , , K represents the total number of pose nodes in the pose node set, and the point cloud matching edge and the visual re-projection edge are set in combination with the map node set , the spatial factor graph is constructed, wherein the point cloud matching edge is used to connect the pose node and the point cloud map node, and the visual re-projection edge is used to connect the pose node and the visual feature map node

[0127] Based on the spatial factor graph, the point cloud matching error and the visual re-projection error are set, wherein the point cloud matching error is used to represent the deviation of the point cloud map node under the pose node and the map node , and the visual re-projection error is used to represent the re-projection deviation of the visual feature map node under the pose node ;

[0128] The total target function is constructed, the Gauss-Newton method is used to iteratively optimize the total target function, and the incremental equation is solved, U represents the Hessian matrix of the total target function, represents the parameter update amount, b represents the gradient vector, the pose node set and the map node set are updated, the sum of squares of the point cloud matching error and the visual re-projection error is minimized by weighted summation, and the optimized pose node set and the optimized map node set are output, and the optimized map node set includes the optimized point cloud map node set and the optimized visual feature map node set ;

[0129] If the optimized point cloud map node and the optimized visual feature map node have a spatial distance less than , a spatial fusion threshold, the optimized point cloud map node and the optimized visual feature map node are spatially fused by using a weighted average method to generate a spatial fusion map node .

[0130] The spatial fusion map node and the corresponding spatial feature descriptor are associated to generate the multi-modal global spatial map.

[0131] The point cloud matching error is calculated according to the following formula:

[0132] In the formula, denotes the transformation matrix corresponding to the pose node . denotes the point cloud map node observed under the pose node .

[0133] The visual re-projection error is calculated according to the following formula:

[0134] In the formula, denotes a perspective projection function; denotes an intrinsic matrix; denotes the visual feature map node observed under the pose node .

[0135] The sum of squares of the point cloud matching error and the visual re-projection error is minimized by weighted summation, and the corresponding calculation formula is as follows:

[0136] In the formula, respectively denote the information matrices of the point cloud matching error and the visual re-projection error;

[0137] The calculation formula of the spatial fusion map node is as follows:

[0138] In the formula, respectively denote the spatial fusion weights corresponding to the optimized point cloud map node and the optimized visual feature map node .

[0139] Firstly, the point cloud map node set contained in the initial spatial point cloud map and the visual feature map node set contained in the spatial visual feature map are acquired, and the two are merged into a unified map node set, thereby realizing node-level integration of the point cloud geometric information and the visual feature information.

[0140] Next, a pose node set composed of multiple pose nodes is set, and a point cloud matching edge and a visual re-projection edge are defined: the point cloud matching edge is used to connect the pose node and the point cloud map node; the visual re-projection edge is used to connect the pose node and the visual feature map node. In combination with the integrated map node set, a spatial factor graph containing the pose node, the map node and the two types of constraint edges is constructed, and the geometric and projection constraints between the nodes are quantified through the graph structure.

[0141] Subsequently, a point cloud matching error and a visual re-projection error are defined: the point cloud matching error is used to measure the spatial deviation of the observed point cloud map node and the map node under the action of the transformation matrix corresponding to the pose node; the visual re-projection error is used to represent the re-projection deviation of the visual feature map node observed under the pose node after being converted by the perspective projection function and the intrinsic matrix. Based on the two types of errors, a total objective function is constructed, and iterative optimization is performed through the Gauss-Newton method: the increment equation composed of the Hessian matrix, the parameter update amount and the gradient vector is solved, the sum of squares of the point cloud matching error and the visual re-projection error is minimized in a weighted summation manner, the pose node set and the map node set are constantly updated, and finally the optimized pose node and the map node set containing the optimized point cloud node and the optimized visual node are output.

[0142] If the spatial distance between the optimized point cloud map node and the visual feature map node is less than the spatial fusion threshold, the two represent the same spatial position, and a weighted average method is used for fusion: according to the confidence of the two types of nodes, corresponding spatial fusion weights are given, a spatial fusion map node is generated through weighted calculation, and information fusion of the geometric and visual features in the same spatial position is realized.

[0143] Finally, the spatial fusion map node is associated with its corresponding spatial feature descriptor, the geometric-visual composite features of the fusion map node and the optimized pose information are integrated, and a multi-modal global spatial map containing accurate spatial geometric structure and rich visual texture information is formed, thereby providing a unified and high-precision spatial reference benchmark for subsequent dynamic target pose estimation.

[0144] As shown in Figure 2 FIG. 1 is a system block diagram of a large space real-time positioning system based on SLAM and visual graphics provided by an embodiment of the present application, the system comprises:

[0145] The data acquisition module is configured to acquire spatial scene image data through a visual sensor, acquire spatial environment point cloud data through a laser radar, and perform distortion correction and grayscale processing on the spatial scene image data, and perform denoising and down-sampling processing on the spatial environment point cloud data.

[0146] The target recognition module is configured to perform dynamic detection on the spatial scene image data based on a deep learning target detection model, recognize a dynamic target and a corresponding target category and target bounding box.

[0147] The point cloud map construction module is configured to eliminate dynamic spatial feature points in the spatial environment point cloud data and retain static spatial feature points based on the target category and the target bounding box through a motion analysis method, and construct an initial spatial point cloud map.

[0148] The visual map construction module is configured to perform ORB feature extraction on the static spatial feature points to obtain spatial feature descriptors, and construct a spatial visual feature map based on the spatial feature descriptors.

[0149] The joint optimization module is configured to perform joint optimization on the initial spatial point cloud map and the spatial visual feature map through a graph optimization algorithm, and generate a multi-modal global spatial map.

[0150] The pose determination module is configured to adopt a visual-inertial odometer framework, combine spatial inertial measurement information of the dynamic target collected by an inertial measurement unit, perform real-time pose estimation on the dynamic target to obtain an initial pose, and perform optimization on the initial pose through a bundle adjustment method based on the multi-modal global spatial map to obtain a final pose.

[0151] Figure 2 The device of the embodiment shown can be used to perform the steps in the method embodiment shown, and the implementation principles and technical effects are similar, which will not be described here. Figure 1 The device of the embodiment shown can be used to perform the steps in the method embodiment shown, and the implementation principles and technical effects are similar, which will not be described here.

[0152] An electronic device includes a memory and a processor, and the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor performs the steps of the large-space real-time positioning method based on SLAM and visual graphics according to any one of the above.

[0153] As Figure 3 shown, a hardware structure schematic diagram of an electronic device is provided, and the electronic device 30 includes a processor 31, a memory 32 and a computer program; wherein

[0154] A memory 32 for storing the computer program, which can also be a flash memory. The computer program is, for example, an application program, a function module, etc. for implementing the above method.

[0155] A processor 31 for executing the computer program stored in the memory to implement each step of the method performed by the device. Details can be referred to the related description in the above method embodiments.

[0156] Optionally, the memory 32 can be independent or integrated with the processor 31.

[0157] When the memory 32 is independent of the processor 31, the device can further comprise:

[0158] A bus 33 for connecting the memory 32 and the processor 31.

[0159] A readable storage medium having a computer program stored therein, the computer program being executed by a processor to implement the steps of the method for large space real-time positioning based on SLAM and visual graph according to any one of the above embodiments.

[0160] The readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transfer of computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general or special purpose computer. For example, the readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). In addition, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in the communication device. The readable storage medium can be a read only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0161] The application further provides a program product comprising execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions to make the device implement the method provided in the various embodiments.

[0162] In the embodiments of the above apparatus, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or can also be any conventional processor, etc. The steps of the method disclosed in the present application can be directly embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0163] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A large space real-time positioning method based on SLAM and visual graphics, characterized in that, The method comprises: acquiring spatial scene image data through a visual sensor and spatial environment point cloud data through a laser radar, and performing distortion correction and grayscale processing on the spatial scene image data and denoising and down-sampling processing on the spatial environment point cloud data; performing dynamic detection on the spatial scene image data based on a deep learning target detection model, identifying dynamic targets and corresponding target categories and target bounding boxes; based on the target categories and the target bounding boxes, removing dynamic spatial feature points in the spatial environment point cloud data and retaining static spatial feature points through a motion analysis method, and constructing an initial spatial point cloud map; performing ORB feature extraction on the static spatial feature points to obtain spatial feature descriptors, and constructing a spatial visual feature map based on the spatial feature descriptors; jointly optimizing the initial spatial point cloud map and the spatial visual feature map through a graph optimization algorithm to generate a multi-modal global spatial map; adopting a visual inertial odometer framework, combining spatial inertial measurement information of the dynamic targets collected by an inertial measurement unit, performing real-time pose estimation on the dynamic targets to obtain an initial pose, and based on the multi-modal global spatial map, optimizing the initial pose through a bundle adjustment method to obtain a final pose; based on the target categories and the target bounding boxes, removing dynamic spatial feature points in the spatial environment point cloud data and retaining static spatial feature points through a motion analysis method, and constructing an initial spatial point cloud map, specifically comprising: setting a bounding box expansion coefficient according to the target categories, and the product of the target bounding box corresponding bounding box region and the bounding box expansion coefficient is a bounding box expansion region, and filtering the spatial environment point cloud data located in the bounding box expansion region to generate a dynamic candidate point cloud set; performing inter-frame motion analysis on dynamic candidate point clouds in the dynamic candidate point cloud set through the motion analysis method, and calculating the spatial position change amount of the dynamic candidate point clouds in two consecutive frames; setting a motion change amount threshold, when the spatial position change amount exceeds the motion change amount threshold, the dynamic candidate point cloud is determined to be the dynamic spatial feature point, otherwise it is determined to be the static spatial feature point; removing the dynamic spatial feature points and retaining the static spatial feature points; splicing the static spatial feature points according to the time sequence, performing down-sampling processing on the static spatial feature points through a voxel filtering algorithm, retaining key geometric features in the static spatial feature points, and generating the initial spatial point cloud map; the ORB feature extraction on the static spatial feature points to obtain spatial feature descriptors, and constructing a spatial visual feature map based on the spatial feature descriptors, specifically comprising: selecting the static spatial feature points with a grayscale change degree greater than a significant grayscale change threshold as effective static feature points; calculating the direction angle of the effective static feature points through a grayscale centroid method, and rotating the 31x31 pixel neighborhood of the effective static feature points to the positive direction corresponding to the direction angle; A 256-bit length spatial feature descriptor is generated by comparing the gray scale of 256 pairs of pixel points in a 31*31 pixel neighborhood of the effective static feature point after rotation. The correspondence between the effective static feature points is established by screening and matching the spatial feature descriptor through Hamming distance calculation, and the spatial visual feature map is constructed. 2.The large space real-time positioning method based on SLAM and visual graph according to claim 1, characterized in that, The deep learning target detection model dynamically detects the spatial scene image data, identifies dynamic targets, corresponding target categories and target bounding boxes, specifically including: The deep learning target detection model uses an improved Faster R-CNN model; The spatial scene image data after distortion correction and grayscale processing is input into the backbone network of the improved Faster R-CNN model, and multi-scale features of the spatial scene image data are extracted through convolution layers and pooling layers, generating a spatial scene feature map F, and the spatial scene feature map F includes the feature maps of dynamic targets and static backgrounds; The spatial scene feature map F is input into the region proposal network of the improved Faster R-CNN model, a spatial scene anchor box of a preset size is generated through a sliding window, and the probability that each spatial scene anchor box belongs to the dynamic target and the parameters of the candidate bounding box are predicted, and the corresponding prediction loss function is set; By minimizing the prediction loss function, the prediction process is optimized and a candidate bounding box set is output; The candidate bounding box in the candidate bounding box set is mapped to the spatial scene feature map F, and the candidate bounding box feature corresponding to the candidate bounding box is extracted through ROIAlign and input into the classification regression head for judgment: A preset category library is obtained, and based on the classifier of the classification regression head, the class matching probability of the dynamic target and each preset category in the preset category library is output, and the preset category corresponding to the maximum class matching probability is selected as the target category; Based on the regressor of the classification regression head, a bounding box correction coefficient is output, and the coordinates of the candidate bounding box are updated to obtain the coordinates of the target bounding box. 3.The large space real-time positioning method based on SLAM and visual graph according to claim 2, characterized in that, The expression of the prediction loss function is as follows: wherein, represents a prediction loss function; represents the total number of spatial scene anchor boxes; represents the number of spatial scene anchor boxes in dynamic targets; represents a cross-entropy loss function for optimizing the classification of dynamic targets and static backgrounds; represents a smooth L1 loss function for optimizing the parameters of candidate bounding boxes; represents the probability that the i-th spatial scene anchor box belongs to a dynamic target; represents the i-th real label when the i-th spatial scene anchor box belongs to a dynamic target, represents the i-th real label when the i-th spatial scene anchor box belongs to a static background, ; represents a balance coefficient, ; represents the parameters of the i-th candidate bounding box, , respectively represent the offset of the center coordinates of the i-th candidate bounding box in the x and y directions, respectively represent the width and height scaling factors of the i-th candidate bounding box; represents the relative parameters of the i-th candidate bounding box to the spatial scene anchor box, , respectively represent the relative offset of the center coordinates of the i-th candidate bounding box in the x and y directions, respectively represent the relative width and height scaling factors of the i-th candidate bounding box. 4.The method of claim 1, wherein, The calculation formula of the direction angle is as follows: wherein denotes a direction angle; denotes a first-order gray moment of the 31x31 pixel neighborhood of the effective static feature point in the x direction, for reflecting a gray distribution barycentric offset of the 31x31 pixel neighborhood of the effective static feature point in the x direction; denotes a first-order gray moment of the 31x31 pixel neighborhood of the effective static feature point in the y direction, for reflecting a gray distribution barycentric offset of the 31x31 pixel neighborhood of the effective static feature point in the y direction; denotes a gray value of a pixel point with coordinates (x, y) in the 31x31 pixel neighborhood of the effective static feature point, arctan() denotes an inverse tangent function; The binary form of the spatial feature descriptor is as follows: where D represents a binary form of spatial feature descriptor; represents the a-th pixel pair in the 31x31 pixel neighborhood of the rotated valid static feature point; represents the a-th pixel pair gray value of the a-th pixel pair 5.The large space real-time positioning method based on SLAM and visual graph according to claim 1, characterized in that, The initial spatial point cloud map and the spatial visual feature map are jointly optimized by the graph optimization algorithm to generate a multi-modal global spatial map, specifically including: obtaining a point cloud map node set of the initial spatial point cloud map , , M represents the total number of point cloud map nodes in the point cloud map node set, and a visual feature map node set of the spatial visual feature map , , R represents the total number of visual feature map nodes in the visual feature map node set, and a map node set is constructed , ; Set of pose nodes , , K denotes the total number of pose nodes in the set of pose nodes, and set of point cloud matching edges and set of visual re-projection edges , in combination with the set of map nodes , construct a spatial factor graph, wherein the point cloud matching edges are used to connect pose nodes and point cloud map nodes, and the visual re-projection edges are used to connect pose nodes and visual feature map nodes; based on the spatial factor graph, set a point cloud matching error and a visual re-projection error wherein the point cloud matching error is used to represent a pose node under a point cloud map node and a map node deviation, the visual re-projection error is used to represent a re-projection deviation of a visual feature map node under a pose node ; constructing a total objective function, iteratively optimizing the total objective function by using a Gauss-Newton method, and solving an increment equation , U represents a Hessian matrix of the total objective function, represents a parameter update amount, b represents a gradient vector, updating the pose node set and the map node set , minimizing a square sum of the point cloud matching error and the visual re-projection error by weighted summation, and outputting an optimized pose node set and an optimized map node set , and the optimized map node set includes an optimized point cloud map node set and an optimized visual feature map node set ; If the spatial distance between the optimized point cloud map node and the optimized visual feature map node is less than , a spatial fusion threshold, a weighted average method is used to perform spatial fusion on the optimized point cloud map node and the optimized visual feature map node , to generate a spatial fusion map node . associating the spatial fusion map node and the corresponding spatial feature descriptors, generating the multi-modal global spatial map.

6. The large space real-time positioning method based on SLAM and visual graphics according to claim 5, characterized in that, The point cloud matching error The formula for calculating the point cloud matching error is as follows: wherein, represents a pose node a corresponding transformation matrix; represents a pose node a point cloud map node observed below; the visual re-projection error The formula for the calculation of the visual re-projection error is as follows: wherein, denotes a perspective projection function; denotes an intrinsic matrix; denotes a pose node a visual feature map node observed by the camera.

7. The large space real-time positioning method based on SLAM and visual graphics according to claim 5, characterized in that, minimizing the point cloud matching error by weighted sum and the squared sum of the visual re-projection error The corresponding computational formula is as follows: wherein, respectively represent the information matrix of the point cloud matching error, the visual re-projection error. The spatial fusion map node The calculation formula is as follows: wherein, respectively represent the optimized point cloud map node , the optimized visual feature map node corresponding spatial fusion weight.

8. A large space real-time positioning system based on SLAM and visual graphics, applied to the large space real-time positioning method based on SLAM and visual graphics as claimed in any one of claims 1-7, characterized in that, The system includes: A data acquisition module is configured to acquire spatial scene image data through a visual sensor, acquire spatial environment point cloud data through a laser radar, and perform distortion correction and grayscale processing on the spatial scene image data, and perform denoising and down-sampling processing on the spatial environment point cloud data; A target identification module is configured to dynamically detect the spatial scene image data based on a deep learning target detection model, identify dynamic targets, corresponding target categories and target bounding boxes. a point cloud map construction module, configured to eliminate dynamic spatial feature points in the spatial environment point cloud data and retain static spatial feature points based on the target category and the target bounding box through a motion analysis method, and construct an initial spatial point cloud map; a visual map construction module, configured to perform ORB feature extraction on the static spatial feature points to obtain spatial feature descriptors, and construct a spatial visual feature map based on the spatial feature descriptors; a joint optimization module, configured to perform joint optimization on the initial spatial point cloud map and the spatial visual feature map through a graph optimization algorithm, and generate a multi-modal global spatial map; a pose determination module, configured to adopt a visual-inertial odometry framework, combine spatial inertial measurement information of the dynamic target collected by an inertial measurement unit, perform real-time pose estimation on the dynamic target to obtain an initial pose, and perform optimization on the initial pose through a bundle adjustment method based on the multi-modal global spatial map to obtain a final pose.

Images