An end-to-end SLAM method based on deep recurrent convolutional neural network

Abstract

The present invention discloses an end-to-end SLAM method based on a deep recurrent convolutional neural network. The method collects image sequence frames in real time through a depth camera, and converts them into a set image size before outputting; creates a convolutional neural network model to receive and process the image sequence frames output by the depth camera, and outputs the extracted feature point cloud coordinates and descriptor information and camera pose information for establishing topological information of the image geometric structure; creates a recurrent neural network model to receive and process the image sequence frames output by the depth camera, and outputs credible semantic areas and object-level landmark information with a credibility above a set first threshold; creates a grid map with semantic information. The present invention constructs a three-dimensional map with semantic recognition and perception based on a deep learning model, and has significant improvements in parameter indicators such as positioning accuracy, map construction accuracy, loop detection accuracy, system efficiency and robustness.

Description

Technical Field

[0001] The present invention relates to the field of spatial positioning technology, and in particular to an end-to-end SLAM method based on a deep recurrent convolutional neural network. Background Art

[0002] SLAM (Simultaneous localization and mapping) technology is the forefront of spatial positioning technology in the visual field. It is mainly used to solve the positioning and map construction problems of robots or cameras when they move in unknown environments. It is widely used in VR / AR, drones, unmanned driving and other fields. SLAM (Simultaneous Localization and Mapping) technology has many implementation methods. According to the type of sensors used, data processing methods and application scenarios, SLAM technology can be divided into the following categories:

[0003] 1. Visual SLAM

[0004] Visual SLAM relies on cameras to obtain image data of the environment and analyzes the feature points in these images to achieve positioning and map construction. This method usually uses monocular, binocular or RGB-D (color image and depth information) cameras.

[0005] 2. Laser SLAM (LiDAR SLAM)

[0006] Laser SLAM is based on a laser radar (LiDAR) sensor, which acquires depth information and distance data of the environment by emitting lasers and measuring their reflection time. This method is often used in scenarios that require high-precision positioning and large-scale map construction.

[0007] 3. Acoustic SLAM

[0008] Acoustic SLAM uses ultrasonic or acoustic wave sensors to measure distance and combines other sensors (such as IMU or visual sensors) to build maps and perform positioning. This method is suitable for scenes with complex lighting environments and is often used in specific fields (such as underwater environments or aerodynamic applications).

[0009] The shortcomings of current visual SLAM technology are mainly in the following aspects:

[0010] (1) Sensitive to lighting changes. Visual SLAM is highly dependent on the quality and stability of ambient light. When the light is insufficient (such as at night or in a dark room) or too strong (such as direct sunlight), it is difficult for the camera to obtain a clear image, which will affect the feature extraction and positioning accuracy of the SLAM system.

[0011] (2) Dynamic scene problem. In dynamic scenes, visual SLAM is easily disturbed by moving objects. For example, if there are a large number of moving objects (such as pedestrians and vehicles) in the images captured by the camera, it may lead to incorrect feature matching and inaccurate map construction.

[0012] (3) Feature-poor scenes. In scenes with few or repeated features (such as white walls, corridors, deserts, etc.), visual SLAM has difficulty extracting enough unique feature points for positioning, which may lead to loss of tracking or increased positioning error.

[0013] (4) Occlusion problem. The camera field of view may be blocked, for example, by the user’s hand, other objects, or the structure of the device itself, which may cause interruption of tracking. Summary of the invention

[0014] The purpose of the present invention is to provide an end-to-end SLAM method based on a deep recurrent convolutional neural network in view of the shortcomings of the prior art.

[0015] To achieve the above object, the present invention provides an end-to-end SLAM method based on a deep recurrent convolutional neural network, comprising:

[0016] The depth camera collects image sequence frames in real time and converts them into the set image size for output;

[0017] Create a convolutional neural network model to receive and process the image sequence frames output by the depth camera, perform feature extraction, image feature matching, and camera pose motion state estimation on the input image sequence frames in sequence, and output the extracted feature point cloud coordinates and descriptor information and camera pose information to establish the topological information of the image geometric structure;

[0018] Create a recurrent neural network model to receive and process the image sequence frames output by the depth camera, to perform image segmentation, dynamic feature removal based on the attention mechanism, and feature matching under semantic constraints on the input image sequence frames in sequence, and output credible semantic areas and object-level landmark information with a credibility above a set first threshold;

[0019] Create a grid map with semantic information as follows:

[0020] Utilize object-level landmark information to obtain pixel-level semantic information, annotate semantic categories on corresponding feature point clouds, and perform incremental Bayesian updates so that point cloud category labels converge to stability as more observations are made, thereby creating a point cloud map with semantic information.

[0021] Each point cloud with semantic information is converted into a bag-of-words vector, which represents the frequency of occurrence of each visual word, and the frequency is weighted by the word frequency; the bag-of-words vector of the point cloud map is matched with the bag-of-words vector of the known object based on the object-level data association of the bag-of-words vector to calculate the similarity, and the object contained in the point cloud map is determined according to the calculated similarity;

[0022] Map the feature point cloud coordinates to a grid system, and then record the occupancy status of the feature point cloud into the grid;

[0023] Define a grid map, map the feature point cloud coordinates to the coordinate system of the grid map, and call the MapCoordinatesToGrid method after obtaining the point cloud coordinate data each time to add the identified and tracked object positions to the grid, thereby constructing a grid map of the entire environment.

[0024] Furthermore, it also includes: graph optimization and closed-loop correction based on loop detection, as follows:

[0025] The similarity between the current image frame and the key frame is determined by calculating the Hamming distance between the feature points of the current image frame and the pre-selected key frame. When the similarity between the current image frame and the key frame exceeds a set second threshold, loop detection is triggered.

[0026] Through the iSAM incremental graph optimization algorithm, a graph containing camera trajectories, feature points, and the constraint relationships between camera trajectories and feature points is constructed, the camera pose, feature points, and edges are calculated, and a residual sum of squares function is constructed to calculate the constraint error. The nodes in the graph are adjusted by minimizing the constraint error, and optimization is performed using the nonlinear least squares method.

[0027] Furthermore, the feature extraction method is specifically as follows:

[0028] Use the convolution operation of the convolution layer to identify the basic features in the image;

[0029] Use pooling layers to reduce the spatial dimensions of basic features;

[0030] Use the fully connected layer to fuse the features output by the pooling layer to generate a more abstract global representation;

[0031] Focusing on different areas of the image based on the spatial attention mechanism, assigning different weights to different areas according to their importance, generating a weight map, and adjusting the contribution of each position of the feature map according to the weight map;

[0032] Focus on the important areas in the image by weighted average or weighted sum, and generate a spatial weight map to emphasize the features of the important areas;

[0033] The final feature representation of the image is generated through the output layer, and the probability distribution after the Sigmoid activation function indicates the category to which the image belongs.

[0034] Furthermore, the image feature matching method is specifically as follows:

[0035]

[0036] in, is the Hamming distance between feature A and feature B, To calculate the feature description in feature A The feature description in feature B Are they equal, n is the number of feature descriptions;

[0037] The AKAZE algorithm can be used to extract feature points of images at different scales, and detect key points based on the region method in the scale space to obtain the coordinates of the feature points;

[0038] AKAZE uses M-LDB descriptor to describe the surrounding area of ​​feature points;

[0039] Feature matching is performed by comparing the M-LDB descriptors between different images, thereby identifying similar areas in the images and constructing the geometric features of the images.

[0040] Furthermore, the camera position motion state estimation method is as follows:

[0041] Initialize camera parameter settings;

[0042] Convert the image into a Mat matrix, calculate the image disparity value according to the stereo matching algorithm, and convert the disparity value into a depth value to obtain the depth map of the image;

[0043] The 3D world coordinate values ​​of the spatial feature points are obtained based on the 2D pixel coordinate values ​​and depth values ​​of the image feature points, and the camera's posture is estimated from the known correspondence between the 2D pixel coordinate values ​​and the 3D world coordinate values ​​through the PnP algorithm; the rotation and translation matrix is ​​converted into a series, the previous frame posture series is compared with the current posture series, and filtered using low-pass and optical flow filtering; a 4×4 transformation matrix is ​​created, with the camera as the center, the camera coordinates are converted into world coordinates and image coordinates in turn, and the camera posture and motion state estimation are adjusted.

[0044] Furthermore, the image segmentation is used to divide the image into multiple block images, and the feature of each block image is used as a time step input of the recurrent neural network model.

[0045] Furthermore, the method of removing dynamic features based on the attention mechanism is as follows:

[0046] Create an LSTM layer to extract the edges and corners of image objects as landmarks, and then add an attention layer, which dynamically calculates the attention weights of each feature based on the current input and hidden state;

[0047] According to the historical pixel semantic category, confidence, reprojection depth error, and photometric error corresponding to the landmark points, the reliability score of the landmark points is continuously updated, and the landmark points with reliability lower than the set third threshold are eliminated; an initial dynamic probability is given to each new landmark point. If the feature point corresponding to the landmark point falls within the contour of the prior dynamic object, its dynamic probability increases according to the weight, and the dynamic probability is propagated between frames, and the feature points with a final dynamic probability higher than the set fourth threshold are eliminated.

[0048] Furthermore, the feature matching method under the semantic constraint is specifically as follows:

[0049] The YOLOX model is used to focus on extracting credible semantic areas in the feature extraction stage, and a multi-association graph is constructed to model the spatial structure and scale hierarchy of each credible area. The matching between image object areas is achieved by minimizing an energy function.

[0050] Beneficial effects: The present invention combines deep recurrent convolutional neural networks with SLAM technology, which not only significantly reduces the impact of environmental texture and light transformation on image feature point extraction, but also can identify image semantic information, understand the outline, type, shape and appearance of objects in the image, and can create maps and real-time positioning even in dynamic environments; the method based on deep recurrent convolutional neural networks significantly improves the intelligent perception ability of visual SLAM, forming a visual semantic SLAM with multiple functions such as geometric measurement, positioning and posture determination, and environmental understanding; compared with traditional visual SLAM methods that only extract and construct planar geometric maps through feature point algorithms, it has significant improvements in parameter indicators such as positioning accuracy, map construction accuracy, loop detection accuracy, system efficiency and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of an end-to-end SLAM method based on a deep recurrent convolutional neural network;

[0052] Figure 2 It is a schematic diagram of the transformation of the object pose from the camera coordinate system to the image coordinate system. DETAILED DESCRIPTION

[0053] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. The embodiments are implemented based on the technical solution of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0054] like Figure 1As shown, an embodiment of the present invention provides an end-to-end SLAM method based on a deep recurrent convolutional neural network, comprising:

[0055] The depth camera collects image sequence frames in real time and converts them into the set image size before outputting them. It is preferred to scale the image to a size of 100×32×1.

[0056] Create a convolutional neural network (CNN) model to receive and process the image sequence frames output by the depth camera, perform feature extraction, image feature matching, and camera pose motion state estimation on the input image sequence frames in sequence, and output the extracted feature point cloud coordinates and descriptor information and camera pose information to establish the topological information of the image geometric structure. The details are as follows:

[0057] 1. Image feature extraction

[0058] Extract key features from images that help distinguish different images or objects. CNN automatically learns these features through multiple layers of convolution and pooling operations. The specific steps are as follows:

[0059] 1) Convolutional layer extracts local features

[0060] In the convolution layer, through the convolution operation, CNN can identify basic features such as edges, corners, textures, etc. in the image. The convolution kernel (or filter) used in each layer of convolution is learnable, and the network will automatically adjust the parameters of the convolution kernel according to the training data, so that the network can extract more distinguishing features.

[0061] 2) Pooling layer dimensionality reduction

[0062] The pooling layer is used to reduce the spatial dimensions (such as the width and height of the image), retain important features, and improve the robustness of the features. The pooling operation can reduce the amount of computation while avoiding overfitting.

[0063] 3) Fully connected layer and feature fusion

[0064] After convolution and pooling operations, the feature map is flattened into a one-dimensional vector and input to the fully connected layer for further processing. The function of the fully connected layer is to fuse the features output by the pooling layer to generate a more abstract global representation.

[0065] 4) Spatial Attention Mechanism

[0066] The spatial attention mechanism focuses on different areas of the image, assigning different weights to different positions according to the importance of the image area, allowing the model to dynamically decide which parts of the information are more important. By generating a weight map (also called an attention map) to indicate which areas should receive more attention, the network adjusts the contribution of each position of the feature map according to the weight map.

[0067] Focusing on important areas in the image by weighted average or weighted sum, generating a spatial weight map to emphasize the features of important areas. Assuming the input X is a feature map, the spatial attention S can be generated by convolution operation:

[0068]

[0069] in, is a 1×1 convolution, used to generate the weights for each spatial position; is the activation function used to normalize the generated weights.

[0070] 5) Output layer

[0071] Finally, the final feature representation of the image is generated through the output layer, and the probability distribution after the Sigmoid activation function indicates the category to which the image belongs.

[0072] Through the above process, CNN can automatically extract important features in the image (such as edges, corners, etc.) and map them to a low-dimensional feature space to facilitate subsequent image matching and classification.

[0073] 2. Image feature matching

[0074] Image feature matching is to compare the similarities between two or more images, identify whether they belong to the same category, or identify the same or similar parts in the images. The specific steps are as follows:

[0075] 1) Calculation of eigenvectors

[0076] After extracting the feature vector of the image through CNN, it can be used to calculate the similarity between images. The similarity between images is determined by the Hamming distance method. If the similarity between the two feature vectors A and B is very high, it means that the images are very similar.

[0077] The calculation formula of Hamming distance is:

[0078]

[0079] in, is the Hamming distance between feature A and feature B, To calculate the feature description in feature A The feature description in feature B Whether they are equal, n is the number of feature descriptions, and the smaller the Hamming distance value, the higher the similarity between feature vectors A and B.

[0080] 2) Key point detection

[0081] The AKAZE algorithm can extract feature points of images at different scales, and has the characteristics of scale invariance and rotation invariance. In the scale space, a region-based method is used to detect key points and obtain the coordinates of the feature points. These key points correspond to local extreme points in the image, that is, they have the maximum or minimum grayscale value in the surrounding area.

[0082] 3) Descriptor Generation

[0083] After detecting the key points, AKAZE uses the M-LDB descriptor to describe the surrounding area of ​​the feature points, which can describe the characteristics of the image.

[0084] 4) Feature matching

[0085] Finally, feature matching is performed by comparing the M-LDB descriptors between different images to identify similar areas in the images and construct the geometric features of the images.

[0086] 3. Camera pose and motion state estimation

[0087] Use CNN to infer the six degrees of freedom (6DoF) of the camera from a single image, namely three translation coordinates (x, y, z) and three rotation angles, namely Euler angles.

[0088] 1) Initialize camera parameter settings

[0089] Initialize the camera focal length, image center point, camera internal matrix, camera field of view, distortion coefficient and other values.

[0090] 2) Calculate disparity map and depth map

[0091] The image is converted into a Mat matrix, and the image disparity value is calculated according to the stereo matching algorithm, that is, the difference in the coordinates of the feature pixels corresponding to the left and right cameras. The disparity value is converted into a depth value according to Formula 1 to obtain the image depth map.

[0092] Formula 1

[0093] Among them, Z is the depth value, B is the camera baseline length, f is the camera focal length, and d is the disparity value.

[0094] 3) Camera pose estimation and coordinate transformation

[0095] The 3D world coordinates (x, y, z) of the spatial feature points can be obtained based on the 2D pixel coordinates (x, y) and depth values ​​of the image feature points. The PnP algorithm is used to estimate the camera's pose from the known 2D pixel coordinates-3D world coordinates. The rotation and translation matrix is ​​converted into a sequence, and the previous frame pose sequence is compared with the current pose sequence and filtered using low-pass and optical flow filtering to reduce noise. A 4×4 transformation matrix is ​​created, with the camera as the center, and the camera coordinates are converted into world coordinates, and then converted into image coordinates to adjust the camera pose and motion state estimation. For details, see Figure 2 , Figure 2 It shows the schematic diagram of the object pose conversion from the camera coordinate system to the image coordinate system, where O w -X w Y w Z w Represents the world coordinate system, describing the position of the camera, in units of m; O c -X c Y c Z c represents the camera coordinate system, with the optical center as the origin and the unit being m; O-XY represents the image coordinate system, with the optical center as the image center and the unit being mm.

[0096] In summary, the CNN model finally outputs the feature point cloud coordinates, feature descriptors, and camera pose information, which are used to establish the topological information of the image geometric structure.

[0097] Create a recurrent neural network model to receive and process the image sequence frames output by the depth camera, perform image segmentation, dynamic feature removal based on the attention mechanism, and feature matching under semantic constraints on the input image sequence frames, and output credible semantic areas and object-level landmark information with a credibility above the set first threshold. The details are as follows:

[0098] 1) Image segmentation

[0099] The image is divided into multiple small blocks (like a grid) and the features of each small block are used as a time step input to the RNN.

[0100] 2) Dynamic feature removal based on attention mechanism

[0101] The attention mechanism allows the model to assign different importance weights to different input features at each time step. This mechanism can dynamically select which features are most important for the prediction of the current moment or the entire sequence, thereby eliminating features. The segmented small blocks of the image are input into the RNN network, and an LSTM layer is created to extract geometric features such as edges and corners of the image objects as landmarks. Then an attention layer is added, which dynamically calculates the attention weights of each feature based on the current input and hidden state.

[0102] According to the historical pixel semantic category, confidence, reprojection depth error, photometric error and other geometric information corresponding to the landmark point, the reliability score of the landmark point is continuously updated, and the landmark points with reliability lower than the set third threshold are eliminated to achieve more accurate positioning, and the coordinates of the landmark points with higher credibility are output. An initial dynamic probability is given to each new landmark point. If the feature point corresponding to the landmark point falls within the contour of the prior dynamic object, its dynamic probability will increase according to the weight, and the dynamic probability will be propagated between frames. Finally, the feature points with dynamic probability higher than the set fourth threshold will be eliminated.

[0103] 3) Feature matching under semantic constraints

[0104] Using specific semantic matching results, the geometry-based feature matching results can be corrected and constrained. The YOLOX model is used to focus on extracting features of credible semantic areas (such as the ground, walls, and desktop logos) during the feature extraction phase, while weakening the focus on untrustworthy areas (such as the sky). A multi-association graph is constructed to model the spatial structure and scale level of each credible area, and matching between image object areas is achieved by minimizing an energy function.

[0105] In summary, the RNN model finally outputs credible semantic areas and object-level landmark information with high credibility, such as object feature point cloud coordinates, object size, shape, etc.

[0106] Create a grid map with semantic information as follows:

[0107] 1) Semantic point cloud map construction

[0108] By marking the image semantic information in the feature point cloud, a semantic point cloud map with category labels can be constructed. By using object-level landmark information to obtain pixel-level semantic information, the semantic categories are marked on the corresponding feature point cloud, and incremental Bayesian updates are performed so that the point cloud category labels can converge to stability as more observations are made, thus creating a point cloud map with semantic information.

[0109] 2) Create object-level data associations based on bag-of-words (BoW) vectors

[0110] The bag-of-words vector clusters the point cloud with semantic information to build a vocabulary. The vocabulary contains all possible "visual words" in the image, which represent a certain type of local features. Each point cloud with semantic information is converted into a bag-of-words vector, which represents the frequency of occurrence of each visual word, and the frequency is weighted using the word frequency (TF-IDF) to reduce the influence of common visual words.

[0111] Object-level data association based on bag-of-words vectors involves matching the BoW vector of an image with the BoW vector of a known object, and calculating the similarity between the BoW vector of the image and the BoW vector of the object in the training set (using Euclidean distance). Based on the size of the similarity, it is determined which object the image is most likely to contain. If the similarity exceeds a preset threshold, the image is considered to contain the object.

[0112] 3) Create a grid map

[0113] The feature point cloud coordinates are mapped to a grid system, and then the occupancy status of these points (such as whether they are occupied by obstacles) is recorded in a grid.

[0114] Then define a grid map. A grid map is a two-dimensional array in which each cell represents the occupancy status of an area. Usually, a grid map uses 0 to represent an idle area and 1 to represent an occupied area, or uses floating probability values ​​to represent different occupancy probabilities. Map the feature point cloud coordinates to the grid coordinate system. This can be achieved through coordinate transformation, converting the world coordinates to the corresponding grid coordinates. Each time after obtaining the point cloud coordinate data, you need to call the MapCoordinatesToGrid method to add the identified and tracked object positions to the grid to update the grid map, so that you can gradually build a grid map of the entire environment.

[0115] Loop detection is used to solve the problem of camera positioning drift caused by accumulated errors during long-term movement. The main goal of loop detection is to identify the places that the camera has previously recorded, and use this information to correct the current positioning of the camera, thereby avoiding serious errors in the map. The embodiment of the present invention also includes graph optimization and closed-loop correction based on loop detection, as follows:

[0116] 1) Keyframe selection

[0117] Loop detection is generally not performed between each frame, but through key frames to reduce the amount of calculation. A fixed threshold is set according to the change in camera posture (translation distance or rotation angle), and frames exceeding the fixed threshold are selected as key frames. These frames can better represent the changes in the environment. By performing loop detection between these key frames, the efficiency can be significantly improved.

[0118] 2) Trigger loop closure based on image matching threshold

[0119] In loop detection, calculating the similarity between the current image and the historical keyframes is the core step. Image similarity is measured based on matching metrics such as the Hamming distance between feature point descriptors based on feature point clouds. When the image similarity between the historical frame and the current frame exceeds the set second threshold, a loop is considered to have occurred.

[0120] 3) Graph optimization and closed-loop correction

[0121] Once a loop is detected, the graph optimization and positioning correction algorithm will be started. Through the iSAM incremental graph optimization algorithm, a graph containing the camera trajectory, feature points and the constraint relationship between them is constructed, the camera pose, feature points and edges (representing the movement between two consecutive frames) are calculated, and the residual square sum function is constructed to calculate the constraint error. The nodes in the graph are adjusted by minimizing these errors and optimized using nonlinear least squares. After the optimization process, the camera pose nodes and feature point nodes will be adjusted to a new position to minimize the error of the entire graph.

[0122] The present invention constructs a three-dimensional map with semantic recognition and perception based on a deep learning model. Compared with the traditional visual SLAM method that only extracts and constructs a planar geometric map through a feature point algorithm, it has significant improvements in positioning accuracy, map construction accuracy, loop detection accuracy, system efficiency, robustness and other parameter indicators, as shown in Table 1:

[0123] Table 1: Parameter comparison table

[0124]

[0125] After testing, compared with the traditional visual SLAM method, the end-to-end SLAM method of deep recurrent convolutional neural network has reduced the error range by more than 10 times, improved the loop detection accuracy by 30%, and increased the response speed by more than 5 times. It has the characteristics of high precision, low latency, and intelligent perception.

[0126] The above is only a preferred embodiment of the present invention. It should be noted that for ordinary technicians in this technical field, other parts not specifically described belong to the prior art or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. An end-to-end SLAM method based on deep recurrent convolutional neural network, characterized in that: include: The depth camera collects image sequence frames in real time and converts them into the set image size for output; Create a convolutional neural network model to receive and process the image sequence frames output by the depth camera, perform feature extraction, image feature matching, and camera pose motion state estimation on the input image sequence frames in sequence, and output the extracted feature point cloud coordinates and descriptor information and camera pose information to establish the topological information of the image geometric structure; Create a recurrent neural network model to receive and process the image sequence frames output by the depth camera, to perform image segmentation, dynamic feature removal based on the attention mechanism, and feature matching under semantic constraints on the input image sequence frames in sequence, and output credible semantic areas and object-level landmark information with a credibility above a set first threshold; Create a grid map with semantic information as follows: Utilize object-level landmark information to obtain pixel-level semantic information, annotate semantic categories on corresponding feature point clouds, and perform incremental Bayesian updates so that point cloud category labels converge to stability as more observations are made, thereby creating a point cloud map with semantic information. Each point cloud with semantic information is converted into a bag-of-words vector, which represents the frequency of occurrence of each visual word, and the frequency is weighted by the word frequency; the bag-of-words vector of the point cloud map is matched with the bag-of-words vector of the known object based on the object-level data association of the bag-of-words vector to calculate the similarity, and the object contained in the point cloud map is determined according to the calculated similarity; Map the feature point cloud coordinates to a grid system, and then record the occupancy status of the feature point cloud into the grid; Define a grid map, map the feature point cloud coordinates to the coordinate system of the grid map, and call the MapCoordinatesToGrid method after obtaining the point cloud coordinate data each time to add the identified and tracked object positions to the grid, thereby constructing a grid map of the entire environment.

2. An end-to-end SLAM method based on a deep recurrent convolutional neural network according to claim 1, characterized in that: Also includes: Graph optimization and closed-loop correction are performed based on loop closure detection, as follows: The similarity between the current image frame and the key frame is determined by calculating the Hamming distance between the feature points of the current image frame and the pre-selected key frame. When the similarity between the current image frame and the key frame exceeds a set second threshold, loop detection is triggered. Through the iSAM incremental graph optimization algorithm, a graph containing camera trajectories, feature points, and the constraint relationships between camera trajectories and feature points is constructed, the camera pose, feature points, and edges are calculated, and a residual sum of squares function is constructed to calculate the constraint error. The nodes in the graph are adjusted by minimizing the constraint error, and optimization is performed using the nonlinear least squares method.

3. An end-to-end SLAM method based on a deep recurrent convolutional neural network according to claim 1, characterized in that: The feature extraction method is as follows: Use the convolution operation of the convolution layer to identify the basic features in the image; Use pooling layers to reduce the spatial dimensions of basic features; Use the fully connected layer to fuse the features output by the pooling layer to generate a more abstract global representation; Focusing on different areas of the image based on the spatial attention mechanism, assigning different weights to different areas according to their importance, generating a weight map, and adjusting the contribution of each position of the feature map according to the weight map; Focus on the important areas in the image by weighted average or weighted sum, and generate a spatial weight map to emphasize the features of the important areas; The final feature representation of the image is generated through the output layer, and the probability distribution after the Sigmoid activation function indicates the category to which the image belongs.

4. An end-to-end SLAM method based on a deep recurrent convolutional neural network according to claim 3, characterized in that: The image feature matching method is as follows: ； in, is the Hamming distance between feature A and feature B, To calculate the feature description in feature A The feature description in feature B Are they equal, n is the number of feature descriptions; The AKAZE algorithm is used to extract feature points of the image at different scales, and the key points are detected based on the region method in the scale space to obtain the coordinates of the feature points; AKAZE uses M-LDB descriptor to describe the surrounding area of ​​feature points; Feature matching is performed by comparing the M-LDB descriptors between different images, thereby identifying similar areas in the images and constructing the geometric features of the images.

5. An end-to-end SLAM method based on a deep recurrent convolutional neural network according to claim 1, characterized in that: The method of estimating the camera position motion state is as follows: Initialize camera parameter settings; Convert the image into a Mat matrix, calculate the image disparity value according to the stereo matching algorithm, and convert the disparity value into a depth value to obtain the depth map of the image; The 3D world coordinate values ​​of the spatial feature points are obtained based on the 2D pixel coordinate values ​​and depth values ​​of the image feature points, and the camera's posture is estimated from the known correspondence between the 2D pixel coordinate values ​​and the 3D world coordinate values ​​through the PnP algorithm; the rotation and translation matrix is ​​converted into a series, the previous frame posture series is compared with the current posture series, and filtered using low-pass and optical flow filtering; a 4×4 transformation matrix is ​​created, with the camera as the center, the camera coordinates are converted into world coordinates and image coordinates in turn, and the camera posture and motion state estimation are adjusted.

6. An end-to-end SLAM method based on a deep recurrent convolutional neural network according to claim 1, characterized in that: The image segmentation is used to divide the image into multiple block images, and the feature of each block image is used as a time step input of the recurrent neural network model.

7. An end-to-end SLAM method based on a deep recurrent convolutional neural network according to claim 6, characterized in that: The method of dynamic feature elimination based on the attention mechanism is as follows: Create an LSTM layer to extract the edges and corners of image objects as landmarks, and then add an attention layer, which dynamically calculates the attention weights of each feature based on the current input and hidden state; According to the historical pixel semantic category, confidence, reprojection depth error, and photometric error corresponding to the landmark points, the reliability score of the landmark points is continuously updated, and the landmark points with reliability lower than the set third threshold are eliminated; an initial dynamic probability is given to each new landmark point. If the feature point corresponding to the landmark point falls within the contour of the prior dynamic object, its dynamic probability increases according to the weight, and the dynamic probability is propagated between frames, and the feature points with a final dynamic probability higher than the set fourth threshold are eliminated.

8. An end-to-end SLAM method based on a deep recurrent convolutional neural network according to claim 6, characterized in that: The feature matching method under the semantic constraint is as follows: The YOLOX model is used to focus on extracting credible semantic areas in the feature extraction stage, and a multi-association graph is constructed to model the spatial structure and scale hierarchy of each credible area. The matching between image object areas is achieved by minimizing an energy function.

Images