A SLAM system based on vehicle-mounted multi-view cameras and deep neural networks

Abstract

This invention discloses a SLAM system based on vehicle-mounted multi-camera and deep neural networks. Multiple cameras are mounted on the vehicle and calibrated to obtain camera intrinsic parameters and transformation matrices between the cameras. A series of video or image sequences are captured as network input. The image sequences are input to a front-end computational visual odometry system, which consists of a feature encoder, a visual similarity calculation module, and an update iteration module. Keyframe sequences are input to a back-end optimization system, which performs global basis comparison (BA) on the entire keyframe sequence. This invention, based on deep learning algorithms, endows the SLAM system with stronger stability and generalization capabilities, enabling stable tracking even in a series of challenging scenarios. The multiple cameras provide the system with lower costs, higher accuracy, and robustness, avoiding system crashes caused by the occlusion of individual sensors. This achieves a more stable and robust SLAM system operation at a lower cost.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a SLAM system based on an in-vehicle multi-view camera and a deep neural network. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) is a technical term, also known as CML or Concurrent Localization and Mapping. The Simultaneous Localization and Mapping problem can be described as follows: A robot starts moving from an unknown location in an unknown environment, performs self-localization based on its location and map during the movement, and simultaneously builds an incremental map based on its self-localization to achieve autonomous localization and navigation of the robot. The SLAM problem can be handled by different sensors, such as vision, LiDAR, IMU, and GNSS sensors.

[0003] Although the technology has made significant progress in recent years, current SLAM algorithms are mainly designed for monocular and binocular cameras. These methods have a small field of view and are easily affected by external interference. When the sensor is blocked, the system will crash and fail in many cases, such as feature point tracking loss, optimization problem divergence and non-convergence, and offset accumulation. This results in low system stability and robustness. Moreover, sensors such as LiDAR and GNSS are expensive, and multi-sensor systems are unaffordable for users. Summary of the Invention

[0004] This invention provides a SLAM system based on a vehicle-mounted multi-view camera and a deep neural network. It can effectively solve the problems mentioned in the background art, which state that current SLAM algorithms are mainly designed for monocular and binocular cameras. These methods have a small field of view, are easily affected by external interference, and will crash when the sensor is blocked. Moreover, they will fail in many cases, such as feature point tracking loss, optimization problem divergence and non-convergence, and offset accumulation, resulting in low system stability and robustness. Furthermore, the high cost of sensors such as LiDAR and GNSS makes multi-sensor systems unaffordable for users.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a SLAM system based on vehicle-mounted multi-view cameras and deep neural networks, including a vehicle-mounted camera frame, which consists of four cameras at the front and rear. The front of the vehicle has one monocular RGB camera responsible for the front field of view of the vehicle, and the rear of the vehicle has one monocular RGB camera responsible for the rear field of view of the vehicle and one binocular RGBD camera with a field of view angled downwards.

[0006] The system includes the following implementation steps:

[0007] S1. Install multiple cameras on the vehicle and calibrate them to obtain the camera intrinsic parameters and the transformation matrix between the multiple cameras;

[0008] S2. Capture a sequence of videos or images as network input;

[0009] S3. Input the image sequence into the front-end computational visual odometry. The front-end consists of a feature encoder, a visual similarity calculation module, and an update iteration module.

[0010] S4. Input the keyframe sequence into the backend for optimization. The backend will perform global BA on the entire keyframe sequence.

[0011] S5. Perform loop closure detection on the keyframe sequence to further optimize the camera pose.

[0012] According to the above technical solution, in S1, when obtaining the camera intrinsic parameters and the transformation matrix between multiple cameras, a high-precision marker scene is first prepared, the vehicle is driven into the center of the scene, and a set of pre-set ChArUco calibration plates at different angles are placed around the scene.

[0013] Specifically, the individual camera is first calibrated. The corner points of the calibration board images acquired by the camera at different angles are detected to obtain the pixel coordinates of the corner points. The physical coordinate values ​​of the corner points are calculated based on the size of the calibration board and the world coordinate origin. The intrinsic parameter matrix and distortion coefficient are then obtained. Next, the extrinsic parameters of the camera are calibrated to obtain the relative attitude between the cameras. The extrinsic parameter calibration of the camera can also be obtained by using the coordinate data of multiple points.

[0014] According to the above technical solution, S2 mainly refers to capturing a series of video or image sequences of road conditions using a camera, and using these as network input.

[0015] According to the above technical solution, in S3, the video or image sequence captured in S2 is mainly input, and the front-end computational vision odometry directly operates on the input video or image sequence to construct a frame-graph to store the co-view relationship between consecutive frames, and iteratively updates and optimizes a key frame sequence and its corresponding pose and depth multiple times.

[0016] First, calculate the image features and corresponding matching feature points between consecutive frames to solve the relative pose and the three-dimensional positional relationship of feature points between two frames. Use a linear motion model to initialize the pose and fix the poses of the first two frames of the image sequence as the initial relative pose. Then calculate the position of the feature of one frame projected onto the other frame image to establish the reprojection error.

[0017] In addition, the photometric error of each pixel caused by camera motion in the two consecutive frames needs to be calculated and complemented by the feature point method. After being fully connected, they are input into the update module together, so that the network can use the smoothness of the motion field to obtain higher robustness. After the front end tracks a new key frame, the distance between them is calculated by calculating the average optical flow of the consecutive key frames, and redundant key frames in the middle are deleted, so as to perform local BA operation.

[0018] According to the above technical solution, in S3, specifically, the feature encoder consists of a feature extraction network and a context network, and is executed only once in the network. The feature extraction network consists of 6 residual blocks and 3 downsampling layers. The 6 residual blocks are specifically: the resolutions are 2 x 1 / 2 original resolution, 2 x 1 / 4 original resolution, and 2 x 1 / 8 original resolution, respectively.

[0019] When a new image arrives, features are first extracted using a feature extraction network to generate an image with a resolution of 1 / 8 of the input image. H ×W×3 →I H / ×W / 8×D The feature map with D set to 256 has the same structure as the context network and the feature extraction network. The difference is that the feature extraction network is responsible for extracting the features of each pixel in the two frames before and after, while the context network only extracts the features of the previous frame.

[0020] According to the above technical solution, in step S3, the visual similarity calculation module constructs a frame-graph to store the co-visibility relationship between consecutive frames. The cost volume of consecutive frames obtained from the previous network is used as the fully relevant volume. To reduce the size of the relevant information tensor, the last two dimensions of the relevant volume are average-pooled with convolutional kernels of sizes 1, 2, 4, and 8, respectively, to construct a 4-layer relevant pyramid, generating relevant information tensors C with resolutions of 1, 1 / 2, 1 / 4, and 1 / 8 of the original resolution. 1 C 2 C 3 C 4 ;

[0021] To facilitate matching, a grid with radius r is used to index the relevant volume at each level using bilinear interpolation. This index is then passed through a fully connected layer and used as the final feature vector. The update and iteration module consists of a 3×3 ConvGRU with a hidden vector h, which can be implemented according to the following formula:

[0022]

[0023] The relevant information tensor c ij After optical flow features are convolved through two layers and then fully connected with context features, they are fed into a ConvGRU. The hidden vector h is then processed. ijAverage pooling is used to extract global context information, and the hidden vector is adjusted in each iteration. Update to obtain according to The optical flow field error correction r can be obtained through two layers of convolution mapping. ij and corresponding confidence level w ij The corrected mapped pixel grid is then represented as:

[0024]

[0025] Next, in the DBA layer, the corrected correlation optical flow field changes are mapped to the camera pose Δp. () and the inverse Δd of depth () The mapping of camera pose and depth can be achieved based on SE(3) using the following cost function:

[0026]

[0027] Next, the Gauss-Newton algorithm is used to solve for the update (Δp, Δd), and Schur is used to decompose the matrix to obtain Δp and Δd respectively, where v and w represent the gradient directions of the camera pose and depth, respectively:

[0028]

[0029] According to the above technical solution, in step S4, the key frame sequence is input to the back-end optimization. The back-end thread constrains the relative poses of multiple cameras and performs global BA optimization on the entire history of key frames through the update iteration module to solve the pose drift and cumulative error caused by camera pose estimation.

[0030] According to the above technical solution, in S4, the prediction of camera pose and depth has the following constraints: optical flow field error constraint, relative pose constraint between the four cameras, and depth constraint of the binocular camera with the field of view facing downward.

[0031] First, the relative poses between the multiple cameras obtained in step S1 and the depth variables obtained by the binocular RGBD camera need to be fixed in the DBA layer of the above update and iteration module, and constraint terms need to be added to the camera pose and depth mapping cost function.

[0032] The true depth variable is used to calculate the squared distance between the measured and predicted values, which is used as a penalty term to constrain the predicted depth. Secondly, during each iteration, the frame-graph is reconstructed based on the optical flow distance between all keyframes and represented as an N*N distance matrix.

[0033] First, add co-view relationships to keyframes that are temporally adjacent. Then, in the order of increasing optical flow, set the constraint distance to 2 and adopt the new co-view relationships from the distance matrix. Then, perform an update iteration module on the entire frame-graph to obtain the globally optimized keyframe camera pose. To recover the pose of non-keyframes, iteratively estimate the optical flow between each keyframe and its adjacent non-keyframes.

[0034] According to the above technical solution, in S5, the camera pose is further optimized by performing closed-loop detection on the key frame sequence. Specifically, the closed-loop detection tracks historical closed-loop frames, establishes the connection between open-loop frames and closed-loop frames in historical time, reduces the drift of closed-loop frames, and makes the accumulated error and uncertainty start accumulating again from the closed-loop frame.

[0035] According to the above technical solution, in S5, loop closure detection is an important module for reducing open-loop errors in the SLAM backend. It uses the common-view features and optical flow field stored in the frame-graph to perform feature matching, find key frames with high similarity but not adjacent in time, and determine whether there is a closed loop path. Then, it combines the depth of the key frame and multi-view information to verify the generated closed loop, eliminate possible erroneous closed loops, and finally integrate the detected closed loops into the existing SLAM system.

[0036] Specifically, this means sending the updated camera pose to the backend optimization module, starting from scratch to accumulate errors, and further improving the overall pose estimation results.

[0037] Compared with the prior art, the beneficial effects of the present invention are: the present invention has a scientific and reasonable structure and is safe and convenient to use.

[0038] 1. Deep learning-based algorithms give SLAM systems stronger stability and generalization capabilities, enabling them to track stably even in a range of challenging scenarios. Multiple cameras provide the system with lower costs, higher accuracy, and robustness, avoiding system crashes caused by the occlusion of individual sensors. This allows SLAM systems to operate more stably and robustly at a lower cost.

[0039] 2. This invention initializes the pose using a linear motion model and fixes the poses of the first two frames of the image sequence as the initial relative pose. Then, it can calculate the position of the feature projection of one frame onto the image of another frame, thereby establishing the reprojection error. It also calculates the photometric error of each pixel caused by camera motion in the two frames and complements it with the feature point method. After full connection, they are input into the update module together, thereby allowing the network to use the smoothness of the motion field to obtain higher robustness.

[0040] 3. This invention uses ChArUco calibration plates at different angles pre-set around the scene to allow each camera to capture the complete calibration plates at different angles, thereby facilitating camera calibration and improving the accuracy of subsequent video or image capture.

[0041] By constraining the optical flow field error, the relative pose of the four cameras, and the depth of the binocular camera with its field of view pointing downwards, the SLAM system becomes more robust through the prediction of camera pose and depth. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a detailed structural diagram of the vehicle-mounted camera frame of the present invention;

[0044] Figure 2 This is a flowchart of the system of the present invention;

[0045] Figure 3 This is a schematic diagram of the feature encoder network structure of the present invention;

[0046] Figure 4 This is a schematic diagram of the BA optimization network of the present invention;

[0047] Figure 5 This is a schematic diagram of the structure of the update and iteration module network of the present invention;

[0048] Figure 6 This is a schematic diagram of the camera pose stitched point cloud obtained by the present invention;

[0049] Figure 7 This is a flowchart illustrating the steps of implementing the present invention. Detailed Implementation

[0050] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0051] Example: Figure 1-7 As shown, the present invention provides a technical solution, a SLAM system based on vehicle-mounted multi-view cameras and deep neural networks, including a vehicle-mounted camera frame, which consists of four cameras at the front and rear. The front of the vehicle has a monocular RGB camera responsible for the front field of view of the vehicle, and the rear of the vehicle has a monocular RGB camera responsible for the rear field of view of the vehicle and a binocular RGBD camera with a field of view that is angled downwards.

[0052] The system includes the following implementation steps:

[0053] S1. Install multiple cameras on the vehicle and calibrate them to obtain the camera intrinsic parameters and the transformation matrix between the multiple cameras;

[0054] S2. Capture a sequence of videos or images as network input;

[0055] S3. Input the image sequence into the front-end computational visual odometry. The front-end consists of a feature encoder, a visual similarity calculation module, and an update iteration module.

[0056] S4. Input the keyframe sequence into the backend for optimization. The backend will perform global BA on the entire keyframe sequence.

[0057] S5. Perform loop closure detection on the keyframe sequence to further optimize the camera pose.

[0058] Based on the above technical solution, in S1, when obtaining the camera intrinsic parameters and the transformation matrix between multiple cameras, a high-precision marker scene is first prepared. The vehicle is driven into the center of the scene. Around the scene are a set of pre-set ChArUco calibration boards at different angles. This is mainly to ensure that each camera can capture the complete calibration boards at different angles so as to facilitate camera calibration.

[0059] Specifically, the individual camera is first calibrated. The corner points of the calibration board images acquired by the camera at different angles are detected to obtain the pixel coordinates of the corner points. The physical coordinate values ​​of the corner points are calculated based on the size of the calibration board and the world coordinate origin. The intrinsic parameter matrix and distortion coefficient are then obtained. Next, the extrinsic parameters of the camera are calibrated to obtain the relative attitude between the cameras. The extrinsic parameter calibration of the camera can also be obtained by using the coordinate data of multiple points.

[0060] Based on the above technical solution, S2 mainly refers to capturing a series of video or image sequences of road conditions using a camera, and using these as network input.

[0061] Based on the above technical solution, in S3, the video or image sequence captured in S2 is mainly input. The front-end computational vision odometry directly operates on the input video or image sequence to construct a frame-graph to store the co-view relationship between consecutive frames, and iteratively updates and optimizes a key frame sequence and its corresponding pose and depth multiple times.

[0062] First, calculate the image features and corresponding matching feature points between consecutive frames to solve the relative pose and the three-dimensional positional relationship of feature points between two frames. Use a linear motion model to initialize the pose and fix the poses of the first two frames of the image sequence as the initial relative pose. Then calculate the position of the feature of one frame projected onto the other frame image to establish the reprojection error.

[0063] In addition, the photometric error of each pixel caused by camera movement in the two consecutive frames needs to be calculated and complemented by the feature point method. After being fully connected, they are input into the update module together, so that the network can use the smoothness of the motion field to obtain higher robustness. After the front end tracks the new key frame, the distance between them is calculated by calculating the average optical flow of the consecutive key frames, and redundant key frames in the middle are deleted, so as to perform local BA operation, BA is the bundle adjustment.

[0064] Based on the above technical solution, in S3, specifically, the feature encoder consists of a feature extraction network and a context network, and is executed only once in the network. The feature extraction network consists of 6 residual blocks and 3 downsampling layers. The 6 residual blocks are specifically: the resolutions are 2 x 1 / 2 original resolution, 2 x 1 / 4 original resolution, and 2 x 1 / 8 original resolution, respectively.

[0065] When a new image arrives, features are first extracted using a feature extraction network to generate an image with a resolution of 1 / 8 of the input image. H ×W×3 →I H / ×W / 8×D The feature map with D set to 256 has the same structure as the context network and the feature extraction network. The difference is that the feature extraction network is responsible for extracting the features of each pixel in the two frames before and after, while the context network only extracts the features of the previous frame.

[0066] Based on the above technical solution, in S3, the visual similarity calculation module constructs a frame-graph to store the co-visibility relationship between consecutive frames. The cost volume of consecutive frames obtained from the previous network is used as the fully relevant volume. To reduce the size of the relevant information tensor, the last two dimensions of the relevant volume are average-pooled with convolutional kernels of sizes 1, 2, 4, and 8, respectively, to construct a 4-layer relevant pyramid, generating relevant information tensors C with resolutions of 1, 1 / 2, 1 / 4, and 1 / 8 of the original resolution. 1 C 2 C 3 C 4 ;

[0067] To facilitate matching, a grid with radius r is used to index the relevant volume at each level using bilinear interpolation. This index is then passed through a fully connected layer and used as the final feature vector. The update and iteration module consists of a 3×3 ConvGRU with a hidden vector h, as shown in the diagram. Figure 4 As shown, the specific network structure is as follows: Figure 5 As shown, c ij Image I obtained from the previous network i with I j The relevant information tensor between the two images represents the degree of visual consistency between them, h. ij It is the hidden vector, gij Image I i Each pixel coordinate grid g in i Based on the currently estimated pose and depth in image I j Pixel mapping on the surface can be achieved using the following formula:

[0068]

[0069] To improve the robustness of the model, the relevant information tensor c is... ij After optical flow features are convolved through two layers and then fully connected with context features, they are fed into a ConvGRU. The hidden vector h is then processed. ij Average pooling is used to extract global context information, and the hidden vector is adjusted in each iteration. Update to obtain according to The optical flow field error correction r can be obtained through two layers of convolution mapping. ij and corresponding confidence level w ij The corrected mapped pixel grid is then represented as:

[0070]

[0071] Next, in the DBA layer, DBA is dense beam adjustment, which maps the corrected correlation optical flow field changes to the camera pose Δp. () and the inverse Δd of depth () For convenience, all subsequent depths are inverses of depth. The mapping between camera pose and depth can be based on SE(3), which is a special Euclidean group and is implemented by the following cost function:

[0072]

[0073] Next, the Gauss-Newton algorithm is used to solve for the update (Δp, Δd), and Schur is used to decompose the matrix to obtain Δp and Δd respectively, where v and w represent the gradient directions of the camera pose and depth, respectively:

[0074]

[0075] Based on the above technical solution, in S4, the key frame sequence is input to the back-end optimization. The back-end thread constrains the relative poses of multiple cameras and performs global BA optimization on the entire history of key frames through the update iteration module to solve the pose drift and cumulative error caused by camera pose estimation.

[0076] Based on the above technical solution, in S4, the prediction of camera pose and depth has the following constraints: optical flow field error constraint, relative pose constraint between the four cameras, and depth constraint of the binocular camera with the field of view facing downward. These constraints make the SLAM system more robust.

[0077] First, the relative poses between the multiple cameras obtained in step S1 and the depth variables obtained by the binocular RGBD camera need to be fixed in the DBA layer of the above update and iteration module, and constraint terms need to be added to the camera pose and depth mapping cost function.

[0078] The true depth variable is used to calculate the squared distance between the measured and predicted values, which is used as a penalty term to constrain the predicted depth. Secondly, during each iteration, the frame-graph is reconstructed based on the optical flow distance between all keyframes and represented as an N*N distance matrix.

[0079] First, add co-view relationships to keyframes that are temporally adjacent. Then, in the order of increasing optical flow, set the constraint distance to 2 and adopt the new co-view relationships from the distance matrix. Then, perform an update iteration module on the entire frame-graph to obtain the globally optimized keyframe camera pose. To recover the pose of non-keyframes, iteratively estimate the optical flow between each keyframe and its adjacent non-keyframes.

[0080] Based on the above technical solution, in S5, closed-loop detection is performed on the key frame sequence to further optimize the camera pose. Specifically, closed-loop detection tracks historical closed-loop frames, establishes the connection between open-loop frames and closed-loop frames in historical time, reduces the drift of closed-loop frames, and makes the accumulated error and uncertainty start accumulating again from the closed-loop frame.

[0081] Based on the above technical solution, in S5, loop closure detection is an important module for reducing open-loop errors in the SLAM backend. It uses the common-view features and optical flow field stored in the frame-graph to perform feature matching, find key frames with high similarity but not adjacent in time, and determine whether there is a closed loop path. Then, it combines the depth of the key frames and multi-view information to verify the generated closed loop, eliminate possible erroneous closed loops, and finally integrate the detected closed loops into the existing SLAM system.

[0082] Specifically, this means sending the updated camera pose to the backend optimization module, starting from scratch to accumulate errors, and further improving the overall pose estimation results.

[0083] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A SLAM system based on vehicle-mounted multi-view cameras and deep neural networks, characterized in that: The vehicle includes a camera frame, which consists of four cameras: a monocular RGB camera at the front of the vehicle, responsible for the front field of view, and a monocular RGB camera at the rear of the vehicle responsible for the rear field of view, as well as a binocular RGBD camera with a field of view angled downwards. The system includes the following implementation steps: S1. Install multiple cameras on the vehicle and calibrate them to obtain the camera intrinsic parameters and the transformation matrix between the multiple cameras; S2. Capture a sequence of videos or images as network input; S3. Input the image sequence into the front-end computational visual odometry. The front-end consists of a feature encoder, a visual similarity calculation module, and an update iteration module. S4. Input the keyframe sequence into the backend for optimization. The backend will perform global BA on the entire keyframe sequence. S5. Perform loop closure detection on keyframe sequences to further optimize camera pose; In S3, the video or image sequence captured in S2 is input, and the front-end computational vision odometry directly operates on the input video or image sequence to construct a frame-graph to store the co-view relationship between consecutive frames, and iteratively updates and optimizes a key frame sequence and its corresponding pose and depth multiple times. First, calculate the image features and corresponding matching feature points between consecutive frames to solve the relative pose and the three-dimensional positional relationship of feature points between two frames. Use a linear motion model to initialize the pose and fix the poses of the first two frames of the image sequence as the initial relative pose. Then calculate the position of the feature of one frame projected onto the other frame image to establish the reprojection error. In addition, the photometric error of each pixel caused by camera movement in the two consecutive frames needs to be calculated and complemented by the feature point method. After being fully connected, they are input into the update module together, so that the network can use the smoothness of the motion field to obtain higher robustness. After tracking a new keyframe at the front end, the distance between the two keyframes is calculated by calculating the average optical flow of the two keyframes, and redundant keyframes in the middle are deleted, so as to perform local BA operation. In S4, the keyframe sequence is input to the backend optimization. The backend thread constrains the relative poses of the multiple cameras and performs global BA optimization on the entire history of the keyframes through the update iteration module to solve the pose drift and cumulative error caused by camera pose estimation. In step S5, loop closure detection is performed on the key frame sequence to further optimize the camera pose. Specifically, the loop closure detection tracks historical loop closure frames, establishes the connection between historical open-loop frames and closed-loop frames, reduces the drift of closed-loop frames, and makes the accumulated error and uncertainty start accumulating again from the closed-loop frame.

2. The SLAM system based on vehicle-mounted multi-view cameras and deep neural networks according to claim 1, characterized in that: In S1, when obtaining camera intrinsic parameters and transformation matrices between multiple cameras, a high-precision marker scene is first prepared. The vehicle is driven into the center of the scene, and a set of pre-set ChArUco calibration boards at different angles are placed around the scene so that each camera can capture the complete calibration boards at different angles for camera calibration. Specifically, the individual camera is first calibrated. The corner points of the calibration board images acquired by the camera at different angles are detected to obtain the pixel coordinates of the corner points. The physical coordinate values ​​of the corner points are calculated based on the size of the calibration board and the world coordinate origin. The intrinsic parameter matrix and distortion coefficient are then obtained. Next, the extrinsic parameters of the camera are calibrated to obtain the relative attitude between the cameras. The extrinsic parameter calibration of the camera can also be obtained by using the coordinate data of multiple points.

3. The SLAM system based on vehicle-mounted multi-view cameras and deep neural networks according to claim 1, characterized in that: S2 refers to capturing a sequence of video or image data of road conditions using a camera and using this as network input.

4. The SLAM system based on a vehicle-mounted multi-view camera and a deep neural network according to claim 1, characterized in that: In S3, specifically, the feature encoder consists of a feature extraction network and a context network, and is executed only once in the network. The feature extraction network consists of 6 residual blocks and 3 downsampling layers. The 6 residual blocks are specifically: two 1 / 2 original resolutions, two 1 / 4 original resolutions and two 1 / 8 original resolutions, respectively. When a new image arrives, features are first extracted using a feature extraction network to generate an image at 1 / 8 the resolution of the input image. The feature map with D set to 256 has the same structure as the context network and the feature extraction network. The difference is that the feature extraction network is responsible for extracting the features of each pixel in the two frames before and after, while the context network only extracts the features of the previous frame.

5. A SLAM system based on a vehicle-mounted multi-view camera and a deep neural network according to claim 4, characterized in that: In step S3, the visual similarity calculation module constructs a frame-graph to store the co-visibility relationship between consecutive frames. The cost volume of consecutive frames obtained from the previous network is used as the fully relevant volume. To reduce the size of the relevant information tensor, the last two dimensions of the relevant volume are average-pooled with convolutional kernels of sizes 1, 2, 4, and 8, respectively, to construct a 4-layer relevant pyramid, generating relevant information tensors with resolutions of 1, 1 / 2, 1 / 4, and 1 / 8 of the original resolution. ; To facilitate matching, a grid of radius r is used to index the relevant volume at each level using bilinear interpolation. This index is then passed through a fully connected layer and used as the final feature vector. The update iteration module consists of a hidden vector h. The ConvGRU configuration can be implemented according to the following formula: ; Tensor of relevant information After optical flow features are convolved through two layers and then fully connected with context features, they are fed into ConvGRU for processing of the hidden vectors. Average pooling is used to extract global context information, and the hidden vector is adjusted in each iteration. Update to obtain ,according to The optical flow field error correction can be obtained through two layers of convolution mapping. and corresponding confidence level The corrected mapped pixel grid is then represented as: ; Next, in the DBA layer, the corrected correlation optical flow field changes are mapped to the camera pose. and depth inverse The mapping of camera pose and depth can be achieved based on SE(3) using the following cost function: ; Then, the Gauss-Newton algorithm is used to solve the update problem. At the same time, the matrix is ​​decomposed using Schur to obtain the values ​​of each matrix. and v and w represent the gradient directions of the camera pose and depth, respectively: 。 6. A SLAM system based on an in-vehicle multi-view camera and a deep neural network according to claim 1, characterized in that: In S4, the prediction of camera pose and depth is subject to the following constraints: optical flow field error constraint, relative pose constraint between the four cameras, and depth constraint of the binocular camera with the field of view facing downward. First, the relative poses between the multiple cameras obtained in step S1 and the depth variables obtained by the binocular RGBD camera need to be fixed in the DBA layer of the above update and iteration module, and constraint terms need to be added to the camera pose and depth mapping cost function. The true depth variable is used to calculate the squared distance between the measured and predicted values, which is used as a penalty term to constrain the predicted depth. Secondly, during each iteration, the frame-graph is reconstructed based on the optical flow distance between all keyframes and represented as an N*N distance matrix. First, add co-view relationships to keyframes that are temporally adjacent. Then, in the order of increasing optical flow, set the constraint distance to 2 and adopt the new co-view relationships from the distance matrix. Then, perform an update iteration module on the entire frame-graph to obtain the globally optimized keyframe camera pose. To recover the pose of non-keyframes, iteratively estimate the optical flow between each keyframe and its adjacent non-keyframes.

7. A SLAM system based on an in-vehicle multi-view camera and a deep neural network according to claim 1, characterized in that: In S5, loop closure detection is a crucial module in the SLAM backend for reducing open-loop errors. It uses co-view features and optical flow fields stored in the frame-graph for feature matching, searches for keyframes with sufficiently high similarity but not adjacent in time, and determines whether a closed loop path exists. Then, it combines the depth of the keyframes and multi-view information to verify the generated loops, eliminating any erroneous loops. Finally, the detected loops are integrated into the existing SLAM system, specifically by sending the updated camera pose to the backend optimization module to start accumulating errors from scratch.

Images