Optical flow semantic SLAM (Simultaneous Localization and Mapping) method for any dynamic object

Abstract

The invention provides an optical flow semantic SLAM (Simultaneous Localization and Mapping) method for any dynamic object, which comprises the following steps of: obtaining continuous RGB (Red, Green and Blue) images, and performing optical flow estimation on a current frame image and a next frame image by using an optical flow estimation module based on deep learning every eight frames of images to generate an optical flow graph; a salient region detection module is used for extracting a dynamic region in the optical flow diagram as a mask, two representative points are found out from the dynamic mask to prompt a semantic segmentation module to perform segmentation, any dynamic object is eliminated, and interference of the dynamic object to a system is eliminated; a lightweight CNN architecture is utilized to extract and match feature points from an image without a dynamic area, the RGB image and an original depth map after the feature points are matched and the dynamic area is removed are sent to an ORB-SLAM2 system, and the positioning and mapping precision is improved by means of the high-quality feature points. The invention provides a more efficient SLAM (Simultaneous Localization and Mapping) method, which has higher positioning and mapping precision and has higher robustness in practical engineering.

Description

Technical Field

[0001] The present invention relates to the field of vision-based positioning and mapping of unmanned vehicles, and in particular to an optical flow semantic SLAM method for arbitrary dynamic objects. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) is a technology that performs both positioning and mapping. Accurate and efficient SLAM technology is a key factor in ensuring accurate navigation of robots, unmanned vehicles and other equipment. Among them, ORB-SLAM2 is a relatively classic visual SLAM algorithm. This algorithm has high positioning and mapping accuracy in simple static scenes with little illumination change, but it is not suitable for complex scenes. For example, dynamic blur caused by large fluctuations in viewing angle seriously affects the quality of ORB feature points; in scenes with dynamic objects, tracking failures or feature point mismatches in dynamic areas often occur. However, the algorithm currently improved by deep learning based on ORB-SLAM2 still has the following defects: (1) It can only recognize pre-trained dynamic object categories. If an unknown dynamic object appears in the scene, it cannot effectively recognize the object and remove the dynamic feature points in its area. (2) The efficiency of feature point extraction is not high, which seriously affects the real-time performance of the system. In addition, the rationality of feature point distribution is poor, which reduces the positioning accuracy of the system. Therefore, based on the above two difficulties, it is necessary to design a more efficient SLAM method to enable unmanned vehicles to have higher positioning and mapping accuracy. This enables it to have higher robustness in practical engineering. Summary of the invention

[0003] In order to overcome the difficulties faced by the above-mentioned unmanned vehicle positioning and mapping technology based on deep learning target detection and traditional feature point extraction in complex environments, the present invention provides an optical flow semantic SLAM method for arbitrary dynamic objects. First, the unmanned vehicle obtains continuous RGB images through an RGB-D depth camera. Every eight frames of images, the optical flow estimation module based on deep learning is used to estimate the optical flow of the current and next frames of images to generate an optical flow map. The dynamic area in the optical flow map is extracted as a mask using a salient area detection module, and two representative points are found in the dynamic mask to prompt the semantic segmentation module to perform segmentation, so as to eliminate any dynamic objects and eliminate the interference of dynamic objects on the system. Then, a lightweight CNN architecture is used to extract and match feature points from the image that eliminates the dynamic area, and the feature points and the processed image are sent to the ORB-SLAM2 system to improve the accuracy of positioning and mapping with high-quality feature points.

[0004] In order to achieve the above technical objectives, the present invention provides the following technical solutions:

[0005] An optical flow semantic SLAM method for arbitrary dynamic objects specifically comprises the following steps:

[0006] S1. Obtain continuous RGB images, and use the deep learning-based optical flow estimation module to perform optical flow estimation on the current and next frames of images every eight frames to generate an optical flow map;

[0007] S2. Use the salient region detection module to extract the dynamic region in the optical flow map as a mask, and find two representative points in the dynamic mask;

[0008] S3, prompting the semantic segmentation module to segment the next frame of the image according to the obtained representative points, thereby eliminating any dynamic objects and obtaining a semantically segmented mask;

[0009] S4, further extracting two representative points from the mask obtained by semantic segmentation in step S3, prompting the semantic segmentation module to segment the dynamic object area of ​​the next frame image, until eight frames later, the representative points obtained by the salient area detection module are reused to prompt the semantic segmentation module to perform segmentation;

[0010] S5. Use a lightweight CNN architecture to extract and match feature points from images with dynamic objects removed;

[0011] S6, send the successfully matched feature points, the RGB image after removing dynamic objects, and the original depth map to the ORB-SLAM2 system, and use the BA algorithm to minimize the reprojection error, track and locate the camera pose of each frame;

[0012] S7, when a new key frame is inserted into the local map, the key frames in the local map and the connection relationship between the key frames are updated, and the poses and map points of the key frames in the current local map are optimized by BA, and finally the redundant key frames and their corresponding map points are deleted;

[0013] S8, using the feature point extraction module to train the bag-of-words model, and perform loop detection, calculate the best candidate frame at the loop, and replace the current frame to correct the posture and local map;

[0014] S9. Use graph optimization to optimize all keyframes and map points.

[0015] Furthermore, in step S1, the optical flow estimation module used is a variant of RAFT.

[0016] More specifically, the specific process of optical flow estimation in step S1 includes:

[0017] S11, feature extraction: down-sample the current and next frame images input through the feature encoding network to extract the feature map with 1 / 8 resolution;

[0018] S12, visual similarity calculation: using the two feature maps obtained in step S11 to calculate the correlation volume, and then using the multi-level correlation of the correlation pyramid to capture larger and smaller pixel displacements, and then generating a new feature map by indexing the features in each level of the correlation pyramid;

[0019] Given the current optical flow estimate (f 1 ,f 2 ), for the current original RGB image I1 and the next frame original RGB image I2, map each pixel x = (u, v) of I1 to I2 to obtain the estimate x' of the next frame image pixel; the mapping relationship is: x' = (u + f 1 (u),v+f 2 (v)); define a local search neighborhood S(x′) around x′ r , the formula is:

[0020] S(x′) r = {x′+Δx|Δx∈R 2 ,||Δx||1≤r};

[0021] Among them, R 2 is a set of two-dimensional integer grid points, r is the search radius on the two-dimensional image plane;

[0022] Based on the local search area S(x′) r For each layer of the correlation pyramid, a feature map is found and then these feature maps are spliced ​​together;

[0023] S13, iterative update; the feature maps obtained from each layer of the correlation pyramid in step S12 are concatenated into a total feature map and input into the update operator with the transformer module as the core to estimate the optical flow estimation sequence {f0,f1,...,f n}; Start from an initial starting point f0, which is directly predicted by reusing the existing context encoder and feeding it with stacked input frames; In each iteration, a flow update direction Δf is generated and added to the current estimate, that is, the formula is expressed as: f k+1 =f k +Δf k ;

[0024] The loss function in the iterative process is the mixed Laplace loss, and the network is trained to predict the mixing parameters of the Laplace distribution to maximize the log-likelihood of the ground truth flow; given the image pair {I1, I2} and the true optical flow value μ gt , the training loss is:

[0025] L prob =-logp θ (μ=μgt |I1,I2);

[0026] Among them, p θ () represents the probability density function of Laplace distribution.

[0027] Furthermore, in step S2, the salient region detection model explicitly decomposes the binary image segmentation task on high-resolution data into a positioning module and a reconstruction module; and adopts a bilateral filter to retain edge information and reduce noise.

[0028] More specifically, step S2 includes the following steps:

[0029] S21, image preprocessing: enhancing the contrast of the optical flow map and using a bilateral filter to preserve edge information and reduce noise;

[0030] S22, build a positioning module; use the Transformer encoder to extract features at different stages; the features extracted at different stages are recorded as: Characteristics of the first three stages are passed to the corresponding decoder stages with lateral connections and are stacked and concatenated at the last encoder block to generate fused features After that, the features are fused It is fed into the classification module, and the hollow spatial convolutional pooling pyramid ASPP module is used to perform multi-context fusion. Compression to obtain compression features In order to transfer to the reconstruction module;

[0031] S23, construct reconstruction module; compress features is the input; for the i-th Transformer encoder extracted Deformable convolution with hierarchical receptive fields and adaptive average pooling layers are used to extract features with receptive fields of various scales; these features extracted by different receptive fields are then concatenated as Then through the 1×1 convolution layer and batch normalization layer, the output features after reconstruction module processing are obtained. The output characteristics Then pass a 1×1 convolution layer to obtain the final prediction image;

[0032] S24, performing bilateral reference; divided into internal reference and external reference; the internal reference is cropped adaptively to avoid adjusting the image size; the external reference uses gradient labels to obtain areas with richer gradient information;

[0033] S25. Set the objective function; use BCE, IoU and SSIM loss functions to supervise at the pixel, region and boundary levels respectively. The formula is expressed as:

[0034]

[0035] Among them, L pixel , L region and L boundary are the losses at the pixel, region, and boundary levels respectively. BCE is the cross entropy loss for the binary classification problem; L IoU is the overlap loss between the predicted segmentation result and the true segmentation label; L SSIM is the structural similarity loss; k1, k2 and k3 are the weights of each loss respectively.

[0036] Furthermore, in step S3, the semantic segmentation module is a variant of EfficientViT, which uses EfficientViT to replace the image encoder of the segmentation everything model SAM, while retaining the structure of the lightweight hint encoder and mask decoder, which is denoted as EfficientViT-SAM.

[0037] More specifically, step S3 specifically includes:

[0038] S31, the input image passes through three convolution blocks and two EfficientViT modules in sequence to obtain the features of five stages, and the fused features of the last three stages are obtained by upsampling and addition, and are fed to the neck including several MBConv blocks, and then fed to the SAM head;

[0039] S32, perform training; the training includes two stages: first, use SAM's image encoder to train EfficientViT-SAM's image encoder, and use L2 loss as the loss function; then train the EfficientViT-SAM end-to-end network on the entire SA-1B dataset;

[0040] S33. Extract representative points of the mask: fit the center line of the mask through a polynomial, and adjust the degree of the polynomial to adjust the fitting degree of the curve; take two points at one-sixth and five-sixths on the center line respectively to prompt the semantic segmentation model to segment the next image, that is, use the semantic segmentation result of the previous frame of the image to prompt the semantic segmentation model to segment the same moving object area in the next frame of the image, so as to realize the semantic association between adjacent images.

[0041] Furthermore, step S5 specifically includes:

[0042] S51, separate key point detection into an independent branch, using 1×1 convolution to perform fast processing on an 8×8 tensor block transformed image;

[0043] S52, designing a matching refinement module, wherein the learning of the module predicts pixel-level offsets by considering only the nearest neighbor pairs at 1 / 8 of the original spatial resolution in the original coarse level features;

[0044] S53, dividing the image into a number of appropriate equal parts according to the density of feature points of the previous frame image, the greater the density, the more parts are divided, and the maximum is 8 equal parts;

[0045] S54, select feature points with better quality in each image block and regard them as a cluster. Calculate the average quality of all feature points in the image, and filter out feature points with poorer quality according to the difference between the quality of each cluster and the average quality according to N=km; where m is the quality difference, k is the proportional constant obtained through experiments, and N is the number of feature points to be filtered out;

[0046] S55. Using the calculated descriptor used to describe the pixel information around the feature point, a bag-of-words model containing environmental semantic information is trained to provide a richer environmental semantic vocabulary for loop detection.

[0047] Based on the above technical solution, the present invention has the following beneficial effects:

[0048] The method proposed in the present invention utilizes the optical flow estimation module, salient region detection module, and semantic segmentation module based on deep learning to jointly eliminate any dynamic objects. In addition, the feature extraction and matching module based on the lightweight CNN network is used to replace the original ORB feature point extraction module in the ORB_SLAM2 system. Combined with a reasonable screening strategy, the image with the dynamic region eliminated is extracted and matched to obtain high-quality feature points, and the feature points are sent to the ORB-SLAM2 system to achieve higher-precision positioning.

[0049] The method proposed in the present invention eliminates the influence of any dynamic objects in the environment on the SLAM system. At the same time, it can efficiently extract high-quality feature points when the camera viewing angle fluctuates violently and the lighting changes significantly, thereby enhancing the reliability of the SLAM front end and further improving the robustness of the entire system in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A block diagram of the optical flow semantic SLAM system for arbitrary dynamic objects provided by the present invention;

[0051] Figure 2 A dynamic object culling block diagram provided by the present invention;

[0052] Figure 3 The feature point extraction and matching block diagram provided by the present invention; DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0054] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0055] Objects in complex scenes faced by existing SLAM technology can be divided into three states: fully dynamic, that is, the object is always moving in the camera's field of view, such as a person walking back and forth; semi-dynamic, that is, the object is sometimes moving and sometimes still in the camera's field of view, such as a person walking from time to time or a table and chair that moves due to external forces; fully static, that is, the object is always still in the camera's field of view, such as a table and chair that is not affected by external forces. In this embodiment, the method proposed in the present invention is based on an optical flow estimation module, a salient area detection module, and a semantic segmentation module to detect moving objects captured by the camera. At the same time, Figure 2 As shown. The elimination of dynamic objects does not focus on the object attributes, but only focuses on the current motion state of the object, so as to increase the utilization rate of the feature points in the image by the system, thereby improving the positioning accuracy. In this embodiment, the method proposed by the present invention is applied to an unmanned vehicle, and the specific process is as follows Figure 1 As shown, an optical flow semantic SLAM method for arbitrary dynamic objects proposed by the present invention is given, comprising the following steps:

[0056] S1. The unmanned vehicle obtains continuous RGB images through the RGB-D depth camera. Every eight frames, the optical flow estimation module based on deep learning is used to estimate the optical flow of the current and next frames to generate an optical flow map.

[0057] As a preferred implementation of step S1, in this embodiment, the optical flow estimation module adopted is a variant of RAFT; optical flow estimation is a basic task in low-level vision, which aims to estimate the 2D motion of each pixel between two adjacent frames, obtain the motion field of the corresponding pixel points, and identify the dynamic area by analyzing the difference in pixel optical flow values ​​between the moving area and the static area in the motion field.

[0058] The specific process of optical flow estimation is:

[0059] S11, feature extraction: down-sample the current and next frame images input through the feature encoding network to extract the feature map with 1 / 8 resolution;

[0060] S12, visual similarity calculation: using the two feature maps obtained in step S11 to calculate the correlation volume, and then using the multi-level correlation of the correlation pyramid to capture larger and smaller pixel displacements, and then generating a new feature map by indexing the features in each level of the correlation pyramid;

[0061] Given the current optical flow estimate (f 1 ,f 2 ), for the current original RGB image I1 and the next frame original RGB image I2, map each pixel x = (u, v) of I1 to I2 to obtain the estimate x' of the next frame image pixel; the mapping relationship is: x' = (u + f 1 (u),v+f 2 (v)); This formula reflects the displacement information of each pixel in image I2 relative to the corresponding pixel in image I1 in the u and v directions.

[0062] Then define a local search neighborhood S(x′) around x′ r , the formula is:

[0063] S(x′) r = {x′+Δx|Δx∈R 2 ,||Δx||1≤r};

[0064] Among them, R 2 is a set of two-dimensional integer grid points, r is the search radius on the two-dimensional image plane;

[0065] Based on the local search area S(x′) r For each layer of the correlation pyramid, a feature map is found and then these feature maps are spliced ​​together;

[0066] S13, iterative update; the feature maps obtained from each layer of the correlation pyramid in step S12 are concatenated into a total feature map and input into the update operator with the transformer module as the core to estimate the optical flow estimation sequence {f0,f1,...,f n}; Start from an initial starting point f0, which is directly predicted by reusing the existing context encoder and feeding it with stacked input frames; In each iteration, a flow update direction Δf is generated and added to the current estimate, that is, the formula is expressed as: f k+1 =f k +Δf k ;

[0067] The loss function in the iterative process is the mixed Laplace loss, and the network is trained to predict the mixing parameters of the Laplace distribution to maximize the log-likelihood of the ground truth flow; given the image pair {I1, I2} and the true optical flow value μ gt , the training loss is:

[0068] L prob =-logp θ (μ=μ gt |I1,I2);

[0069] Among them, p θ () represents the probability density function of Laplace distribution.

[0070] S2. Use the salient region detection module to extract the dynamic region in the optical flow map as a mask, and find two representative points in the dynamic mask;

[0071] As a preferred implementation of step S2, in this embodiment, the salient region detection model explicitly decomposes the binary image segmentation task on high-resolution data into a positioning module and a reconstruction module; and uses a bilateral filter to retain edge information and reduce noise; the specific process includes:

[0072] S21, image preprocessing: enhancing the contrast of the optical flow map and using a bilateral filter to preserve edge information and reduce noise;

[0073] S22, build a positioning module; use the Transformer encoder to extract features at different stages; the features extracted at different stages are recorded as: Characteristics of the first three stages are passed to the corresponding decoder stages with lateral connections and are stacked and concatenated at the last encoder block to generate fused features After that, the features are fused It is fed into the classification module, and the hollow spatial convolutional pooling pyramid ASPP module is used to perform multi-context fusion. Compression to obtain compression features In order to transfer to the reconstruction module;

[0074] S23, construct reconstruction module; compress features is the input; compressed features It contains the features extracted at different stages in step S22, so in the reconstruction module, the features extracted by the i-th Transformer encoder can be Deformable convolution with hierarchical receptive fields and adaptive average pooling layers are used to extract features with receptive fields of various scales; these features extracted by different receptive fields are then concatenated as Then through the 1×1 convolution layer and batch normalization layer, the output features after reconstruction module processing are obtained. The output characteristics Then pass a 1×1 convolution layer to obtain the final prediction image;

[0075] S24, performing bilateral reference; divided into internal reference and external reference; the internal reference is cropped adaptively to avoid adjusting the image size; the external reference uses gradient labels to obtain areas with richer gradient information;

[0076] S25. Set the objective function; use BCE, IoU and SSIM loss functions to supervise at the pixel, region and boundary levels respectively. The formula is expressed as:

[0077]

[0078] Among them, L pixel , and L boundary are the losses at the pixel, region, and boundary levels respectively. BCE is the cross entropy loss for the binary classification problem; L IoU is the overlap loss between the predicted segmentation result and the true segmentation label; L SSIM is the structural similarity loss; k1, k2 and k3 are the weights of each loss respectively.

[0079] The salient area detection module detects the dynamic area by detecting the more significant area in the optical flow map generated by the optical flow estimation module, that is, the area with a large color difference from the surrounding background. In the case of fluctuations in the camera viewing angle or large changes in ambient light, the optical flow map obtained by the optical flow estimation module will contain areas with incorrect estimates. These areas are characterized by messy light blocks, which have a serious impact on the identification of dynamic object areas in the optical flow map. When only a part of the object moves, in the optical flow map obtained by optical flow estimation, the transition between the moving area and the static area is not obvious. The present invention further processes the optical flow map to enhance its contrast, and uses a bilateral filter to retain edge information and reduce noise. Finally, the salient area detection module can effectively avoid the problem of false detection or inaccurate detection caused by traditional edge detection algorithms, and segment the significant areas in the optical flow map to obtain dynamic object areas.

[0080] S3, prompting the semantic segmentation module to segment the next frame of the image according to the obtained representative points, thereby eliminating any dynamic objects and obtaining a semantically segmented mask;

[0081] Since the optical flow estimation module and the salient area detection module are relatively time-consuming, they affect the real-time performance of the SLAM system to a certain extent. Therefore, a semantic segmentation module with better real-time performance is adopted, and the dynamic area is segmented through the periodic prompt points of the above two modules. The semantic segmentation module designs three prompting methods for segmentation. The first is the box prompt: the object to be segmented is selected by the box. Under this prompting method, the limitations of semantic segmentation accuracy are very large. Its accuracy is seriously dependent on the quality of the box, that is, it can only be correctly and completely segmented when the box just surrounds the area where the object is located. If the box is too large, it will cause the segmentation of redundant objects, and if the box is too small, it will cause the object to be completely segmented. The second is the point prompt: according to the location of one or more points, the entire area of ​​the selected object is automatically segmented. The third is a mixed prompt of points and boxes: under the premise of box prompts, point prompts are further used for segmentation. This method retains the limitations of box prompts and also loses real-time performance. In summary, the present invention uses point prompts to semantically segment dynamic objects.

[0082] More specifically, in this embodiment, the semantic segmentation module used is a variant of EfficientViT, which uses EfficientViT to replace the image encoder of the segmentation model SAM, while retaining the structure of the lightweight hint encoder and mask decoder, which is recorded as EfficientViT-SAM. The specific process includes:

[0083] S31, the input image passes through three convolution blocks and two EfficientViT modules in sequence to obtain the features of five stages, and the fused features of the last three stages are obtained by upsampling and addition, and are fed to the neck including several MBConv blocks, and then fed to the SAM head;

[0084] S32, perform training; the training includes two stages: first, use SAM's image encoder to train EfficientViT-SAM's image encoder, and use L2 loss as the loss function; then train the EfficientViT-SAM end-to-end network on the entire SA-1B dataset;

[0085] S33. Extract representative points of the mask: fit the center line of the mask through a polynomial, and adjust the degree of the polynomial to adjust the fitting degree of the curve; take two points at one-sixth and five-sixths on the center line respectively to prompt the semantic segmentation model to segment the next image, that is, use the semantic segmentation result of the previous frame of the image to prompt the semantic segmentation model to segment the same moving object area in the next frame of the image, so as to realize the semantic association between adjacent images.

[0086] S4, further extracting two representative points from the mask obtained by semantic segmentation in step S3, prompting the semantic segmentation module to segment the dynamic object area of ​​the next frame image, until eight frames later, the representative points obtained by the salient area detection module are reused to prompt the semantic segmentation module to perform segmentation;

[0087] S5. Use a lightweight CNN architecture to extract and match feature points from images with dynamic objects removed;

[0088] Complex environments can have a serious impact on traditional ORB feature point extraction. For example, large camera angle fluctuations can aggravate the dynamic blur of the image, and obvious lighting changes can reduce the correlation between two frames. In order to improve the number and quality of feature points extracted in complex environments, this embodiment uses a deep learning-based feature point extraction and matching module to extract feature points from images after dynamic object removal. This module uses the Xfeat model as its core to improve the speed of feature point extraction and matching. Figure 3 As shown in the figure, this module reduces the interference of light changes and perspective fluctuations in complex environments on feature point extraction, extracts high-quality and effective feature points, and improves the positioning accuracy of the unmanned vehicle in the SLAM process.

[0089] In this embodiment, the preferred implementation of step S5 is:

[0090] S51, separate key point detection into an independent branch, using 1×1 convolution to perform fast processing on an 8×8 tensor block transformed image;

[0091] S52, designing a matching refinement module, wherein the learning of the module predicts pixel-level offsets by considering only the nearest neighbor pairs at 1 / 8 of the original spatial resolution in the original coarse level features;

[0092] S53, dividing the image into a number of appropriate equal parts according to the density of feature points of the previous frame image, the greater the density, the more parts are divided, and the maximum is 8 equal parts;

[0093] S54, select feature points with better quality in each image block and regard them as a cluster. Calculate the average quality of all feature points in the image, and filter out feature points with poorer quality according to the difference between the quality of each cluster and the average quality according to N=km; where m is the quality difference, k is the proportional constant obtained through experiments, and N is the number of feature points to be filtered out;

[0094] S55. Using the calculated descriptor used to describe the pixel information around the feature point, a bag-of-words model containing environmental semantic information is trained to provide a richer environmental semantic vocabulary for loop detection.

[0095] In the method proposed by the present invention, the above steps S1-S5 are all only processed for RGB images, because the depth map is to provide depth values ​​for the feature points extracted from the processed RGB image, and the unprocessed areas in the depth map will not participate in the pose estimation process, so there is no need to process them at the same time. In addition, the depth map can also be obtained through an RGB-D depth camera.

[0096] S6, send the successfully matched feature points and the RGB image after removing dynamic objects and the original depth image to the ORB-SLAM2 system, and use the BA algorithm to minimize the reprojection error, track and locate the camera pose of each frame;

[0097] S7, when a new key frame is inserted into the local map, the key frames in the local map and the connection relationship between the key frames are updated, and the poses and map points of the key frames in the current local map are optimized by BA, and finally the redundant key frames and their corresponding map points are deleted;

[0098] S8. Use the feature point extraction module to train the bag-of-words model, perform loop detection, and correct the estimated pose and local map;

[0099] S9. Use graph optimization to optimize all keyframes and map points.

[0100] In summary, the optical flow semantic SLAM method for arbitrary dynamic objects proposed in the present invention is a more efficient SLAM method, which can enable unmanned vehicles to have higher positioning and mapping accuracy, and at the same time it can have higher robustness in actual engineering.

[0101] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0102] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. An optical flow semantic SLAM method for arbitrary dynamic objects, characterized in that: The specific steps include: S1. Obtain continuous RGB images, and use the deep learning-based optical flow estimation module to perform optical flow estimation on the current and next frames of images every eight frames to generate an optical flow map; S2. Use the salient region detection module to extract the dynamic region in the optical flow map as a mask, and find two representative points in the dynamic mask; S3, prompting the semantic segmentation module to segment the next frame of the image according to the obtained representative points, thereby eliminating any dynamic objects and obtaining a semantically segmented mask; S4, further extracting two representative points from the mask obtained by semantic segmentation in step S3, prompting the semantic segmentation module to segment the dynamic object area of ​​the next frame image, until eight frames later, the representative points obtained by the salient area detection module are reused to prompt the semantic segmentation module to perform segmentation; S5. Use a lightweight CNN architecture to extract and match feature points from images with dynamic objects removed; S6, send the successfully matched feature points, the RGB image after removing dynamic objects, and the original depth map to the ORB-SLAM2 system, and use the BA algorithm to minimize the reprojection error, track and locate the camera pose of each frame; S7, when a new key frame is inserted into the local map, the key frames in the local map and the connection relationship between the key frames are updated, and the poses and map points of the key frames in the current local map are optimized by BA, and finally the redundant key frames and their corresponding map points are deleted; S8, using the feature point extraction module to train the bag-of-words model, and perform loop detection, calculate the best candidate frame at the loop, and replace the current frame to correct the posture and local map; S9. Use graph optimization to optimize all keyframes and map points.

2. An optical flow semantic SLAM method for arbitrary dynamic objects according to claim 1, characterized in that: The optical flow estimation module used in step S1 is a new variant of RAFT.

3. An optical flow semantic SLAM method for arbitrary dynamic objects according to claim 2, characterized in that: The specific process of optical flow estimation in step S1 includes: S11, feature extraction: down-sample the current and next frame images input through the feature encoding network to extract the feature map with 1 / 8 resolution; S12, visual similarity calculation: using the two feature maps obtained in step S11 to calculate the correlation volume, and then using the multi-level correlation of the correlation pyramid to capture larger and smaller pixel displacements, and then generating a new feature map by indexing the features in each level of the correlation pyramid; Given the current optical flow estimate (f 1 ,f 2 ), for the current original RGB image I1 and the next frame original RGB image I2, map each pixel x = (u, v) of I1 to I2 to obtain the estimate x' of the next frame image pixel; the mapping relationship is: x' = (u + f 1 (u),v+f 2 (v)); define a local search neighborhood S(x′) around x′ r , the formula is: S(x′) r ={x′+Δx∣Δx∈R 2 ,||Δx||1≤r}; Among them, R 2 is a set of two-dimensional integer grid points, r is the search radius on the two-dimensional image plane; Based on the local search area S(x′) r For each layer of the correlation pyramid, a feature map is found and then these feature maps are spliced ​​together; S13, iterative update; the feature maps obtained from each layer of the correlation pyramid in step S12 are concatenated into a total feature map and input into the update operator with the transformer module as the core to estimate the optical flow estimation sequence {f0,f1,...,f n }; Start from an initial starting point f0, which is directly predicted by reusing the existing context encoder and feeding it with stacked input frames; In each iteration, a flow update direction Δf is generated and added to the current estimate, that is, the formula is expressed as: f k+1 =f k +Δf k ; The loss function in the iterative process is the mixed Laplace loss, and the network is trained to predict the mixing parameters of the Laplace distribution to maximize the log-likelihood of the ground truth flow; given the image pair {I1, I2} and the true optical flow value μ gt , the training loss is: L prob =-logp θ (μ=μ gt ∣I1,I2); Among them, p θ () represents the probability density function of Laplace distribution.

4. The optical flow semantic SLAM method for arbitrary dynamic objects according to claim 1, characterized in that: In step S2, the salient region detection model explicitly decomposes the binary image segmentation task on high-resolution data into a positioning module and a reconstruction module; and adopts a bilateral filter to retain edge information and reduce noise.

5. The optical flow semantic SLAM method for arbitrary dynamic objects according to claim 4, characterized in that: Step S2 specifically includes: S21, image preprocessing: enhancing the contrast of the optical flow map and using a bilateral filter to preserve edge information and reduce noise; S22, build a positioning module; use the Transformer encoder to extract features at different stages; the features extracted at different stages are recorded as: Characteristics of the first three stages are passed to the corresponding decoder stages with lateral connections and are stacked and concatenated at the last encoder block to generate fused features After that, the features are fused It is fed into the classification module, and the hollow spatial convolutional pooling pyramid ASPP module is used to perform multi-context fusion. Compression to obtain compression features In order to transfer to the reconstruction module; S23, construct reconstruction module; compress features is the input; for the i-th Transformer encoder extracted Deformable convolution with hierarchical receptive fields and adaptive average pooling layers are used to extract features with receptive fields of various scales; these features extracted by different receptive fields are then concatenated as Then through the 1×1 convolution layer and batch normalization layer, the output features after reconstruction module processing are obtained. The output characteristics Then pass a 1×1 convolution layer to obtain the final prediction image; S24, performing bilateral reference; divided into internal reference and external reference; the internal reference is cropped adaptively to avoid adjusting the image size; the external reference uses gradient labels to obtain areas with richer gradient information; S25. Set the objective function; use BCE, IoU and SSIM loss functions to supervise at the pixel, region and boundary levels respectively. The formula is expressed as: Among them, L pixel , L region and L boundary are the pixel, region, and boundary level losses respectively. BCE is the cross entropy loss for the binary classification problem; L IoU is the overlap loss between the predicted segmentation result and the true segmentation label; L SSIM is the structural similarity loss; k1, k2 and k3 are the weights of each loss respectively.

6. The optical flow semantic SLAM method for arbitrary dynamic objects according to claim 1, characterized in that: In step S3, the semantic segmentation module is a variant of EfficientViT, which uses EfficientViT to replace the image encoder of the segmentation everything model SAM, while retaining the structure of the lightweight hint encoder and mask decoder, which is denoted as EfficientViT-SAM.

7. The optical flow semantic SLAM method for arbitrary dynamic objects according to claim 6, characterized in that: Step S3 specifically includes: S31, the input image passes through three convolution blocks and two EfficientViT modules in sequence to obtain the features of five stages, and the fused features of the last three stages are obtained by upsampling and addition, and are fed to the neck including several MBConv blocks, and then fed to the SAM head; S32, perform training; the training includes two stages: first, use SAM's image encoder to train EfficientViT-SAM's image encoder, and use L2 loss as the loss function; then train the EfficientViT-SAM end-to-end network on the entire SA-1B dataset; S33. Extract representative points of the mask: fit the center line of the mask through a polynomial, and adjust the degree of the polynomial to adjust the fitting degree of the curve; take two points at one-sixth and five-sixths on the center line respectively to prompt the semantic segmentation model to segment the next image, that is, use the semantic segmentation result of the previous frame of the image to prompt the semantic segmentation model to segment the same moving object area in the next frame of the image, so as to realize the semantic association between adjacent images.

8. The optical flow semantic SLAM method for arbitrary dynamic objects according to claim 1, characterized in that: Step S5 specifically includes: S51, separate key point detection into an independent branch, using 1×1 convolution to perform fast processing on an 8×8 tensor block transformed image; S52, designing a matching refinement module, wherein the learning of the module predicts pixel-level offsets by considering only the nearest neighbor pairs at 1 / 8 of the original spatial resolution in the original coarse level features; S53, dividing the image into a number of appropriate equal parts according to the density of feature points of the previous frame image, the greater the density, the more parts are divided, and the maximum is 8 equal parts; S54, select feature points with better quality in each image block and regard them as a cluster. Calculate the average quality of all feature points in the image, and filter out feature points with poorer quality according to the difference between the quality of each cluster and the average quality according to N=km; where m is the quality difference, k is the proportional constant obtained through experiments, and N is the number of feature points to be filtered out; S55. Using the calculated descriptor used to describe the pixel information around the feature point, a bag-of-words model containing environmental semantic information is trained to provide a richer environmental semantic vocabulary for loop detection.

Images