A simultaneous localization and mapping method based on fast-scnn and deepfillv2
By combining Fast-SCNN and Deepfillv2 networks with multi-view geometry, dynamic regions are accurately removed and lost background information is repaired, solving the problem of low tracking success rate of visual SLAM systems in dynamic environments and achieving system stability and information integrity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2023-07-21
- Publication Date
- 2026-05-15
AI Technical Summary
Existing visual SLAM systems struggle to accurately remove dynamic regions in dynamic environments, resulting in low tracking success rates and an inability to effectively repair lost background information.
We employ a Fast-SCNN semantic segmentation network and a Deepfillv2 background restoration network combined with a multi-view geometry method. We use semantic segmentation to label potential dynamic regions, and use the multi-view geometry method to remove dynamic feature points. Finally, we use the Deepfillv2 network to restore the lost background information.
This improves the tracking success rate of the SLAM system in dynamic environments and ensures the integrity of image information and the stability of the system.
Smart Images
Figure CN117115418B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of autonomous navigation for mobile robots, and in particular, it is a method for simultaneous localization and mapping based on Fast-SCNN semantic segmentation network and Deepfillv2 background restoration network. Background Technology
[0002] SLAM technology plays a crucial role in the field of robotics, enabling robots to autonomously perceive, understand, and navigate their environment, significantly enhancing their autonomy and adaptability. Visual SLAM, which utilizes cameras as sensors, has also seen widespread development, achieving excellent results in static environments.
[0003] However, traditional visual SLAM relies on static environments. When faced with environments containing a large number of dynamic objects, the performance of visual SLAM algorithms will significantly decrease. The application scope of traditional SLAM is limited by real-world dynamic objects.
[0004] With the development of computer vision and deep learning, more and more scholars are trying to add semantic segmentation networks to SLAM systems to eliminate potential dynamic targets by imbuing environmental objects with semantic information. However, eliminating dynamic regions can remove effective background information from the image, leading to decreased system stability or even tracking failure.
[0005] CN11351664A discloses a visual SLAM method based on semantic segmentation of dynamic points. This patent employs a Mask R-CNN segmentation network combined with a multi-view geometric algorithm to remove dynamic feature points from the image, improving the system's accuracy in dynamic environments. However, the aforementioned method only uses the remaining static feature points for subsequent tracking and mapping, which can lead to tracking loss when the number of feature points is small.
[0006] CN115187614A discloses a real-time simultaneous localization and mapping method based on the STDC semantic segmentation network. This patent uses the STDC segmentation network to remove dynamic regions of the image and uses static regions for tracking and mapping. However, this method also fails to repair missing regions, and the problem of lost tracking trajectories still exists.
[0007] CN112435262A discloses a dynamic environment information detection method based on semantic segmentation networks and multi-view geometry, comprising the following steps: camera calibration and image distortion removal; acquisition and input of environmental images; segmentation of the input image using a semantic segmentation network to obtain object masks and achieve preliminary dynamic segmentation; extraction of ORB feature points from the input image and calculation of descriptors; detection and removal of dynamic feature points using a combination of multi-view geometry and semantic information; matching of ORB feature points to obtain robot pose information; judgment and insertion of keyframes; point cloud processing through a local mapping thread to obtain a sparse point cloud map; and optimization of pose and correction of drift errors using loop closure detection. This patent removes dynamic feature points, retaining only static areas for tracking and mapping. In environments with many dynamic areas, there are too few static feature points, leading to tracking loss. This invention, however, can repair lost background areas, thereby increasing the information available for system tracking and improving the tracking success rate. Summary of the Invention
[0008] This invention aims to solve the problems of the prior art. It proposes a simultaneous localization and mapping method based on Fast-SCNN and Deepfillv2. The technical solution of this invention is as follows:
[0009] A simultaneous localization and mapping method based on Fast-SCNN and Deepfillv2 includes the following steps:
[0010] S1. Acquire environmental information using an RGB-D camera;
[0011] S2. Use the ORB feature extraction algorithm to extract feature points from RGB-D image information;
[0012] S3. Using the multi-view geometry method, points exceeding the threshold are identified as dynamic feature points by the difference between feature points between two frames of images.
[0013] S4. Obtain semantic information of the image using the Fast-SCNN dual-branch inverse residual connection network, and use the semantic information to label potential dynamic regions;
[0014] S5. Combine the dynamic feature points determined by the multi-view geometry method with the potential dynamic regions labeled by the Fast-SCNN network to identify and eliminate the true dynamic regions.
[0015] S6. Use the Deepfillv2 background restoration network to restore the background information of the removed parts of the image;
[0016] S7. The completed image is localized and navigated using the ORB-SLAM3 algorithm. ORB-SLAM3 is a real-time SLAM algorithm that includes a tracking thread, a local mapping thread, and a loop closure detection thread.
[0017] Furthermore, step S2 utilizes the ORB feature extraction algorithm to extract feature points from the RGB-D image information, specifically including:
[0018] Key points are obtained using the FAST algorithm;
[0019] Then, the gray-scale centroid of the image patch is used to add rotation descriptions to the feature points;
[0020] Finally, the BRIEF descriptor is used to describe the feature points.
[0021] Furthermore, step S3 utilizes a multi-view geometry method to determine points exceeding a threshold as dynamic feature points based on the difference between feature points in two frames of images. Specifically, this includes:
[0022] The spatial coordinates of point P in historical frame L1 are:
[0023] P = [X, Y, Z] T
[0024] The camera motion between historical frame L1 and current frame L2 is represented as follows:
[0025] s1X1=KP
[0026] s²X²=K(RP+t)
[0027] In the formula: K represents the camera intrinsic parameters, s1 and s2 represent the depth values of P, and R and t are the rotation and translation motion of the camera between the historical frame L1 and the current frame L2.
[0028] Represented in homogeneous coordinates:
[0029] p1 = KP
[0030] p2 = K(RP + t)
[0031] Pick:
[0032] x1 = K -1 p2
[0033] x2 = K -1 p2
[0034] In the formula: p1 and p2 represent the projected coordinates of spatial point P on L1 and L2, respectively, and x1 and x2 represent the coordinates of p1 and p2 in the normalized coordinate system, respectively. Substituting them, we get:
[0035] x2=Rx1+t
[0036] The transpose of x2 is obtained by multiplying both sides. have to:
[0037]
[0038] Taking the inner product with x2, we get:
[0039]
[0040] Substitute p1 and p2:
[0041]
[0042] Then the essential matrix E and the fundamental matrix F can be expressed as:
[0043] E=t^R
[0044] F = K -T EK -1
[0045]
[0046] The coordinates x1 and x2 of p1 and p2 in the normalized plane can be expressed as:
[0047] x1 = [u1, v1, 1] T
[0048] x2 = [u2, v2, 1] T
[0049] In the formula: u and v are pixel coordinates, then the epipolar line l1 is represented as
[0050]
[0051] The distance D from p2 to the polar line l2 is expressed as:
[0052]
[0053] Set a distance threshold g. If the distance D from p2 to the epipolar line l2 is greater than g, the projection point is determined to be a dynamic feature point. If the distance D is less than g, the projection point is determined to be a static feature point.
[0054] Furthermore, the Fast-SCNN semantic segmentation network in step S4 is a high-efficiency real-time semantic segmentation network, whose structure includes a downsampling module and a feature extraction module. Specifically, Fast-SCNN uses depthwise separable convolutions to encode the image in the downsampling module, and utilizes an inverse residual network in the feature extraction module to guide the low-level learning of spatial detail features.
[0055] Furthermore, the downsampling module specifically utilizes three depthwise separable convolutions to obtain feature maps of different receptive fields. The stride of each of the three convolutional layers is 2, and its computational cost is:
[0056] C_depthwise=(k×k×n×H×W×n)+(1×1×n×H×W×m)
[0057] In the formula: C is the computational cost, k is the kernel size, H is the input feature length, W is the input feature width, n is the number of input feature channels, and m is the number of output channels.
[0058] Furthermore, the feature extraction module employs an inverse residual network to extract image detail features, and uses two identical inverse residual convolutions with a stride of 1 to deepen the network. An inverse residual network is a network that uses inverse residual connections; it extracts detail features and reduces computation by shrinking, expanding, and then shrinking the number of channels in the input feature module through inverse residual connections.
[0059] Furthermore, step S5, which combines multi-view geometry with semantic segmentation network to remove dynamic feature regions, specifically involves: first, marking potential dynamic regions in the semantic segmentation map; then calculating the proportion of dynamic feature points in the dynamic region determined by multi-view geometry to the total number of feature points; setting a threshold, and if the proportion exceeds the threshold, removing the region, otherwise retaining it; setting the threshold to 0.3.
[0060] Furthermore, the Deepfillv2 background inpainting network in step S6 specifically comprises an image inpainter and a discriminator. The image inpainter uses gated convolution for inpainting, extracting input features and texture information. By automatically learning a mask, it controls the selection and combination of features, thereby completing the background inpainting. The discriminator is a conditional generative adversarial network (GAN)-based discriminator structure. The GAN includes generation and discrimination modules, which learn to fit the corresponding generation through mutual game theory, helping the model better judge the difference between the generated image and the real image, thus improving the quality and credibility of the generated image.
[0061] Furthermore, step S7 uses the ORB-SLAM3 algorithm to perform localization and navigation on the completed image, specifically including:
[0062] Tracking thread: Find and match local map features, use bundle adjustment optimization (BA) algorithm to minimize reprojection error, and determine the camera position in each frame;
[0063] Local mapping thread: Optimizes camera pose using local BA algorithm;
[0064] Loop closure detection thread: Eliminates accumulated errors through pose optimization; after pose optimization, starts the global BA algorithm thread to calculate the optimal structure and motion structure of the entire system.
[0065] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the simultaneous localization and mapping method based on Fast-SCNN and Deepfillv2 as described in any one of the claims.
[0066] The advantages and beneficial effects of this invention are as follows:
[0067] 1. This invention addresses the problem that SLAM systems integrating semantic segmentation networks cannot accurately remove dynamic regions in dynamic environments and have low tracking success rates. It proposes a simultaneous localization and mapping method based on Fast-SCNN semantic segmentation and Deepfillv2 background restoration. Firstly, the invention uses Fast-SCNN semantic segmentation to label potential dynamic objects (steps 4 and 5), using three depthwise separable convolutions to obtain feature maps with different receptive fields, and employing an inverse residual network to extract detailed image features. Secondly, since semantic segmentation networks can only identify potential dynamic objects and cannot determine whether an object is in motion, this invention uses a multi-view geometry method combined with semantic segmentation to remove dynamic objects. Multi-view geometry is used to determine dynamic feature points, and then the dynamic feature points in the semantically segmented region are compared with all feature points in that region. If the comparison exceeds a threshold, the region is removed. This method is more accurate than using only semantic segmentation to remove dynamic regions. Finally, the invention uses the Deepfillv2 background restoration network to restore the image of the removed dynamic regions, using a gated convolutional network to fill in the missing background areas. This method can improve the tracking success rate of the system. Attached Figure Description
[0068] Figure 1 This invention provides a preferred embodiment of a method framework for simultaneous localization and mapping based on a Fast-SCNN semantic segmentation network and a Deepfillv2 background restoration network. Detailed Implementation
[0069] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.
[0070] The technical solution of the present invention to solve the above-mentioned technical problems is:
[0071] like Figure 1As shown, a method for simultaneous localization and mapping based on Fast-SCNN semantic segmentation network and Deepfillv2 background inpainting network includes:
[0072] S1 uses an RGB-D camera to acquire environmental information.
[0073] S2, preprocess the input image and extract feature points using the ORB algorithm. Use the FAST algorithm to obtain key points; the steps are: 1. Calculate the gray value I of each pixel in the image. P 2. Set a threshold T (take I) P 30%); 3. Using this pixel as the center, take 16 points within a circle with a radius of 3; 4. If 12 of the 16 points have a grayscale range greater than I... P +T or less than I P If -T is selected, the point is determined to be a key point of FAST.
[0074] Then, the gray-level centroid of the image patch is used to add a rotation description to the feature points, and the matrix of the image patch is defined as follows:
[0075]
[0076]
[0077] In the formula: I(x,y) is the gray value at pixel (x,y) in the image, m x With m y Represents moments in the horizontal direction and moments in the vertical direction;
[0078] The direction of the feature point is defined as:
[0079]
[0080] Finally, the BRIEF descriptor is used to describe the feature point; N pairs of points are randomly selected around feature point P, and their gray values are compared:
[0081]
[0082] In the formula: P(x) and P(y) are the gray values at points x and y, respectively;
[0083] Therefore, the BRIEF descriptor is represented as:
[0084]
[0085] In the formula: n represents the nth descriptor of the feature point.
[0086] S3 uses multi-view geometry to determine dynamic feature points.
[0087] The spatial coordinates of point P in historical frame L1 are:
[0088] P = [X, Y, Z] T
[0089] The camera motion between historical frame L1 and current frame L2 can be represented as:
[0090] s1X1=KP
[0091] s²X²=K(RP+t)
[0092] In the formula: K represents the camera intrinsic parameters, s1 and s2 represent the depth values of P, and R and t are the rotation and translation motion of the camera between the historical frame L1 and the current frame L2.
[0093] Represented in homogeneous coordinates:
[0094] p1 = KP
[0095] p2 = K(RP + t)
[0096] Pick:
[0097] x1 = K -1 p2
[0098] x2 = K -1 p2
[0099] In the formula: x1 and x2 represent the coordinates of p1 and p2 in the normalized coordinate system, respectively. Substituting them, we get:
[0100] x2=Rx1+t
[0101] Multiply both sides simultaneously have to:
[0102]
[0103] Taking the inner product with x2, we get:
[0104]
[0105] Substitute p1 and p2:
[0106]
[0107] Then the essential matrix E and the fundamental matrix F can be expressed as:
[0108] E=t^R
[0109] F = K -T EK -1
[0110]
[0111] The coordinates x1 and x2 of p1 and p2 in the normalized plane can be expressed as:
[0112] x1 = [u1, v1, 1] T
[0113] x2 = [u2, v2, 1] T
[0114] In the formula: u and v are pixel coordinates, then the epipolar line l1 can be expressed as
[0115]
[0116] Therefore, the distance D from p2 to the epipolar line l2 can be expressed as:
[0117]
[0118] A distance threshold g is set. If the distance D from p2 to the epipolar line l2 is greater than g, the projected point is determined to be a dynamic feature point; if the distance D is less than g, the projected point is determined to be a static feature point. This paper sets this threshold to 0.8.
[0119] S4 utilizes the Fast-SCNN semantic segmentation network to acquire semantic information from the image and label potential dynamic regions. The Fast-SCNN semantic segmentation network uses depthwise separable convolutions to encode the image in the downsampling module, and in the feature extraction module, it uses an inverse residual network to guide the low-level learning of spatial detail features. Specifically, the downsampling module uses three depthwise separable convolutions to obtain feature maps with different receptive fields, with each of the three convolutional layers having a stride of 2. The inverse residual network extracts image detail features, and two identical inverse residual convolutions with a stride of 1 are used to deepen the network to acquire more image detail features. The computational cost of the depthwise separable convolution is:
[0120] C_depthwise=(k×k×n×H×W×n)+(1×1×n×H×W×m)
[0121] In the formula: C is the computational cost, k is the kernel size, H is the input feature length, W is the input feature width, n is the number of input feature channels, and m is the number of output channels.
[0122] S5 combines the dynamic feature points determined by the multi-view geometry method with the potential dynamic regions labeled by the Fast-SCNN semantic segmentation network to remove real dynamic regions. First, the potential dynamic regions in the semantic segmentation map are marked; then, the proportion of dynamic feature points determined by the multi-view geometry method in the dynamic region is calculated to the total number of feature points; a threshold is set, and if the proportion exceeds the threshold, the region is removed, otherwise it is retained; the threshold is set to 0.3.
[0123] S6. The Deepfillv2 background restoration network is used to restore the background information of the removed parts of the image. This network uses a gated convolution to extract the input features and texture information. By automatically learning a mask, it controls the selection and combination of features, thereby completing the background restoration of the image.
[0124] S7. The completed image is used for localization and navigation using the ORB-SLAM3 algorithm to obtain a SLAM system trajectory tracking map and a dense environmental map, specifically including:
[0125] Tracking thread: Find and match local map features, use bundle adjustment optimization (BA) algorithm to minimize reprojection error, and determine the camera position in each frame;
[0126] Local mapping thread: Optimizes camera pose using local BA algorithm;
[0127] Loop closure detection thread: Eliminates accumulated errors through pose optimization; after pose optimization, starts the global BA algorithm thread to calculate the optimal structure and motion structure of the entire system.
[0128] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the simultaneous localization and mapping method based on Fast-SCNN and Deepfillv2 as described in any one of the claims.
[0129] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.
[0130] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0131] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0132] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.
Claims
1. A method for simultaneous localization and mapping based on Fast-SCNN and Deepfillv2, characterized in that, Includes the following steps: S1. Acquire environmental information using an RGB-D camera; S2. Use the ORB feature extraction algorithm to extract feature points from RGB-D image information; S3. Using the multi-view geometry method, points exceeding the threshold are identified as dynamic feature points by the difference between feature points between two frames of images. S4. Obtain semantic information of the image using the Fast-SCNN dual-branch inverse residual connection network, and use the semantic information to label potential dynamic regions; S5. Combine the dynamic feature points determined by the multi-view geometry method with the potential dynamic regions labeled by the Fast-SCNN network to identify and eliminate the true dynamic regions. S6. Use the Deepfillv2 background restoration network to restore the background information of the removed parts of the image; S7. The completed image is localized and navigated using the ORB-SLAM3 algorithm. ORB-SLAM3 is a real-time SLAM algorithm that includes a tracking thread, a local mapping thread, and a loop closure detection thread. The Fast-SCNN semantic segmentation network in step S4 is an efficient real-time semantic segmentation network. Its structure includes a downsampling module and a feature extraction module. In the downsampling module, Fast-SCNN uses depthwise separable convolution to encode the image, and in the feature extraction module, it uses an inverse residual network to guide the low-level learning of spatial details. The downsampling module specifically utilizes three depthwise separable convolutions to obtain feature maps of different receptive fields. The stride of each of the three convolutional layers is 2, and its computational complexity is: In the formula: C is the computational cost, k is the kernel size, H is the input feature length, W is the input feature width, n is the number of input feature channels, and m is the number of output channels; The feature extraction module uses an inverse residual network to extract image detail features, and uses two identical inverse residual networks with a stride of 1 to deepen the network depth. The inverse residual network is a network that uses an inverse residual connection method. It extracts detail features and reduces the amount of computation by shrinking, expanding and shrinking the number of channels in the input feature module. Step S5, which combines multi-view geometry with semantic segmentation network to remove dynamic feature regions, specifically involves: first, marking potential dynamic regions in the semantic segmentation map; then calculating the proportion of dynamic feature points in the dynamic region determined by multi-view geometry to the total number of feature points; setting a threshold, and if the proportion exceeds the threshold, removing the region, otherwise retaining it; the threshold is set to 0.
3.
2. The method for simultaneous localization and mapping based on Fast-SCNN and Deepfillv2 according to claim 1, characterized in that, Step S2 uses the ORB feature extraction algorithm to extract feature points from the RGB-D image information, specifically including: Key points are obtained using the FAST algorithm; Then, the gray-scale centroid of the image patch is used to add rotation descriptions to the feature points; Finally, the BRIEF descriptor is used to describe the feature points.
3. The simultaneous localization and mapping method based on Fast-SCNN and Deepfillv2 according to claim 1, characterized in that, Step S3 utilizes a multi-view geometry method to determine points exceeding a threshold as dynamic feature points based on the difference between feature points in two frames of images. Specifically, it includes: spatial point The spatial coordinates of historical frame L1 are: The camera motion between historical frame L1 and current frame L2 is represented as follows: In the formula: K represents the camera intrinsic parameters, s1 and s2 represent the depth values of P, and R and t are the rotation and translation motion of the camera between the historical frame L1 and the current frame L2. Represented in homogeneous coordinates: Pick: In the formula: p1 and p2 represent spatial points respectively. The projected coordinates on L1 and L2, where x1 and x2 represent the coordinates of p1 and p2 in the normalized coordinate system, are obtained by substituting them into the equations: The transpose of x2 is obtained by multiplying both sides. have to: Taking the inner product with x2, we get: Substitute p1 and p2: Then the essential matrix E and the fundamental matrix F can be expressed as: The coordinates x1 and x2 of p1 and p2 in the normalized plane can be expressed as: In the formula: u and v are pixel coordinates, then the epipolar line l1 is represented as Distance from p2 to polar line l2 Represented as: Set a distance threshold g. If the distance D from p2 to epipolar line l2 is greater than g, the projection point is determined to be a dynamic feature point. If the distance D is less than g, the projection point is determined to be a static feature point.
4. The simultaneous localization and mapping method based on Fast-SCNN and Deepfillv2 according to claim 1, characterized in that, The Deepfillv2 background restoration network in step S6 is specifically as follows: The Deepfillv2 network structure includes an image inpainter and a discriminator; the image inpainter uses a gated convolution for restoration, which is used to extract the input features and texture information. By automatically learning a mask, it controls the selection and combination of features, thereby completing the background restoration of the image; the discriminator is a discriminator structure based on a conditional generative adversarial network. The generative adversarial network includes two modules: generation and discrimination. Through mutual game learning, it fits the corresponding generation and helps the model better judge the difference between the generated image and the real image, thereby improving the quality and credibility of the generated image.
5. The simultaneous localization and mapping method based on Fast-SCNN and Deepfillv2 according to claim 1, characterized in that, Step S7 uses the ORB-SLAM3 algorithm to locate and navigate the completed image, specifically including: Tracking thread: Find and match local map features, use bundle adjustment optimization (BA) algorithm to minimize reprojection error, and determine the camera position in each frame; Local mapping thread: Optimizes camera pose using local BA algorithm; Loop closure detection thread: Eliminates accumulated errors through pose optimization; after pose optimization, starts the global BA algorithm thread to calculate the optimal structure and motion structure of the entire system.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the simultaneous localization and mapping method based on Fast-SCNN and Deepfillv2 as described in any one of claims 1 to 5.