Semantic SLAM method combining instance segmentation and optical flow estimation

Abstract

The invention discloses a semantic SLAM method combining instance segmentation and optical flow estimation, and belongs to the technical field of robot positioning and navigation.The method comprises the steps that a dynamic region detection thread, a semantic map building thread and a point-line feature fusion method are fused into an ORB-SLAM 3 system; the dynamic object detection thread is combined with an improved YOLOv8-seg semantic segmentation network and an optical flow estimation network RAFT, images are collected through an RGBD camera, point feature extraction and line feature extraction are carried out on the obtained images, the input images are segmented through the semantic segmentation network, masks of objects are obtained, area masks are obtained through the optical flow estimation network, and the dynamic object detection thread is obtained. And combining semantic segmentation and optical flow estimation masks to determine a final dynamic region, and finally removing point features and line features in the dynamic region. According to the method, the precision and robustness of the SLAM system in a dynamic environment and a weak texture environment are improved by combining semantic information and optical flow information to remove dynamic objects and fusing point and line features, and the method can be applied to real-time semantic map construction in a dynamic scene.

Description

Technical Field

[0001] The present invention relates to the technical field of robot positioning and navigation, and more specifically, to a semantic SLAM method combining instance segmentation and optical flow estimation. Background Art

[0002] Visual Simultaneous Localization and Mapping (V-SLAM) is a technology that uses visual sensors to simultaneously perform localization and mapping in an unknown environment. This technology captures continuous image data and combines algorithms such as feature extraction, matching, and motion estimation to achieve real-time perception and modeling of the environment. With the development of computer vision and sensor technology, vision-based SLAM (VSLAM) has gradually become the mainstream. VSLAM uses a camera to capture visual data, extracts feature points in the image, estimates the position and orientation of the device, and updates the map simultaneously. Due to its advantages such as low cost, high precision, and no need for pre-arranging the scene, this technology has been widely applied in fields such as augmented reality, robot navigation, driverless vehicles, and unmanned aerial vehicles. Currently, most SLAM systems are based on the assumption of a static environment, but there are moving objects in the real environment, such as moving pedestrians, moving cars, etc. If the number of such dynamic objects in the scene is large, it will seriously affect the positioning accuracy and robustness of the SLAM system. When the robot is in an environment with weak texture, it will face the problem of insufficient point features, which will also lead to a decrease in the positioning accuracy of the SLAM system. Therefore, reducing the interference of moving objects and improving the positioning accuracy and robustness in complex environments are challenges faced by the visual SLAM field and are also current research hotspots.

[0003] With the rapid development of the deep learning field, more and more research has introduced deep learning into the SLAM field to improve the accuracy and robustness of the SLAM system. The DS-SLAM algorithm proposed by Yu et al. uses the SegNet network to segment the dynamic object area and eliminates all the dynamic feature points in this area to improve the accuracy and robustness of the system; DM-SLAM uses semantic segmentation information and optical flow information to eliminate the influence of dynamic objects in the scene. However, the above SLAM algorithms either have the phenomenon of under-segmentation or directly discard the area where the dynamic object is located, resulting in insufficient feature points in the system, thus affecting the robustness of the system. Therefore, integrating deep learning methods into the SLAM system is of great significance for improving the system in a dynamic environment. Summary of the Invention

[0004] The technical problem to be solved by the present invention is how to use the method of combining deep learning and optical flow estimation to reduce the impact of dynamic objects in the environment on the accuracy and robustness of the system, and solve the problems of low pose estimation accuracy and poor robustness of the SLAM system caused by insufficient feature quantity in weak texture environments.

[0005] To solve the above technical problems, the present invention provides the following technical solutions:

[0006] A semantic SLAM method combining instance segmentation and optical flow estimation, comprising the following steps:

[0007] Step 1: Obtain the RGB image and depth image of the image frame through an RGB-D camera;

[0008] Step 2: Send the RGB image obtained in Step 1 into the tracking thread and the dynamic area detection thread respectively;

[0009] Step 3: Send the RGB image obtained in Step 1 into the tracking thread to extract ORB point features and line features;

[0010] Step 4: Perform semantic segmentation on the RGB image obtained in Step 1 through an improved YOLOv8-seg segmentation network, perform dilation operations on the predicted dynamic areas to fill holes and expansion operations to generate a prior dynamic object mask mask 1 And assign values, assign 1 to dynamic objects and 0 to static objects;

[0011] Step 5: Predict the dynamic area of the RGB image obtained in Step 1 through an optical flow estimation network and generate a dynamic area mask mask 2 ;

[0012] Step 6: Combine the prior dynamic object contour mask obtained by the improved semantic segmentation network in Step 3 with the dynamic object mask obtained by the optical flow estimation in Step 4; more accurately detect dynamic objects, and finally remove the point features and line features of the dynamic target area;

[0013] Step 7: Combine the semantic information in Step 4 to construct a dense point cloud map;

[0014] Step 8: Perform subsequent pose estimation and loop detection on the image frame with only static features obtained in Step 6

[0015] Furthermore, in Step 3, the extraction and matching of point features adopt the method of the ORB-SLAM3 algorithm, use ORB features for feature point extraction, including detecting the position of OrientedFAST corner points and calculating BRIEF descriptors. By calculating the Hamming distance between descriptors, the best matching pairs are found. The specific steps include:

[0016] Calculate the IDs of all grid columns and rows within a circle centered at the feature point with a radius of r.

[0017] Traverse all grids within the circular area and check whether the feature points within each grid meet the conditions.

[0018] Find the point with the minimum Hamming distance of the descriptor and perform verification.

[0019] Construct a rotation histogram and filter the matching points.

[0020] Furthermore, in step 3, line features are used to solve the shortage of point features in a weak texture environment to improve the robustness and accuracy of the system. The specific method is as follows:

[0021] A line has four degrees of freedom in space. The Plücker coordinate method and the orthonormal representation method are used to represent line features. The Plücker coordinate method consists of six parameters, which are used for initialization, transformation, and projection respectively. A three-dimensional space line can be represented by the Plücker coordinate method as

[0022] L = (n T , v T ) T

[0023] where v ∈ R 3 represents the direction vector of the straight line, and n ∈ R 3 represents the normal vector of the plane π determined by L and the origin of the camera coordinate system. Although the Plücker coordinate method can simply represent the transformation process, due to its over-parameterization, it is not easy to directly optimize.

[0024] The orthonormal representation method consists of four parameters for optimization. A three-dimensional space line can be represented by the Plücker coordinate method as:

[0025] L = (U, W)

[0026] where U ∈ SO(3) and W ∈ SO(2). SO(n) is a special orthogonal group containing the rotation matrix R, and its definition is as follows: SO(n) = {R ∈ R n×n ∣ RR T = I, det(R) = 1}.

[0027] The equation of the space line from the Plücker coordinate system to the orthogonal coordinate system can be represented as:

[0028]

[0029] where L P represents the line feature represented by the Plücker coordinate method, and L O represents the line feature represented by the orthonormal representation method;

[0030] First, calculate the gradient magnitude and horizontal line angle θ of the pixel (x, y) for the input image:

[0031]

[0032] where I(x, y) is the pixel value, g x (x, y) is the gradient value, θ(x, y) is the horizontal line angle, the gradient value of the pixel is non - negative, and the range of the horizontal line angle is 0° to 360°; then extract the line features through edge detection, and use the least - squares method to fit the generated edge pixel chain to extract line segments. Edge detection is divided into four steps: First, input a grayscale image, and use a Gaussian filter to smooth the image to suppress noise; calculate the gradient intensity and direction of each pixel point in the image according to the Sobel operator. If the calculated gradient is greater than the gradient threshold, it is an edge pixel; in the gradient map, select the pixel point with the largest gradient value as the "anchor point", and these anchor points are usually potential line endpoints; connect these anchor points to form an initial edge pixel chain, and these chains intuitively correspond to the object boundaries in the image; finally, check the extracted line segments, keep the appropriate line features, and eliminate the inappropriate line features; the specific steps are as follows: calculate the NFA value of the extracted line segment. NFA represents the probability that i pixel points are selected from a line segment of length n and their gradients are in the same direction. If NFA(n, k)>1, keep the line segment. If NFA(n, k)≤1, the line segment is considered valid, otherwise it is considered an invalid line segment and is eliminated;

[0033]

[0034] where N 4 represents the number of potential line segments, n represents the length of the line segment, k represents the number of pixel points whose gradient angles are calculated along the line segment and are consistent with the line segment direction, and p represents the probability that the gradients of the pixels are in the same direction; the EDLines algorithm has three main parameters Ng, Na, and Ni, which represent the gradient threshold, anchor point threshold, and scanning interval value respectively. Set Ng to 32 to extract more obvious edge line segments; set Na to 16 to make the detected line segments smoother and reduce the influence of small fluctuations and noise; set Ni to 4 to eliminate short segments, and set the line spacing to 14 pixels to make the line segments evenly distributed to ensure that the line segment distribution is not too dense.

[0036] Furthermore, a focused linear attention mechanism LFT is added in step 4. The specific method is as follows: The focused linear attention mechanism introduces a focusing function:

[0037] Sim(Q i ,K j )=φ p (Q i )φp (K j ) T

[0038]

[0039] x **p denotes taking the p - th power of x bit - by - bit. This focusing function can make the inner product of similar vectors smaller, thereby achieving focusing and enhancing the model expression ability of the segmentation network. In addition, general linear attention cannot obtain a full - rank attention matrix because the rank of the linear attention matrix is limited by the smaller of the dimension d of each head and the number of features N:

[0040] rank(φ(Q)φ(K) T )≤min{rank(φ(Q)),rank(φ(K))}

[0041] ≤min{N,d},

[0042] Generally speaking, d is smaller than N. Therefore, usually the linear attention matrix cannot reach the full rank N, which means there is redundancy between the weights of different rows. Since the output of self - attention is the weighted combination of these weights for the same set of values, the homogenization of weights will inevitably lead to a decrease in the diversity of the model output, thereby affecting the model performance. The focused linear attention mechanism uses lightweight depthwise convolution to restore the rank of the attention matrix, making the linear attention matrix reach full rank.

[0043] O=φ(Q)φ(K) T V+DWC(V)

[0044] DWC is a special attention, where each query only focuses on several spatially adjacent features instead of all values. At this time, even if the outputs corresponding to two queries in linear attention are similar, the model can still obtain different outputs from the local features proposed by DWC, thus ensuring feature diversity.

[0045] Furthermore, the added LFT focused linear attention mechanism in step 4 is added after the c2f module of the head - layer network. After adding the LFT attention mechanism, the number of layers of the head - layer network needs to be modified to ensure the normal operation of the segmentation network.

[0046] Furthermore, in step 5, the optical flow estimation network is used to detect the dynamic object objects missed by the segmentation network and generate a dynamic object mask mask 2 , and the specific steps are as follows:

[0047] For two consecutive input RGB images I1 and I 2 , a dense optical flow displacement field (f 1 , f 2 ) is obtained through the optical flow estimation algorithm RAFT, where I 2 The corresponding position of each pixel is (u′, v′) = (u + f 1 (u), v + f 2 (v)), and the optical flow motion amount of each corresponding pixel is S(f 1 (u), f 2 (v)), and then the magnitude of the optical flow is calculated Then it is normalized within the value range of [0, 1] to obtain Then the global optimal threshold determined by the Otsu method is used to divide the dynamic region and the static region, and then the image is assigned values, the dynamic region is assigned 1, the static region is assigned 0, and a mask mask is generated 2 = OTSU(255×e 1 ).

[0048] Furthermore, in step 6, the prior dynamic object contour mask information obtained by semantic segmentation is combined with the dynamic object mask obtained by the optical flow estimation network; finally, the point features and line features in the dynamic target area are removed, and the specific method is as follows:

[0049] The mask obtained by the instance segmentation network YOLOv8-seg is combined with the mask obtained by the optical flow estimation network RAFT to obtain the final mask mask = mask 1 × mask 2 , if the mask mask at the position where the feature points and feature lines are located is 0, it is a static feature and is retained for subsequent pose estimation and mapping. If the mask at the position where the feature points and feature lines are located is 1, it is a dynamic feature and is removed.

[0050] Furthermore, in step 7, only the image frames with static features are used for subsequent pose estimation and map construction.

[0051] Compared with the existing well-known technologies, the beneficial effects that the present invention can achieve are at least as follows:

[0052] Through the combination of semantic information and optical flow information, the present invention can more accurately remove dynamic objects in the scene, improving the accuracy and robustness of the SLAM system in dynamic environments. The method of fusing point-line features can ensure the robustness of the SLAM system when operating in weak-texture and high-dynamic environments. Additionally, a focused linear attention mechanism is added to the semantic segmentation network YOLOV8-seg, improving the segmentation accuracy of the segmentation network. By adopting the parallel operation mode of point-line features, the present invention can ensure high precision and good real-time performance of the system in weak-texture and high-dynamic environments. Description of the Drawings

[0053] Figure 1 It is the overall flowchart of the present invention;

[0054] Figure 2 It is the framework diagram of the instance segmentation network of the present invention;

[0055] Figure 3 It is the flowchart for removing dynamic regions of the present invention. Detailed Embodiments

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the specification. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0057] Embodiment 1: A semantic SLAM method combining instance segmentation and optical flow estimation provided by an embodiment of the present invention, in conjunction with the attached Figure 1 , this embodiment adopts five-thread parallelism: dynamic region detection, tracking, local mapping, loop detection, and semantic mapping threads. Among them, the tracking thread includes a point feature extraction module, a line feature extraction module, an initialization and relocalization module, and a local tracking module. The dynamic region detection thread includes an improved instance segmentation network module and an optical flow estimation network module. The line feature extraction module extracts line features through the EDLines algorithm, obtains the semantic mask of the object through the instance segmentation network, and then obtains the region mask through the optical flow estimation network. By combining the masks obtained from the segmentation network and the optical flow estimation network, high-quality dynamic region detection is achieved, and point feature information and line feature information are extracted from the image, and the point features and line features in the dynamic region are removed. It includes the following steps:

[0058] In this embodiment, first, multiple consecutive color images and depth images are obtained by an RGBD camera, encapsulated into frames, and respectively input into the front-end tracking thread and the dynamic region detection thread. Both the front-end tracking thread and the dynamic region detection thread use two-way branches for parallel processing.

[0059] Combined with the attached Figure 2 The figure shows the instance segmentation network framework diagram. To ensure real-time performance and segmentation effect, a one-stage segmentation network YOLOV8-seg is adopted, and a focused linear attention mechanism LFT is added to the YOLOV8-seg network framework, effectively improving the accuracy and performance of the segmentation network.

[0060] This instance segmentation model is trained on a subset of COCOtrain2017 and can distinguish 20 classes of objects. The mean IoU (Intersection over Union) of the instance segmentation model is 58.6, and the global pixelwise acc is 92.3, with good real-time performance and segmentation effect.

[0061] The instance segmentation network detects all instances in the image according to prior information and obtains the mask of each object in the image 1 , assign values to the segmented masks, assign 1 to dynamic objects and 0 to static objects, and achieve preliminary dynamic segmentation.

[0062] The consecutive two RGB images I 1 and I 2 are input into the optical flow estimation network RAFT, and a dense optical flow displacement field (f 1 , f 2 ) is obtained through the optical flow estimation algorithm RAFT, where the position corresponding to each pixel of I 2 is (u′, v′) = (u + f 1 (u), v + f 2 (v)), and the optical flow movement amount of each corresponding pixel is S(f 1 (u), f 2 (v)), then the magnitude of the optical flow is calculated Then it is normalized within the value range of [0, 1] to obtain Then the global optimal threshold determined by the maximum inter-class variance method is used to divide the dynamic region and the static region, and then values are assigned to the image region mask, 1 is assigned to the dynamic region and 0 is assigned to the static region, and a mask mask 2 = OTSU(255×e 1 ) is generated.

[0063] Combine the mask obtained by the instance segmentation network YOLOV8-seg and the mask obtained by the optical flow estimation network RAFT to obtain the final mask mask = mask1 ×mask 2 , if the position mask is 0, it is a static area, if the position mask is 1, it is a dynamic area.

[0064] Point feature extraction and line feature extraction are performed on the image after dynamic area is removed. ORB feature points are used for point feature extraction, and EDLines line feature extraction is used for line feature extraction.

[0065] The input grayscale image is smoothed using a Gaussian filter to suppress noise; the gradient strength and direction of each pixel in the image is calculated based on the Sobel operator. If the calculated gradient is greater than the gradient threshold, it is an edge pixel; in the gradient map, the pixel with the largest gradient value is selected as the "anchor point", which is usually the potential endpoint of the straight line; the adjacent anchor points are connected, and the generated initial line segments are fitted and screened using the least squares method to remove unnecessary intermediate points to make the line segments smoother and more accurate.

[0066] The specific steps of EDLines straight line feature matching are:

[0067] Calculate the NFA value of the extracted line segment. NFA represents the probability that the gradients of i pixels are in the same direction when selecting i pixels from a line segment of length n. If NFA(n,k)>1, the line segment is retained. If NFA(n,k)≤1, the line segment is considered valid. Otherwise, it is considered invalid and is discarded.

[0068]

[0069] Where N 4 Represents the number of potential line segments, n represents the length of the line segment, k represents the calculation of the gradient angle of each pixel along the line segment and the number of pixels consistent with the direction of the line segment, and p represents the probability that the gradients of the pixels are in the same direction.

[0070] The EDLines algorithm has three main parameters: Ng, Na, and Ni, which represent the values ​​of gradient threshold, anchor threshold, and scanning interval, respectively. In this paper, Ng is set to 32 to extract more obvious edge segments; Na is set to 16 to make the detected segments smoother and reduce the influence of small fluctuations and noise; Ni is set to 4 to eliminate short segments. The line spacing is set to 14 pixels to make the line segments evenly distributed to ensure that the line segments are not too densely distributed.

[0071] Combine the semantic information in step 4 to build a dense point cloud map;

[0072] The image frames containing only static features are used for initial camera pose estimation and loop closure detection.

[0073] Beneficial effects of this example: Through the combination of semantic information and optical flow information, the present invention improves the accuracy and robustness of the SLAM system in dynamic environments. The method of fusing point-line features can ensure the robustness of the SLAM system when operating in weak-texture and high-dynamic environments. Additionally, a focused linear attention mechanism is added to the semantic segmentation network YOLOV8n-seg, improving the segmentation accuracy of the segmentation network. By adopting the parallel operation mode of point-line features, the present invention can ensure high precision and good real-time performance of the system in weak-texture and high-dynamic environments.

[0074] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, rather than limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. A semantic SLAM method combining instance segmentation and optical flow estimation, characterized in that: The following steps are involved: Step 1: Obtain the RGB image and depth image of the image frame through the RGB-D camera; Step 2: Send the RGB image obtained in step 1 to the tracking thread and the dynamic area detection thread respectively; Step 3: Send the RGB image obtained in step 1 to the tracking thread to extract ORB point features and line features; Step 4: The RGB image obtained in step 1 is semantically segmented through the improved YOLOv8-seg segmentation network, and the predicted dynamic area is dilated to fill the holes and extended to generate a priori dynamic object mask mask1 and assign values, with dynamic objects assigned 1 and static objects assigned 0; Step 5: Use the RGB image obtained in step 1 to predict the dynamic area through the optical flow estimation network and generate the dynamic area mask mask2; Step 6: Combine the priori dynamic object contour mask obtained by the improved semantic segmentation network in step 3 with the dynamic object mask obtained by the optical flow estimation in step 4; detect dynamic objects more accurately, and finally remove the point features and line features of the dynamic target area; Step 7: Combine the semantic information from step 4 to build a dense point cloud map; Step 8: The image frame with only static features obtained in step 6 is used for subsequent pose estimation and loop closure detection.

2. The semantic SLAM method for joint instance segmentation and optical flow estimation according to claim 1, characterized in that: The YOLOv8-seg instance segmentation network introduces a focused linear attention module. Since the self-attention mechanism directly performs global self-attention calculations, which often results in excessive computational complexity, the general linear attention mechanism either has low performance or slow inference speed. The focused linear attention mechanism has strong model expression capabilities and fast inference speed. The focused linear attention mechanism introduces a focusing function: Sim(Q i ,K j )=φ p (Q i )φ p (K j ) T x **p Indicates that x is raised to the power of p. Changing the focusing function can make the inner product of similar vectors smaller, thereby achieving focusing and enhancing the model expression ability of the segmentation network. In addition, general linear attention cannot obtain a full-rank attention matrix because the rank of the linear attention matrix will be limited by the smaller of the dimension d of each head and the number of features N: rank(φ(Q)φ(K) T )≤min{rank(φ(Q)),rank(φ(K))} ≤min{N,d}, Generally speaking, d is smaller than N. Therefore, the linear attention matrix cannot reach the full rank N. This means that there is redundancy between the weights of different rows. Since the output of self-attention is the weighted combination of these weights on the same set of values, the homogenization of weights will inevitably lead to a decrease in the diversity of model outputs, which in turn affects model performance. The focused linear attention mechanism uses a lightweight depthwise convolution to restore the rank of the attention matrix, making the linear attention matrix reach full rank. O=φ(Q)φ(K) T V+DWC(V) DWC is a special type of attention in which each query only focuses on a few spatially adjacent features rather than all values. At this time, even if the outputs corresponding to two queries in linear attention are similar, the model can still obtain different outputs from the local features proposed by DWC, thereby ensuring feature diversity.

3. The semantic SLAM method of combining instance segmentation and optical flow estimation according to claim 1, characterized in that: The optical flow estimation network RAFT uses optical flow information to estimate the motion state of the object and generate a mask. The specific steps are: From the two consecutive RGB images I1 and I2 input in step 1, a dense optical flow displacement field (f 1 ,f 2 ), where the position of each pixel in I2 is (u′,v′)=(u+f 1 (u),v+f 2 (v)), the corresponding optical flow motion of each pixel is S(f 1 (u),f 2 (v)), and then calculate the size of the optical flow Then normalize it in the range of [0,1] to get Then, the global optimal threshold determined by the maximum inter-class variance method is used to divide the dynamic area and the static area, and then the image is assigned a value, the dynamic area is assigned a value of 1, the static area is assigned a value of 0, and a mask mask2 = OTSU (255×e1) is generated.

4. The semantic SLAM method of combining instance segmentation and optical flow estimation according to claim 1, characterized in that: The mask obtained by the instance segmentation network YOLOv8-seg and the mask obtained by the optical flow estimation network RAFT are combined to obtain the final mask mask = mask1×mask2. If the position mask where the feature points and feature lines fall is 0, it is a static feature and is retained for subsequent pose estimation and mapping. If the position mask where the feature points and feature lines fall is 1, it is a dynamic feature and is removed.

5. The semantic SLAM method of combining instance segmentation and optical flow estimation according to claim 1, characterized in that: In order to solve the problem of insufficient number of point features in weak texture and complex dynamic environment, the point-line feature fusion method is adopted. In the front-end tracking thread, the extraction and detection of line features are integrated, including edge detection, line segment extraction and line segment verification. The specific steps are as follows: a line has four degrees of freedom in space, and the line features are represented by Plücker coordinates and standard orthogonal representation. The Plücker coordinates method consists of six parameters, which are used for initialization, transformation and projection respectively. A three-dimensional space line can be represented by the Plücker coordinates method as follows: L=(n T ,v T ) T Where v∈R 3 Represents the direction vector of the line, n∈R 3 Represents the normal vector of the plane π determined by L and the origin of the camera coordinate system. Although the Plücker coordinate method can concisely represent the transformation process, it is not easy to directly optimize due to its over-parameterization. The standard orthogonal representation consists of four parameters for optimization. A three-dimensional space line can be expressed as follows using the Plücker coordinate method: L=(U,W) Where U∈SO(3), W∈SO(2), SO(n) is a special orthogonal group containing the rotation matrix R, which is defined as follows: SO(n)={R∈R n×n ∣RR T =I, det(R) = 1}; the equation of the straight line from the Plücker coordinate system to the orthogonal coordinate system can be expressed as: Where, L P represents the line feature represented by the Plücker coordinate method, L O represents line features expressed in standard orthogonal representation; First, calculate the gradient size and horizontal angle θ of the pixel (x, y) of the input image: Where I(x,y) is the pixel value, g x (x, y) is the gradient value, θ(x, y) is the horizontal angle, the pixel gradient value is non-negative, and the horizontal angle ranges from 0° to 360°; then edge detection is used to extract straight line features, and the generated edge pixel chain is fitted using the least squares method to extract line segments. Edge detection is divided into four steps: first, a grayscale image is input and the image is smoothed using a Gaussian filter to suppress noise; the gradient intensity and direction of each pixel in the image are calculated based on the Sobel operator. If the calculated gradient is greater than the gradient threshold, it is an edge pixel; in the gradient map, the pixel with the largest gradient value is selected as the "anchor point", which is usually a potential straight line endpoint; Connect these anchor points to form the initial edge pixel chain, which intuitively corresponds to the object boundary in the image; finally, the extracted line segments are tested, suitable line features are retained, and inappropriate line features are removed; the specific steps are as follows: calculate the NFA value of the extracted line segment, NFA represents the probability that the gradients of i pixels are in the same direction when the length of the line segment is n, if NFA(n,k)>1, the line segment is retained, if NFA(n,k)≤1, the line segment is considered valid, otherwise it is considered invalid and removed; Where N 4 represents the number of potential line segments, n represents the length of the line segment, k represents the calculation of the gradient angle of each pixel along the line segment and the number of pixels consistent with the direction of the line segment, and p represents the probability that the gradients of the pixels are in the same direction; the EDLines algorithm has three main parameters Ng, Na, and Ni, which represent the values ​​of the gradient threshold, anchor point threshold, and scanning interval, respectively. Ng is set to 32 to extract more obvious edge segments; Na is set to 16 to make the detected segments smoother and reduce the influence of small fluctuations and noise; Set Ni to 4 to eliminate short segments and set the Line Spacing to 14 pixels to evenly distribute the segments to ensure that the line segments are not too densely distributed.

Images