A sequence image segmentation-based slam method

Abstract

The application provides a SLAM method based on sequence image segmentation, and belongs to the technical field of space positioning. The sequence image is divided into multiple sub-sequences, instance segmentation is performed on the first frame image in each sub-sequence, a target detection frame under the image is obtained, the position information of the target detection frame under the image is used to determine the approximate position of the target detection frame in the next frame or any frame image, the image in the corresponding target detection frame in each frame image is obtained, instance segmentation is performed on the image in the target detection frame, and finally, the dynamic feature points are removed by using an optical flow detection method based on the instance segmentation result, so that the subsequent target area is synchronously positioned and mapped. Compared with the segmentation processing of the whole image in the prior art, the size of the segmented image is obviously reduced, the image instance segmentation speed is effectively improved, and the subsequent SLAM mapping work efficiency is improved.

Description

Technical Field

[0001] This invention relates to a SLAM method based on sequence image segmentation, belonging to the field of spatial localization technology. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a research area that has received a lot of attention in recent years. Its main purpose is to solve the problem of simultaneously estimating the camera position and pose and the 3D structure of the scene from a set of features detected from multiple images. By detecting dynamic targets, it can complete the localization and mapping tasks in dynamic environments.

[0003] Currently, many existing general frameworks and algorithms still have some problems: First, many algorithms focus on the geometric relationships of images in different spatiotemporal contexts, while ignoring the semantic information of key targets in the environment, which is detrimental to environmental understanding. Second, many algorithms can only achieve excellent localization and mapping results in static environments, but when an unknown number or type of dynamic targets appear in the experimental environment, the accuracy of localization and mapping will be greatly reduced. To solve these two problems, namely how to reduce the impact of dynamic target feature extraction in SLAM in dynamic environments, segmentation algorithms are integrated into the SLAM process. For example, in the patent CN111797688A entitled "A Visual SLAM Method Based on Optical Flow and Semantic Segmentation", a Mask R-CNN network is used to segment the input multi-frame images, and dynamic features are extracted from the segmentation results. The feature points after removing the dynamic effects are used in the subsequent SLAM system, which can effectively eliminate the influence of dynamic objects on SLAM localization and mapping. However, this patent segments each frame of the image to obtain the static region and the predicted dynamic region of each frame. SLAM mainly removes feature points from the dynamic region and segments each frame of the image, resulting in an excessive amount of segmentation data and low image segmentation speed, which seriously affects the efficiency of SLAM. Summary of the Invention

[0004] The purpose of this invention is to provide a SLAM method based on sequence image segmentation to solve the problem of low SLAM efficiency caused by the large amount of data in sequence image segmentation in the prior art.

[0005] This invention proposes a SLAM method based on sequence image segmentation, which includes the following steps:

[0006] 1) Use a depth camera to acquire a sequence of images of the target area, as well as the camera pose information and ORB feature points of each frame at the corresponding time in the sequence of images.

[0007] 2) Divide the acquired sequence images into multiple subsequences, and perform instance segmentation on the first frame image in each subsequence to obtain the target detection box information in the first frame image of each subsequence; using the target detection box information, depth information, and camera pose information at the corresponding time in the image, calculate the position information and size of the target detection box in the first frame image;

[0008] 3) Based on the position and size of the target detection box in the first frame image and the pose information of the camera in the first and second frames, calculate the approximate position of the target detection box in the second frame image; the position of the target detection box in any other frame image in the subsequence is determined by using the camera pose information of the current frame and the camera pose information, target detection box position and size of any frame in the previous few frames.

[0009] 4) Based on the approximate location of the target detection box in each frame image, extract the image within the target detection box in each frame image as the target image in each frame image, and superimpose the target image into a blank image of a set size to obtain a target image of a set size; perform instance segmentation on the target image of the set size, and put the instance segmentation result into the original image of the corresponding frame to obtain the instance segmentation target of each frame image.

[0010] 5) Using optical flow detection and instance segmentation of each frame of image, determine whether the obtained ORB feature points are dynamic feature points, remove dynamic feature points, and use the remaining feature points for tracking and mapping.

[0011] This invention divides a sequence of images into multiple subsequences. For the first frame of each subsequence, instance segmentation is performed to obtain the target detection box within that image. Based on the position and size of the target detection box and the corresponding camera pose information from two previous frames, the approximate position of the target detection box in the next frame or any subsequent frame is determined. The approximate position of the target detection box in each frame is determined using the camera pose information of the current frame and the camera pose information, target detection box position and size from any of the previous frames. This invention uses this method to obtain the approximate position of the target detection box in each frame, thereby obtaining the image within the corresponding target detection box in each frame, i.e., the target information. Instance segmentation is then performed on the image within the target detection box. Compared to existing technologies that segment the entire image in each frame, this significantly reduces the amount of data for image segmentation, effectively improving the speed of image instance segmentation and increasing work efficiency. Finally, based on the instance segmentation results, optical flow detection is used to remove dynamic feature points, improving the efficiency of subsequent synchronous localization and mapping of the target area.

[0012] Furthermore, each subsequence contains several frames of images arranged in chronological order, with the first frame being the earliest image in each subsequence.

[0013] The images in each subsequence are arranged in chronological order. By utilizing the temporal correlation between images, the position of the target detection box in the next frame can be easily estimated based on the position of the target detection box in the previous frame.

[0014] Furthermore, the first frame of each subsequence is segmented using a Mask R-CNN network.

[0015] Further, in step 4), the extracted target image is placed at the center of a blank image of a set size to generate a target image of a set size.

[0016] Furthermore, the target images of a set size in each subsequence are segmented using the Mask R-CNN network, wherein the central region of the target image of the set size is selected as the selection region of the anchor point in the Mask R-CNN network.

[0017] The extracted target images are all placed in the center of a blank image of a set size, which facilitates the selection of anchor point regions during subsequent image instance segmentation. It is no longer necessary to segment the entire image. Using the central region as the anchor point selection region in Mask R-CNN can reduce the number of anchor points and the number of generated anchor boxes, thereby improving segmentation efficiency.

[0018] Furthermore, in step 3), the position of the target detection box in the next frame image is determined by a point mapping method.

[0019] Point mapping can quickly determine the position of a point in the current frame image in the next frame image.

[0020] Furthermore, the method for determining ORB dynamic feature points is as follows: if the detected feature point is not included in the instance segmentation target, the feature point is a static feature point; if the feature point is included in the instance segmentation target and the target is static, the feature point is a static feature point; if the feature point is included in the instance segmentation target and the target is dynamic, the feature point is a dynamic feature point.

[0021] The relationship between ORB feature points and segmentation targets is determined through the above process. Feature points not included in the target are background feature points, which can be used for tracking and reconstruction. Feature points included in the target and the target is static can be used for tracking and target reconstruction, but not for scene reconstruction. Feature points included in the target and the target is dynamic cannot be used for tracking and reconstruction and are discarded. Attached Figure Description

[0022] Figure 1 This is a flowchart of the SLAM system based on sequence image segmentation of the present invention;

[0023] Figure 2 This is a flowchart of the sequence image segmentation method of the present invention;

[0024] Figure 3(a) is a schematic diagram of the original RPN anchor point generation;

[0025] Figure 3(b) is a schematic diagram of RPN anchor point generation according to the present invention;

[0026] Figure 4(a) is a schematic diagram of the first anchor point generation strategy used in the experimental example of the present invention;

[0027] Figure 4(b) is a schematic diagram of the second anchor point generation strategy used in the experimental example of the present invention;

[0028] Figure 4(c) is a schematic diagram of the third anchor point generation strategy used in the experimental example of the present invention;

[0029] Figure 4(d) is a schematic diagram of the fourth anchor point generation strategy used in the experimental example of the present invention. Detailed Implementation

[0030] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0031] This invention proposes a SLAM method based on sequence image segmentation, specifically as follows: Figure 1 As shown. This invention implements the entire process based on the ORB-SLAM3 system. First, it acquires a sequence of images of the target area and the camera pose at the corresponding time, and obtains the ORB features of each frame. Then, the sequence of images is divided into multiple subsequences. Instance segmentation is performed on the first frame of each subsequence. The obtained target detection box information and the camera pose at different time points are used to determine the target images of the remaining frames. The target images of each frame are then segmented to obtain the final instance segmentation results for all images. After obtaining the sequence segmentation results, dynamic feature points are removed using optical flow detection. The removed feature points are then passed to the subsequent tracking and mapping process. To ensure the robust operation of the entire system, as follows... Figure 1 As shown, apart from the Opticalflow Consistency Check and Screening feature points processes, the remaining processes still use the ORB-SLAM3 system's processes and corresponding algorithms, maintaining consistency with ORB-SLAM3.

[0032] Step 1. Obtain data

[0033] Sequential images of the target area are acquired using a depth camera mounted on a fixed platform (drone or unmanned vehicle). This depth camera can be a monocular camera, a stereo camera, or an RGB-D camera. Simultaneously, the sensor pose at each time step can be estimated using an IMU mounted on the platform. ORB feature points on each frame also need to be acquired, and these ORB feature points utilize... Figure 1 The ORB Extract module can be used to obtain ORB features, which can be achieved through ORB feature extraction methods (such as the Fast+BRIEF algorithm).

[0034] Step 2. Instance Segmentation

[0035] The acquired image sequence is segmented to identify specific objects within the images. For example... Figure 2 As shown, the sequence image segmentation method consists of the following steps: ① Divide the sequence image into several subsequences; ② Perform instance segmentation on the first frame of each subsequence to obtain target detection box information and mask information; ③ Calculate the position and size of the target detection box in the current frame using the camera pose information, depth information, and detection box information; ④ Read the next frame image in the subsequence, and calculate the approximate position of the target detection box in the current frame image using the calculated position and size information and the camera pose information of the previous and next frames; ⑤ Extract the target image under the corresponding target detection box for each frame image, and place it in a blank image of a set size to generate a target image of a set size; ⑥ Input the target image of the set size in each subsequent frame into the improved segmentation network for segmentation to obtain the final detection box information and target mask results.

[0036] In step ①, the sequence of images is divided into multiple subsequences according to their chronological order. Each subsequence contains several frames of images arranged chronologically, with the first frame being the earliest image in each subsequence. In this embodiment, each subsequence contains 10 images. Alternatively, the number of images in each subsequence can be determined based on the actual number of sequence images acquired.

[0037] In step ②, instance segmentation is performed on the first frame image in each subsequence to obtain the target detection bounding box information of that image, such as... Figure 2 As shown, the bounding boxes containing the two instance targets are obtained. In this embodiment, Mask R-CNN is used for instance segmentation. Other implementations may also use Deep Snake, Transformer, etc., for instance segmentation.

[0038] In step ③, the position and size of the target detection box in the current frame are calculated using the camera's pose information, depth information, and detection box information, as shown in formulas (1) and (2). The corner coordinates of the target detection box in the current frame can be obtained to determine the position and size of the detection box, for example, by using the coordinates of the upper left corner and the lower right corner, or by using the coordinates of the lower left corner and the upper right corner.

[0039] In step ④, the approximate location of the target detection box in subsequent frames is determined using point mapping. Given the pose information of the cameras in the preceding and following frames, point mapping is used to calculate the estimated position of the point in the current frame image based on the point information from the previous frame and the pose information of the cameras in the preceding and following frames. For example, P1 and P2 represent the same point in the previous and current frames, respectively.

[0040] P1=[u1,v1,d1],P2=[u2,v2,d2] (1)

[0041] Where u and v represent image pixel coordinates, and d represents the corresponding depth value. Based on the camera intrinsic parameters and formula (2), the camera coordinates of this point in both camera poses can be calculated:

[0042]

[0043] Where (x) c ,y c f is the principal point of the camera. x and f y For the camera's focal length, and These represent the camera coordinates of the point in the two camera poses, respectively. The relationship between the two camera poses can be expressed by formula (3):

[0044]

[0045] Where R represents the rotation matrix and t represents the translation matrix, and R and t can be obtained from the pose extrinsic parameters of the cameras in the preceding and following frames.

[0046] For example, the detection bounding box Box1(L1,R1) obtained from the detection and segmentation of the first frame image, assuming L1(u L1 ,v L1 ) and R1(u R1 ,v R1 ) represent the pixel coordinates of the top left and bottom right corners of the target detection box in the first frame image, respectively. The approximate positions of the target detection box in the second frame image can be obtained using formulas (1)-(3): BOX2((L2,R2)), L2(u L2 ,v L2 ) and R2(u R2 ,v R2The numbers () represent the pixel coordinates of the top-left and bottom-right corners of the target detection box in the second frame image, respectively. Similarly, to determine the approximate location of the target detection box in the i-th frame image, it is determined using the camera pose information corresponding to the i-th frame image and the camera pose information, target detection box position information, and size of any frame from the preceding frames. For example, the approximate location of the target detection box in the 5th frame image can be determined using any frame from the first to the fourth frame. Furthermore, the approximate location of the target detection box in all images except the first frame can be determined using the first frame image or the adjacent previous frame image.

[0047] In step ⑤, based on the approximate location of the target detection box in each frame of the image calculated in step ④, the target image at the corresponding location is extracted, such as... Figure 2 As shown, two target images can be extracted from each frame. The extracted target images are then placed within a blank image of a set size. In this embodiment, the blank image is 256 pixels × 256 pixels. In other embodiments, the size of the blank image can be determined according to actual conditions. In this embodiment, the extracted target image of each frame is placed in the center of the 256 pixel × 256 pixel blank image, resulting in a 256 pixel × 256 pixel target image for each frame. In other embodiments, it can also be placed in the upper left or upper right corner, etc.

[0048] In steps ⑥ and ②, this embodiment uses Mask R-CNN for instance segmentation. Mask R-CNN is a two-stage detection method. The anchor boxes generated by the two-stage method are mapped to regions of the feature map, and then these regions are re-input into the fully connected layer for classification and regression. Since a large number of anchor boxes are generated, and each anchor box needs to be mapped and classified and regressed through the fully connected layer, the two-stage method is time-consuming and cannot meet the time efficiency requirements of sequential image processing in SLAM. Therefore, in step ⑥, instance segmentation is no longer performed on the entire image frame. Instead, candidate proposal boxes are directly generated from the target image of the specified size in step ⑥ using the RPN layer in the Mask R-CNN network. This reduces the number of initial anchor points and the number of candidate proposal boxes, thereby reducing the time consumed by the candidate proposal box mapping, classification, and regression processes, and improving segmentation efficiency.

[0049] Figure 3(a) shows the basic structure of the RPN layer in the Mask R-CNN network. Its input is the feature map, and its output is the candidate region, represented by coordinate data (x, y, w, h). x and y represent the coordinates of the top-left corner of the candidate box, and w and h represent the width and height of the rectangle. In the original Mask R-CNN, five different sizes and three aspect ratios of rectangles were selected as initial candidate boxes. The points in Figure 3(a) are called anchors, which essentially refer to the mapping points of the center of the current sliding window on the feature map in the original pixel space. The ResNet algorithm iterates through five feature layers (P2-P6), selecting anchor points at each layer with a certain stride. For each anchor point, three anchor boxes with aspect ratios of 2:1, 1:1, and 1:2 are generated, corresponding to the dimensions of the feature layer. Then, all generated anchor boxes are classified and regressed, categorized as either foreground or background. k anchor boxes generate 2k scores, and the regression operation yields 4k scores corresponding to x, y, w, and h. Finally, the Intersection over Union (IOU) value is used to select 128 positive and 128 negative samples. Their corresponding scores are used to calculate the cross-entropy loss function, which is then used for network training. In the original network, the stride for searching anchor points is 1, meaning that every point in each feature layer is used as an anchor point to generate an anchor box. Taking layer P2 as an example, if the original input image is 512x512, then the size of the feature map of layer P2 is 128x128. The stride of this feature layer relative to the original image is 4, and the number of anchor boxes on this feature layer is 128x128x3. The size of the anchor boxes is determined by the scaling size of the feature layer.

[0050] To reduce the number of anchor boxes generated and improve the network speed, the characteristics of potential dynamic targets in the image sequence are considered: ① The time interval between the acquisition of each frame in the image sequence limits the movement range of the dynamic target between frames; ② Since the camera and the dynamic target are moving simultaneously, and the range of movement is limited, the size of the dynamic target will also change in the next frame, but the range of change is also limited. Therefore, based on the motion characteristics of the potential moving target, this invention only generates anchor boxes for the central region of the target image of a set size, that is, only for the target image of the set size in step ⑤. If the target image of the set size is in the central region of a blank image, only the central region is processed. As shown in Figure 3(b), the figure shows the anchor point selection areas of the P6, P5, and P4 feature layers when the input image size is 512x512, represented by the central black part (the P3 and P2 feature layers are not shown due to the number of grids). The feature map sizes for layers P2-P6 are 128x128, 64x64, 32x32, 16x16, and 8x8, respectively. A new anchor point selection rule is adopted, reducing the number of anchor boxes to 1 / 8 of the original. The anchor box sizes for the five feature layers are 24, 48, 96, 192, and 384, respectively. Alternatively, the number of anchor boxes in each layer can be determined based on actual needs, as long as it covers the target image of a set size.

[0051] The obtained instance segmentation results are directly translated and placed into the original image of the corresponding frame to obtain the instance segmentation results of each frame image under each subsequence, which is the target information.

[0052] Step 3. Optical Flow Dynamic Analysis

[0053] After detecting the target in the image, in order to achieve subsequent target synchronous localization and mapping, it is necessary to determine whether the ORB feature points in step 1 are dynamic feature points. The relationship between feature points and targets after instance segmentation can be divided into the following categories: ① The feature point is not included in the target, the feature point is a background feature point, and can be used for tracking and reconstruction; ② The feature point is included in the target, and the target is static, the feature point can be used for tracking and target reconstruction, but not for scene reconstruction; ③ The feature point is included in the target, and the target is dynamic, the feature point cannot be used for tracking and reconstruction, and should be discarded.

[0054] This invention employs optical flow to analyze the motion of feature points. First, it determines the dynamic state of the moving points. If a target contains a certain number of feature points, it is considered a dynamic target. The pyramid LK optical flow method is based on three assumptions: constant brightness, minimal object motion between adjacent frames, and consistent optical flow within the neighborhood. The first step is to calculate the optical flow pyramid to obtain the motion velocity of each feature point in the x and y directions. The average motion velocity of feature points within the background and target regions is then calculated. If the difference between the average motion velocities of feature points in the background and target regions exceeds a certain value, the target is considered dynamic.

[0055] Use I L and J L Let u represent the L-th layer pyramid image of images I and J, where u = [u x ,u y ] T Let I be the pixel coordinates tracked in I, then in I L Middle seat u L =[u x L ,u y L ] T =u / 2 L The idea behind the gold standard pyramid optical flow method is to first calculate the optical flow magnitude at the highest layer, then use the result as the initial value for the optical flow of the next layer, and so on, until the optical flow magnitude of the last layer is calculated as the final result. Let's assume the initial value of the optical flow magnitude at layer L+1, i.e., the optical flow magnitude of layer L, is g. L =[g x L ,g y L ] T We need to find pixel v in image J. L =u L +d L =[u x L +d x L +d y L ] T The point is chosen to minimize the difference in grayscale values ​​between the two. Then d can be... L =[d x L ,d y L ] T Defined as optical flow.

[0056] To solve for the optical flow, the solution process is transformed into an optimization problem, minimizing the optimization function in equation (4):

[0057]

[0058] Where ε L (d L ) represents the difference in grayscale values, i.e., the objective function, w x ,w y Representing the neighborhood range, we can solve for:

[0059] d L =G L-1 b L (5)

[0060]

[0061]

[0062] in Let δI represent the gradients of the image in the x and y directions, respectively. L (x,y)=I L (x,y)-J L (x,y) represents the image difference. The optical flow magnitude d of layer L is calculated. L Then, this value is used as the initial value for the L-1 layer, and the optical flow of the L-1 layer is calculated, and so on.

[0063] This invention utilizes instance segmentation and optical flow dynamic analysis of the aforementioned image sequence to achieve... Figure 1 The Opticalflow Consistency Check and Screening feature points in the code remove ORB dynamic feature points, and input the remaining ORB static feature points into the input. Figure 1 In the ORB-SLAM3 system, subsequent local map tracking and new key frame decision are performed to achieve local mapping, loop closure, map fusion, and global map optimization, thus completing scene map reconstruction. This invention adds two processes to the ORB-SLAM3 system framework: Optical flow Consistency Check and Screening feature points. The remaining processes are consistent with the ORB-SLAM3 system, and the corresponding algorithms remain unchanged. Therefore, this system will not be described in detail here.

[0064] To better illustrate the effects of the present invention, the SLAM method of the present invention is now experimentally verified.

[0065] Two sets of image sequences, each containing 100 frames, were selected from the TUM dataset for experimentation. The main objects in the images were people and chairs. This experiment tested the selection of anchor point generation strategies and the effectiveness of the proposed joint segmentation method for image sequences, and compared four anchor generation strategies. Taking an 8×8 feature map as an example, the four different strategies are shown in Figures 4(a), 4(b), 4(c), and 4(d). To verify the effectiveness of the proposed joint segmentation method for image sequences, three sets of image sequences were tested, and the accuracy and speed of target segmentation were analyzed.

[0066] The results are shown in Table 1. The proposed sequence image segmentation method is nearly 3 times faster than the original Mask R-CNN; furthermore, it also improves accuracy, proving the effectiveness of the proposed strategy. The anchor point generation strategy shown in Figure 4(d) achieves the highest segmentation accuracy of 84.5%, but requires more time. The anchor point generation strategy in Figure 4(c) has the shortest segmentation time, but lower accuracy. Furthermore, the segmentation accuracy of the anchor point generation strategies in Figures 4(a) and 4(b) is similar, but the segmentation time of the anchor point generation strategy in Figure 4(b) is shorter. Therefore, 4(b) is the optimal strategy.

[0067] Table 1

[0068]

Claims

1. A SLAM method based on sequence image segmentation, characterized in that, The method includes the following steps: 1) Use a depth camera to acquire a sequence of images of the target area, as well as the camera pose information and ORB feature points of each frame at the corresponding time in the sequence of images. 2) Divide the acquired sequence images into multiple subsequences, and perform instance segmentation on the first frame image in each subsequence to obtain the target detection box information in the first frame image of each subsequence; Using the target detection box information, depth information, and camera pose information at the corresponding time in the image, the position information and size of the target detection box in the first frame image are calculated; 3) Based on the position and size of the target detection box in the first frame image and the pose information of the camera in the first and second frames, calculate the approximate position of the target detection box in the second frame image; the position of the target detection box in any other frame image in the subsequence is determined by using the camera pose information of the current frame and the camera pose information, target detection box position and size of any frame in the previous few frames. 4) Based on the approximate location of the target detection box in each frame image, extract the image within the target detection box in each frame image as the target image in each frame image, and superimpose the target image into a blank image of a set size to obtain a target image of a set size; perform instance segmentation on the target image of the set size, and put the instance segmentation result into the original image of the corresponding frame to obtain the instance segmentation target of each frame image. 5) Using optical flow detection and instance segmentation of each frame of image, determine whether the obtained ORB feature points are dynamic feature points, remove dynamic feature points, and use the remaining feature points for tracking and mapping.

2. The SLAM method based on sequence image segmentation according to claim 1, characterized in that, Each subsequence contains several frames of images arranged in chronological order, with the first frame being the earliest image in each subsequence.

3. The SLAM method based on sequence image segmentation according to claim 1, characterized in that, The first frame of each subsequence is segmented using a Mask R-CNN network.

4. The SLAM method based on sequence image segmentation according to claim 1, characterized in that, In step 4), the extracted target image is placed at the center of a blank image of a set size to generate a target image of a set size.

5. The SLAM method based on sequence image segmentation according to claim 4, characterized in that, In each subsequence, the target image of a set size is segmented using the Mask R-CNN network, wherein the central region of the target image of the set size is selected as the anchor point selection region in the Mask R-CNN network.

6. The SLAM method based on sequence image segmentation according to claim 1, characterized in that, In step 3), the position of the target detection box in the next frame image is determined by the point mapping method.

7. The SLAM method based on sequence image segmentation according to claim 1, characterized in that, The method for determining ORB dynamic feature points is as follows: if the detected feature point is not included in the instance segmentation target, the feature point is a static feature point; if the feature point is included in the instance segmentation target and the target is static, the feature point is a static feature point; if the feature point is included in the instance segmentation target and the target is dynamic, the feature point is a dynamic feature point.

Images