DSO Algorithm Based on Local Map Common View Window Optimization

Abstract

This invention discloses a Local Map Shared-View Window (DSO) algorithm based on a local map shared-view window optimization algorithm, comprising the following steps: obtaining an initial sliding window; calculating the image frames within the initial sliding window frame by frame based on time interval, shared-visibility criteria, and semantic information; deleting image frames within the initial sliding window whose calculation results do not meet a first preset condition, and reselecting keyframes in the sliding window to replace the deleted image frames based on a combination of time interval, shared-visibility criteria, and semantic information; and performing PBA optimization based on the updated keyframes in the sliding window to construct a map. This invention utilizes dynamic window optimization to select not only several keyframes close to the current time when constructing a local map, but also several other keyframes with high shared-view points and the same semantic information as the current keyframe, ultimately establishing a persistent and globally consistent map.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to the DSO algorithm based on the local map shared-view window optimization algorithm. Background Technology

[0002] Visual Odometry (DSO) is a monocular sparse direct visual odometry algorithm, a representative algorithm in visual SLAM, also known as the direct method. It mainly consists of a front-end thread for keyframe extraction and a back-end thread for map generation, without loop closure detection or map reuse. From the back-end perspective, DSO uses a sliding window composed of several keyframes as its back-end. This sliding window exists throughout the entire visual odometry (VO) process. The front-end uses certain conditions (such as excessive camera movement) to determine whether a new frame can be inserted as a new keyframe into the back-end; the back-end uses certain conditions to remove (marginalize using Schur complement) extra frames. DSO attempts to project map points from each old keyframe into the new keyframe, forming residual terms. Simultaneously, it extracts immature points from the new keyframes, hoping they will evolve into normal map points.

[0003] In direct visual odometry (BA), the optimization objective is to minimize photometric error. This is because direct methods do not rely on the more traditional geometric reprojection error based on feature point matching; instead, they directly process image pixel values ​​and minimize the intensity differences of reprojected points. Therefore, in direct methods, BA is also called PBA (Photometric Bundle Adjustment). However, current PBA-based methods can only perform visual odometry (VO) because they use a sliding window approach to select only a few time-proximity keyframes to accurately estimate camera pose, marginalizing map points that have left the field of view. This results in VO only building a temporary map and maintaining a local state, rather than building and maintaining a persistent and globally consistent map. Therefore, if the camera revisits an already mapped area, PBA cannot reuse the map points that have been marginalized and must recreate them. This is a serious limitation: the system cannot utilize and maintain information from existing map points, leading to motion drift and global map inconsistencies. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a DSO algorithm based on a local map co-view window optimization algorithm. This invention utilizes dynamic window optimization to select keyframes for constructing a local map. It not only chooses several keyframes close to the current time but also selects other keyframes with high co-view points and the same semantic information as the current keyframe, ultimately establishing a persistent and globally consistent map.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] The DSO algorithm based on the local map common-view window optimization algorithm includes the following steps:

[0007] Step S1: Obtain an initial sliding window, and determine N consecutive image frames in the video image frame sequence based on the initial sliding window;

[0008] Step S2: Calculate the image frames within the initial sliding window frame by frame based on time interval, co-visibility criterion, and semantic information;

[0009] Step S3: Determine whether the calculation result of the image frame in the initial sliding window meets the first preset condition. If yes, determine that the current image frame is a key frame, keep the current image frame in the sliding window, and proceed to step S4; otherwise, determine that the current image frame is not a key frame, delete the current image frame from the sliding window, and proceed to step S5.

[0010] Step S4: Determine whether the current image frame is the oldest image frame in the initial sliding window. If yes, proceed to step S8; otherwise, proceed to step S2 and continue to calculate the next image frame adjacent to the current image frame in the initial sliding window.

[0011] Step S5: Extract the associated frame set, which includes M consecutive image frames before and / or after the initial sliding window;

[0012] Step S6: Calculate the image frames in the associated frame set frame by frame based on time interval, co-visibility criterion, and semantic information;

[0013] Step S7: Determine whether the calculation result of the image frame in the associated frame meets the second preset condition. If yes, determine the current image frame in the associated frame set as the key frame, replace the current image frame deleted in the sliding window with the current image frame in the associated frame set, and proceed to step S4; otherwise, proceed to step S6 and continue to calculate the next image frame adjacent to the current image frame in the associated frame set.

[0014] Step S8: Perform PBA optimization based on the keyframes in the current sliding window to construct a map.

[0015] Preferably, the calculation formulas in steps S2 and S6 are as follows:

[0016]

[0017] In the formula s(I i ) represents the set of keyframes, Δt(I) i ,I0) and α1 represent the time difference and weight between the selected frame and the tracking frame, respectively; Δs(Ii α0 and α2 represent the difference in co-viewpoint between the selected frame and the tracking frame, and their respective weights; o(I i α0 and α3 represent the common semantic observation objects and weights of the selected frame and the tracking frame, respectively. n1 and n2 represent the total number of frames and the number of common semantic observation objects set when selecting keyframes, respectively. α1+α2+α3=1.

[0018] Preferably, the tracking frame is the first frame within the initial sliding window or a key frame that has already been selected.

[0019] Preferably, the common viewpoint is determined by the visibility s of map points. u The visibility s was measured. u The calculation formula is as follows:

[0020] s u =∑min(p z / p′ z ,1)N

[0021] In the formula, N represents the total number of visible points in the selected frame, and p z , p′ z These are the point inverse depths of the selected frame and the tracking frame, respectively.

[0022] Preferably, the time difference is calculated based on the timestamp inherent in the frame.

[0023] Preferably, the detection result of the semantic object is output by the deep learning algorithm deeplabv3+.

[0024] Preferably, the common semantic observation object and weight are set dynamically according to the differences in different scenarios.

[0025] Preferably, the sliding window contains 5 to 7 keyframes.

[0026] Based on the above technical solution, the beneficial effects of the present invention are:

[0027] 1. The Local Map Covisibility Window (LMCW) method proposed in this invention selects key frames in a sliding window based on a combination of time interval, covisibility criteria, and semantic information. It can utilize dynamic window optimization to select key frames to build a local map. In addition to selecting several key frames close to the current time, it also selects several other key frames with high covisibility and the same semantic information as the current key frame, ultimately building a persistent and globally consistent map.

[0028] 2. This invention can dynamically adjust common semantic observation objects and different weights according to the differences in different scenarios, so that the invention has editability when there are large differences in scenarios, thereby improving the success rate of map construction. Attached Figure Description

[0029] Figure 1 This is a flowchart of the DSO algorithm based on the local map shared-view window optimization algorithm in one embodiment;

[0030] Figure 2 This is a schematic diagram of keyframe selection in a sliding window in one embodiment;

[0031] Figure 3 This is a result image of semantic segmentation in one embodiment;

[0032] Figure 4 This is an experimental result diagram of the DSO algorithm based on the local map shared-view window optimization algorithm in one embodiment. Detailed Implementation

[0033] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0034] like Figure 1 As shown, this embodiment provides a DSO algorithm based on a local map shared-view window optimization algorithm, including the following steps:

[0035] Step S1: Obtain the initial sliding window, and determine the six consecutive image frames [I0, I1, I2, I3, I4, I5] in the image frame sequence of the video based on the initial sliding window;

[0036] Step S2: Calculate the image frames within the initial sliding window frame by frame based on time interval, co-visibility criterion, and semantic information;

[0037] Step S3: Determine whether the calculation result of the image frame in the initial sliding window meets the first preset condition. If yes, determine that the current image frame is a key frame, keep the current image frame in the sliding window, and proceed to step S4; otherwise, determine that the current image frame is not a key frame, delete the current image frame from the sliding window, and proceed to step S5.

[0038] Step S4: Determine whether the current image frame is the oldest image frame in the initial sliding window. If yes, proceed to step S8; otherwise, proceed to step S2 and continue to calculate the next image frame adjacent to the current image frame in the initial sliding window.

[0039] Step S5: Extract the associated frame set, which includes M consecutive image frames before and / or after the initial sliding window;

[0040] Step S6: Calculate the image frames in the associated frame set frame by frame based on time interval, co-visibility criterion, and semantic information;

[0041] Step S7: Determine whether the calculation result of the image frame in the associated frame meets the second preset condition. If yes, determine the current image frame in the associated frame set as the key frame, replace the current image frame deleted in the sliding window with the current image frame in the associated frame set, and proceed to step S4; otherwise, proceed to step S6 and continue to calculate the next image frame adjacent to the current image frame in the associated frame set.

[0042] Step S8: Perform PBA optimization based on the keyframes in the current sliding window to construct a map.

[0043] In this embodiment, the traditional DSO algorithm's PBA scheme, when using the sliding window method, selects consecutive image frames [I0, I1, I2, I3, I4, I5] as frames within the sliding window for optimization. However, two of these six frames (image frame I4 and image frame I5) are invalid frames, and using the sliding window in this case can even negatively impact the optimization. Using this algorithm, it can be seen that compared to image frames [I7, I10, I11, I12, I13] and those close to image frame [I10] ([I4, I5, I6, I8, I9]), although image frames [I4, I5, I6, I8, I9] have a temporal advantage, their shared viewpoint ratio and the proportion of identical semantic information are relatively low. Therefore, image frames [I4, I5, I6, I8, I9] will not be included in the sliding window. Figure 2 As shown.

[0044] Based on three aspects—time interval, co-visibility criterion, and semantic information—keyframes are reselected and added to the sliding window. This allows PBA to select not only keyframes close to the current time when building a local map, but also other keyframes with high co-visibility and semantic information, even if they are temporally distant, such as frames [I7, I10]. The sliding window frames will then become [I0, I1, I2, I3, I7, I10]. PBA can then reuse information from existing map points while optimizing temporally nearby and distant map points and keyframes, thereby reducing pose estimation drift and building a persistent and globally consistent map.

[0045] In one embodiment of the DSO algorithm based on local map shared-view window optimization, the keyframe calculation formula is as follows:

[0046]

[0047] In the formula s(I i ) represents the set of keyframes, Δt(I) i,I0) and α1 represent the time difference and weight between the selected frame and the tracking frame, respectively; Δs(I i α0 and α2 represent the difference in co-viewpoint between the selected frame and the tracking frame, and their respective weights; o(I i α0 and α3 represent the common semantic observation objects and weights of the selected frame and the tracking frame, respectively. n1 and n2 represent the total number of frames and the number of common semantic observation objects set when selecting keyframes, respectively. α1 + α2 + α3 = 1. Among them, the tracking frame is the first frame in the initial sliding window or the already selected keyframe.

[0048] In this embodiment, the common viewpoint is determined by the visibility s of the map point. u The visibility s was measured. u The calculation formula is as follows:

[0049] s u =∑min(p z / p′ z ,1) / N

[0050] In the formula, N represents the total number of visible points in the latest keyframe, p z , p′ z These are the inverse depths of the points in the latest keyframe and the tracking frame, respectively.

[0051] The preset thresholds for both the first and second preset conditions are pre-set based on actual applications. As can be seen from the keyframe calculation formula, the preset threshold range in the first preset condition should be less than 1 in actual applications. The second preset condition is to reduce the selection criteria for keyframes, so it is the threshold in the first preset condition multiplied by a weight coefficient, which is less than or equal to 1.

[0052] In this embodiment, the detection result of the semantic object is output by the deep learning algorithm deeplabv3+.

[0053] Based on the deeplabv3+ semantic segmentation algorithm fused in this algorithm, tests were conducted. First, an RGB-D camera was used to move at a constant speed around the conference room, such as... Figure 3 As shown in the image, the semantically segmented results have been collected. The semantically segmented objects in this experiment are shown in Table 1.

[0054] Table 1 Experimental Results

[0055]

[0056] Then, considering the elongated shape of the indoor testing scenario, weights α1 = 0.3, α2 = 0.4, and α3 = 0.3 were set, and "door" and "box" were set as common semantic observation objects for keyframes. However, this is not the only option; the setting of common semantic observation objects and weights can be dynamically adjusted according to the differences in different scenarios, so that the global map can have better scene adaptability when building in different scenarios.

[0057] Finally, the above inputs are fed into the DSO algorithm for map construction, such as... Figure 4 As shown, the two marker lines clearly span a long time interval, enabling keyframe selection based on shared viewpoints and semantic information.

[0058] The above description is merely a preferred embodiment of the DSO algorithm based on the local map shared-view window optimization algorithm disclosed in this invention, and is not intended to limit the scope of protection of the embodiments in this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments in this specification should be included within the scope of protection of the embodiments in this specification.

Claims

1. A DSO algorithm based on a local map shared-view window optimization algorithm, characterized in that, Includes the following steps: Step S1: Obtain an initial sliding window, and determine N consecutive image frames in the video image frame sequence based on the initial sliding window; Step S2: Calculate the image frames within the initial sliding window frame by frame based on time interval, co-visibility criterion, and semantic information; Step S3: Determine whether the calculation result of the image frame in the initial sliding window meets the first preset condition. If yes, determine that the current image frame is a key frame, keep the current image frame in the sliding window, and proceed to step S4; otherwise, determine that the current image frame is not a key frame, delete the current image frame from the sliding window, and proceed to step S5. Step S4: Determine whether the current image frame is the oldest image frame in the initial sliding window. If yes, proceed to step S8; otherwise, proceed to step S2 and continue to calculate the next image frame adjacent to the current image frame in the initial sliding window. Step S5: Extract the associated frame set, which includes M consecutive image frames before and / or after the initial sliding window; Step S6: Calculate the image frames in the associated frame set frame by frame based on time interval, co-visibility criterion, and semantic information; Step S7: Determine whether the calculation result of the image frame in the associated frame meets the second preset condition. If so, determine the current image frame in the associated frame set as the key frame, replace the current image frame deleted in the sliding window with the current image frame in the associated frame set, and proceed to step S4. Otherwise, proceed to step S6 to continue calculating the next image frame adjacent to the current image frame in the associated frame set; Step S8: Perform PBA optimization based on the keyframes in the current sliding window to construct a map.

2. The DSO algorithm based on local map shared-view window optimization algorithm according to claim 1, characterized in that, The calculation formulas in steps S2 and S6 are as follows: In the formula s(I i ) represents the set of keyframes, Δt(I) i ,I0) and α1 represent the time difference and weight between the selected frame and the tracking frame, respectively; Δs(I i α0 and α2 represent the difference in co-viewpoint between the selected frame and the tracking frame, and their respective weights; o(I i α0 and α3 represent the common semantic observation objects and weights of the selected frame and the tracking frame, respectively. n1 and n2 represent the total number of frames and the number of common semantic observation objects set when selecting keyframes, respectively. α1+α2+α3=1.

3. The DSO algorithm based on local map shared-view window optimization algorithm according to claim 2, characterized in that, The tracking frame is either the first frame within the initial sliding window or a key frame that has already been selected.

4. The DSO algorithm based on local map shared-view window optimization algorithm according to claim 3, characterized in that, The common viewpoint is determined by the visibility s of the map point. u The visibility s was measured. u The calculation formula is as follows: In the formula, N represents the total number of visible points in the selected frame, and p z , p′ z These are the point inverse depths of the selected frame and the tracking frame, respectively.

5. The DSO algorithm based on local map shared-view window optimization algorithm according to claim 2, characterized in that, The time difference is calculated based on the timestamp inherent in the frame.

6. The DSO algorithm based on local map shared-view window optimization algorithm according to claim 2, characterized in that, The detection results of the semantic objects are output by the deep learning algorithm deeplabv3+.

7. The DSO algorithm based on local map shared-view window optimization algorithm according to claim 2, characterized in that, The common semantic observation objects and weights are set dynamically based on the differences in different scenarios.

8. The DSO algorithm based on local map shared-view window optimization algorithm according to claim 1, characterized in that, The sliding window contains 5 to 7 keyframes.

Images