A semantic map optimization method and system based on SLAM mapping frame selection incorporating semantic information

Abstract

This invention provides a semantic map optimization method and system that integrates semantic information to select SLAM mapping frames. Based on various semantic segmentation errors in dynamic scenes, including unsegmented static objects, processing errors between static objects, and incorrect segmentation of static objects, it employs corresponding models and compensation terms to evaluate the richness and accuracy of image semantics, assigning a score to each image in the sequence, and finally selecting the image with the highest score as the mapping frame to construct the semantic map. This invention primarily provides a mapping frame selection mechanism for visual SLAM systems, helping them select image frames with more complete semantic information, thereby obtaining a more stable and richer octree semantic map. The core of this invention is to establish corresponding models by analyzing actual problems in the scene and integrating semantic information into the visual SLAM mapping frame selection mechanism.

Description

Technical Field

[0001] This invention describes a semantic map optimization scheme that integrates semantic information for selecting mapping frames in a visual SLAM system, belonging to the fields of artificial intelligence robots and computer vision. Background Technology

[0002] Visual Simultaneous Localization and Mapping (SLAM) technology utilizes image information for visual localization and constructs environmental maps. It boasts advantages such as low cost, accurate localization, and rich perceptual information, and is widely used in navigation and positioning. With the development of industries such as autonomous driving, AR / VR, and intelligent unmanned systems, visual semantic SLAM has emerged as a further development based on visual SLAM. This technology aims to integrate semantic information with traditional visual SLAM to improve localization accuracy, system robustness, and the intelligence of the map.

[0003] Visual semantic map construction is a crucial task in visual semantic SLAM, aiming to create a map that not only describes the geometry of the environment but also facilitates scene understanding. This allows robots to not only determine "whether an object exists at a certain spatial location" but also "what that object is," effectively enhancing the robot's intelligence. To achieve this goal, a common approach is to extract pixel-level semantic information from images using semantic segmentation and label the semantic categories on the corresponding map point cloud, forming a semantic information mapping from two-dimensional image pixels to three-dimensional landmark points. Therefore, the completeness and accuracy of semantic information in a semantic map heavily depend on the effectiveness of image semantic segmentation.

[0004] In traditional feature-based visual SLAM systems, the front end is responsible for selecting keyframes with rich features and low redundancy to construct a sparse map, which is then used for pose estimation, such as ORB-SLAM2. Many visual semantic mapping methods build upon this foundation, retaining the keyframe selection mechanism of traditional visual SLAM but combining the image semantic segmentation information of these keyframes with the pose and depth information acquired by visual SLAM for semantic mapping. For example, DS-SLAM adds a semantic segmentation thread and an octree semantic map construction thread to ORB-SLAM2. The octree map is a three-dimensional spatial occupancy grid map that continuously divides the space into cubic voxels and updates the occupancy state of these voxels, thus describing the free-roaming and obstacle regions of the space. Semantic labels are then annotated on the voxels to form the semantic octree map. However, taking DS-SLAM as an example, this visual SLAM method, based on an improvement of ORB-SLAM2, uses keyframes originally intended for localization for mapping, without considering the semantic information required for semantic mapping. This leads to widespread errors and omissions of semantic information in some selected keyframes, consequently resulting in errors and omissions of semantic information in the semantic map. Furthermore, since the semantic information in the semantic map is accumulated and annotated from the semantic information of image frames, erroneous segmentation results also contribute to poor stability of the semantic annotation. Figure 1 As shown, this image sequence represents the semantic segmentation results of selected keyframes from DS-SLAM running on the TUM RGB-D dataset walk_xyz sequence. One image failed to correctly segment the computer on the right due to human occlusion, leading to... Figure 2 The semantic map constructed in the image shows a large area of ​​missing computer-generated semantic annotations on the right side. This indicates that many dynamic objects in reality can occlude static backgrounds, leading to a loss of semantic information in the map, a problem that traditional visual SLAM keyframe selection mechanisms do not address. Summary of the Invention

[0005] This invention addresses the problem of visual semantic mapping in dynamic scenes by designing a semantic mapping keyframe selection mechanism to replace the original pose estimation keyframes for constructing semantic maps, thereby obtaining semantic maps with richer and more stable semantic information.

[0006] To achieve the above objectives, the technical solution proposed in this invention is a semantic map optimization method that integrates semantic information to select SLAM mapping frames. Based on various situations of semantic segmentation errors in dynamic scenes, including unsegmented static objects, processing errors between static objects, and incorrect segmentation of static objects, the method uses corresponding models and compensation terms to evaluate the richness and accuracy of image semantics, assigns a score to each image in the sequence, and finally selects the image with the highest score as the mapping frame to achieve the construction of a semantic map.

[0007] Furthermore, for cases where static objects are not segmented, the total score is calculated by comparing the semantic information richness of the images, as follows:

[0008] Suppose there are m classes of static objects a1, a2, ..., a3 in the image. m The image sequence has a total of k frames. The number of pixels of the nth category in the i-th image is denoted as p. ni The total number of pixels of the nth category in frame k can be calculated as follows: The average number is

[0009] For each category in each image, an offset value is set. The offset(ni) is defined as the proportion of the difference between the pixel value of the nth class object in the i-th image and the mean value of the pixel value of that class.

[0010]

[0011] Simultaneously calculate the pixel sum of all static objects in k frames.

[0012]

[0013] To obtain the proportion of a single category n, pro(n),

[0014]

[0015] The coefficient of a single class n is denoted as fac(n).

[0016]

[0017] The total score, grade, of the semantic information of the i-th image is obtained and denoted as grade. i ,

[0018] grade i (n) = fac(n) × offset(n)

[0019]

[0020] Among them, grade i(n) represents the score of the semantic information of the i-th image in a single category n;

[0021] Select the image with the highest score as the initial mapping frame, denoted as .

[0022] max(grade1, grade2, ..., grade k )

[0023] Here, max() means taking the maximum value.

[0024] Furthermore, the filtering model is optimized to address errors in handling static objects and errors in segmenting static objects, as follows:

[0025] Construct an r×t matrix, where r and t represent the length and width of the image. At the pixel coordinates (p, q) in this matrix, there exists an m+1 dimensional vector (a1, a2, ..., a...). m b), where each value in the vector represents the cumulative observation frequency of that pixel with respect to a certain category, where the observation frequencies of static object pixel categories are a1, a2, ..., a m , corresponding to m categories; the observation frequency of useless background and dynamic object categories is represented by b, corresponding to the (m+1)th category;

[0026] Initially, the observation frequency is initialized to 0. As the image sequence is input sequentially, the pixel category at that coordinate in each input image will be associated with a1, a2, ..., a m For a given observation frequency in b, increment the corresponding observation frequency by one. After reading all k frames of the sequence, the vector at (p,q) will record the frequency of each target type at that point. The observation frequency with the highest frequency is denoted as a. x :

[0027] a x =max(a1,a2,…,a m b)

[0028] A threshold parameter ε is set to determine the semantic consistency of a pixel in a k-frame image sequence. If:

[0029] a x ≥k×ε

[0030] This point is considered category x, and the frequency of being classified as any other static object category is considered as the number of incorrect observations (error). ni This represents the sum of frequencies at which pixels in the i-th image that are ultimately classified as belonging to the n-th category are observed as pixels belonging to other categories. This error is subtracted from the total score (grade) of each image. ni The modified offset value is:

[0031]

[0032] And when

[0033] a x <k×ε

[0034] This point is considered either a dynamic target or background, and the frequency of determining the effective static object category is considered as redundant observations. ni , representing the sum of frequencies of pixels in the i-th image that were ultimately judged as invalid moving targets or backgrounds but were observed as valid static object categories. This portion of 'sur' needs to be subtracted from the total score 'grade' for each image. ni The modified offset value is:

[0035]

[0036] Finally, the final score for the i-th image is:

[0037]

[0038] Furthermore, the randomness of errors can be reduced by extending the observation sequence length. Suppose that the segmented image sequence of K = K1 frames is extended, and the number of images in the extended sequence is denoted as K = K2 and K = K3, respectively. The corresponding weights are calculated based on the K value of the image sequence.

[0039] When K = K1, a set of grade values ​​is obtained, and the corresponding K2 is calculated simultaneously:

[0040]

[0041] K2=K1 / ε

[0042] When K = K2, a set of grade values ​​is obtained, and the corresponding K3 is calculated simultaneously:

[0043]

[0044] K3 = K2 / ε

[0045] When K = K3, the final set of grade values ​​is obtained:

[0046]

[0047] The final score for each image is as follows:

[0048]

[0049]

[0050] Finally, the maximum score (grade) in the image sequence is selected. final The corresponding image frame is used as the mapping frame:

[0051] grade final =max(grade(1),grade(2),…,grade(k)).

[0052] On the other hand, the present invention also provides a semantic map optimization system for filtering SLAM mapping frames by fusing semantic information, which is used to implement the semantic map optimization method for filtering SLAM mapping frames by fusing semantic information as described above.

[0053] Furthermore, it includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a semantic map optimization method for filtering SLAM mapping frames by fusing semantic information, as described above.

[0054] Alternatively, it may include a readable storage medium storing a computer program that, when executed, implements a semantic map optimization method for filtering SLAM mapping frames by fusing semantic information, as described above.

[0055] This invention optimizes the completeness and stability of semantic information in visual semantic mapping. The semantic mapping keyframe selection mechanism designed in this invention analyzes various problem scenarios and error conditions, establishes targeted models, measures the richness and stability of image semantic information, and ultimately selects reliable image frames for mapping. Applying this invention to visual semantic mapping can effectively address the loss of semantic information caused by occlusion of moving objects in dynamic scenes, resulting in a semantic map with richer and more stable semantic annotation information.

[0056] Compared with the prior art, the improvement of the present invention is as follows: 1. Based on the visual SLAM keyframe, the present invention adds a parallel mapping frame branch by adding extra threads and modules. In the mapping frame branch, a mapping frame filtering method is designed. This method uses semantic information as the filtering basis, which can remove image frames with large missing or incorrect semantic information from the image sequence and obtain image frames with richer and more complete semantic information.

[0057] 2. The mapping frame filtering mechanism proposed in this invention can filter image frames with more complete and richer semantic information from the image sequence. These frames will be used as image frames for visual SLAM to build semantic maps, thereby obtaining semantic maps with richer and more complete semantic information.

[0058] The present invention is simple and convenient to implement, highly practical, and solves the problems of low practicality and inconvenience in actual application of related technologies. It can improve user experience and has significant market value. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the keyframe segmentation results selected by existing DS-SLAM in the TUM RGBD walk xyz sequence. The image with boxed annotations shows that the right computer was not accurately segmented under occlusion conditions.

[0060] Figure 2 This is a schematic diagram of the existing DS-SLAM semantic mapping results for the TUM RGBD walk xyz sequence. The computer semantic information on the right side is largely missing.

[0061] Figure 3 This diagram illustrates common errors in existing semantic segmentation techniques in dynamic scenarios.

[0062] Figure 4 This is a flowchart illustrating the implementation of the present invention in a visual SLAM system.

[0063] Figure 5 These are comparison images of the effects of embodiments of the present invention, where the left image is the map construction result without using mapping frames, the middle image is the original image result, and the right image is the map construction result using mapping frames. Detailed Implementation

[0064] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0065] See Figure 4 This invention proposes a semantic map optimization method that integrates semantic information to filter SLAM mapping frames. To achieve frame filtering for visual semantic mapping, a filtering branch is added in parallel to the traditional visual SLAM system branches, specifically serving the construction of the semantic map. This invention fully considers various scenarios of semantic segmentation errors in dynamic scenes, proposing different models and compensation terms to evaluate the richness and accuracy of image semantics, assigning a score to each image in the sequence, and finally selecting the image with the highest score as the mapping frame.

[0066] This invention first analyzes common scenarios of semantic segmentation errors in the scene, such as... Figure 3 As shown:

[0067] (1) Static objects are not separated, meaning they are treated as background or dynamic objects.

[0068] (2) Errors in processing static objects, such as processing static object a as b.

[0069] (3) Static objects are incorrectly segmented, that is, the background or dynamic objects are segmented as static objects.

[0070] Among them, (1) is the most common error situation that this invention focuses on. Therefore, the strategy design of this invention is mainly based on situation (1), and optimized and compensated according to (2) and (3).

[0071] For an input image sequence, this invention calculates the total score by comparing the semantic information richness of the images. Assume there are m classes of static objects a1, a2, ..., a... m The image sequence has a total of k frames. The number of pixels of the nth category in the i-th image is denoted as p. ni Therefore, the total number of pixels of the nth category in frame k can be calculated as follows: The average number is

[0072] For each category in each image, this invention sets an offset value, defined as: the proportion of the difference between the pixel value of the nth category of objects in the i-th image and the mean value of that category of pixels.

[0073]

[0074] Simultaneously, this invention calculates the sum of all valid pixels (i.e., pixels of static objects) in k frames:

[0075]

[0076] Based on this, the proportion pro(n) of a single category n can be obtained in this invention:

[0077]

[0078] Considering that categories with a larger proportion have more pixels, meaning they have a smaller negative impact on image construction, this invention assigns a larger coefficient fac to categories with a smaller proportion of pixels. The coefficient for a single category n is denoted as fac(n):

[0079]

[0080] Finally, this invention can obtain the total score grade of the semantic information of the i-th image, denoted as grade. i :

[0081] grade i (n) = fac(n) × offset(n)

[0082]

[0083] Among them, grade i (n) represents the score of a single category n for the semantic information of the i-th image.

[0084] Select the image with the highest score as the initial mapping frame:

[0085] max(grade1, grade2, ..., grade k )

[0086] Here, max() means taking the maximum value.

[0087] Subsequently, this invention optimizes the screening model based on the scenarios in (2) and (3). This invention constructs an r×t matrix, where r and t represent the length and width of the image, and the image pixel coordinates (p, q) in this matrix correspond to an m+1 dimensional vector (a1, a2, ..., a...). m b), where each value in the vector represents the cumulative observation frequency of that pixel with respect to a certain category, where the observation frequencies of static object pixel categories are a1, a2, ..., a m There are m categories (1, 2, ..., m); the observation frequency of useless background and dynamic object categories is represented by b, corresponding to the (m+1)th category. Initially, the observation frequencies are all initialized to 0. As the image sequence is input sequentially, the pixel category at that coordinate of each input image will be associated with a1, a2, ..., a... m For a given observation frequency in b, the corresponding observation frequency is incremented by one. After reading all k frames of the sequence, the vector at (p,q) records the frequency of each target type at that point. The observation frequency with the highest frequency is denoted as a. x :

[0088] a x =max(a1,a2,…,a m b)

[0089] This invention provides a threshold parameter ε to determine the semantic consistency of a pixel in a k-frame image sequence. This parameter is an empirical parameter, and in the embodiment, ε = 0.67 was obtained through experimental statistics. In specific implementation, different parameters can be set according to different numbers of semantic categories.

[0090] if:

[0091] a x ≥k×ε

[0092] Therefore, this point is considered category x, and the frequency of being classified as any other static object category is considered as the number of incorrect observations (error). ni Error is the sum of the frequencies at which pixels ultimately classified as the nth category in the i-th image are observed as pixels in other categories. This error needs to be subtracted from the total score (grade) of each image. ni Therefore, the modified offset value is:

[0093]

[0094] And when

[0095] a x <k×ε

[0096] Therefore, this point is considered either a dynamic target or background, and the frequency of determining the effective static object category is considered as a redundant observation count. ni In other words, in the i-th image, the sum of the frequencies at which pixels ultimately judged as invalid moving targets or backgrounds are observed as valid static objects is the sum of the frequencies of those pixels. This sum needs to be subtracted from the total score (grade) of each image. ni The modified offset value is:

[0097]

[0098] Finally, the final score for the i-th image is:

[0099]

[0100] Considering that the method of this invention primarily removes data with significant contrast in the data queue, the highest grade in a set of data is not necessarily the one with the highest segmentation accuracy. However, by appropriately expanding this queue, the randomness of errors will be further reduced as the number of images in the queue increases. Based on this, this invention further optimizes the original method by expanding the segmented image sequence of frames K=K1, with the expanded sequence images denoted as K=K2 and K=K3, respectively, and calculating the corresponding weights based on the K values ​​of the image sequences. Considering that the previously given empirical threshold parameter ε is positively correlated with the accuracy of the segmentation model—that is, the more accurate the semantic segmentation model, the larger the corresponding ε, and the more reliable the obtained semantic information—the optimization in this section reduces the randomness of errors by expanding the length of the observation sequence. The more reliable the obtained semantic information, the smaller the number of images required for each expansion. Therefore, the final solution is as follows:

[0101] When K = K1, this invention obtains a set of grade values ​​and simultaneously calculates the corresponding K2:

[0102]

[0103] K2=K1 / ε

[0104] When K = K2, this invention obtains a set of grade values ​​and simultaneously calculates the corresponding K3:

[0105]

[0106] K3 = K2 / ε

[0107] When K = K3, this invention obtains the final set of grade values:

[0108]

[0109] The final score obtained by this invention for each image is:

[0110]

[0111]

[0112] Finally, the maximum score (grade) in the image sequence is selected. final The corresponding image frame is used as the mapping frame:

[0113] grade final =max(grade(1),grade(2),…,grade(k))

[0114] Using the mapping frames in this invention for visual semantic mapping can effectively improve the quality of semantic information in maps, mainly in the following aspects.

[0115] First, the mapping frame algorithm is effective for filtering high-accuracy segmentation results. As shown in Table 1, semantic segmentation is performed on a set of sequences, and the segmentation results are compared with the ground truth to obtain the accuracy mIoU. At the same time, the mapping frame filtering method of this invention is used to score the semantic richness and accuracy of the image. It can be observed that the results with higher segmentation accuracy also have relatively higher scores, which proves that this invention effectively evaluates the quality of image semantic information without obtaining the ground truth. As shown in Table 2, the mapping frame filtering and traditional keyframes are combined with the classic semantic segmentation method, respectively. The accuracy of the image segmentation results after filtering and without filtering is compared. It can be seen that the segmentation accuracy of the filtered mapping frames is significantly improved.

[0116] Table 1 Correlation between Mapping Frame Scores and Segmentation mIoU

[0117] image P1 P2 P3 P4 P5 P6 <![CDATA[grade final ]]> -1.375 -1.506 -2.831 -2.327 -1.421 -1.417 mIou 0.576 0.718 0.392 0.259 0.592 0.483

[0118] Table 2 Comparison of segmentation effects between original keyframes and optimized mapping frames

[0119]

[0120] During the construction of the octree semantic map, the system continuously projects pixels into space using pose and depth information to form a point cloud, and then continuously divides the space into voxel blocks according to the occupancy state of the point cloud. Semantic labels are also annotated on these voxel blocks during this process. The stability and completeness of the map construction process can be evaluated based on the volatility of voxel semantic category label updates and the comparison between map semantic information and actual scene semantic information. As shown in Table 3, maps using the mapping frames exhibit higher completeness and lower volatility.

[0121] Table 3 Map Construction Quality Comparison

[0122]

[0123] See Figure 5 The left image shows the map construction result without using mapping frames, the middle image shows the original map result, and the right image shows the map construction result using mapping frames. The map built by this invention in dynamic scenes has more complete semantic annotations.

[0124] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0125] In some possible embodiments, a semantic map optimization system for filtering SLAM mapping frames by fusing semantic information is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the semantic map optimization method for filtering SLAM mapping frames by fusing semantic information as described above.

[0126] In some possible embodiments, a semantic map optimization system for filtering SLAM mapping frames by fusing semantic information is provided, including a readable storage medium on which a computer program is stored. When the computer program is executed, it implements the semantic map optimization method for filtering SLAM mapping frames by fusing semantic information as described above.

[0127] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A semantic map optimization method that integrates semantic information to filter SLAM mapping frames, characterized in that: Based on various scenarios of semantic segmentation errors in dynamic scenes, including failure to segment static objects, processing errors between static objects, and incorrect segmentation of static objects, corresponding models and compensation terms are used to evaluate the richness and accuracy of image semantics. Each image in the sequence is scored, and the image with the highest score is finally selected as the mapping frame to realize the construction of a semantic map. For cases where static objects are not segmented, the total score is calculated by comparing the semantic information richness of the images. The implementation method is as follows: Suppose there are m types of static objects in the image. The image sequence has k frames. The number of pixels of the nth category in the i-th image is denoted as . The total number of pixels of the nth category in frame k can be calculated as follows: The average number is ; An offset value is set for each category of each image. ,definition It is the proportion of the difference between the pixel value of the nth object class and the mean value of the pixel value of the i-th image. Simultaneously calculate the pixels of all static objects in k frames. Obtain the proportion of a single category n The coefficient of a single category n is denoted as The total score for obtaining the semantic information of the i-th image. , recorded as in, The score for a single category n of semantic information for the i-th image; Select the image with the highest score as the initial mapping frame, denoted as . Where max() represents taking the maximum value; To address errors in handling static objects and errors in segmenting static objects, the filtering model is optimized as follows. Construct an r×t matrix, where This represents the length and width of the image, specifically the pixel coordinates within the matrix. This corresponds to an m+1 dimensional vector. Each value in the vector represents the cumulative observation frequency of that pixel with respect to a certain category, where the observation frequencies for static object pixel categories are respectively... , corresponding to m categories; the observation frequency of useless background and dynamic object categories is represented as This corresponds to the (m+1)th category; Initially, the observation frequency is initialized to 0. As the image sequence is input sequentially, the pixel category at that coordinate in each input image will be... For a given observation frequency, increment the corresponding observation frequency by one. After the frame sequence is read, in The vector at point 1 records the frequency of each type of target at that point. The frequency of the highest observation is denoted as . : Set threshold parameters Used to determine the semantic consistency of a pixel in a k-frame image sequence, if: This point is considered category x, and the frequency of being classified as any other static object category is considered as the number of incorrect observations. , indicating the first The image was ultimately determined to be number one. The pixels of each category are observed as the sum of the frequencies of pixels of other categories, and this portion is subtracted from the total score (grade) of each image. The modified offset value is: And when This point is considered either a dynamic target or background, and the frequency of determining the effective static object category is considered as redundant observations. , indicating the first The sum of the frequencies at which pixels in an image are ultimately classified as invalid moving targets or background are observed as valid static objects is subtracted from the total score (grade) for each image. The modified offset value is: Finally, the final score for the i-th image is: 。 2. The semantic map optimization method for filtering SLAM mapping frames by fusing semantic information according to claim 1, characterized in that: To reduce the randomness of errors, the segmented image sequence of K=K1 frames is expanded, with the number of expanded image sequences denoted as K=K2 and K=K3, respectively. The corresponding weights are then calculated based on the K value of the image sequence. At that time, a set was obtained The value is calculated simultaneously. : At that time, a set was obtained The value is calculated simultaneously. : When the last set is obtained. value: The final score for each image is as follows: Finally, the maximum score in the image sequence is selected. The corresponding image frame is used as the mapping frame: 。 3. A semantic map optimization system that integrates semantic information to filter SLAM mapping frames, characterized in that: This method is used to implement the semantic map optimization method for filtering SLAM mapping frames by fusing semantic information as described in any one of claims 1-2.

4. The semantic map optimization system for filtering SLAM mapping frames by fusing semantic information according to claim 3, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the semantic map optimization method for filtering SLAM mapping frames by fusing semantic information as described in any one of claims 1-2.

5. The semantic map optimization system for filtering SLAM mapping frames by fusing semantic information according to claim 3, characterized in that: The method includes a readable storage medium on which a computer program is stored. When the computer program is executed, it implements the semantic map optimization method for filtering SLAM mapping frames by fusing semantic information as described in any one of claims 1-2.

Images