A semantic segmentation based visual odometry method and system

Abstract

The application belongs to the technical field of image processing, and discloses a visual odometer method and system based on semantic segmentation. The method comprises the following steps: converting a video to be processed into multiple pictures arranged in time sequence; performing semantic segmentation and visual odometer extraction feature points and descriptors on each picture; performing feature matching to calculate the conversion matrix between the pictures; projecting the picture at time t+1 in the time period onto the picture at time t using the conversion matrix between the pictures to calculate the displacement of each feature pixel point from the initial time to time N, and using the displacement to determine whether the feature pixel point is a feature pixel point on a moving object, if yes, deleting the feature pixel point in the picture, and if no, keeping the feature pixel point in the picture, wherein N is the total number of times. Through the application, the problem that dynamic moving objects cannot be accurately and highly precisely removed from a video is solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field related to image processing, and more particularly to a visual odometry method and system based on semantic segmentation. BACKGROUND

[0002] Front-end odometry is an important part of the SLAM (Simultaneous Localization and Mapping) system, which is mainly divided into two categories, namely visual odometry and laser odometry. Visual odometry estimates the camera pose and motion between adjacent frames through images, and restores the spatial structure of the scene; compared with laser odometry, visual odometry performs better in a larger scale environment, has more map information in map construction, is easier to realize loop detection to eliminate cumulative errors, and has lower cost.

[0003] Visual odometry has a wide range of applications in automatic driving, autonomous exploration and three-dimensional reconstruction. The core of visual odometry is to extract and match feature points, and the commonly used method is feature point method. Feature point method is to extract corner points with strong recognition in images. Feature points are composed of key points and descriptors. Key points refer to the position of feature points in images, and descriptors are usually a vector that describes the information of pixels around the key points according to human-designed measurements. After decades of development of SLAM technology, ORB features have good quality and performance in current SLAM solutions, but ORB feature points are greatly affected by image quality. Surface features, especially in environments with unobvious textures and dynamic environments, will affect feature point extraction and matching, resulting in errors in feature point matching and uneven distribution of matching points in the front-end odometry, thus poor effect and ultimately affecting the accuracy of positioning.

[0004] Traditional SLAM field attempts to introduce semantic information on its basis, so that the machine can understand the environment more deeply. The existing SLAM technology uses semantic information mainly in dynamic environments. Semantic segmentation objects are segmented with camera data, dynamic obstacles are detected and removed, and the accuracy of state estimation is improved in dynamic scenes, but the existing technology simply divides dynamic objects, roughly considers vehicles as dynamic objects and removes them, and does not distinguish whether the object itself is dynamic, which results in the removal of too many semantic masks in crowded scenes with a large number of vehicles, so that the visual odometry cannot extract enough feature points from the current frame, and the positioning accuracy is reduced. Therefore, there is an urgent need for a method for removing moving objects from video with high precision. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a visual odometer method and system based on semantic segmentation, which solves the problem of being unable to accurately and highly precisely remove dynamic moving objects from a video.

[0006] To achieve the above object, according to one aspect of the present application, a visual odometer method based on semantic segmentation is provided, which comprises the following steps:

[0007] converting a video to be processed into a plurality of pictures arranged in time sequence;

[0008] For each picture, a pre-trained semantic segmentation network is used to obtain the coordinates of each pixel point in the picture and the environmental label of the pixel point; meanwhile, the picture is input into a visual odometer for feature extraction to obtain a feature pixel point in the picture and a descriptor corresponding to the feature pixel point, thereby obtaining the coordinates, the environmental label and the descriptor corresponding to the feature pixel point in each picture;

[0009] For different pictures, the environmental label and the descriptor of the feature pixel point in each picture are used for feature matching to obtain the matching feature pixel points of each feature pixel point in different pictures, and the coordinates of the matching feature pixel points are used to calculate the conversion matrix between pictures;

[0010] the picture at time t+1 in the time period is projected onto the picture at time t by using the conversion matrix between pictures, so as to calculate the displacement of each feature pixel point from the initial time to time N, and the displacement is used to judge whether the feature pixel point is on a moving object, if yes, the feature pixel point is deleted in the picture, and if not, the feature pixel point is retained in the picture, N being the total number of times.

[0011] Further preferably, before the picture is input into the semantic segmentation network, the picture is sequentially subjected to picture compression and Gaussian smoothing.

[0012] Further preferably, the pixels after picture compression are calculated according to the following formula:

[0013]

[0014] wherein I'(x',y') is the pixel value at pixel position (x',y') in the compressed picture, I(x',y') is the pixel value of the region corresponding to pixel position (x',y') in the picture before compression, and θ is the direction of the pixel centroid of the region corresponding to pixel position (x',y') in the picture before compression.

[0015] Further preferably, the pre-trained semantic segmentation network adopts a deep learning model, which comprises a plurality of convolutional layers.

[0016] Further preferably, the expansion rates of the first three convolutional layers in the deep learning model are as follows:

[0017] θ1 = θ2 - 2σ

[0018]

[0019] θ3 = θ2 + 2σ

[0020] where [·] is the smallest integer greater than ·, θ1 is the expansion rate of the first layer, θ2 is the expansion rate of the second layer, θ3 is the expansion rate of the third layer, w is the height of the picture, and σ is the aspect ratio of the picture.

[0021] Further preferably, the feature matching is performed according to the following steps:

[0022] For a group of feature pixel points with the same environment label in different pictures, the Hamming distance between the feature pixel point descriptors of the group is calculated, and the Hamming distance is compared with a preset threshold. If the Hamming distance is less than or equal to the preset threshold, the group of feature pixel points is matched, otherwise, the group of feature pixel points is not matched.

[0023] Further preferably, the preset threshold is calculated according to the following formula:

[0024]

[0025] where M is the preset threshold, L min is the minimum Hamming distance, a is the proportion of vegetation and water area in the picture, and b is the proportion of road and building in the picture.

[0026] Further preferably, in the feature matching, if the feature pixel point to be matched is an edge point, a Laplacian operator sharpening is performed on the region where the edge point is located to make the edge clearer.

[0027] Further preferably, whether a feature pixel point is a pixel point on a moving object is determined by calculating a loss function value between the displacement of a dynamic environment label and the displacement of a static environment label. If the calculated loss function value is greater than a preset acceptable threshold, the object with the dynamic environment label is a moving object, otherwise, the object is not a moving object.

[0028] According to another aspect of the present application, a system of the above-mentioned semantic segmentation-based visual odometry method is provided, characterized in that the system comprises a video processing module, a semantic segmentation module, a visual odometry module, a feature matching module, and a dynamic and static detection module, wherein:

[0029] The video processing module is configured to convert a video into a plurality of pictures arranged in time sequence.

[0030] The semantic segmentation module is used to obtain the coordinates of each pixel point in the picture and the environmental label of the pixel point.

[0031] The visual odometer is used to perform feature extraction to obtain feature pixel points and descriptors corresponding to the feature pixel points.

[0032] The feature matching module is used to match the feature pixel points between different pictures.

[0033] The dynamic and static detection module is used to determine whether the feature pixel points in the picture are feature pixel points on a moving object.

[0034] Overall, compared with the prior art, the above technical scheme conceived by the present application has the following beneficial effects:

[0035] 1. In the present application, the semantic segmentation network is used for environmental label recognition, the visual odometer is used to obtain feature pixel points and descriptors, then the feature matching is performed to calculate the moving distance of different environmental labels, and it is determined whether the feature pixel points are points on a moving object, thereby improving the accuracy of the determination, and the method can accurately identify the object which has a dynamic environmental label but does not move at the current time, thereby avoiding the error caused by directly determining the moving object based on the environmental label and improving the identification precision.

[0036] 2. In the present application, the picture is compressed before semantic segmentation, and the picture size is reduced, thereby effectively reducing the calculation amount when performing semantic segmentation and feature pixel point extraction and matching, improving the real-time performance of the system, and effectively avoiding the appearance of wave or stripe-shaped artifacts in the reduced image based on the region interpolation picture reduction method, and the characteristics of the image edge are preserved to the greatest extent, thereby improving the accuracy of subsequent image processing.

[0037] 3. In the present application, the feature pixel points with the same environmental label are selected for matching when performing feature matching, and only the feature pixel points with the same environmental label are screened, thereby effectively avoiding the error matching of the feature pixel points and improving the speed of feature pixel point matching.

[0038] 4. Compared with the traditional semantic simultaneous localization and mapping algorithm, the present application distinguishes between dynamic and static in the dynamic environment, has higher robustness of the algorithm to the environment, and avoids the decrease in the accuracy of the visual odometer state estimation caused by the removal of too many regions in the traditional algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of a semantic segmentation-based visual odometer method constructed according to the preferred embodiment of the present application;

[0040] Figure 2 is a flowchart of semantic segmentation constructed according to the preferred embodiment of the present application;

[0041] Figure 3 is a flow chart of feature matching constructed according to a preferred embodiment of the present application;

[0042] Figure 4 is a flow chart of dynamic-static matching constructed according to a preferred embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0044] As shown in Figure 1 , the visual odometer method based on semantic segmentation according to the present application comprises the following steps:

[0045] Step 1, video stream with timestamp is video decoded, the video stream is decoded into pictures with timestamp, and a picture compression sub is performed;

[0046] Further, the picture compression sub is to compress the size of a single frame picture, and to perform smoothing processing on the compressed picture.

[0047] In an embodiment of the present application, an unmanned aerial vehicle takes pictures in the air using a monocular camera, and a video stream with timestamp is obtained;

[0048] The picture is compressed to obtain the size and the pixels at different positions; the picture size 1920x1080 is reduced to 960x540, a picture reduction method based on improved edge feature region interpolation is designed, each pixel in the compressed picture is regarded as a representative of a region of the picture before compression, and the features of the original picture are preserved to the greatest extent;

[0049] The compression ratio is set, the picture size 1920x1080 is reduced to 960x540, and the compression ratio in width and height is both 2; for the pixel position (x', y') in the compressed picture and the pixel value I'(x', y'), the region B corresponding to the picture before compression before compression is calculated as [2x', 2(x'+1))x[2y', 2(y'+1)), and the pixel value of the pixels in the region is I(x', y');

[0050] The moment of the region is defined as m p，q =∑ x′，y′∈B x p y qI(x',y'),p,q={0,1}

[0051] Calculate the pixel centroid direction of the region θ=arctan(m 01 / m 10 );

[0052] The original picture is regionally compressed, and the corresponding pixel value is as follows:

[0053]

[0054] Step two, using a semantic segmentation network, performing semantic segmentation on the compressed picture, and storing the semantic segmentation results, including environment labels, region coordinates, etc. At the same time of semantic segmentation, the visual odometry works to extract feature pixels, and the same picture is processed with semantic segmentation. Feature pixels are extracted using ORB features, and the descriptors of the feature pixels are calculated.

[0055] The semantic segmentation network performs segmentation on a single picture to label the pixel points, and labels the environment label information of the pixel points. The environment label includes vegetation, building, water area, road, vehicle, etc.

[0056] In an embodiment of the present application, as Figure 2 shown, a semantic segmentation network based on a DeepLab V3+ deep learning model with adaptive dilation rate is designed to perform semantic segmentation on common ground environment features in the compressed picture, including vegetation, building, water area, road and vehicle.

[0057] The size of the picture is w, h, and the aspect ratio is where [·] represents the smallest integer greater than ·. The dilation rates θ1, θ2, θ3 of the first three layers of the N×N convolution layer in the semantic segmentation network are assigned values. In this embodiment, a 3×3 convolution layer is used, as follows:

[0058]

[0059] θ1=θ2-2σ;

[0060] θ3=θ2+2σ;

[0061] where [·] represents the smallest integer greater than ·.

[0062] Through semantic segmentation, the pixel position and the environment label category of each pixel point p in a single picture i are obtained During semantic segmentation, visual odometry operates concurrently, processing the same image frame as semantic segmentation. The image is input into visual odometry, which outputs feature pixels and their corresponding descriptors. Feature pixels are extracted using ORB features, and then descriptors are calculated by combining the environment label with the feature pixel descriptors. Merged feature pixels Perform feature pixel matching.

[0063] Step 3: Use the descriptors and semantic segmentation results to match feature pixels in different frames; perform pose estimation on the matched feature pixels to obtain the transformation matrix between different frames;

[0064] In one embodiment of the present invention, such as Figure 3 As shown, the feature pixels between different frames are utilized. The information is matched. When initially screening feature pixel pairs, the search is based on the environment label. Only those with the same environment label are screened. This method can effectively avoid incorrectly matched feature pixel pairs and improve the speed of feature pixel matching. After the initial screening of feature pixels based on semantic information and the gray values ​​of the four pixels around the feature pixel, different matching strategies are used according to the semantically segmented regions because the features in different regions are different.

[0065] For building and road areas with clear textures and distinct features, matching is performed by comparing the Hamming distance between descriptors with a threshold T.

[0066] For low-texture areas such as vegetation and water:

[0067] (1) For feature pixel matching within the region, an adaptive threshold M is used to ensure that feature pixels with relatively obvious features are matched in low-texture regions, while ensuring that there are enough feature pixel pairs when matching multiple frames of images.

[0068] The threshold M is adjusted based on the proportions of vegetation, water, roads, and buildings in the image. The proportion of vegetation and water in the image is a%, and the proportion of roads and buildings is b%. The minimum Hamming distance calculated during keypoint matching is L. min Adjust the threshold

[0069] (2) For the points of the edge of the region, first, the Laplacian operator sharpening is carried out on the edge region, so that the low-texture region has stronger edges and clearer details, and when the feature pixel point matching is carried out again, better matching effect is obtained. According to the above matching method, enough correct matching feature pixel point pairs between frames can be ensured, and higher accuracy of the subsequent conversion matrix is ensured.

[0070] Step four, the picture at t+1 time in the time period is projected onto the picture at t time by using the conversion matrix between the pictures, so as to calculate the displacement of each feature pixel point from the initial time to N time, and the displacement is used to judge whether the feature pixel point is on the moving object, if yes, the feature pixel point is deleted in the picture, if not, the feature pixel point is retained in the picture, and N is the total number of times. The dynamic and static detection of the object that may be dynamic in the original picture is realized, the dynamic region is removed, and the static region is retained.

[0071] In an embodiment of the present application, as shown in Figure 4 the feature pixel points of the environment label of the last frame in the current time sequence are selected respectively, and the positions of the pixel points are obtained, the vehicle: the building: The conversion matrix result obtained by state estimation is sequentially re-projected into the previous frame in the time sequence until the first frame is re-projected to obtain the coordinates of the vehicle: the building: At this time, the real coordinates of the vehicle: the building: are compared,

[0072] The loss function between the displacements of the vehicle and the building is calculated, a threshold T is set, when the difference between the two loss functions is greater than T, the vehicle at this time is considered to be a dynamic vehicle, then the pixel points of the vehicle are removed, and conversely, the region is retained; In the crowded scene of the vehicle, more accurate state estimation and positioning are obtained.

[0073] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A visual odometry method based on semantic segmentation, characterized in that, The method includes the following steps: Convert the video to be processed into multiple images arranged in chronological order; For each image, the coordinates and environment label of each pixel in the image are obtained using a pre-trained semantic segmentation network. At the same time, the image is input into a visual odometry system for feature extraction to obtain the feature pixels and their corresponding descriptors. Thus, the coordinates, environment label, and descriptor of each feature pixel in the image are obtained. For different images, feature matching is performed using the environment label and descriptor of the feature pixels in each image to obtain the feature pixels that match each feature pixel in different images. The transformation matrix between the images is then calculated using the coordinates of the matched feature pixels. The transformation matrix between the images is used to project the image at time t+1 in the time interval onto the image at time t. The displacement of each feature pixel from the initial time to time N is calculated. The displacement is used to determine whether the feature pixel is a feature pixel on the moving object. If it is, the feature pixel is deleted from the image. If not, the feature pixel is retained in the image. N is the total number of time intervals. The feature matching is performed according to the following steps: For a set of feature pixels with the same environment label in different images, calculate the Hamming distance between the descriptors of the set of feature pixels, and compare the Hamming distance with a preset threshold. If the Hamming distance is less than or equal to the preset threshold, the set of feature pixels is matched; otherwise, they are not matched. The preset threshold is calculated according to the following formula: M = max(1.5 ×min(1+ ,2)),30×min(1+ ,2))) Where M is a preset threshold, is the minimum Hamming distance, a is the proportion of vegetation and water in the image, and b is the proportion of roads and buildings in the image.

2. The visual odometry method based on semantic segmentation as described in claim 1, characterized in that, Before the images are input into the semantic segmentation network, they are compressed and Gaussian smoothed sequentially.

3. The visual odometry method based on semantic segmentation as described in claim 2, characterized in that, The pixels of the compressed image are calculated according to the following formula: （ ， ）= （ （ ， ）× + （ +1， ）× + （ ， +1）× + （ +1， +1）× ） in, ( , ) is the pixel position in the compressed image ( , The pixel value at ) ( , ) is the pixel position in the image before compression. , ) corresponds to the pixel value of the region, where θ is the pixel value in the image before compression relative to the pixel position ( , The direction of the centroid of the pixels in the corresponding region.

4. The visual odometry method based on semantic segmentation as described in claim 1, characterized in that, The pre-trained semantic segmentation network employs a deep learning model, which includes multiple convolutional layers.

5. A visual odometry method based on semantic segmentation as described in claim 4, characterized in that, The dilation rates of the first three convolutional layers in the deep learning model are as follows: = - 2p = [(( -3) / 2)+1] = + 2σ Where [·] is the smallest integer greater than ·. It is the expansion rate of the first layer. It is the expansion rate of the second layer. It is the expansion rate of the third layer. σ is the height of the image, and σ is the aspect ratio of the image.

6. The visual odometry method based on semantic segmentation as described in claim 1, characterized in that, In feature matching, if the feature pixel to be matched is an edge point, the Laplacian operator is used to sharpen the region where the edge point is located, making the edge clearer.

7. The visual odometry method based on semantic segmentation as described in claim 1, characterized in that, To determine whether a feature pixel is a pixel on a moving object, the loss function value between the displacement of the dynamic environment label and the displacement of the static environment label is calculated. If the calculated loss function value is greater than a preset acceptable threshold, the object with the dynamic environment label is a moving object; otherwise, the object is not a moving object.

8. A system utilizing the visual odometry method based on semantic segmentation as described in any one of claims 1-7, characterized in that, The system includes a video processing module, a semantic segmentation module, a visual odometry module, a feature matching module, and a motion / static detection module, among which: The video processing module is used to convert the video into multiple images arranged in chronological order; The semantic segmentation module is used to obtain the coordinates of each pixel in the image and the environment label of that pixel; The visual odometry is used to extract features to obtain feature pixels and descriptors corresponding to the feature pixels. The feature matching module is used to match feature pixels between different images; The static and dynamic detection module is used to determine whether the feature pixels in the image are feature pixels on a moving object.

Images