Visual odometer method and system based on semantic segmentation

Abstract

The invention belongs to the related technical field of image processing, and discloses a visual odometer method and system based on semantic segmentation. The method comprises the following steps: converting a video to be processed into a plurality of pictures arranged according to a time sequence; performing semantic segmentation on each picture, and extracting feature points and descriptors by a visual odometer; performing feature matching to calculate a conversion matrix between the pictures; and projecting the picture at the t + 1 moment in the time period to the picture at the t moment by using the conversion matrix among the pictures, calculating the displacement of each feature pixel point from the initial moment to the N moment, judging whether the feature pixel point is the feature pixel point on the moving object by using the displacement, if so, deleting the feature pixel point in the picture, and if not, deleting the feature pixel point in the picture. If not, the feature pixel point is reserved in the picture, and N is the total number of moments. According to the invention, the problem that the dynamic moving object cannot be accurately removed from the video with high precision is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to image processing, and more specifically, relates to a visual odometer method and system based on semantic segmentation. Background Art

[0002] Front-end odometer is an important component of SLAM (Simultaneous Localization and Mapping) system, which is mainly divided into two categories, namely visual odometer and laser odometer. Visual odometer estimates the camera pose and motion through images between adjacent frames and restores the spatial structure of the scene; compared with laser odometer, visual odometer performs better in larger scale environments, has more map information in map construction, is easier to implement loop detection to eliminate cumulative errors, and has lower cost.

[0003] Visual odometry has a wide range of applications in the fields of autonomous driving, autonomous exploration, and 3D reconstruction. The core of visual odometry is to extract and match feature points, and its commonly used method is the feature point method. The feature point method is to extract highly recognizable corner points in the image. Feature points are composed of key points and descriptors. Key points refer to the position of feature points in the image. Descriptors are usually vectors that describe the information of pixels around key points based on artificially designed measurements. After decades of development of SLAM technology, ORB features have good quality and performance in current SLAM solutions, but ORB feature points are greatly affected by image quality. Surface features, especially in environments with unclear textures and dynamic environments, will affect the extraction and matching of feature points. As a result, when extracting feature points during the flight of unmanned aerial vehicles, the characteristics of different ground environments and dynamic objects in the ground environment will cause the front-end odometer feature point matching errors and uneven distribution of matching points, resulting in poor results and ultimately affecting the accuracy of positioning.

[0004] The traditional SLAM field attempts to introduce semantic information on its basis, so that the machine can understand the environment more deeply. The existing SLAM technology uses semantic information mainly in dynamic environments. The semantic segmentation object is segmented from the camera data, and dynamic obstacles are detected and removed, which improves the accuracy of state estimation in dynamic scenes. However, the existing technology simply divides dynamic objects, roughly considers vehicles as dynamic objects and removes them, and does not distinguish whether the object itself is dynamic and removes it roughly. As a result, too many semantic masks are removed in crowded scenes with a large number of vehicles, so that the visual odometer cannot extract enough feature points from the current frame, resulting in a decrease in positioning accuracy. Therefore, there is an urgent need for a high-precision method for removing moving objects from videos. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a visual odometer method and system based on semantic segmentation to solve the problem that dynamic moving objects cannot be removed from videos accurately and with high precision.

[0006] To achieve the above object, according to one aspect of the present invention, a visual odometer method based on semantic segmentation is provided, the method comprising the following steps:

[0007] Convert the video to be processed into multiple pictures arranged in chronological order;

[0008] For each image, the pre-trained semantic segmentation network is used to obtain the coordinates of each pixel in the image and the environmental label of the pixel; at the same time, the image is input into the visual odometer for feature extraction to obtain the feature pixels in the image and the descriptors corresponding to the feature pixels, thereby obtaining the coordinates, environmental labels and descriptors corresponding to the feature pixels in each image;

[0009] For different pictures, feature matching is performed using the environment labels and descriptors of the feature pixels in each picture, so as to obtain the feature pixels that each feature pixel matches in different pictures, and the transformation matrix between each picture is calculated using the coordinates of the matched feature pixels;

[0010] The transformation matrix between each picture is used to project the picture at time t+1 in the time period onto the picture at time t, so as to calculate the displacement of each feature pixel from the initial time to time N. The displacement is used to determine whether the feature pixel is a feature pixel on the moving object. If so, the feature pixel is deleted from the picture. If not, the feature pixel is retained in the picture. N is the total number of time points.

[0011] Further preferably, before the image is input into the semantic segmentation network, the image is compressed and Gaussian smoothed in sequence.

[0012] Further preferably, the pixels of the compressed image are calculated according to the following formula:

[0013]

[0014] Among them, I′(x′, y′) is the pixel value at the pixel position (x′, y′) in the compressed image, I(x′, y′) is the pixel value of the area corresponding to the pixel position (x', y′) in the image before compression, and θ is the pixel centroid direction of the area corresponding to the pixel position (x'', y′) in the image before compression.

[0015] Further preferably, the pre-trained semantic segmentation network adopts a deep learning model, which includes multiple convolutional layers.

[0016] Further preferably, the expansion rates of the first three convolutional layers in the deep learning model are as follows:

[0017] θ1=θ2-2σ

[0018]

[0019] θ3=θ2+2σ

[0020] Where [·] is the smallest integer greater than ·, θ1 is the dilation rate of the first layer, θ2 is the dilation rate of the second layer, θ3 is the dilation rate of the third layer, w is the height of the image, and σ is the aspect ratio of the image.

[0021] Further preferably, the feature matching is performed according to the following steps:

[0022] For a group of feature pixels with the same environmental label in different images, the Hamming distance between the descriptors of the group of feature pixels is calculated, and the Hamming distance is compared with a preset threshold. If it is less than or equal to the preset threshold, the group of feature pixels matches, otherwise it does not match.

[0023] Further preferably, the preset threshold is calculated according to the following formula:

[0024]

[0025] Among them, M is the preset threshold, L min is the minimum Hamming distance, a is the proportion of vegetation and water in the picture, and b is the proportion of roads and buildings in the picture.

[0026] Further preferably, during feature matching, if the feature pixel point to be matched is an edge point, Laplacian sharpening is performed on the area where the edge point is located to make the edge clearer.

[0027] Further preferably, whether the feature pixel point is a pixel point on a moving object is determined by calculating the loss function value between the displacement of the dynamic environment label and the displacement of the static environment label. If the calculated loss function value is greater than a preset acceptable threshold, the object with the dynamic environment label is a moving object, otherwise the object is not a moving object.

[0028] According to another aspect of the present invention, a system for the above-mentioned visual odometer method based on semantic segmentation is provided, characterized in that the system includes a video processing module, a semantic segmentation module, a visual odometer module, a feature matching module and a dynamic and static detection module, wherein:

[0029] The video processing module is used to convert the video into a plurality of pictures arranged in chronological order;

[0030] The semantic segmentation module is used to obtain the coordinates of each pixel in the image and the environmental label of the pixel;

[0031] The visual odometer is used to perform feature extraction to obtain feature pixels and descriptors corresponding to the feature pixels;

[0032] The feature matching module is used to match feature pixels between different pictures;

[0033] The dynamic and static detection module is used to determine whether the characteristic pixel points in the image are characteristic pixel points on a moving object.

[0034] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0035] 1. The present invention adopts a semantic segmentation network to identify environmental labels, adopts a visual odometer to obtain feature pixels and descriptors, and then calculates the moving distances of different environmental labels after feature matching, and judges whether the feature pixels are points on a moving object, thereby improving the accuracy of judgment. For objects with dynamic environmental labels but not moving at the current moment, the above method can be used to accurately identify them, avoiding the error caused by directly judging moving objects through environmental labels, thereby improving the recognition accuracy;

[0036] 2. The present invention compresses the image before semantic segmentation to reduce the image size, effectively reducing the amount of calculation during semantic segmentation and feature pixel extraction and matching, improving the real-time performance of the system. At the same time, the image reduction method based on regional interpolation effectively avoids the appearance of ripples or stripes in the reduced image, and preserves the image edge features to the greatest extent, thereby improving the accuracy of subsequent image processing;

[0037] 3. When performing feature matching, the present invention first selects feature pixels with the same environmental labels for matching, and only the same environmental labels are screened. This method can effectively avoid incorrectly matched feature pixel pairs and improve the speed of feature pixel matching;

[0038] 4. Compared with the traditional semantic simultaneous localization and mapping algorithm in a dynamic environment, the present invention distinguishes between dynamic and static, making the algorithm more robust to the environment and avoiding the reduction in the accuracy of visual odometer state estimation caused by excluding too many areas in the traditional algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of a visual odometer method based on semantic segmentation constructed according to a preferred embodiment of the present invention;

[0040] Figure 2 is a flowchart of semantic segmentation constructed according to a preferred embodiment of the present invention;

[0041] Figure 3 is a flow chart of feature matching constructed according to a preferred embodiment of the present invention;

[0042] Figure 4 It is a schematic diagram of the dynamic and static matching process constructed according to the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0044] like Figure 1 As shown, the visual odometer method based on semantic segmentation described in the present invention comprises the following steps:

[0045] Step 1: Decode the video stream with the timestamp, decode the video stream into a picture with the timestamp, and perform picture compression;

[0046] Furthermore, the image compressor compresses the size of a single frame of an image and performs smoothing on the compressed image.

[0047] In one embodiment of the present invention, the UAV uses a monocular camera to shoot in the air, and the video with a time stamp;

[0048] Compress the image to obtain the size and pixels at different positions; reduce the image size from 1920×1080 to 960×540, design an image reduction method based on regional interpolation with improved edge characteristics, and regard each pixel in the compressed image as a representative of a region of the image before compression, so as to retain the characteristics of the original image to the greatest extent;

[0049] Set the compression ratio, reduce the image size from 1920×1080 to 960×540, and the compression ratios in width and height are both 2; for the pixel position (x′, y′) in the compressed image, the pixel value is I′(x′, y′); calculate the area B corresponding to the image before compression as [2x′, 2(x′+1))×[2y′, 2(y′+1)), and the pixel value of the pixels in the area is I(x′, y′);

[0050] Define the moment of the region as m p，q =∑ x′，y′∈B x p y qI(x′,y′),p,q={0,1};

[0051] The pixel centroid direction of the calculation area is θ = arctan (m 01 / m 10 );

[0052] The original image is compressed in a certain area, and the corresponding pixel values ​​are as follows:

[0053]

[0054] Step 2: Use the semantic segmentation network to perform semantic segmentation on the compressed image, and store the results of the semantic segmentation, including environmental labels, area coordinates, etc.; while semantic segmentation is in progress, the visual odometer works simultaneously to extract feature pixels, and processes the same frame of the image as the semantic segmentation, extracts feature pixels using ORB features, and calculates the descriptors of the feature pixels;

[0055] The semantic segmentation network segments and marks pixels in a single frame image, annotating the environmental label information of the pixels; environmental labels include vegetation, buildings, waters, roads, vehicles, etc.

[0056] In one embodiment of the present invention, Figure 2 As shown, a semantic segmentation network based on the Deeplab V3+ deep learning model with adaptive expansion rate is designed to perform semantic segmentation on common ground environment features in compressed images, including vegetation, buildings, waters, roads, and vehicles.

[0057] The image has a width and height of w and h, and an aspect ratio of Where [·] represents the smallest integer greater than ·, and assigns the expansion rates θ1, θ2, and θ3 of the first three layers of the N×N convolutional layer in the semantic segmentation network; in this embodiment, a 3×3 convolutional layer is used, as follows:

[0058]

[0059] θ1=θ2-2σ;

[0060] θ3=θ2+2σ;

[0061] Here, [·] represents the smallest integer greater than ·.

[0062] Obtain the pixel position of each pixel p in a single frame image i through semantic segmentation and environmental label categories When semantic segmentation is working, the visual odometry works at the same time, processing the same frame of image with semantic segmentation. The image is input into the visual odometry, and the feature pixels and the descriptors corresponding to the feature pixels are output. The feature pixels are extracted using ORB features, and then the environment label is calculated with the feature pixel descriptor to obtain the descriptor. Merged feature pixels Perform feature pixel matching.

[0063] Step 3: Use the results of descriptors and semantic segmentation to match feature pixels between pictures in different frames; estimate the pose of the matched feature pixels to obtain the transformation matrix between pictures in different frames;

[0064] In one embodiment of the present invention, Figure 3 As shown, the feature pixels between different frames are used When performing preliminary screening for matching of feature pixel pairs, the environment labels are searched, and only the same environment labels are screened. This method can effectively avoid incorrect matching of feature pixel pairs and improve the matching speed of feature pixels. After preliminary screening of feature pixels based on semantic information and the grayscale values ​​of the four pixels around the feature pixels, different matching strategies are used according to the semantically segmented areas in traditional methods because of the different features in different areas.

[0065] For buildings and road areas with clear texture and obvious features, matching is performed by comparing the Hamming distance between descriptors with the threshold T;

[0066] For low-texture areas such as vegetation and water:

[0067] (1) For the matching of feature pixels in the region, an adaptive threshold M is used to ensure that the feature pixels with relatively obvious features in the low-texture region are matched and that there are enough feature pixel pairs when matching multiple frames of images;

[0068] The setting of threshold M is adjusted according to the proportion of vegetation, water area, road and building in the image. The proportion of vegetation and water area in the image is a%, and the proportion of road and building area in the image is b%. The minimum Hamming distance calculated when key points are matched is L min , adjust the threshold

[0069] (2) For the points at the edge of the region, the edge region is first sharpened using the Laplacian operator to make the low-texture region have stronger edges and clearer details. This allows for better matching results when performing feature pixel matching. The above matching method ensures that there are enough correctly matched feature pixel pairs between frames, ensuring higher accuracy in the subsequent transformation matrix.

[0070] Step 4: Use the transformation matrix between each image to project the image at time t+1 in the time period onto the image at time t, so as to calculate the displacement of each characteristic pixel from the initial time to time N, and use the displacement to determine whether the characteristic pixel is a characteristic pixel on a moving object. If so, delete the characteristic pixel in the image, if not, keep the characteristic pixel in the image, and N is the total number of moments. This realizes dynamic and static detection of objects that may be dynamic in the original image, removes the dynamic area, and retains the static area.

[0071] In one embodiment of the present invention, Figure 4 As shown in the figure, feature pixels with environment labels of vehicles and buildings (static roads are also acceptable) in the last frame of the current time series are selected, and the positions of their pixels are obtained. Vehicles: architecture: The transformation matrix results obtained by state estimation are reprojected to the previous frame in the time series in sequence until they are reprojected to the first frame to obtain the coordinate vehicle: architecture: At this time and its real coordinates architecture: For comparison,

[0072] Calculate the loss function between the vehicle and building displacement, set the threshold T, and when the difference between the two loss functions is greater than T, the vehicle is considered to be a dynamic vehicle, and the pixel points of the vehicle are removed. Otherwise, the area here is retained; in the scene of crowded vehicles, more accurate state estimation and positioning are achieved.

[0073] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual odometry method based on semantic segmentation, characterized in that: The method comprises the following steps: Convert the video to be processed into multiple pictures arranged in chronological order; For each image, the pre-trained semantic segmentation network is used to obtain the coordinates of each pixel in the image and the environmental label of the pixel; at the same time, the image is input into the visual odometer for feature extraction to obtain the feature pixels in the image and the descriptors corresponding to the feature pixels, thereby obtaining the coordinates, environmental labels and descriptors corresponding to the feature pixels in each image; For different pictures, feature matching is performed using the environment labels and descriptors of the feature pixels in each picture, so as to obtain the feature pixels that each feature pixel matches in different pictures, and the transformation matrix between each picture is calculated using the coordinates of the matched feature pixels; The transformation matrix between each picture is used to project the picture at time t+1 in the time period onto the picture at time t, so as to calculate the displacement of each feature pixel from the initial time to time N. The displacement is used to determine whether the feature pixel is a feature pixel on the moving object. If so, the feature pixel is deleted from the picture. If not, the feature pixel is retained in the picture. N is the total number of time points.

2. The visual odometer method based on semantic segmentation according to claim 1, characterized in that: Before the image is input into the semantic segmentation network, the image is compressed and Gaussian smoothed in turn.

3. The visual odometer method based on semantic segmentation as claimed in claim 2, characterized in that: The pixels of the compressed image are calculated according to the following formula: Among them, I′(x′, y′) is the pixel value at the pixel position (x′, y′) in the compressed image, I(x′, y′) is the pixel value of the area corresponding to the pixel position (x′, y′) in the image before compression, and θ is the pixel centroid direction of the area corresponding to the pixel position (x′, y′) in the image before compression.

4. The visual odometer method based on semantic segmentation according to claim 1, characterized in that: The pre-trained semantic segmentation network adopts a deep learning model, which includes multiple convolutional layers.

5. The visual odometer method based on semantic segmentation as claimed in claim 4, characterized in that: The expansion rates of the first three convolutional layers in the deep learning model are as follows: θ1=θ2-2σ θ3=θ2+2σ Where [·] is the smallest integer greater than ·, θ1 is the dilation rate of the first layer, θ2 is the dilation rate of the second layer, θ3 is the dilation rate of the third layer, w is the height of the image, and σ is the aspect ratio of the image.

6. The visual odometer method based on semantic segmentation according to claim 1, characterized in that: The feature matching is performed according to the following steps: For a group of feature pixels with the same environmental label in different images, the Hamming distance between the descriptors of the group of feature pixels is calculated, and the Hamming distance is compared with a preset threshold. If it is less than or equal to the preset threshold, the group of feature pixels matches, otherwise it does not match.

7. The visual odometer method based on semantic segmentation according to claim 6, characterized in that: The preset threshold is calculated according to the following formula: Among them, M is the preset threshold, L min is the minimum Hamming distance, a is the proportion of vegetation and water in the picture, and b is the proportion of roads and buildings in the picture.

8. The visual odometer method based on semantic segmentation according to claim 1, characterized in that: During feature matching, if the feature pixel point to be matched is an edge point, Laplace operator sharpening is performed on the area where the edge point is located to make the edge clearer.

9. The visual odometer method based on semantic segmentation according to claim 1, characterized in that: To determine whether a feature pixel is a pixel on a moving object, the loss function value between the displacement of the dynamic environment label and the displacement of the static environment label is calculated. If the calculated loss function value is greater than the preset acceptable threshold, the object with the dynamic environment label is a moving object, otherwise the object is not a moving object.

10. A system using the visual odometer method based on semantic segmentation according to any one of claims 1 to 9, characterized in that: The system includes a video processing module, a semantic segmentation module, a visual odometer module, a feature matching module, and a dynamic and static detection module, among which: The video processing module is used to convert the video into a plurality of pictures arranged in chronological order; The semantic segmentation module is used to obtain the coordinates of each pixel in the image and the environmental label of the pixel; The visual odometer is used to perform feature extraction to obtain feature pixels and descriptors corresponding to the feature pixels; The feature matching module is used to match feature pixels between different pictures; The dynamic and static detection module is used to determine whether the characteristic pixel points in the image are characteristic pixel points on a moving object.

Images