Dynamic environment navigation method and system based on semantic and geometric fusion information

Abstract

The present invention discloses a dynamic environment navigation method and system based on semantic and geometric fusion information, which relates to the technical field of dynamic environment intelligent navigation, including: obtaining continuous frame images of a dynamic environment and preprocessing them; inputting the preprocessed continuous frame images into an improved target detection model to identify the categories of objects in the dynamic environment; extracting and matching feature points in the preprocessed continuous frame images, using an improved GC-RANSAC algorithm to distinguish inliers and inliers in the matched feature points, and calculating a fundamental matrix based on the inliers; based on the fundamental matrix, using epipolar constraints and multi-view geometric constraints, combined with the semantic information of the identified object category, screening and determining static feature points among the feature points; estimating the camera pose based on the screened static feature points, then optimizing them using a BA optimization algorithm, constructing an environment map based on the optimized results, and performing dynamic environment navigation. The present invention can achieve precise positioning and navigation in dynamic environments.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic environment intelligent navigation, and in particular to a dynamic environment navigation method and system based on semantic and geometric fusion information. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, simultaneous localization and mapping (SLAM) technology has rapidly developed and been widely used in fields such as mobile robots, drones, self-driving cars, augmented reality, and virtual reality. With the improvement of computing power and the advancement of sensor technology, SLAM technology has gradually evolved from the initial 2D SLAM based on lidar to multimodal fusion technologies such as visual SLAM and RGB-D SLAM. Among them, visual SLAM has become a hot topic of research and application due to its low hardware cost and wide applicability. However, traditional SLAM technology still faces challenges in dealing with dynamic environments, weak texture scenes, and real-time performance, including:

[0004] (1) Most SLAM systems are based on the assumption of a static environment. However, when faced with a dynamic environment, dynamic objects such as pedestrians and vehicles that frequently appear in actual scenes will cause a large number of feature matching errors, resulting in serious deviations in the system's state estimation, affecting the accuracy of trajectory estimation and the robustness of the system.

[0005] (2) When dealing with dynamic objects, existing SLAM algorithms lack specialized detection and processing methods for dynamic objects, resulting in an inability to effectively distinguish between dynamic and static objects. Consequently, dynamic feature points are often mistakenly used for pose estimation, causing trajectory drift and positioning errors. Therefore, when dealing with object detection in dynamic environments, deep learning models are currently commonly used for detection. However, the high computational requirements of these models pose a huge challenge to embedded systems and real-time applications. Most dynamic SLAM methods, while pursuing accuracy, often sacrifice real-time performance, making it difficult to meet the needs of application scenarios that require rapid response, such as mobile robots and self-driving cars.

[0006] (3) Existing visual SLAM systems have difficulty extracting sufficiently effective feature points in weak texture environments, drastic lighting changes (such as overexposure or low light), and blurred images, which significantly affects the positioning accuracy and system stability. Summary of the Invention

[0007] To address the deficiencies of the above-mentioned prior art, the present invention provides a dynamic environment navigation method and system based on semantic and geometric fusion information. By extracting effective feature points from preprocessed continuous frame images of the dynamic environment, an improved GC-RANSAC algorithm (graph cut-random sampling consensus algorithm) is adopted to classify and match the feature points in combination with geometric, spatial, and temporal consistency, thereby improving the accuracy of state estimation. In addition, a method and system for accurately identifying static feature points for pose estimation is adopted by combining semantic information with polar geometric constraints and multi-view geometric constraints, thereby avoiding trajectory drift and positioning errors, reducing dependence on deep learning models, effectively improving the generalization ability and real-time performance of the system, and realizing precise SLAM positioning and navigation in dynamic environments.

[0008] In a first aspect, the present invention provides a dynamic environment navigation method based on semantic and geometric fusion information.

[0009] A dynamic environment navigation method based on semantic and geometric fusion information, comprising:

[0010] Acquire continuous frame images of a dynamic environment; the images include RGB images and depth images;

[0011] Performing image preprocessing on continuous frame images;

[0012] The pre-processed continuous frame images are input into the improved target detection model to identify the categories of objects in the dynamic environment;

[0013] Extract feature points from pre-processed continuous frame images. After feature point matching, use the improved GC-RANSAC algorithm to distinguish inliers and inliers among the matched feature points, and calculate the fundamental matrix based on the inliers.

[0014] Based on the basic matrix, epipolar constraints and multi-view geometric constraints are used to jointly identify the semantic information of the object category to screen and determine the static feature points among the feature points;

[0015] The camera pose is estimated based on the filtered static feature points, and then optimized using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

[0016] In a second aspect, the present invention provides a dynamic environment navigation system based on semantic and geometric fusion information.

[0017] A dynamic environment navigation system based on semantic and geometric fusion information, comprising:

[0018] An image acquisition module is used to acquire continuous frame images of a dynamic environment; the images include RGB images and depth images;

[0019] An image preprocessing module, used for performing image preprocessing on continuous frame images;

[0020] The object category recognition module is used to input the preprocessed continuous frame images into the improved target detection model to identify the category of objects in the dynamic environment;

[0021] The image feature point extraction and matching module is used to extract feature points from pre-processed continuous frame images. After feature point matching, the improved GC-RANSAC algorithm is used to distinguish the inliers and inliers in the matched feature points, and the basic matrix is ​​calculated based on the inliers.

[0022] The static feature point screening module is used to screen and determine the static feature points among the feature points based on the basic matrix, epipolar constraints and multi-view geometric constraints, and the semantic information of the recognized object category;

[0023] The pose optimization and navigation module is used to estimate the camera pose based on the filtered static feature points, and then optimize it using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

[0024] In a third aspect, the present invention further provides an electronic device comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned dynamic environment navigation method based on semantic and geometric fusion information when executing the executable instructions stored in the memory.

[0025] In a fourth aspect, the present invention further provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned dynamic environment navigation method based on semantic and geometric fusion information.

[0026] In a fifth aspect, the present invention also provides a computer program product, which includes executable instructions, and the executable instructions are stored in a computer-readable storage medium; wherein, when the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned dynamic environment navigation method based on semantic and geometric fusion information is implemented.

[0027] One or more of the above technical solutions have the following beneficial effects:

[0028] 1. The present invention provides a dynamic environment navigation method and system based on semantic and geometric fusion information to improve the positioning accuracy in dynamic environments, handle the interference of dynamic objects, and ensure the real-time performance of the system. First, through image preprocessing such as dark light environment image enhancement, high-quality image input is provided for subsequent image feature point extraction and matching in low-light environments; the effective feature points of the preprocessed dynamic environment continuous frame images are extracted, and the improved GC-RANSAC algorithm (graph cut-random sampling consensus algorithm) is used to convert the separation problem of feature point inliers and outliers into an energy minimization problem, and the feature points are distinguished by combining geometric, spatial, and temporal consistency, providing an accurate guarantee for the polar constraint, and the camera posture can still be accurately estimated when dynamic objects exist, effectively improving the accuracy of state estimation, and thus improving the overall positioning accuracy of the system; the static feature points are accurately identified by combining semantic information with polar geometric constraints and multi-view geometric constraints for pose estimation, avoiding trajectory drift and positioning error, reducing dependence on deep learning models, effectively improving the generalization ability and real-time performance of the system, and realizing SLAM precise positioning and navigation in dynamic environments.

[0029] 2. In order to effectively reduce the interference of dynamic objects on the SLAM system, the present invention combines the feature pyramid network (FPN) and the compression and excitation module (SE) on the basis of the lightweight target detection model MobileNetV3-SSD to improve the detection ability of small targets and complex scenes. Compared with complex deep learning models, the lightweight network reduces the computational complexity and significantly improves the real-time processing capability, which is suitable for resource-constrained embedded devices; the improved model is used to quickly extract semantic information in dynamic scenes, and the semantic information is extracted through the lightweight object detection model. Different categories of objects have different dynamic characteristics, and the movement probability is assigned to each feature point according to the category of the object; then the geometric information is obtained through the polar constraint and multi-view geometric constraint, and the feature points are classified in combination with the semantic information and geometric information to filter out dynamic feature points. This method shows stronger adaptability and generalization ability in complex dynamic scenes, can effectively filter out the interference of dynamic objects, and ensure high-precision positioning and mapping. It is especially suitable for scenes with high requirements on real-time and robustness, such as unmanned driving and mobile robots.

[0030] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1This is a flow chart of a dynamic environment navigation method based on semantic and geometric fusion information according to an embodiment of the present invention;

[0033] Figure 2 This is a flowchart of image preprocessing in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the results of filtering out inliers using the improved GC-RANSAC algorithm in an embodiment of the present invention;

[0035] Figure 4 Schematic diagram of target detection results in an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of epipolar constraint in an embodiment of the present invention;

[0037] Figure 6 This is a flow chart of dynamic feature point elimination and static feature point screening in an embodiment of the present invention;

[0038] Figure 7 This is a schematic diagram of the multi-view geometric constraints in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that the following detailed descriptions are exemplary only and are intended to describe specific embodiments and provide further explanation of the present invention, and are not intended to limit the exemplary embodiments according to the present invention. Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0040] Example 1

[0041] This embodiment provides a dynamic environment navigation method based on semantic and geometric fusion information, which can achieve high-precision, high-perception, and high-robust real-time perception and positioning, and realize accurate navigation in dynamic environments, such as Figure 1 As shown, the method includes:

[0042] Step S1: Acquire continuous frame images of a dynamic environment. In this embodiment, an RGB-D camera or a webcam is used to acquire continuous frame RGB images of a dynamic environment and their corresponding depth images.

[0043] Step S2: performing image preprocessing on the continuous frame images.

[0044] Considering that the existing visual SLAM system is difficult to extract sufficient and effective feature points in weak texture environment, drastic lighting changes and blurred images, this embodiment first performs image preprocessing on the acquired continuous frame RGB images, while keeping the computational overhead low, enhancing the image brightness, contrast and detail visibility, suppressing noise and artifacts, ensuring that the enhanced image is natural and efficient, thereby improving the subsequent extraction and matching of image feature points. The specific steps of the image preprocessing are as follows: Figure 2 As shown, including:

[0045] Step S2.1: Perform color space conversion on the original RGB image obtained, converting the original RGB image from the RGB space to the Lab color space to obtain a Lab image.

[0046] Specifically, the original RGB image Convert from RGB space to Lab (brightness, a aisle, b Channel) color space, the brightness channel component in Lab space Indicates brightness, which is independent of color components. Brightness can be adjusted independently without affecting color information, allowing for more accurate brightness adjustment. Represents the coordinates of the pixel in the image.

[0047] Step S2.2: For the brightness channel component in the Lab image, a multi-scale Retinex algorithm is used to perform multi-scale brightness enhancement, and the enhanced brightness component is then subjected to contrast correction, adaptive brightness adjustment, and denoising processing in sequence.

[0048] Specifically, the multi-scale Retinex algorithm is used to perform multi-scale enhancement on the luminance channel component (hereinafter referred to as the luminance component): First, Gaussian filters of different scales are used to generate low-frequency components ; Then, calculate the log odds ; Secondly, the multi-scale results are combined to obtain the enhanced brightness component . Processing through the Retinex algorithm can enhance brightness while suppressing uneven lighting and low-light noise.

[0049] Secondly, the enhanced brightness component is processed by histogram equalization (CLAHE). Local contrast enhancement is performed to complete contrast correction. Among them, CLAHE can limit the possibility of excessive contrast enhancement, preserve details and suppress noise.

[0050] After that, adaptive brightness adjustment is performed. To avoid overexposure or darkening, the global brightness mean is calculated. and standard deviation , normalize the brightness component and it can be expressed as:

[0051] ;

[0052] in, and is an adjustable coefficient that controls the contrast range and global brightness.

[0053] Finally, the enhanced brightness component Bilateral filtering denoising is performed to smooth noise while retaining edge details, which can effectively reduce high-frequency noise commonly found in low-light images.

[0054] Step S2.3: Based on the adjusted brightness channel component, the enhanced image is reconstructed by merging it with other channel components in the original Lab color space.

[0055] Specifically, the adjusted brightness component With the original Lab space a and b The channel components are merged to reconstruct the enhanced image.

[0056] Step S2.4: Based on the reconstructed Lab image, perform color space conversion to obtain an enhanced RGB image.

[0057] Step S2.5: Use the Laplace operator to perform edge extraction and edge superposition on the enhanced RGB image to obtain a preprocessed image.

[0058] Specifically, the Laplacian operator is applied to the enhanced RGB image to perform edge extraction, and the edge extraction result is superimposed on the image to further enhance the edge features.

[0059] Step S3: input the preprocessed continuous frame images into the improved target detection model to identify the category of objects in the dynamic environment.

[0060] In this embodiment, MobileNetV3-SSD is used as the target detection model, and an improved target detection model is constructed by introducing FPN and SE modules. This can significantly improve the accuracy and robustness of target detection, especially in small target detection and dynamic scenes. Among them, FPN (Feature Pyramid Network) enhances the detection ability of targets of different sizes through multi-scale feature fusion; SE module further optimizes the suppression of background noise and strengthens key features through adaptive channel weights, thereby reducing false detection and missed detection rates; in addition, the computational overhead of the introduced FPN and SE modules is relatively low, and the overall performance remains lightweight and real-time. Through the above-mentioned improved MobileNetV3-SSD target detection network, target detection can be performed on images captured by the camera, and dynamic objects in the scene, such as pedestrians and vehicles, can be identified. By detecting dynamic objects in real time, mismatching and tracking errors caused by dynamic objects in SLAM can be effectively avoided.

[0061] Specifically, the improved MobileNetV3-SSD target detection model mainly includes: MobileNet V3 backbone network, FPN module, SE module and SSD module.

[0062] First, MobileNet is a lightweight convolutional neural network designed specifically for resource-constrained devices. It uses depthwise separable convolution to reduce the number of model parameters and computational cost. Depthwise separable convolution divides traditional convolution into two steps: depthwise convolution and pointwise convolution. Spatial convolution is performed independently on each input channel to capture spatial features, and 1×1 pointwise convolution is performed on all channels to integrate channel information. By using this depthwise separable convolution to extract features, the amount of computation can be further reduced.

[0063] In this example, the preprocessed RGB image is input into the object detection model. It first passes through the MobileNet V3 backbone network, undergoing multiple layers of depthwise separable convolution to extract a multi-scale feature map that integrates spatial and channel information. The MobileNet V3 network consists of 29 layers, 17 of which are the main network, and the remaining 12 layers are derived from the 17th layer. This design ensures lightweight architecture while extracting higher-quality features.

[0064] Secondly, the SE module is also introduced in the above-mentioned object detection model for adaptive channel weight optimization. In this embodiment, a SE module (compression and excitation module) is inserted after the inverted residual block (Inverted Residual Block) of each layer of the MobileNet V3 backbone network, which can significantly improve the expressiveness of the feature map and further optimize the detection accuracy. The SE module adaptively enhances key features and optimizes the feature map through the following three steps, including:

[0065] (1) Perform global average pooling on the feature map through global pooling to extract the global information of the feature map;

[0066] (2) Generate the weight of each channel through a two-layer fully connected network, learn the importance of each channel, and generate the importance weight of each channel;

[0067] (3) Adjust the activation value of each channel according to the generated weight, enhance key features, suppress irrelevant information, and complete feature recalibration;

[0068] (4) Output the feature map after SE module optimization processing. This feature map has stronger expression ability and is helpful for subsequent multi-scale feature fusion and target detection.

[0069] Then, based on the optimized multi-scale feature maps, FPN is introduced into the object detection model to provide multi-scale fusion features. This effectively improves the performance of the SSD detection head, making it better suited to multi-scale object detection. FPN is a multi-scale feature fusion technology that combines shallow high-resolution features with deep semantic features to enhance the detection of small objects and objects of varying sizes.

[0070] In this embodiment, based on the 6-layer feature maps of different scales output by the MobileNetV3 backbone network, FPN is fused into high-quality feature maps. The feature maps cover everything from high-resolution shallow features to low-resolution deep semantic features, enabling the subsequent detection head (i.e., SSD module) to efficiently detect targets at all scales. This feature fusion includes:

[0071] (1) The multi-scale feature map is input into FPN. FPN fuses the shallow high-resolution features with the deep high-semantic features through top-down and lateral connections, and outputs the multi-scale fusion feature map after feature fusion. This high-quality feature map not only retains the detail information but also has rich semantic information.

[0072] (2) The output of the multi-scale fusion feature map after FPN fusion can enhance the detection ability of targets of different scales, especially the recognition effect of small targets.

[0073] Finally, the multi-scale fused feature map output by the FPN is input to the SSD (Single Shot Multibox Detector) module. The SSD module is a lightweight object detection framework that can apply convolution kernels of different sizes and ratios to feature maps at different levels to detect objects of different scales. In this embodiment, the object detection process of the multi-scale detection head is as follows:

[0074] (1) Apply convolution kernels of different sizes and proportions to the fused feature maps of each scale to meet the detection requirements of targets of different sizes;

[0075] (2) The SSD module generates predictions on each feature layer and outputs the bounding box location and category probability of the target;

[0076] (3) Finally, the module outputs the detected bounding boxes and category labels of dynamic objects (such as pedestrians, vehicles, etc.) in the scene, thereby achieving real-time target detection.

[0077] Preferably, the improved lightweight object detection model is pre-trained using the annotated COCO dataset (existing dataset). The dataset category can be directly used for SLAM target recognition tasks in indoor environments. During the training process, a loss function of the model is constructed, and the dataset is continuously iterated and trained until the loss function is minimized, completing the model training. The loss function constructed includes position loss and category loss, which can be expressed as:

[0078] ;

[0079] in, is the position loss, which is used to measure the position difference between the predicted bounding box and the true bounding box; is the category loss, which is used to measure the difference between the predicted category and the true category; It is a balance parameter used to adjust the relative weight of position loss and category loss, and is usually set to 1.

[0080] Furthermore, the localization loss is used to optimize the position and size of the predicted bounding box to make it closer to the real bounding box. The localization loss is defined as:

[0081] ;

[0082] in, is the number of positive samples; To predict bounding box parameters (including center coordinates, width, height, etc.); is the true bounding box parameter; Defined as:

[0083] ;

[0084] When the error is small, use Loss can help stabilize training, and when the error is large, The loss reduces sensitivity to outliers.

[0085] In addition, the confidence loss is used to optimize the category prediction of each anchor box or target box. Considering the imbalance of positive and negative samples in static and dynamic environments, and to avoid the shortcomings of cross entropy loss in dealing with category imbalance, this embodiment adopts the following category loss function to reduce the contribution of easy-to-classify samples to the loss, thereby focusing on difficult-to-classify samples. The category loss can be expressed as:

[0086] ;

[0087] in, is the number of positive samples; is a collection of categories, including background categories; For the Samples are predicted as categories probability; For category The weights are used to balance the importance of different categories; This is an adjustment parameter used to adjust the attention level of difficult and easy samples. This category loss is achieved by introducing an adjustment factor , reducing the focus on easy-to-classify samples and enhancing the learning effect on difficult-to-classify samples, thereby improving the overall detection performance.

[0088] The design of the above loss function can effectively balance the bounding box positioning accuracy and category classification accuracy, especially in multi-scale and dynamic scenes, improve the detection accuracy and robustness of the model, and ensure the accuracy of the final semantic classification. Finally, the above pre-trained model is used to extract the semantic information of the object category in the dynamic environment, such as Figure 4 As shown in the figure, the model can effectively identify various categories of dynamic objects in indoor navigation environments, such as person, chair, bottle, etc., and can significantly improve detection accuracy and robustness of dynamic scenes while ensuring real-time performance.

[0089] Step S4: extracting feature points from the pre-processed continuous frame images, matching the feature points, using the improved GC-RANSAC algorithm to distinguish the inliers and inliers in the matched feature points, and calculating the basic matrix based on the inliers.

[0090] In this embodiment, the ORB algorithm (a feature detection and feature description algorithm) is used to extract feature points and their descriptors in the image, that is, the corner points are first detected by FAST (i.e., a fast detection algorithm), and then the direction of the corner points is calculated based on the local gradient information, and a rotation-invariant BRIEF descriptor is generated; for the extracted feature points, brute force matching or the efficient FLANN matching algorithm (fast nearest neighbor matching algorithm) is used to match the feature points of the two frames of images, the Hamming distance is used to measure the similarity of the descriptors, and the feature points with high matching quality are screened out through a ratio test, so as to obtain the matching feature points of two consecutive frames.

[0091] Afterwards, the improved GC-RANSAC algorithm is used to process the continuous frame images and estimate the basic matrix, which provides good accuracy for the subsequent epipolar constraints and effectively reduces the interference of dynamic objects on the pose estimation. Specifically, for a given set of 2D to 2D matching feature points in two consecutive frames of images, , using the fundamental matrix Calculate the geometric relationship between the matched feature points and the epipolar lines, where the feature points The corresponding epipolar line satisfy: .

[0092] If the feature point If a point lies on its corresponding epipolar line, it is considered to be a static point, i.e., an inlier. Conversely, if it deviates far from the epipolar line, it is likely to be a dynamic point, i.e., an outlier. Therefore, the distance between the feature point and the epipolar line can be used to preliminarily classify the inliers and outliers.

[0093] Furthermore, an improved GC-RANSAC algorithm is used to transform the problem of dividing the inliers and outliers in the feature points into an energy minimization problem. Combining geometric, spatial, and temporal consistency, the inliers in the feature points are identified through solution calculation. This method can effectively improve the accuracy of inlier recognition.

[0094] Specifically, based on the geometric relationship between the matching feature points and the basic matrix, as well as the spatial and temporal consistency of adjacent feature points, an energy function is constructed, which can be expressed as:

[0095] ;

[0096] in, represents the label assignment of the matching point set, and G is the adjacency graph; 、 Represent feature points 、 Label assignment; is the adjustment parameter; the unary term Represents the geometric relationship between the matching points and the basic matrix, which is defined as:

[0097] ;

[0098] ;

[0099] in, Represents the fundamental matrix The angle parameter, Represents the geometric distance between the matching point (i.e., matching feature point) and the basic matrix, is the preset threshold of the internal and external points, function is the Gaussian kernel function, Represents matching point pairs is an interior point, otherwise it is an exterior point.

[0100] Typically, in the GC-RANSAC algorithm, the pairwise terms It is only used to describe the spatial consistency of adjacent feature points and is defined as:

[0101] ;

[0102] Furthermore, considering that the movement of objects is often continuous, the continuity of the object's motion trajectory can often provide more contextual information, and the temporal consistency between adjacent feature points can help infer whether they belong to the same object or whether they are dynamic feature points. To this end, this embodiment improves the existing GC-RANSAC algorithm for dynamic scenes. By introducing temporal consistency on the basis of geometric and spatial consistency, the motion trend and speed of feature points between consecutive frames are considered, so that dynamic points and static points can be distinguished more accurately.

[0103] Specifically, based on the original spatial consistency, temporal consistency is introduced , so that it not only depends on the geometric relationship, but also considers the dynamic characteristics in time. The pair term can be expressed as:

[0104] ;

[0105] in, and Indicates a point and In chronological order, is a temporal consistency function, which is used to measure the time difference or motion consistency between adjacent points. and come from the same object and move in unison, then ; If point and from different objects or have inconsistent motions, then , that is, the time series consistency is proportional to the time difference; The weight hyperparameter representing the temporal consistency term is used to adjust the impact of temporal consistency on the overall energy function.

[0106] By introducing temporal consistency, the GC-RANSAC algorithm's performance in dynamic scenes can be further improved, especially when matching points with motion features. By considering the temporal dimension, the motion patterns between adjacent points can be better captured, thereby improving the accuracy of inlier recognition and the ability to remove dynamic feature points.

[0107] In this embodiment, according to experience, and By solving the minimization energy function, the optimal label allocation L of the feature points is calculated, and the optimal label allocation L is used to distinguish the internal and external points. That is, if the feature point Tags , then it means that the feature point Consistent with other points, this feature point is an interior point; otherwise, , it means that the feature point is consistent with other points. By adopting the improved GC-RANSAC algorithm, the internal points and external points can be distinguished more effectively, such as Figure 3 As shown in Figure 1, (a) is the static matching feature point determined without the improved GC-RANSAC algorithm, and (b) is the static matching feature point determined with the improved GC-RANSAC algorithm (it should be clear that the images used in the diagrams of this embodiment are all public images in the existing data set). After the improved GC-RANSAC filtering, the inliers are almost all static feature points and can be used to accurately calculate the basic matrix. Subsequently, the basic matrix is ​​estimated based on the identified inliers. , thus providing accurate prior information for the epipolar constraint, ensuring that the dynamic feature point removal algorithm has higher accuracy.

[0108] Step S5: Based on the basic matrix, epipolar constraints and multi-view geometry constraints are used to jointly identify the semantic information of the object category to screen and determine the static feature points among the feature points.

[0109] This embodiment combines geometric constraints (including epipolar constraints and multi-view geometric constraints) with semantic information (the above-mentioned target detection results) to suppress dynamic feature points, eliminate dynamic feature points and accurately screen static feature points, while reducing the dependence on deep learning, such as Figure 6 As shown, the specific steps include:

[0110] Step S5.1: Assign a weight to each feature point of each object based on the object category identified in the dynamic environment. Feature points corresponding to highly dynamic objects are assigned high weights, while feature points corresponding to less dynamic or static objects are assigned low weights.

[0111] Step S5.2: Based on the basic matrix, use the epipolar constraint to calculate the distance from each feature point to the epipolar line, and calculate the weighted distance based on the distance and weight of each feature point, and preliminarily screen out static feature points based on the weighted distance.

[0112] Specifically, in a dynamic environment, considering that the existing target detection network can only identify the category of objects but cannot identify the motion state of these objects, in order to accurately eliminate dynamic points in stationary objects, this embodiment makes full use of polar line geometry to further subdivide the static and dynamic states of the feature points of dynamic objects.

[0113] First, the distance between the feature points and the epipolar lines is calculated, and the feature points exceeding the threshold are regarded as dynamic points. In the initial pose estimation part, the basic matrix is ​​obtained by using the improved GC-RANSAC algorithm in the previous step. , providing precise support for epipolar constraints in subsequent steps.

[0114] like Figure 5 As shown, the geometric relationship in the polar constraint is solved from the perspective of algebraic geometry, where and Indicates the optical center of the camera, the straight line and is the polar line in the frame, and the matching feature points of the spatial point P corresponding to the previous frame and the current frame map are and , which can be expressed as:

[0115] ;

[0116] The corresponding homogeneous coordinates are:

[0117] ;

[0118] in, 、 and 、 is the pixel coordinate value of the feature point in the image coordinate system.

[0119] Through the basic matrix Calculate the epipolar line under the current frame , the formula is as follows:

[0120] ;

[0121] In the above formula, , , represents a line vector; is the basic matrix, which can be expressed as:

[0122] ;

[0123] In the above formula, There are 9 unknown numbers.

[0124] The epipolar constraint can be expressed as:

[0125] ;

[0126] Define feature points To the extreme line The offset distance between for:

[0127] ;

[0128] like is a dynamic space point, then point The offset distance is:

[0129] ;

[0130] like is a static space point, then point The offset distance is:

[0131] ;

[0132] when When it is a static space feature point, its corresponding feature point Should be located on the polar line However, due to the noise in real scenes, the offset distance is usually greater than zero but lower than the empirical threshold. . On point Move to point When , the corresponding feature point With polar line The distance between , by comparing the distance and experience value threshold , the dynamic state of the feature points can be determined.

[0133] Furthermore, in order to avoid excessive reliance on deep learning in dynamic feature discrimination, this embodiment proposes a dynamic feature suppression method based on geometric constraints. First, semantic information is obtained through the target detection model, and the semantic information is used as the prior knowledge of the object category in the system. Since objects of different categories have different motion characteristics, the system assigns a weight value to each feature point according to the object category. Specifically, the weight value The range is set between 0 and 1. For highly dynamic objects (such as vehicles, pedestrians, etc.), the weight value is set to 0 because of its high possibility of movement. Set to 0.75 to 1.0; for low-dynamic or static objects (such as buildings, tables, etc.), the possibility of movement is small, and the weight value is Set it to 0 to 0.25.

[0134] Secondly, the epipolar constraint is used to calculate the distance from the feature point to the corresponding epipolar line , and combined with the weight value of the feature point An evaluation is performed and compared with the empirical threshold. If the distance is greater than the empirical threshold, the feature point is considered to be a dynamic feature point; otherwise, if the distance is less than the empirical threshold, the static feature point is preliminarily screened out.

[0135] Step S5.3: For the static feature points initially screened out, use multi-view geometric constraints to analyze their angle and depth changes in adjacent frames, determine their dynamic state, and then perform re-screening to eliminate dynamic feature points and determine the final static feature points to improve the stability and robustness of the system.

[0136] Taking into account the current point Move to point along the epipolar direction When , the corresponding feature point Also falls on the extreme line Therefore, in this case, relying solely on the epipolar constraint cannot effectively identify these dynamic objects. To solve this problem, this embodiment introduces multi-view geometry constraints on the basis of the epipolar constraint to further detect dynamic feature points that have been missed.

[0137] Specifically, if there are dynamic objects in the scene, the depth values ​​and angles of the corresponding feature points in the current frame and the reference frame will change significantly. Based on this phenomenon, these dynamic feature points can be identified by depth constraints and angle constraints. The principle is as follows: Figure 7 shown.

[0138] First, the five key frames with the highest consistency with the current frame are selected as reference frames. Determined by the following formula:

[0139] ;

[0140] in, is the consistency measure between frames, 、 are the increments of position and attitude respectively.

[0141] By calculating the depth value of the key point in the current frame And the projected depth value of the key frame in the current frame Get depth information, that is, depth difference , which can be defined as:

[0142] ;

[0143] Parallax angle yes and The angle between the back projections of It can be expressed as:

[0144] ;

[0145] Where, the first perspective vector Represents the vector from the camera position to the feature point in the current frame, the second perspective vector Represents the vector from the camera position to the feature point in the key frame, is the dot product of the two view vectors in the current frame and the historical frame, and Respectively represent the modulus lengths of the two view vectors.

[0146] Combining the above depth information and perspective information, if or , then the feature point is regarded as a dynamic feature point under multi-view geometry constraints, where and Represent the depth threshold and angle threshold respectively.

[0147] By applying multi-view geometric constraints to analyze the angle and depth changes of feature points in adjacent frames, we can further judge and filter static feature points. Through this criterion, we can accurately judge the status of all feature points in the current frame one by one.

[0148] Step S6: Estimate the camera pose based on the filtered static feature points, optimize using the BA optimization algorithm, build an environment map based on the optimized results, and perform dynamic environment navigation.

[0149] In this embodiment, according to the feature points extracted based on the RGB image mentioned above, combined with the depth image, the initial three-dimensional coordinates of these feature points, i.e., the three-dimensional landmark point coordinates, are directly obtained; then, the screened static feature points are combined with the camera intrinsic parameters to adopt the camera pose estimation method to estimate the 6-DOF pose of the camera; then, the observation data of the feature points in multiple frames are combined with the camera pose, and the accuracy of the three-dimensional coordinates of the feature points is corrected using the depth information; finally, the three-dimensional landmark point coordinates and the camera pose are jointly optimized through the bundled optimization (BA optimization) algorithm to reduce the projection error and improve the accuracy.

[0150] Specifically, the BA optimization method is used to optimize the camera's 6-DOF pose and 3D landmark points simultaneously. BA optimization is a technology that extracts the optimal 3D model and camera parameters (including intrinsic and extrinsic parameters) in visual reconstruction. By adjusting the camera pose and the spatial position of the feature points, the bundles of light rays reflected from each feature point can converge to the camera's optical center after adjustment, thereby optimizing the poses of multiple cameras and the spatial coordinates of the landmark points under the poses. In SLAM, the BA optimization algorithm is used to optimize the camera pose and the spatial position of the feature points, as follows:

[0151] First, according to the optimized variable pose and landmark coordinates , construct the reprojection error function as:

[0152] ;

[0153] Where, Represents the observation value, that is, in the pose and point The projection observation projection of ; Represents the projection function of the model, which is used to project the point coordinates in 3D space onto the image plane.

[0154] Secondly, the Gauss-Newton method is used to solve the objective function and accurately estimate the camera pose for each frame. (including rotation matrix and translation vector) and 3D landmark point coordinates , which can be used to further build more accurate environment maps and perform dynamic environment navigation.

[0155] Example 2

[0156] This embodiment provides a dynamic environment navigation system based on semantic and geometric fusion information, including:

[0157] An image acquisition module is used to acquire continuous frame images of a dynamic environment; the images include RGB images and depth images;

[0158] An image preprocessing module, used for performing image preprocessing on continuous frame images;

[0159] The object category recognition module is used to input the preprocessed continuous frame images into the improved target detection model to identify the category of objects in the dynamic environment;

[0160] The image feature point extraction and matching module is used to extract feature points from pre-processed continuous frame images. After feature point matching, the improved GC-RANSAC algorithm is used to distinguish the inliers and inliers in the matched feature points, and the basic matrix is ​​calculated based on the inliers.

[0161] The static feature point screening module is used to screen and determine the static feature points among the feature points based on the basic matrix, epipolar constraints and multi-view geometric constraints, and the semantic information of the recognized object category;

[0162] The pose optimization and navigation module is used to estimate the camera pose based on the filtered static feature points, and then optimize it using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

[0163] Example 3

[0164] This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for implementing the above method provided in this embodiment when executing the executable instructions stored in the memory.

[0165] Example 4

[0166] This embodiment further provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will be caused to execute the above method provided in this embodiment.

[0167] Example 5

[0168] This embodiment provides a computer program product including executable instructions, which are computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method provided in this embodiment.

[0169] The steps involved in the above embodiments 2 to 5 correspond to those in embodiment 1. For detailed implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media that includes one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to perform any method of the present invention.

[0170] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0171] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention is described in conjunction with the accompanying drawings, it does not limit the scope of protection of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A dynamic environment navigation method based on semantic and geometric fusion information, characterized in that: include: Acquire continuous frame images of dynamic environment; The image includes an RGB image and a depth image; Perform image preprocessing on continuous frame images, including: Performing color space conversion on the original RGB image, converting the original RGB image from the RGB space to the Lab color space, and obtaining a Lab image; For the brightness channel component of Lab image, the multi-scale Retinex algorithm is used to perform multi-scale brightness enhancement, and the enhanced brightness component is then subjected to contrast correction, adaptive brightness adjustment and denoising. Based on the adjusted brightness channel component, the enhanced image is reconstructed by merging it with other channel components in the original Lab color space; Based on the reconstructed Lab image, the enhanced RGB image is obtained through color space conversion; The Laplace operator is used to perform edge extraction and edge superposition on the enhanced RGB image to obtain the preprocessed image; The pre-processed continuous frame images are input into the improved target detection model to identify the categories of objects in the dynamic environment; Extract feature points from pre-processed continuous frame images. After feature point matching, use the improved GC-RANSAC algorithm to distinguish between inliers and outliers in the matched feature points, and calculate the fundamental matrix based on the inliers. The improved GC-RANSAC algorithm transforms the problem of dividing feature points into inliers and outliers into an energy minimization problem. Combining geometric, spatial, and temporal consistency, the inliers in the feature points are identified through solution calculation, including: Based on the geometric relationship between the matching feature points and the basic matrix, as well as the spatial and temporal consistency of adjacent feature points, the energy function is constructed as follows: ； in, represents the label assignment of the matching point set, and G is the adjacency graph; is the adjustment parameter; the unary term Represents the geometric relationship between the matching point and the basic matrix; pairwise It is used to describe the spatial consistency and temporal consistency of adjacent feature points: ； in, Represents the geometric relationship between the matching points and the basic matrix, 、 Represent feature points 、 Label assignment, and Indicates a point and In chronological order, is a temporal consistency function, which is used to measure the time difference or motion consistency between adjacent points. and come from the same object and move in unison, then ; If point and from different objects or have inconsistent motions, then ; represents the weight hyperparameter of the temporal consistency term; The goal is to minimize the energy function, calculate the optimal label allocation L of the feature points, and use the optimal label allocation L to distinguish internal points from external points; Based on the fundamental matrix, epipolar constraints and multi-view geometry constraints are used to jointly identify the semantic information of the object category to screen and determine the static feature points among the feature points, including: Assign a weight to each feature point of each object based on the object category in the dynamic environment. Feature points corresponding to highly dynamic objects are given high weights, while feature points corresponding to low dynamic or static objects are given low weights. According to the basic matrix, the distance from each feature point to the epipolar line is calculated using the epipolar constraint; According to the distance and weight of each feature point, the weighted distance is calculated, and the weighted distance is compared with the empirical threshold to preliminarily screen out static feature points; For the static feature points initially screened, multi-view geometric constraints are used to screen them again to determine the final static feature points to avoid feature mismatching in dynamic environments due to the frequent appearance of dynamic objects in the scene; The camera pose is estimated based on the filtered static feature points, and then optimized using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

2. A dynamic environment navigation method based on semantic and geometric fusion information as claimed in claim 1, characterized in that: MobileNetV3-SSD is used as the target detection model, and the FPN module and SE module are introduced to build an improved target detection model; The preprocessed RGB image is input into the improved object detection model, which first passes through the MobileNet V3 backbone network and undergoes multiple layers of depthwise separable convolution to extract multi-scale feature maps that integrate spatial and channel information. The feature map of each scale is input into the SE module to learn and generate the importance weight of each channel, and the optimized feature map is obtained by channel weighting; The optimized multi-scale feature map is input into the FPN module, which fuses the shallow high-resolution features with the deep high-semantic features, and outputs the multi-scale fusion feature map after feature fusion; The multi-scale fusion feature map is input into the SSD module, which detects and generates predictions on each layer of feature maps and outputs the bounding box and category label of the target.

3. A dynamic environment navigation method based on semantic and geometric fusion information as claimed in claim 2, characterized in that: The loss function of the improved target detection model includes position loss and category loss; The position loss for: ； In the above formula, is the number of positive samples; To predict bounding box parameters, including center coordinates, width, and height; is the true bounding box parameter; function ; The class loss is: ； In the above formula, is the number of positive samples; is a collection of categories; For the Samples are predicted as categories probability; For category The weight of is the adjustment parameter.

4. A dynamic environment navigation system based on semantic and geometric fusion information, characterized in that: include: An image acquisition module is used to acquire continuous frame images of a dynamic environment; The image includes an RGB image and a depth image; The image preprocessing module is used to perform image preprocessing on continuous frame images, including: Performing color space conversion on the original RGB image, converting the original RGB image from the RGB space to the Lab color space, and obtaining a Lab image; For the brightness channel component of Lab image, the multi-scale Retinex algorithm is used to perform multi-scale brightness enhancement, and the enhanced brightness component is then subjected to contrast correction, adaptive brightness adjustment and denoising. Based on the adjusted brightness channel component, the enhanced image is reconstructed by merging it with other channel components in the original Lab color space; Based on the reconstructed Lab image, the enhanced RGB image is obtained through color space conversion; The Laplace operator is used to perform edge extraction and edge superposition on the enhanced RGB image to obtain the preprocessed image; The object category recognition module is used to input the preprocessed continuous frame images into the improved target detection model to identify the category of objects in the dynamic environment; The image feature point extraction and matching module is used to extract feature points from pre-processed continuous frame images. After feature point matching, the improved GC-RANSAC algorithm is used to distinguish between inliers and outliers in the matched feature points, and the basic matrix is ​​calculated based on the inliers. The improved GC-RANSAC algorithm is used to transform the problem of dividing the feature points into inliers and outliers into an energy minimization problem. Combining geometric, spatial, and temporal consistency, the inliers in the feature points are identified through solution calculation, including: Based on the geometric relationship between the matching feature points and the basic matrix, as well as the spatial and temporal consistency of adjacent feature points, the energy function is constructed as follows: ； in, represents the label assignment of the matching point set, and G is the adjacency graph; is the adjustment parameter; the unary term Represents the geometric relationship between the matching point and the basic matrix; pairwise It is used to describe the spatial consistency and temporal consistency of adjacent feature points: ； in, Represents the geometric relationship between the matching points and the basic matrix, 、 Represent feature points 、 Label assignment, and Indicates a point and In chronological order, is a temporal consistency function, which is used to measure the time difference or motion consistency between adjacent points. and come from the same object and move in unison, then ; If point and from different objects or have inconsistent motions, then ; represents the weight hyperparameter of the temporal consistency term; The goal is to minimize the energy function, calculate the optimal label allocation L of the feature points, and use the optimal label allocation L to distinguish internal points from external points; The static feature point screening module is used to screen and determine static feature points among the feature points based on the fundamental matrix, epipolar constraints, and multi-view geometric constraints, combined with the semantic information of the recognized object category, including: Assign a weight to each feature point of each object based on the object category in the dynamic environment. Feature points corresponding to highly dynamic objects are given high weights, while feature points corresponding to low dynamic or static objects are given low weights. According to the basic matrix, the distance from each feature point to the epipolar line is calculated using the epipolar constraint; According to the distance and weight of each feature point, the weighted distance is calculated, and the weighted distance is compared with the empirical threshold to preliminarily screen out static feature points; For the static feature points initially screened, multi-view geometric constraints are used to screen them again to determine the final static feature points to avoid feature mismatching in dynamic environments due to the frequent appearance of dynamic objects in the scene; The pose optimization and navigation module is used to estimate the camera pose based on the filtered static feature points, and then optimize it using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

5. An electronic device, characterized in that: include: a memory for storing executable instructions; The processor is configured to implement the dynamic environment navigation method based on semantic and geometric fusion information as described in any one of claims 1 to 3 when executing the executable instructions stored in the memory.

6. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause the processor to execute the executable instructions to implement the dynamic environment navigation method based on semantic and geometric fusion information as described in any one of claims 1 to 3.

7. A computer program product, characterized in that The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the dynamic environment navigation method based on semantic and geometric fusion information according to any one of claims 1 to 3 is implemented.

Images