Dynamic environment navigation method and system based on semantic and geometric fusion information

Abstract

The invention discloses a dynamic environment navigation method and system based on semantic and geometric fusion information, and relates to the technical field of dynamic environment intelligent navigation, and the method comprises the following steps: obtaining and preprocessing continuous frame images of a dynamic environment; inputting the preprocessed continuous frame images into an improved target detection model, and identifying the type of an object in the dynamic environment; feature points in the preprocessed continuous frame images are extracted and matched, an improved GC-RANSAC algorithm is adopted to distinguish inner and outer points in the matched feature points, and a basic matrix is calculated according to the inner points; screening and determining static feature points in the feature points according to the basic matrix by adopting epipolar constraint and multi-view geometric constraint and combining the recognized object category semantic information; and estimating a camera pose based on the screened static feature points, performing optimization by using a BA optimization algorithm, constructing an environment map according to an optimized result, and performing dynamic environment navigation. According to the invention, accurate positioning and navigation in a dynamic environment can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic environment intelligent navigation, and in particular to a dynamic environment navigation method and system based on semantic and geometric fusion information. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, simultaneous localization and mapping (SLAM) technology has been rapidly developed and widely used in the fields of mobile robots, drones, self-driving cars, augmented reality and virtual reality. With the improvement of computing power and the advancement of sensor technology, SLAM technology has gradually developed from the initial 2D SLAM based on lidar to multimodal fusion technologies such as visual SLAM and RGB-D SLAM. Among them, visual SLAM has become a hot spot for research and application due to its low hardware cost and wide applicability. However, traditional SLAM technology still faces challenges in dealing with dynamic environments, weak texture scenes and real-time performance, including: (1) Most SLAM systems are based on the assumption of a static environment. However, when faced with a dynamic environment, dynamic objects such as pedestrians and vehicles that frequently appear in actual scenes will cause a large number of feature matching errors, resulting in serious deviations in the system's state estimation, affecting the accuracy of trajectory estimation and the robustness of the system. (2) In terms of processing dynamic objects, existing SLAM algorithms lack specialized detection and processing methods for dynamic objects, resulting in the inability to effectively distinguish between dynamic and static objects. As a result, dynamic feature points are often mistakenly used for pose estimation, causing trajectory drift and positioning errors. For this reason, when processing object detection in a dynamic environment, deep learning models are currently usually used for detection. However, the high computing requirements of these models pose huge challenges to embedded systems and real-time applications. Most dynamic SLAM methods often sacrifice real-time performance while pursuing accuracy, making it difficult to meet the needs of application scenarios that require rapid response, such as mobile robots and self-driving cars. (3) Existing visual SLAM systems have difficulty extracting sufficiently effective feature points in weak texture environments, drastic lighting changes (such as overexposure or low light), and blurred images, which significantly affects the positioning accuracy and system stability. Summary of the invention

[0004] To address the deficiencies of the above-mentioned prior art, the present invention provides a dynamic environment navigation method and system based on semantic and geometric fusion information, which extracts effective feature points from preprocessed continuous frame images of the dynamic environment, adopts an improved GC-RANSAC algorithm (graph cut-random sampling consensus algorithm), and classifies and matches the feature points in combination with geometric, spatial, and temporal consistency to improve the accuracy of state estimation. It also uses a combination of semantic information and polar geometric constraints and multi-view geometric constraints to accurately identify static feature points for pose estimation, avoid trajectory drift and positioning errors, reduce dependence on deep learning models, effectively improve the generalization ability and real-time performance of the system, and realize SLAM precise positioning and navigation in dynamic environments.

[0005] In a first aspect, the present invention provides a dynamic environment navigation method based on semantic and geometric fusion information.

[0006] A dynamic environment navigation method based on semantic and geometric fusion information, comprising: Acquire continuous frame images of a dynamic environment; the images include RGB images and depth images; Perform image preprocessing on continuous frame images; The preprocessed continuous frame images are input into the improved target detection model to identify the categories of objects in the dynamic environment; Extract feature points from continuous frame images after preprocessing. After feature point matching, use the improved GC-RANSAC algorithm to distinguish the internal and external points in the matched feature points, and calculate the basic matrix based on the internal points. According to the basic matrix, epipolar constraints and multi-view geometric constraints are used to jointly identify the semantic information of the object category to screen and determine the static feature points among the feature points; The camera pose is estimated based on the screened static feature points, and then optimized using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

[0007] In a second aspect, the present invention provides a dynamic environment navigation system based on semantic and geometric fusion information.

[0008] A dynamic environment navigation system based on semantic and geometric fusion information, comprising: An image acquisition module is used to acquire continuous frame images of a dynamic environment; the images include RGB images and depth images; An image preprocessing module, used for performing image preprocessing on continuous frame images; An object category recognition module is used to input the preprocessed continuous frame images into the improved target detection model to identify the category of objects in a dynamic environment; The image feature point extraction and matching module is used to extract the feature points in the continuous frame images after preprocessing. After the feature points are matched, the improved GC-RANSAC algorithm is used to distinguish the internal and external points in the matching feature points, and the basic matrix is ​​calculated based on the internal points. The static feature point screening module is used to screen and determine the static feature points among the feature points based on the basic matrix, using polar constraints and multi-view geometric constraints, and combining the semantic information of the recognized object category; The pose optimization and navigation module is used to estimate the camera pose based on the screened static feature points, and then optimize it using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

[0009] In a third aspect, the present invention further provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned dynamic environment navigation method based on semantic and geometric fusion information when executing the executable instructions stored in the memory.

[0010] In a fourth aspect, the present invention further provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned dynamic environment navigation method based on semantic and geometric fusion information.

[0011] In a fifth aspect, the present invention also provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned dynamic environment navigation method based on semantic and geometric fusion information is implemented.

[0012] One or more of the above technical solutions have the following beneficial effects: 1. The present invention provides a dynamic environment navigation method and system based on semantic and geometric fusion information to improve the positioning accuracy in dynamic environments, handle the interference of dynamic objects, and ensure the real-time performance of the system. First, through image preprocessing such as dark light environment image enhancement, high-quality image input is provided for subsequent image feature point extraction and matching in low-light environments; the effective feature points of the preprocessed dynamic environment continuous frame images are extracted, and the improved GC-RANSAC algorithm (graph cut-random sampling consistency algorithm) is used to convert the separation problem of the feature point inside and outside points into an energy minimization problem, and the feature points are distinguished by combining geometry, space, and time sequence consistency, providing an accurate guarantee for the polar line constraint, and the camera posture can still be accurately estimated when dynamic objects exist, effectively improving the accuracy of state estimation, and thus improving the overall positioning accuracy of the system; the static feature points are accurately identified by combining semantic information with polar line geometric constraints and multi-view geometric constraints for pose estimation, avoiding trajectory drift and positioning errors, reducing dependence on deep learning models, effectively improving the generalization ability and real-time performance of the system, and realizing SLAM precise positioning and navigation in dynamic environments.

[0013] 2. In order to effectively reduce the interference of dynamic objects on the SLAM system, the present invention combines the feature pyramid network (FPN) and the compression and excitation module (SE) on the basis of the lightweight target detection model MobileNetV3-SSD to improve the detection ability of small targets and complex scenes. Compared with the complex deep learning model, the lightweight network reduces the computational complexity and significantly improves the real-time processing capability, which is suitable for resource-constrained embedded devices; the improved model is used to quickly extract semantic information in dynamic scenes, and the semantic information is extracted through the lightweight object detection model. According to the different dynamic characteristics of objects of different categories, the movement probability is assigned to each feature point according to the category of the object; then the geometric information is obtained through the polar constraint and the multi-view geometric constraint, and the feature points are classified in combination with the semantic information and the geometric information to filter out the dynamic feature points. This method shows stronger adaptability and generalization ability in complex dynamic scenes, can effectively filter out the interference of dynamic objects, and ensure high-precision positioning and mapping. It is especially suitable for scenes with high requirements on real-time and robustness, such as unmanned driving and mobile robots.

[0014] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0016] Figure 1A flowchart of a dynamic environment navigation method based on semantic and geometric fusion information according to an embodiment of the present invention; Figure 2 is a flowchart of image preprocessing in an embodiment of the present invention; Figure 3 This is a schematic diagram of the result of selecting inliers using the improved GC-RANSAC algorithm in an embodiment of the present invention; Figure 4 is a schematic diagram of target detection results in an embodiment of the present invention; Figure 5 is a schematic diagram of epipolar constraints in an embodiment of the present invention; Figure 6 It is a flow chart of dynamic feature point elimination and static feature point screening in an embodiment of the present invention; Figure 7 It is a schematic diagram of multi-view geometric constraints in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0018] Embodiment 1 This embodiment provides a dynamic environment navigation method based on semantic and geometric fusion information, which can achieve high-precision, high-perception, and high-robust real-time perception and positioning, and realize accurate navigation in dynamic environments, such as Figure 1 As shown, the method includes: Step S1, acquiring continuous frame images of a dynamic environment. In this embodiment, an RGB-D camera or a camera is used to collect continuous frame RGB images of a dynamic environment and their corresponding depth images.

[0019] Step S2: performing image preprocessing on the continuous frame images.

[0020] Considering that the existing visual SLAM system is difficult to extract sufficient effective feature points in the weak texture environment, drastic lighting changes and blurred images, this embodiment first performs image preprocessing on the acquired continuous frame RGB images, while keeping the computational overhead low, enhancing the image brightness, contrast and detail visibility, suppressing noise and artifacts, ensuring that the enhanced image is natural and efficient, thereby improving the subsequent extraction and matching of image feature points. The specific steps of the image preprocessing are as follows Figure 2 As shown, including: Step S2.1, performing color space conversion on the acquired original RGB image, converting the original RGB image from the RGB space to the Lab color space, and acquiring a Lab image.

[0021] Specifically, the original RGB image Convert from RGB space to Lab (brightness, a aisle, b Channel) color space, the brightness channel component in Lab space Represents brightness, which is independent of color components. Brightness can be adjusted independently without affecting color information, and brightness can be adjusted more accurately. Represents the coordinates of the pixel points in the image.

[0022] Step S2.2: For the brightness channel component in the Lab image, a multi-scale Retinex algorithm is used to perform multi-scale brightness enhancement, and the enhanced brightness component is then subjected to contrast correction, adaptive brightness adjustment and denoising in sequence.

[0023] Specifically, the multi-scale Retinex algorithm is used to perform multi-scale enhancement on the brightness channel component (hereinafter referred to as the brightness component): first, Gaussian filters of different scales are used to generate low-frequency components ; Then, calculate the log ratio ; Secondly, the multi-scale results are combined to obtain the enhanced brightness component . Through the Retinex algorithm, it can enhance the brightness while suppressing uneven lighting and low-light noise.

[0024] Secondly, the enhanced brightness component is processed by histogram equalization (CLAHE). Local contrast enhancement is performed to complete contrast correction, among which CLAHE can limit the possibility of over-enhancement of contrast, preserve details and suppress noise.

[0025] After that, adaptive brightness adjustment is performed. To avoid overexposure or darkening, the global brightness mean is calculated. and standard deviation , the brightness component is normalized and adjusted, which can be expressed as: ; in, and is an adjustable factor that controls the contrast range and global brightness.

[0026] Finally, the enhanced brightness component Bilateral filtering is used to denoise the image, which can smooth out the noise while retaining edge details, and can effectively reduce the high-frequency noise commonly found in low-light images.

[0027] Step S2.3: Based on the adjusted brightness channel component, the enhanced image is reconstructed by merging it with other channel components in the original Lab color space.

[0028] Specifically, the adjusted brightness component With the original Lab space a and b The channel components are merged to reconstruct the enhanced image.

[0029] Step S2.4: Based on the reconstructed Lab image, the enhanced RGB image is obtained by color space conversion.

[0030] Step S2.5: Use the Laplace operator to perform edge extraction and edge superposition on the enhanced RGB image to obtain a preprocessed image.

[0031] Specifically, the Laplacian operator is applied to the enhanced RGB image to perform edge extraction, and the edge extraction result is superimposed on the image to further enhance the edge feature.

[0032] Step S3: input the preprocessed continuous frame images into the improved target detection model to identify the category of objects in the dynamic environment.

[0033] In this embodiment, MobileNetV3-SSD is used as the target detection model, and an improved target detection model is constructed by introducing FPN and SE modules, which can significantly improve the accuracy and robustness of target detection, especially in small target detection and dynamic scenes. Among them, FPN (Feature Pyramid Network) enhances the detection ability of targets of different sizes through multi-scale feature fusion; SE module further optimizes the suppression of background noise and strengthens key features through adaptive channel weights, thereby reducing false detection and missed detection rates; in addition, the computational overhead of the introduced FPN and SE modules is small, and the overall performance remains lightweight and real-time. Through the above-mentioned improved MobileNetV3-SSD target detection network, target detection can be performed on the images captured by the camera, and dynamic objects such as pedestrians and vehicles in the scene can be identified. By detecting dynamic objects in real time, mismatching and tracking errors caused by dynamic objects in SLAM can be effectively avoided.

[0034] Specifically, the improved MobileNetV3-SSD target detection model mainly includes: MobileNet V3 backbone network, FPN module, SE module and SSD module.

[0035] First, MobileNet is a lightweight convolutional neural network designed for resource-constrained devices, which uses depth-separable convolution to reduce the number of model parameters and computational costs. Depth-separable convolution divides traditional convolution into two steps: depth-wise convolution and point-wise convolution, that is, spatial convolution is performed independently on each input channel to capture spatial features, and 1×1 point-wise convolution is performed on all channels to integrate channel information. By using this depth-separable convolution to extract features, the amount of computation can be further reduced.

[0036] In this embodiment, the preprocessed RGB image is input into the target detection model, and firstly passes through the MobileNet V3 backbone network, and extracts the multi-scale feature map integrating spatial and channel information through multiple layers of deep separable convolution. The MobileNetV3 network contains 29 layers in total, of which the main network has 17 layers, and the remaining 12 layers are derived networks of the 17th layer. This design ensures that it can extract higher quality features while being lightweight.

[0037] Secondly, in the above target detection model, an SE module is also introduced for adaptive channel weight optimization. In this embodiment, an SE module (compression and excitation module) is inserted after the inverted residual block (Inverted Residual Block) of each layer of the MobileNet V3 backbone network, which can significantly improve the expressiveness of the feature map and further optimize the detection accuracy. The SE module adaptively enhances key features and optimizes the feature map through the following three steps, including: (1) Perform global average pooling on the feature map through global pooling to extract the global information of the feature map; (2) Generate the weight of each channel through a two-layer fully connected network, learn the importance of each channel, and generate the importance weight of each channel; (3) Adjust the activation value of each channel according to the generated weight, enhance key features, suppress irrelevant information, and complete feature recalibration; (4) Output the feature map after optimization by the SE module. This feature map has stronger expressive power and is helpful for subsequent multi-scale feature fusion and target detection.

[0038] Then, for the optimized multi-scale feature map, FPN is also introduced into the target detection model to provide multi-scale fusion features, which can effectively improve the performance of the SSD detection head and enable it to better adapt to the detection needs of multi-scale targets. FPN is a multi-scale feature fusion technology that can combine shallow high-resolution features and deep semantic features to enhance the detection ability of small targets and targets of different sizes.

[0039] In this embodiment, based on the output of 6 layers of feature maps of different scales by the MobileNetV3 backbone network, FPN is used to fuse high-quality feature maps, which cover from high-resolution shallow features to low-resolution deep semantic features, so that the subsequent detection head (i.e., SSD module) can efficiently detect targets at various scales. The feature fusion includes: (1) The multi-scale feature map is input into FPN. FPN fuses the shallow high-resolution features with the deep high-semantic features through top-down and lateral connections, and outputs the multi-scale fused feature map after feature fusion. This high-quality feature map retains the detail information and has rich semantic information. (2) The output of the multi-scale fusion feature map after FPN fusion can enhance the detection ability of objects of different scales, especially improve the recognition effect of small objects.

[0040] Finally, the multi-scale fusion feature map output by FPN is input to the SSD (Single Shot Multibox Detector) module, which is a lightweight target detection framework that can apply convolution kernels of different sizes and proportions to feature maps at different levels to detect targets of different scales. In this embodiment, the target detection process of the multi-scale detection head is: (1) Apply convolution kernels of different sizes and proportions to the fused feature maps of each scale to meet the detection requirements of targets of different sizes; (2) The SSD module generates predictions on each feature layer and outputs the bounding box location and category probability of the target; (3) Finally, the module outputs the detected bounding boxes and category labels of dynamic objects (such as pedestrians, vehicles, etc.) in the scene, thereby achieving real-time target detection.

[0041] Preferably, the improved lightweight target detection model is pre-trained using the annotated COCO dataset (existing dataset), and the trained dataset category can be directly used for SLAM target recognition tasks in indoor environments; during the training process, the loss function of the model is constructed, and the dataset is continuously iterated for training until the loss function is minimized, completing the model training. The constructed loss function includes position loss and category loss, which can be expressed as: ; in, is the position loss, which is used to measure the position difference between the predicted bounding box and the true bounding box; It is the category loss, which is used to measure the difference between the predicted category and the true category; It is a balancing parameter used to adjust the relative weight of position loss and category loss, and is usually set to 1.

[0042] Furthermore, the localization loss is used to optimize the position and size of the predicted bounding box to make it closer to the real bounding box. The localization loss is defined as: ; in, is the number of positive samples; To predict bounding box parameters (including center coordinates, width, height, etc.); is the true bounding box parameter; Defined as: ; When the error is small, use The loss can help stabilize the training, while when the error is large, The loss reduces sensitivity to outliers.

[0043] In addition, the confidence loss is used to optimize the category prediction of each anchor box or target box. Considering the imbalance of positive and negative samples in dynamic and static situations, in order to avoid the shortcomings of cross entropy loss in the face of category imbalance, this embodiment adopts the following category loss function, which can reduce the contribution of easy-to-classify samples to the loss, thereby focusing on difficult-to-classify samples. The category loss can be expressed as: ; in, is the number of positive samples; is a collection of categories, including background categories; For the Samples are predicted as categories The probability of For Category The weights are used to balance the importance of different categories; This is an adjustment parameter used to adjust the attention level of difficult and easy samples. This category loss is achieved by introducing an adjustment factor , reducing the focus on easy-to-classify samples and enhancing the learning effect on difficult-to-classify samples, thereby improving the overall detection performance.

[0044] The design of the above loss function can effectively balance the bounding box positioning accuracy and category classification accuracy, especially in multi-scale and dynamic scenes, improve the detection accuracy and robustness of the model, and ensure the accuracy of the final semantic classification. Finally, the above pre-trained model is used to extract the semantic information of the object category in the dynamic environment, such as Figure 4As shown in the figure, the model can effectively identify various categories of dynamic objects in indoor navigation environments, such as person, chair, bottle, etc., and can significantly improve the detection accuracy and robustness of dynamic scenes while ensuring real-time performance.

[0045] Step S4, extracting feature points from the preprocessed continuous frame images, and after feature point matching, using an improved GC-RANSAC algorithm to distinguish the inliers and inliers in the matched feature points, and calculating a basic matrix based on the inliers.

[0046] In this embodiment, the ORB algorithm (a kind of feature detection and feature description algorithm) is used to extract feature points and their descriptors in the image, that is, the corner points are first detected by FAST (i.e., fast detection algorithm), and then the direction of the corner points is calculated according to the local gradient information, and a rotation-invariant BRIEF descriptor is generated; for the extracted feature points, brute force matching or the efficient FLANN matching algorithm (fast nearest neighbor matching algorithm) is used to match the feature points of the two frames of images, the Hamming distance is used to measure the similarity of the descriptors, and the feature points with high matching quality are screened out through a ratio test, so as to obtain the matching feature points of two consecutive frames.

[0047] Afterwards, the improved GC-RANSAC algorithm is used to process the continuous frame images and estimate the basic matrix, which provides good accuracy for the subsequent epipolar constraints and effectively reduces the interference of dynamic objects on pose estimation. Specifically, for a given set of 2D to 2D matching feature points in two continuous frames, , using the fundamental matrix Calculate the geometric relationship between the matched feature points and the epipolar lines, where the feature points The corresponding polar line satisfy: .

[0048] If the feature point If a point is located on its corresponding epipolar line, it is considered to be a static point, i.e., an interior point; on the contrary, if it deviates far from the epipolar line, it is likely to be a dynamic point, i.e., an exterior point. Therefore, the distance from the feature point to the epipolar line can be calculated to preliminarily divide the interior and exterior points.

[0049] Furthermore, an improved GC-RANSAC algorithm is used to transform the problem of dividing the inliers and outliers in the feature points into an energy minimization problem. The inliers in the feature points are identified through calculation and solution by combining the geometric, spatial and temporal consistency. This method can effectively improve the accuracy of inlier identification.

[0050] Specifically, based on the geometric relationship between the matching feature points and the basic matrix, as well as the spatial and temporal consistency of adjacent feature points, an energy function is constructed, which can be expressed as: ; in, represents the label assignment of the matching point set, and G is the adjacency graph; , Represents the feature points , Label assignment of is the adjustment parameter; the unary term Represents the geometric relationship between the matching points and the basic matrix, which is defined as: ; ; in, Represents the basic matrix The angle parameter, Represents the geometric distance between the matching point (i.e., matching feature point) pair and the basic matrix, is the preset threshold of internal and external points, function is the Gaussian kernel function, Represents matching point pairs is an interior point, otherwise it is an exterior point.

[0051] Usually, in the GC-RANSAC algorithm, the pairwise terms It is only used to describe the spatial consistency of adjacent feature points and is defined as: ; Furthermore, considering that the movement of an object is often continuous, the continuity of the object's motion trajectory can often provide more contextual information, and the temporal consistency between adjacent feature points can help infer whether they belong to the same object or are dynamic feature points. To this end, this embodiment improves the existing GC-RANSAC algorithm for dynamic scenes. By introducing temporal consistency based on geometric and spatial consistency, the motion trend and speed of feature points between consecutive frames are considered, so that dynamic points and static points can be distinguished more accurately.

[0052] Specifically, based on the original spatial consistency, temporal consistency is introduced , so that it not only depends on the geometric relationship, but also considers the dynamic characteristics in time. The pair can be expressed as: ; in, and Indicate point and In chronological order, is a temporal consistency function, which is used to measure the time difference or motion consistency between adjacent points. and come from the same object and move in unison, then ; If point and from different objects or have inconsistent motion, then , that is, the time series consistency is proportional to the time difference; The weight hyperparameter representing the temporal consistency term is used to adjust the impact of temporal consistency on the overall energy function.

[0053] By introducing temporal consistency as described above, the performance of the GC-RANSAC algorithm in dynamic scenes can be further improved, especially when dealing with matching points with motion features. By considering the time dimension, the motion pattern between adjacent points can be better captured, thereby improving the accuracy of inlier recognition and the ability to remove dynamic feature points.

[0054] In this embodiment, according to experience, and , by solving the minimization energy function, the optimal label allocation L of the feature point is calculated, and the optimal label allocation L is used to distinguish the internal and external points. That is, if the feature point Tags , then it means that the feature point Consistent with other points, this feature point is an interior point; otherwise, , it means that the feature point is consistent with other points. By adopting the improved GC-RANSAC algorithm, the internal points and external points can be distinguished more effectively, such as Figure 3 As shown in the figure, (a) is the static matching feature point determined without using the improved GC-RANSAC algorithm, and (b) is the static matching feature point determined by using the improved GC-RANSAC algorithm (it should be clear that the images used in the diagrams of this embodiment are all public images in the existing data set). After the improved GC-RANSAC filtering, the inliers are almost all static feature points, which can be used to accurately calculate the basic matrix. Subsequently, the basic matrix is ​​estimated based on the distinguished and identified inliers. , thus providing accurate prior information for the polar line constraint, ensuring that the dynamic feature point removal algorithm has higher accuracy.

[0055] Step S5: According to the basic matrix, epipolar constraints and multi-view geometric constraints are used to jointly identify the semantic information of the object category to screen and determine the static feature points among the feature points.

[0056] This embodiment combines geometric constraints (including epipolar constraints and multi-view geometric constraints) with semantic information (the above-mentioned target detection results) to suppress dynamic feature points, remove dynamic feature points and accurately screen static feature points, while reducing the reliance on deep learning. Figure 6 As shown, the specific steps include: Step S5.1: assign a weight to each feature point of each object according to the object category identified in the dynamic environment, wherein feature points corresponding to highly dynamic objects are assigned high weights, and feature points corresponding to low dynamic or static objects are assigned low weights.

[0057] Step S5.2: According to the basic matrix, the distance from each feature point to the epipolar line is calculated using the epipolar line constraint, and the weighted distance is calculated according to the distance and weight of each feature point, and the static feature points are preliminarily screened out according to the weighted distance.

[0058] Specifically, in a dynamic environment, considering that the existing target detection network can only identify the category of objects but cannot identify the motion state of these objects, in order to accurately eliminate dynamic points in stationary objects, this embodiment makes full use of polar line geometry to further subdivide the static and dynamic states of the feature points of dynamic objects.

[0059] First, the distance between the feature point and the epipolar line is calculated, and the feature points exceeding the threshold are regarded as dynamic points. In the initial pose estimation part, the basic matrix is ​​obtained by using the improved GC-RANSAC algorithm in the previous step , providing precise support for epipolar constraints in subsequent steps.

[0060] like Figure 5 As shown, the geometric relationship in the polar constraint is solved from the perspective of algebraic geometry, where and Represents the optical center of the camera, the straight line and is the polar line in the frame, and the matching feature points of the spatial point P corresponding to the previous frame and the current frame map are and , which can be expressed as: ; The corresponding homogeneous coordinates are: ; in, , and , is the pixel coordinate value of the feature point in the image coordinate system.

[0061] Through the basic matrix Calculate the epipolar line for the current frame , the formula is as follows: ; In the above formula, , , represents a line vector; is the basic matrix, which can be expressed as: ; In the above formula, There are 9 unknown numbers.

[0062] The epipolar constraint can be expressed as: ; Define feature points To the pole The offset distance between for: ; like is a dynamic space point, then point The offset distance is: ; like is a static space point, then the point The offset distance is: ; when When it is a static spatial feature point, its corresponding feature point Should be located at the pole However, due to the influence of noise in real scenes, the offset distance is usually greater than zero but lower than the empirical threshold. . On point Move to point When , the corresponding feature point With polar line The distance between , by comparing the distance With experience threshold , the dynamic state of the feature points can be determined.

[0063] Furthermore, in order to avoid over-reliance on deep learning in dynamic feature discrimination, this embodiment proposes a dynamic feature suppression method based on geometric constraints. First, semantic information is obtained through the target detection model, and the semantic information is used as the prior knowledge of the object category in the system. Since objects of different categories have different motion characteristics, the system assigns a weight value to each feature point according to the object category. Specifically, the weight value The range is set between 0 and 1. For highly dynamic objects (such as vehicles, pedestrians, etc.), the weight value is set to 0 because of their high probability of movement. Set to 0.75 to 1.0; for low-dynamic or static objects (such as buildings, tables, etc.), the possibility of movement is small, and the weight value is Set it to 0 to 0.25.

[0064] Secondly, the epipolar constraint is used to calculate the distance from the feature point to the corresponding epipolar line , and combined with the weight value of the feature point An evaluation is performed and compared with the empirical threshold. If the distance is greater than the empirical threshold, the feature point is considered to be a dynamic feature point; conversely, if the distance is less than the empirical threshold, the static feature points are preliminarily screened out.

[0065] Step S5.3: For the static feature points initially screened out, use multi-view geometric constraints to analyze their angle and depth changes in adjacent frames, determine their dynamic state, and then perform another screening to eliminate dynamic feature points and determine the final static feature points to improve the stability and robustness of the system.

[0066] Considering the point Move to point along the epipolar direction When , the corresponding feature point Also falls on the polar line Therefore, in this case, relying solely on the epipolar constraint cannot effectively identify these dynamic objects. To solve this problem, this embodiment introduces multi-view geometry constraints on the basis of the epipolar constraint to further detect dynamic feature points that are missed.

[0067] Specifically, if there are dynamic objects in the scene, the depth values ​​and angles of the corresponding feature points in the current frame and the reference frame will change significantly. Based on this phenomenon, these dynamic feature points can be identified by depth constraints and angle constraints. The principle is as follows: Figure 7 shown.

[0068] First, five key frames with the highest consistency with the current frame are selected as reference frames. Determined by the following formula: ; in, is the consistency measure between frames, , are the increments of position and attitude respectively.

[0069] By calculating the depth value of the key point in the current frame and the projected depth value of the keyframe in the current frame Get depth information, that is, depth difference , which can be defined as: ; Parallax Angle yes and The angle between the back projections of It can be expressed as: ; In the formula, the first perspective vector Represents the vector from the camera position to the feature point in the current frame, the second perspective vector Represents the vector from the camera position to the feature point in the key frame, is the dot product of the two view vectors in the current frame and the historical frame, and Respectively represent the modulus lengths of the two view vectors.

[0070] Combining the above depth information and perspective information, if or , then the feature point is regarded as a dynamic feature point under multi-view geometric constraints, where and Represent the depth threshold and angle threshold respectively.

[0071] By applying multi-view geometric constraints to analyze the angle and depth changes of feature points in adjacent frames, static feature points can be further judged and screened. Through this criterion, the status of all feature points in the current frame can be accurately judged one by one.

[0072] Step S6: estimate the camera pose based on the screened static feature points, optimize using the BA optimization algorithm, build an environment map based on the optimized results, and perform dynamic environment navigation.

[0073] In this embodiment, according to the feature points extracted based on the RGB image and combined with the depth image, the initial three-dimensional coordinates of these feature points, that is, the three-dimensional landmark point coordinates, are directly obtained; then, the screened static feature points are combined with the camera intrinsic parameters and the camera pose estimation method is used to estimate the 6-DOF pose of the camera; then, the observation data of the feature points in multiple frames are combined with the camera pose, and the depth information is used to correct the accuracy of the three-dimensional coordinates of the feature points; finally, the three-dimensional landmark point coordinates and the camera pose are jointly optimized through the bundling optimization (BA optimization) algorithm to reduce projection errors and improve accuracy.

[0074] Specifically, the BA optimization method is used to optimize the camera's 6-DOF pose and 3D landmark points at the same time. BA optimization is a technology that extracts the optimal 3D model and camera parameters (including intrinsic and extrinsic parameters) in visual reconstruction. By adjusting the camera pose and the spatial position of the feature points, the light (bundles of lightrays) reflected from each feature point can converge to the camera optical center after adjustment, thereby optimizing the poses of multiple cameras and the spatial coordinates of the landmark points under the poses. In SLAM, the BA optimization algorithm is used to optimize the camera pose and the spatial position of the feature points, as follows: First, according to the optimized variable pose and the coordinates of the landmark points , construct the reprojection error function as: ; In the formula, Represents the observation value, that is, in the pose and Point The projection observation projection of ; The projection function of the model is used to project the point coordinates in the 3D space onto the image plane.

[0075] Secondly, the Gauss-Newton method is used to solve the objective function and accurately estimate the position of the camera in each frame. (including rotation matrix and translation vector) and 3D landmark point coordinates , which can be further used to build more accurate environmental maps and perform dynamic environmental navigation.

[0076] Embodiment 2 This embodiment provides a dynamic environment navigation system based on semantic and geometric fusion information, including: An image acquisition module is used to acquire continuous frame images of a dynamic environment; the images include RGB images and depth images; An image preprocessing module, used for performing image preprocessing on continuous frame images; An object category recognition module is used to input the preprocessed continuous frame images into the improved target detection model to identify the category of objects in a dynamic environment; The image feature point extraction and matching module is used to extract the feature points in the continuous frame images after preprocessing. After the feature points are matched, the improved GC-RANSAC algorithm is used to distinguish the internal and external points in the matching feature points, and the basic matrix is ​​calculated based on the internal points. The static feature point screening module is used to screen and determine the static feature points among the feature points based on the basic matrix, using polar constraints and multi-view geometric constraints, and combining the semantic information of the recognized object category; The pose optimization and navigation module is used to estimate the camera pose based on the screened static feature points, and then optimize it using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

[0077] Embodiment 3 This embodiment provides an electronic device, including: a memory, used to store executable instructions; and a processor, used to implement the above method provided in this embodiment when executing the executable instructions stored in the memory.

[0078] Embodiment 4 This embodiment also provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will be caused to execute the above method provided in this embodiment.

[0079] Embodiment 5 This embodiment provides a computer program product, which includes an executable instruction, which is a computer instruction; the executable instruction is stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instruction from the computer-readable storage medium and the processor executes the executable instruction, the electronic device executes the above method provided in this embodiment.

[0080] The steps involved in the above embodiments 2 to 5 correspond to the method embodiment 1. For the specific implementation, please refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0081] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0082] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not a limitation of the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A dynamic environment navigation method based on semantic and geometric fusion information, characterized in that: include: Acquire continuous frame images of a dynamic environment; the images include RGB images and depth images; Perform image preprocessing on continuous frame images; The preprocessed continuous frame images are input into the improved target detection model to identify the categories of objects in the dynamic environment; Extract feature points from continuous frame images after preprocessing. After feature point matching, use the improved GC-RANSAC algorithm to distinguish the internal and external points in the matched feature points, and calculate the basic matrix based on the internal points. According to the basic matrix, epipolar constraints and multi-view geometric constraints are used to jointly identify the semantic information of the object category to screen and determine the static feature points among the feature points; The camera pose is estimated based on the screened static feature points, and then optimized using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

2. A dynamic environment navigation method based on semantic and geometric fusion information as claimed in claim 1, characterized in that: The image preprocessing comprises: Performing color space conversion on the acquired original RGB image, converting the original RGB image from the RGB space to the Lab color space, and acquiring a Lab image; For the brightness channel component in Lab image, multi-scale Retinex algorithm is used to perform multi-scale brightness enhancement, and then contrast correction, adaptive brightness adjustment and denoising are performed on the enhanced brightness component in turn; Based on the adjusted brightness channel component, the enhanced image is reconstructed by combining it with other channel components in the original Lab color space; Based on the reconstructed Lab image, the enhanced RGB image is obtained through color space conversion; The Laplace operator is used to perform edge extraction and edge superposition on the enhanced RGB image to obtain the preprocessed image.

3. A dynamic environment navigation method based on semantic and geometric fusion information as claimed in claim 1, characterized in that: The improved GC-RANSAC algorithm is used to transform the problem of dividing the inner and outer points in the feature points into an energy minimization problem. The inner points in the feature points are identified through solving and calculation by combining geometric, spatial, and temporal consistency, including: Based on the geometric relationship between the matching feature points and the basic matrix, as well as the spatial and temporal consistency of adjacent feature points, an energy function is constructed as follows: ； in, represents the label assignment of the matching point set, and G is the adjacency graph; is the adjustment parameter; the unary term Represents the geometric relationship between the matching point and the basic matrix; pair terms It is used to describe the spatial consistency and temporal consistency of adjacent feature points: ； in, Represents the geometric relationship between the matching points and the basic matrix, , Represents the feature points , The label distribution of and Indicate point and In chronological order, is a temporal consistency function, which is used to measure the time difference or motion consistency between adjacent points. and come from the same object and move in unison, then ; If point and from different objects or have inconsistent motion, then ; represents the weight hyperparameter of the temporal consistency term; The solution is performed with the goal of minimizing the energy function, and the optimal label allocation L of the feature points is calculated. The optimal label allocation L is used to distinguish internal points from external points.

4. A dynamic environment navigation method based on semantic and geometric fusion information as claimed in claim 1, characterized in that: MobileNetV3-SSD is used as the target detection model, and the FPN module and SE module are introduced to build an improved target detection model; The preprocessed RGB image is input into the improved object detection model, first passing through the MobileNet V3 backbone network, and then through multiple layers of deep separable convolution to extract a multi-scale feature map that integrates spatial and channel information; The feature map of each scale is input into the SE module to learn and generate the importance weight of each channel, and the optimized feature map is obtained by channel weighting; The optimized multi-scale feature map is input into the FPN module to fuse the shallow high-resolution features with the deep high-semantic features, and output the multi-scale fusion feature map after feature fusion; The multi-scale fusion feature map is input into the SSD module, which detects and generates predictions on each layer of the feature map and outputs the bounding box and category label of the target.

5. A dynamic environment navigation method based on semantic and geometric fusion information as claimed in claim 4, characterized in that: The loss function of the improved target detection model includes position loss and category loss; The position loss for: ； In the above formula, is the number of positive samples; To predict bounding box parameters, including center coordinates, width, and height; is the real bounding box parameter; function ; The class loss is: ； In the above formula, is the number of positive samples; is a collection of categories; For the Samples are predicted as categories probability; For Category The weight of To adjust the parameters.

6. A dynamic environment navigation method based on semantic and geometric fusion information as claimed in claim 1, characterized in that: The method uses polar constraints and multi-view geometric constraints based on the basic matrix, combines the semantic information of the recognized object category, and screens and determines the static feature points in the feature points, including: According to the object category in the identified dynamic environment, a weight is assigned to each feature point of each object; According to the basic matrix, the distance from each feature point to the epipolar line is calculated using the epipolar line constraint; According to the distance and weight of each feature point, the weighted distance is calculated, and the static feature points are preliminarily screened out according to the weighted distance; For the static feature points initially screened out, multi-view geometric constraints are used to perform another screening to determine the final static feature points.

7. A dynamic environment navigation system based on semantic and geometric fusion information, characterized in that: include: An image acquisition module is used to acquire continuous frame images of a dynamic environment; the images include RGB images and depth images; An image preprocessing module, used for performing image preprocessing on continuous frame images; An object category recognition module is used to input the preprocessed continuous frame images into the improved target detection model to identify the category of objects in a dynamic environment; The image feature point extraction and matching module is used to extract the feature points in the continuous frame images after preprocessing. After the feature points are matched, the improved GC-RANSAC algorithm is used to distinguish the internal and external points in the matching feature points, and the basic matrix is ​​calculated based on the internal points. The static feature point screening module is used to screen and determine the static feature points among the feature points based on the basic matrix, using polar constraints and multi-view geometric constraints, and combining the semantic information of the recognized object category; The pose optimization and navigation module is used to estimate the camera pose based on the screened static feature points, and then optimize it using the BA optimization algorithm. According to the optimized results, the environment map is constructed and dynamic environment navigation is performed.

8. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the dynamic environment navigation method based on semantic and geometric fusion information as described in any one of claims 1 to 6 when executing the executable instructions stored in the memory.

9. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause the processor to execute the executable instructions to implement the dynamic environment navigation method based on semantic and geometric fusion information as described in any one of claims 1-6.

10. A computer program product, characterized in that The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the dynamic environment navigation method based on semantic and geometric fusion information described in any one of claims 1 to 6 is implemented.

Images