Dynamic vision-based positioning and mapping method, system, device and medium

Abstract

The application discloses a positioning and mapping method and system based on dynamic vision, equipment and a medium, and relates to the technical field of simultaneous localization and mapping. The method comprises the following steps: extracting feature points based on dynamic detection information of a dynamic object to obtain a target feature set; compensating a motion trajectory of the dynamic object based on a predicted pose of a vision sensor to obtain an object motion state; determining an object detection optimization variable based on the object motion state, the target feature set and a motion parameter of the vision sensor; and fusing the object detection optimization variable to perform real-time detection on the dynamic object, obtaining a real-time detection result of the dynamic object and generating a static map to which the object belongs. The application can accurately and real-timely track the motion trajectory of the dynamic object, thereby improving the positioning accuracy and real-timeliness of the dynamic object, and further improving the dynamic obstacle detection efficiency and the static map construction accuracy.

Description

Technical Field

[0001] This application relates to the field of simultaneous localization and mapping (SLAM) technology, specifically to a method, system, device, and medium for localization and mapping based on dynamic vision. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a technology that enables robots or autonomous vehicles to navigate autonomously in unknown environments. Traditional SLAM systems are usually based on the assumption of a static environment, that is, that all objects in the environment are considered to be stationary. However, in real-world application scenarios, such as urban roads, shopping malls, and stations, there are a large number of dynamic objects (pedestrians, vehicles, animals, etc.), which can seriously affect the localization accuracy and map quality of SLAM systems. Currently, conventional dynamic SLAM methods mainly use the following strategies to handle dynamic objects: (1) a motion consistency test-based method, which identifies dynamic regions by detecting feature points that do not conform to the camera motion model; (2) a semantic segmentation-based method, which uses a deep learning network to perform pixel-level segmentation of images and identify potential dynamic object categories.

[0003] (3) Methods based on optical flow or scene flow detect dynamic regions by analyzing pixel motion.

[0004] However, these methods have the following limitations: First, motion consistency-based methods require sufficient static feature points to estimate camera motion, which can easily fail in scenes with a large proportion of dynamic objects. Second, semantic segmentation-based methods are computationally intensive, making it difficult to meet real-time requirements, and they also require predefined categories of dynamic objects, lacking flexibility. Third, optical flow-based methods are sensitive to rapid motion and occlusion, making them prone to false detections. Finally, most existing methods can only detect dynamic regions and lack the ability to accurately locate and track dynamic objects.

[0005] The patent "A Multi-Sensor Fusion SLAM Method and System for Dynamic Environments" (Publication No. CN118209101 A) discloses a method for acquiring sensor data from inertial sensors, wheel speedometers, visual cameras, and LiDAR. After pre-integrating the sensor data from the inertial and wheel speedometers, it uses loosely coupled pose estimation and employs geometric methods to detect and filter dynamic points. The visual camera sensor data undergoes semantic segmentation, and the detection results are clustered. Furthermore, reprojection error estimation is used to filter dynamic points during pose estimation, obtaining the current optimal pose. Static point clouds are extracted from the LiDAR sensor data, and the static point clouds are associated with visual feature depth. Based on the inertial sensor pre-integration residuals, wheel speedometer pre-integration residuals, static point reprojection residuals, and edge detection residuals, a pose map is output through sliding window optimization, loop closure detection, and pose map optimization. However, this scheme still has limitations in handling dynamic objects. Its geometric method, which judges dynamics based on the distance threshold between feature points in two frames, has a large error range, thus interfering with pose optimization and resulting in poor map reliability. Summary of the Invention

[0006] The purpose of this application is to address the problem that existing technologies suffer from low object positioning accuracy due to their inability to track the motion trajectory of dynamic objects, resulting in low efficiency in dynamic obstacle detection and poor mapping. This application proposes a dynamic vision-based positioning and mapping method, system, device, and medium. By detecting and extracting feature points from dynamic objects to compensate for and predict their motion state, and then fusing visual sensor parameters to optimize the detection of dynamic objects, real-time positioning and tracking of dynamic objects are achieved, thereby improving the efficiency of dynamic obstacle detection and the accuracy of static map construction.

[0007] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0008] In a first aspect, embodiments of this application provide a localization and mapping method based on dynamic vision. The method includes: extracting feature points based on dynamic detection information of dynamic objects to obtain a target feature set; compensating for the motion trajectory of dynamic objects based on pose prediction by a visual sensor to obtain the motion state of the objects; determining object detection optimization variables based on the object motion state, the target feature set, and motion parameters of the visual sensor; fusing the object detection optimization variables to perform real-time detection of dynamic objects, obtaining real-time detection results of dynamic objects, and generating a static map to which the objects belong.

[0009] This scheme improves mapping accuracy and completeness by acquiring target feature sets to provide sufficient and reliable feature support for subsequent static map construction. Predictive poses from visual sensors can predict the motion state of dynamic objects in the next moment, solving the problem of inaccurate estimation of the motion state of dynamic objects relative to the world coordinate system due to ignoring the influence of camera motion. This improves dynamic object tracking accuracy and further ensures obstacle detection effectiveness and mapping reliability. By fusing various optimization variables, the scheme can effectively and accurately estimate the trajectory of dynamic objects in complex environments, improving positioning accuracy. Simultaneously, the accurate dynamic object detection results are used to inversely affect target feature extraction, thereby improving the accuracy of target features and ensuring the reliability of the static map corresponding to the dynamic object's trajectory.

[0010] Preferably, the step of extracting feature points based on the dynamic detection information of dynamic objects to obtain a target feature set includes:

[0011] A dynamic detection model is constructed using an improved DETR framework based on historical dynamic detection images. Based on the dynamic detection model, the acquired images of dynamic objects are identified to obtain a set of dynamic object detection images containing the category of the dynamic object and the bounding box of the identifier, which is used as dynamic detection information. Based on the dynamic detection information, feature regions are identified in each image, and feature points are extracted based on the feature regions to obtain the target feature set.

[0012] Preferably, the dynamic detection model constructed using the improved DETR framework based on historical dynamic detection images includes:

[0013] Object type labeling is performed on historical dynamic detection images. The labeled images are then converted to obtain an image set containing target categories and bounding boxes. This image set is used as input parameters for a convolutional neural network for model pre-training, and a semantic feature layer is selected to output a feature map. A dimensionality-reducing encoder encodes the feature map to generate feature vectors. The dual-branch network of the decoder performs object type prediction and object bounding box labeling on the feature vectors, outputting the target object type and object detection box of the feature map to construct a basic detection model. Based on the target object type and object bounding box, a bipartite graph matching loss function is used to optimize the basic detection model to obtain the dynamic detection model.

[0014] Preferably, the step of identifying feature regions for each image based on the dynamic detection information and extracting feature points based on the feature regions to obtain a target feature set includes: identifying regions for each dynamic detection image in the dynamic detection information, dividing the image region into static feature regions and dynamic feature regions; extracting first feature points based on the static feature regions; identifying latent static sub-regions in the dynamic feature regions based on the temporal synchronous motion characteristics of the dynamic feature regions, extracting second feature points based on the latent static sub-regions; and constructing a target feature set based on the first feature points and the second feature points.

[0015] Preferably, the step of compensating for the motion trajectory of a dynamic object based on the predicted pose of a visual sensor to obtain the object's motion state includes: obtaining the predicted pose of the visual sensor at future moments based on the pose transformation status of the visual sensor; extracting object detection boxes based on the dynamic detection information, compensating the detection boxes of the dynamic object in consecutive frames according to the predicted pose, and thus obtaining the visual residual of the dynamic object in consecutive frames; calculating the spatial motion velocity of the dynamic object based on the visual residual, and then predicting the position of the object detection boxes at future moments according to the spatial motion velocity, so as to obtain a motion state including the object's motion velocity and position.

[0016] Preferably, determining the object detection optimization variables based on the object's motion state, target feature set, and visual sensor motion parameters includes: extracting the object's position information in consecutive frames based on the object's motion state, calculating the motion residual between consecutive frames, and using it as the first object detection optimization variable; extracting the object's motion parameters based on the object's motion state, fusing the observations of the object detection box to construct the object detection error, and determining the second object detection optimization variable; performing feature point reprojection based on the target feature set, obtaining the static reprojection error of each static feature point, and determining the third object detection optimization variable; obtaining the visual sensor's positioning error based on the GPS positioning information and actual displacement information, and using it as the fourth object detection optimization variable; and determining the fifth object detection optimization variable based on the visual sensor's predicted pose error.

[0017] Preferably, the step of fusing object detection optimization variables to perform real-time detection of dynamic objects, obtaining real-time detection results of dynamic objects, and generating a static map to which the objects belong includes: fusing the first object detection optimization variable, the second object detection optimization variable, the third object detection optimization variable, the fourth object detection optimization variable, and the fifth object detection optimization variable to establish an optimization objective function; solving the optimization objective function to obtain the real-time pose of the visual sensor and the real-time detection information of the dynamic objects under the current pose, thereby obtaining the motion trajectory of the dynamic objects and generating the corresponding static map.

[0018] Secondly, embodiments of this application provide a localization and mapping system based on dynamic vision, comprising: a dynamic object detection module for extracting feature points based on the dynamic detection information of dynamic objects to obtain a target feature set; an object motion tracking module for compensating the motion trajectory of dynamic objects based on pose prediction by a visual sensor to obtain the motion state of the objects; an optimization variable acquisition module for determining object detection optimization variables based on the object motion state, the target feature set, and motion parameters of the visual sensor; and a detection mapping optimization module for fusing the object detection optimization variables to perform real-time detection of dynamic objects, obtaining real-time detection results of dynamic objects, and generating a static map to which the objects belong.

[0019] Thirdly, embodiments of this application provide a computer device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to execute the program stored in the memory to implement the steps of the dynamic vision-based localization and mapping method described in the first aspect above.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the dynamic vision-based localization and mapping method described in the first aspect above.

[0021] The beneficial effects of this application are:

[0022] 1. By extracting and supplementing static features, the problem of loss of effective feature points caused by interference from dynamic regions is overcome, providing sufficient and reliable feature support for subsequent static map construction, thereby ensuring the accuracy and completeness of the map construction;

[0023] 2. The predicted pose of the visual sensor can predict the motion state of the dynamic object in the next moment, which solves the problem of inaccurate estimation of the motion state of the dynamic object relative to the world coordinate system due to ignoring the influence of the camera's own motion. This improves the tracking accuracy of the dynamic object and further ensures the obstacle detection effect and the reliability of the mapping.

[0024] 3. By integrating various optimization variables, the motion trajectory of dynamic objects in complex environments can be effectively and accurately estimated, improving positioning accuracy. At the same time, based on the accurate dynamic object detection results, the target feature extraction is back-acted, thereby improving the accuracy of target features and ensuring the reliability of the static map corresponding to the motion trajectory of dynamic objects. Attached Figure Description

[0025] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0026] Figure 1 A flowchart of a dynamic vision-based localization and mapping method provided in an embodiment of this application.

[0027] Figure 2 This is a schematic diagram of dynamic object detection and static map provided in an embodiment of this application.

[0028] Figure 3 This is a schematic diagram of a system module for a localization and mapping method based on dynamic vision provided in an embodiment of this application.

[0029] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of this application and are only used to explain this application. They do not limit the scope of protection of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Some of the terms or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0032] DETR framework: It is an end-to-end object detection framework based on Transformer. Its structure includes a CNN backbone network for extracting feature maps of the input image, a Transformer encoder, a decoder and a detection head. It integrates CNN feature extraction, Transformer encoder and decoder to directly generate detection results from the input image, simplifying the detection process.

[0033] MobileNetV3 is a lightweight neural network model that improves upon the MnasNet structure through Neural Architecture Search (NAS) technology. It combines depthwise separable convolutions and linear bottleneck inverse residual structures, achieving a 3.2% improvement in ImageNet classification accuracy compared to MobileNetV2 while maintaining the same computational cost.

[0034] ResNet (convolutional layer): refers to a residual network used to extract features from input data;

[0035] Transformer: A deep learning model with self-attention mechanism at its core, enabling efficient parallel computing;

[0036] ORB feature extraction algorithm (Oriented FAST and Rotated BRIEF) is a feature extraction algorithm that combines FAST keypoint detection and BRIEF descriptor. It is mainly used for feature point detection and description in image processing.

[0037] IMU integration: The measurement data of the IMU (Inertial Measurement Unit) is converted into physical quantities such as position, velocity, and attitude through mathematical methods. Using the raw data from the integrating gyroscope and accelerometer, combined with the noise model and bias correction, the motion trajectory of the device in three-dimensional space is calculated.

[0038] Example 1: As Figure 1 As shown, a localization and mapping method based on dynamic vision includes steps S1-S4, wherein:

[0039] S1. Extract feature points based on the dynamic detection information of dynamic objects to obtain the target feature set.

[0040] Specifically, step S1 includes:

[0041] S11. A dynamic detection model is constructed based on historical dynamic detection images using an improved DETR framework;

[0042] S12. Based on the dynamic detection model, identify the acquired images of dynamic objects, obtain a set of dynamic object detection images containing the category of dynamic objects and the bounding box of the identification, and use it as dynamic detection information.

[0043] S13. Based on the dynamic detection information, perform feature region identification on each image, and extract feature points based on the feature regions to obtain the target feature set.

[0044] Specifically, step S11 includes:

[0045] Object type labeling is performed based on historical dynamic detection images. The labeled images are then converted to obtain an image set containing target categories and bounding boxes.

[0046] The image set is used as input parameters for the convolutional neural network to pre-train the model, and the semantic feature layer is selected to output the feature map.

[0047] The encoder performs feature encoding on the feature map based on dimensionality reduction to generate feature vectors;

[0048] Based on the dual-branch network of the decoder, the object type prediction and object bounding box annotation are performed on the feature vectors respectively, and the target object type and object detection box of the feature map are output to build a basic detection model.

[0049] Based on the target object type and object bounding box, the basic detection model is optimized using a bipartite graph matching loss function to obtain the dynamic detection model.

[0050] In this embodiment, the convolutional neural network used for model pre-training based on the image set is MobileNetV3, which replaces the original ResNet convolutional layers in the DETR network. This reduces computational cost while accurately extracting multi-scale features of the image, thereby improving the efficiency of dynamic object detection. The dimensionality reduction encoder refers to reducing the Transformer dimension and the number of Transformer layers, specifically reducing the hidden dimension of the original DETR from 256 to 128, the number of attention heads from 8 to 4, and the number of layers in both the encoder and decoder from 6 to 3. This reduces the processing time per frame for dynamic object detection data and improves processing efficiency.

[0051] In some embodiments, within the improved DETR framework, the MobileNetV3 backbone network extracts multi-scale feature maps from image data acquired by the visual sensor, which are then flattened in space to form a one-dimensional feature vector sequence. Since the Transformer itself lacks the ability to perceive sequence order, a learnable positional encoding is superimposed on each feature vector. This encoding contains the two-dimensional spatial coordinates of the feature in the original image. The feature sequence infused with positional information is input into the Transformer encoder. Based on the encoder's internal multi-head self-attention mechanism, the association weight between any two feature vectors in the feature sequence is calculated. This mechanism allows each feature vector to aggregate information from the entire image, resulting in a feature vector containing global image context information. This feature vector is used as the first input to the decoder. Simultaneously, the number of probes used to extract specific information is dynamically set according to the scene detection speed and accuracy requirements, and these probes are used as the second input to the decoder. The probes interact with the feature vector output by the encoder using a cross-attention mechanism to obtain a prediction result for the dynamic object. This prediction result is an embedding vector containing object-related information. The Transformer decoder includes a dual-branch network: a classification branch network and a regression branch network. The dual-branch network decodes the embedding vectors and outputs the type information of the dynamic object and the bounding box of the dynamic object.

[0052] Furthermore, in this embodiment, when training the detection model, a bipartite graph matching loss function is used to optimize the model. The bipartite graph matching loss function includes at least bounding box loss and classification loss. The bounding box loss is calculated by predicting the coordinates of the bounding box and the ground truth box, and the classification loss is calculated by predicting the class and the ground truth class. Then, the model parameters are adjusted and optimized based on the bounding box loss and the classification loss.

[0053] Specifically, step S13 includes:

[0054] For each dynamic detection image in the dynamic detection information, region recognition is performed to divide the image region into static feature regions and dynamic feature regions;

[0055] Extracting the first feature point based on the static feature region;

[0056] Based on the temporal synchronous motion characteristics of the dynamic feature region, the hidden static sub-regions in the dynamic feature region are identified, and the second feature points are extracted based on the hidden static sub-regions.

[0057] Construct a target feature set based on the first feature point and the second feature point.

[0058] In this embodiment, region identification is performed on each detected image containing an object bounding box. A dynamic mask is set to mark dynamic regions as 1 and the remaining regions in the image as static regions as 0. Morphological dilation is applied to the dynamic mask to improve the feature extraction quality of dynamic and static boundary regions. For static regions, all feature points within the region are extracted based on the ORB feature extraction algorithm, i.e., key points that meet the definition of corner points within the image region and have stable orientations. A rotation-invariant binary descriptor is generated for each key point, which contains position and orientation, and is used as a static feature point (the first feature point in this embodiment).

[0059] Furthermore, the dynamic region is validated to determine whether there is a hidden static sub-region, that is, a static background region in a dynamic object. If it exists, feature points within the hidden static sub-region are extracted, used as second feature points, and merged with the first feature points to obtain the final target feature set.

[0060] In some optional embodiments, the latent static sub-regions in the dynamic feature region are identified based on the temporal synchronous motion characteristics of the dynamic feature region, specifically including:

[0061] Perform superpixel segmentation on the dynamic region to obtain several dynamic sub-regions;

[0062] The pixel offset of all pixels in the dynamic sub-region is calculated based on the motion transformation matrix of the visual sensor between consecutive frames. The motion residual of the pixel is calculated based on the actual motion of the pixel and the pixel offset. If the residual is less than a preset residual threshold, the corresponding dynamic sub-region is determined as an initial static sub-region.

[0063] Based on the initial static sub-region, extract the sub-region texture features and edge features, calculate the texture feature histogram variance based on the regional texture features, and obtain the average edge length based on the edge features. When the texture feature histogram variance is greater than the variance threshold and the average edge length is greater than the edge length threshold, the initial static sub-region is regarded as the latent static sub-region.

[0064] Among them, the motion residual of a pixel is calculated by comparing the actual motion of the pixel with the pixel offset. This residual is the Euclidean distance between the actual motion of the pixel and the pixel offset. The pixel offset refers to the predicted motion position of the pixel.

[0065] It should be noted that by using dynamic masking to divide image regions, accurate segmentation of static regions is achieved, avoiding interference from false feature points in dynamic regions on static mapping. Feature points of static regions in the detected image are extracted. Considering that dynamic objects are not entirely devoid of static parts, effective static feature points in dynamic regions are filtered out. By identifying static sub-regions and extracting features, the loss of static feature points can be reduced, especially in scenes where dynamic objects account for a high proportion. This supplements the completeness of static feature points, overcoming the problem of incomplete mapping caused by discarding dynamic regions in traditional SLAM systems. This provides an effective and sufficient set of feature points for subsequent static map construction, improving the reliability, accuracy, and completeness of the static map.

[0066] S2. Based on the visual sensor, the motion trajectory of the dynamic object is compensated to obtain the motion state of the object.

[0067] Specifically, step S2 includes:

[0068] Based on the pose transformation situation of the visual sensor, the predicted pose of the visual sensor at future moments is obtained.

[0069] Based on the dynamic detection information, the object detection box is extracted, and the detection box of the dynamic object in the continuous frames is compensated according to the predicted pose, so as to obtain the visual residual of the dynamic object in the continuous frames.

[0070] The spatial velocity of the dynamic object is calculated based on the visual residual, and then the position of the object detection box at a future time is predicted based on the spatial velocity, so as to obtain the motion state containing the object's velocity and position.

[0071] In some embodiments, the visual sensor is a device capable of acquiring visual information about dynamic objects, such as a camera.

[0072] In this embodiment, based on IMU integration and the current pose of the visual sensor, the relative motion state of the visual sensor at the current moment is obtained. Then, based on the relative motion state and the current pose, the predicted pose of the visual sensor at the next moment is obtained. For detected dynamic objects, whose two-dimensional bounding box information has been obtained, the two-dimensional bounding box of the object at the next moment can be predicted based on the predicted pose of the visual sensor through the device motion compensation function, thereby obtaining the visual residual of the dynamic object in consecutive frames (i.e., the current frame and the next frame).

[0073] The device motion compensation function is primarily used to calculate new bounding boxes generated solely by the movement of the vision sensor. If the moving object is stationary, its bounding box at the current moment will appear in a new position on the image at the next moment due to the movement of the vision sensor.

[0074] Therefore, in this embodiment, the bounding box (i.e., detection box) of a dynamic object in the future is predicted based on the predicted pose of the visual sensor to compensate for changes in the bounding box. The visual residual of the object in consecutive frames is obtained by calculating the difference between the current bounding box position and the predicted position of the bounding box in the future. The magnitude of the visual residual can then indicate whether the object is moving independently, stationary, or moving in the same direction and at the same speed as the visual sensor. Furthermore, the 3D motion velocity (i.e., spatial motion velocity) of the dynamic object is estimated based on the visual residual. The predicted pose of the visual sensor can be initially estimated using IMU integration.

[0075] Furthermore, by combining the spatial motion speed with the current position of the detection box of the dynamic object, the position of the detection box at the next moment is obtained, thereby compensating for the motion information of the object on the image plane, completing the acquisition of the motion state including the motion speed and position of the dynamic object, and improving the estimation accuracy of the motion state of the dynamic object.

[0076] In this embodiment, by obtaining the position of the bounding box of a dynamic object at the next moment, the position sequence of the object's boundary center point in the image coordinate system as time changes can be obtained. Thus, the size change of the bounding box at each position in the position sequence reflects whether the object is moving closer to or further away from the visual sensor. Combined with the predicted pose of the visual sensor, the motion information of the object in the image plane is compensated, and the true motion state of the object in three-dimensional space, including speed and direction, can be obtained, ultimately achieving continuous tracking of the object.

[0077] It's important to note that by acquiring the object's true motion state in 3D space, we ensure that the bounding box projected onto the 2D image matches the bounding box actually detected by the DETR network. Simultaneously, this ensures the accuracy of continuous object tracking. Specifically, based on the object's spatial velocity and the current position of the detection box, we obtain the box's position at the next moment. This requires that the tracked object be the same dynamic object, and that the object's motion state (e.g., velocity) should not undergo drastic changes over consecutive moments. Acquiring the object's true motion state in 3D space allows us to compare it with historical motion states to determine if changes exceed a preset range. If they do, it indicates a tracking error, allowing for timely adjustments to the visual sensor's pose and ensuring the accuracy of continuous object tracking.

[0078] S3. Determine the object detection optimization variables based on the object's motion state, target feature set, and visual sensor motion parameters.

[0079] Specifically, step S3 includes:

[0080] Based on the object's motion state, extract the object's position information in consecutive frames, calculate the object's motion residual between consecutive frames, and use it as the first object detection optimization variable;

[0081] Based on the object's motion state, the motion parameters of the object are extracted, and the observations of the object detection box are fused to construct the object detection error, so as to determine the second object detection optimization variable;

[0082] Based on the target feature set, feature point reprojection is performed to obtain the static reprojection error of each static feature point, so as to determine the third object detection optimization variable.

[0083] The positioning error of the visual sensor is obtained by combining GPS positioning information and actual displacement information from the visual sensor, and is used as the fourth object detection optimization variable.

[0084] The prediction pose error based on the visual sensor determines the optimization variables for the fifth object detection.

[0085] In this embodiment, in traditional SLAM, camera localization, dynamic object detection, and static mapping are often independent modules with no information sharing. For example, camera localization errors are not fed back to dynamic object tracking, and dynamic detection results are not used to optimize static mapping, leading to inconsistent outputs from the three. Inaccurate camera positioning can cause a mismatch between the bounding boxes of dynamic objects and the map, resulting in poor overall performance. Therefore, this embodiment considers object motion residuals in consecutive frames, object detection errors, static reprojection errors, visual sensor localization errors, and predicted pose errors. By fusing these variables, the independent optimization of each submodule of the SLAM system is transformed into global collaborative optimization. This solves the cascading problems of inaccurate dynamic object tracking, visual sensor localization drift, and static map distortion, ensuring the accuracy of dynamic obstacle detection, improving detection efficiency, and simultaneously enhancing the reliability of the static map.

[0086] Among these, each variable also serves as an optimization constraint. The first object detection optimization variable serves as a constraint on the accuracy of continuous object tracking, ensuring that the same object is being tracked. The second object detection optimization variable serves as a consistency constraint for object motion state detection, associating the predicted motion state (velocity, position) of dynamic objects with the visual detection results, ensuring that the 2D bounding box observations of objects detected by DETR on the current frame image are consistent with the 2D bounding boxes converted from the 3D points projected onto the image plane according to the object size. The third object detection optimization variable serves as a static feature point extraction constraint, ensuring that all feature points are static feature points, eliminating interference from dynamic feature points. The fourth object detection optimization variable refers to the IMU pre-integration constraint, used as a relative motion constraint between visual sensor poses, ensuring that the relative motion between visual sensor poses is consistent with the definition of visual inertial odometry, i.e., constraining the position, attitude velocity, and IMU acceleration and angular velocity bias of the visual sensor. The fifth object detection optimization variable serves as a global absolute position constraint for the visual sensor, used to suppress long-term drift of the visual sensor.

[0087] S4. Combine object detection optimization variables to perform real-time detection of dynamic objects, obtain real-time detection results of dynamic objects, and generate a static map to which the object belongs.

[0088] Specifically, step S4 includes:

[0089] An optimization objective function is established by integrating the first object detection optimization variable, the second object detection optimization variable, the third object detection optimization variable, the fourth object detection optimization variable, and the fifth object detection optimization variable.

[0090] The optimization objective function is solved to obtain the real-time pose of the visual sensor and the real-time detection information of the dynamic object under the current pose, thereby obtaining the motion trajectory of the dynamic object and generating the corresponding static map.

[0091] In this embodiment, an optimization objective function is established by minimizing the sum of all variables. Based on the objective function, the optimal pose, velocity, IMU bias of the visual sensor, static map points, and motion state of the dynamic object are obtained. At the same time, the motion trajectory of the object is formed according to the motion state of the dynamic object. The dynamic object trajectory is further used for motion prediction and obstacle avoidance. Static map points and static features are used for loop closure detection to avoid mismatches caused by dynamic objects.

[0092] In some examples, the output of a SLAM system includes at least: real-time camera pose, static environment map (e.g., ... Figure 2 As shown), the dynamic object detection results (including category, position, and velocity).

[0093] This application also provides a dynamic vision-based localization and mapping system corresponding to the dynamic vision-based localization and mapping method, such as... Figure 3 As shown, the system includes:

[0094] The dynamic object detection module is used to extract feature points based on the dynamic detection information of dynamic objects and obtain the target feature set.

[0095] The object motion tracking module is used to compensate for the motion trajectory of dynamic objects based on the pose prediction of the visual sensor and to obtain the motion state of the objects.

[0096] The variable acquisition module is used to determine the object detection optimization variables based on the object's motion state, target feature set, and visual sensor motion parameters.

[0097] The detection and mapping optimization module is used to integrate object detection optimization variables to perform real-time detection of dynamic objects, obtain real-time detection results of dynamic objects, and generate static maps to which the objects belong.

[0098] As an optional implementation, the dynamic object detection module specifically includes: a first detection unit, used to construct a dynamic detection model based on historical dynamic detection images using an improved DETR framework; a second detection unit, used to identify the acquired images of dynamic objects based on the dynamic detection model, and obtain a set of dynamic object detection images containing the category of the dynamic object and the bounding box of the identifier, which is used as dynamic detection information; and a third detection unit, used to identify feature regions for each image based on the dynamic detection information, and extract feature points based on the feature regions to obtain a target feature set.

[0099] Specifically, the first detection unit includes: labeling object types based on historical dynamic detection images; converting the format of the labeled images to obtain an image set containing target categories and bounding boxes; using the image set as input parameters for a convolutional neural network for model pre-training, and selecting semantic feature layers to output feature maps; performing feature encoding on the feature maps based on a dimensionality reduction encoder to generate feature vectors; performing object type prediction and object bounding box labeling on the feature vectors according to the dual-branch network of the decoder, and outputting the target object type and object detection boxes of the feature maps to construct a basic detection model; and optimizing the basic detection model based on the target object type and object bounding boxes using a bipartite graph matching loss function to obtain the dynamic detection model.

[0100] Specifically, the third detection unit includes: performing region recognition on each dynamic detection image in the dynamic detection information, dividing the image region into static feature regions and dynamic feature regions; extracting a first feature point based on the static feature region; identifying the hidden static sub-regions in the dynamic feature region based on the temporal synchronous motion characteristics of the dynamic feature region, extracting a second feature point based on the hidden static sub-regions; and constructing a target feature set based on the first feature point and the second feature point.

[0101] As an optional implementation, the object motion tracking module specifically includes: obtaining the predicted pose of the visual sensor at future moments based on the pose change status of the visual sensor; extracting object detection boxes based on the dynamic detection information, compensating the detection boxes of the dynamic object in consecutive frames according to the predicted pose, and thus obtaining the visual residual of the dynamic object in consecutive frames; calculating the spatial motion velocity of the dynamic object based on the visual residual, and then predicting the position of the object detection boxes at future moments according to the spatial motion velocity, so as to obtain the motion state including the object's motion velocity and position.

[0102] As an optional implementation, the optimization variable acquisition module specifically includes: extracting the object's position information in consecutive frames based on the object's motion state, calculating the motion residual between consecutive frames, and using it as the first object detection optimization variable; extracting the object's motion parameters based on the object's motion state, fusing the observation values ​​of the object detection box to construct the object detection error, and determining the second object detection optimization variable; performing feature point reprojection based on the target feature set, obtaining the static reprojection error of each static feature point, and determining the third object detection optimization variable; obtaining the positioning error of the visual sensor based on the GPS positioning information and actual displacement information of the visual sensor, and using it as the fourth object detection optimization variable; and determining the fifth object detection optimization variable based on the predicted pose error of the visual sensor.

[0103] As an optional implementation, the detection mapping optimization module specifically includes: establishing an optimization objective function by fusing the first object detection optimization variable, the second object detection optimization variable, the third object detection optimization variable, the fourth object detection optimization variable, and the fifth object detection optimization variable; solving the optimization objective function to obtain the real-time pose of the visual sensor and the real-time detection information of the dynamic object under the current pose, thereby obtaining the motion trajectory of the dynamic object and generating the corresponding static map.

[0104] This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 41, a communication interface 42, a memory 43, and a communication bus, wherein the processor 41, the communication interface 42, and the memory 43 communicate with each other through the communication bus.

[0105] Memory 43 is used to store computer programs;

[0106] The processor 41, when executing the program stored in the memory 43, implements a dynamic vision-based localization and mapping method.

[0107] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0108] Communication interface 42 is used for communication between the above-mentioned electronic device and other devices.

[0109] The memory 43 may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory 43 may also be at least one storage device located remotely from the aforementioned processor.

[0110] The processor 41 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0111] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a dynamic vision-based localization and mapping method.

[0112] The above-described embodiments are preferred embodiments of this application and are not intended to limit the specific scope of this application. The scope of this application includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape, structure, and method of this application are within the protection scope of this application.

Claims

1. A method for localization and mapping based on dynamic vision, characterized in that: The method comprises the following steps: extracting feature points based on dynamic detection information of a dynamic object to obtain a target feature set; compensating a motion trajectory of the dynamic object based on a predicted pose of a visual sensor to obtain an object motion state; determining an object detection optimization variable based on the object motion state, the target feature set and a motion parameter of the visual sensor; fusing the object detection optimization variable to perform real-time detection on the dynamic object to obtain a real-time detection result of the dynamic object and generate a static map to which the object belongs.

2. The dynamic vision based localization and mapping method of claim 1, wherein: The method of extracting feature points based on dynamic detection information of a dynamic object to obtain a target feature set comprises: constructing a dynamic detection model based on historical dynamic detection images using an improved DETR framework; identifying a collected image of the dynamic object based on the dynamic detection model to obtain a dynamic object detection image set containing a category of the dynamic object and an identified bounding box, and taking the dynamic object detection image set as dynamic detection information; identifying a feature region of each image based on the dynamic detection information, and extracting feature points based on the feature region to obtain a target feature set.

3. The dynamic vision based localization and mapping method of claim 2, wherein: The method of constructing a dynamic detection model based on historical dynamic detection images using an improved DETR framework comprises: annotating object types based on historical dynamic detection images, converting the annotated images into a format to obtain an image set containing target categories and bounding boxes; taking the image set as an input parameter of a convolutional neural network to pre-train a model, and screening out a semantic feature layer to output a feature map; encoding the feature map based on a dimension-reduced encoder to generate a feature vector; outputting a target object type of the feature map and an object detection box based on a double-branch network of a decoder to respectively predict an object type and annotate an object bounding box of the feature vector, so as to construct a basic detection model; optimizing the basic detection model based on the target object type and the object bounding box using a bipartite graph matching loss function to obtain the dynamic detection model.

4. The dynamic vision based localization and mapping method of claim 2, wherein: The method of identifying a feature region of each image based on the dynamic detection information and extracting feature points based on the feature region to obtain a target feature set comprises: identifying a region of each dynamic detection image in the dynamic detection information, and dividing an image region into a static feature region and a dynamic feature region; extracting a first feature point based on the static feature region; identifying a hidden static sub-region in the dynamic feature region based on a time-synchronous motion characteristic of the dynamic feature region, and extracting a second feature point based on the hidden static sub-region; constructing a target feature set based on the first feature point and the second feature point.

5. The dynamic vision based localization and mapping method of claim 1, wherein: The method of compensating a motion trajectory of a dynamic object based on a predicted pose of a visual sensor to obtain an object motion state comprises: obtaining a predicted pose of the visual sensor at a future time based on a pose transformation state of the visual sensor; extracting an object detection box based on the dynamic detection information, compensating the object detection box of the dynamic object in consecutive frames based on the predicted pose, and then obtaining a visual residual of the dynamic object in the consecutive frames; calculating a spatial motion speed of the dynamic object based on the visual residual, and then predicting a position of the object detection box at the future time based on the spatial motion speed to obtain a motion state containing the object motion speed and the position.

6. The dynamic vision based localization and mapping method of any one of claims 4 or 5, wherein: The object detection optimization variable is determined based on the object motion state, the target feature set, and the visual sensor motion parameter, and includes: The object motion state is used to extract the position information of the object in consecutive frames, calculate the motion residual of the object between the consecutive frames, and take the motion residual as a first object detection optimization variable; The object motion state is used to extract the motion parameter of the object, fuse the observation value of the object detection frame, construct the object detection error, and determine a second object detection optimization variable; The target feature set is used to perform feature point re-projection, obtain the static re-projection error of each static feature point, and determine a third object detection optimization variable; The GPS positioning information and the actual displacement information of the visual sensor are used to obtain the positioning error of the visual sensor, and take the positioning error as a fourth object detection optimization variable; The prediction pose error of the visual sensor is used to determine a fifth object detection optimization variable.

7. The dynamic vision based localization and mapping method of claim 6, wherein: The dynamic object is detected in real time by fusing the object detection optimization variables, real-time detection results of the dynamic object are obtained, and a static map to which the object belongs is generated, and includes: An optimization objective function is established by fusing the first object detection optimization variable, the second object detection optimization variable, the third object detection optimization variable, the fourth object detection optimization variable, and the fifth object detection optimization variable; The optimization objective function is solved, real-time pose of the visual sensor and real-time detection information of the dynamic object in the current pose are obtained, the motion trajectory of the dynamic object is further obtained, and a corresponding static map is generated.

8. A dynamic vision based positioning and mapping system, characterized in that: The method is suitable for the dynamic vision-based positioning and mapping method in any one of claims 1-7, and includes: A dynamic object detection module is configured to extract feature points based on dynamic detection information of the dynamic object, and obtain a target feature set; An object motion tracking module is configured to compensate the motion trajectory of the dynamic object based on a predicted pose of the visual sensor, and obtain an object motion state; An optimization variable acquisition module is configured to determine an object detection optimization variable based on the object motion state, the target feature set, and a motion parameter of the visual sensor; A detection and mapping optimization module is configured to detect the dynamic object in real time by fusing the object detection optimization variable, obtain real-time detection results of the dynamic object, and generate a static map to which the object belongs.

9. A computer device, comprising: The method includes: A processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus; the memory is configured to store a computer program; and the processor is configured to execute the program stored in the memory, and implement the steps of the dynamic vision-based positioning and mapping method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the dynamic vision-based positioning and mapping method in any one of claims 1-7.

Images