Positioning and mapping method, system and equipment based on dynamic vision and medium

Abstract

The invention discloses a positioning and mapping method, system and device based on dynamic vision and a medium, and relates to the technical field of simultaneous positioning and mapping, and the method comprises the steps: carrying out the feature point extraction based on the dynamic detection information of a dynamic object, and obtaining a target feature set; compensating the motion track of the dynamic object based on the visual sensor prediction pose, and obtaining the motion state of the object; determining an object detection optimization variable based on the object motion state, the target feature set and the visual sensor motion parameters; performing real-time detection on the dynamic object by fusing the object detection optimization variable, obtaining a real-time detection result of the dynamic object and generating a static map to which the object belongs; according to the method, the motion trail of the dynamic object can be accurately tracked in real time, so that the positioning precision and the real-time performance of the dynamic object are improved, and the dynamic obstacle detection efficiency and the static map construction precision are further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of simultaneous localization and mapping, in particular to a positioning and mapping method, system, device and medium based on dynamic vision. BACKGROUND

[0002] Simultaneous Localization and Mapping (SLAM) is a technology that allows robots or autonomous vehicles to navigate in unknown environments autonomously. Traditional SLAM systems are usually based on the assumption of a static environment, that is, all objects in the environment are considered to be static. However, in actual application scenarios, such as urban roads, shopping malls, stations, etc., there are a large number of dynamic objects (pedestrians, vehicles, animals, etc.), which will seriously affect the positioning accuracy and map quality of the SLAM system. The current conventional dynamic SLAM method mainly uses the following strategies to deal with dynamic objects: (1) a method based on motion consistency checking, which detects feature points that do not conform to the camera motion model to identify dynamic regions; (2) a method based on semantic segmentation, which uses a deep learning network to perform pixel-level segmentation on images to identify potential dynamic object categories; (3) a method based on optical flow or scene flow, which detects dynamic regions by analyzing pixel motion.

[0003] However, these methods have the following limitations: First, the method based on motion consistency requires sufficient static feature points to estimate camera motion, which is prone to failure in scenes with a large proportion of dynamic objects. Second, the method based on semantic segmentation has a large amount of calculation and is difficult to meet the real-time requirement, and needs to predefine dynamic object categories, lacking flexibility. Third, the method based on optical flow is sensitive to fast motion and occlusion, and is prone to false detection. Finally, most existing methods can only detect dynamic regions, lacking the ability to accurately locate and track dynamic objects.

[0004] The patent "Multi-sensor fusion SLAM method and system applied to dynamic environment" with publication number CN118209101 A discloses obtaining sensor data of inertial sensors, wheel speed meters, visual cameras and laser radars, pre-integrating the sensor data of the inertial sensors and the wheel speed meters, detecting and filtering out dynamic points by using a geometric method through loose coupling pose estimation, performing semantic segmentation on the sensor data of the visual camera, clustering the detection results, and filtering out dynamic points in the pose estimation process according to re-projection error estimation to obtain the current optimal pose, extracting static point clouds from the sensor data of the laser radar, associating visual feature depth using the static point clouds, and outputting a pose graph through sliding window optimization, loop detection and pose graph optimization according to inertial sensor pre-integration residual error, wheel speed meter pre-integration residual error, static point re-projection residual error and marginalization residual error. However, this scheme still has limitations in processing dynamic objects. The geometric method has a large error range in judging dynamicity through the distance threshold of feature points between two frames, thereby interfering with pose optimization and resulting in poor map reliability. SUMMARY

[0005] The purpose of the present application is to address the problem of low object positioning accuracy due to the inability to track the state of dynamic object motion trajectories in the prior art, resulting in low dynamic obstacle detection efficiency and poor mapping effect. The present application proposes a positioning and mapping method, system, device and medium based on dynamic vision, which extracts feature points by detecting dynamic objects to compensate and predict the motion state of dynamic objects, and then optimizes the detection of dynamic objects by fusing visual sensor parameters, realizes real-time positioning and tracking of dynamic objects, and improves the dynamic obstacle detection efficiency and the static map construction accuracy.

[0006] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows: In a first aspect, the embodiments of the present application provide a positioning and mapping method based on dynamic vision, which comprises: extracting feature points based on dynamic detection information of dynamic objects to obtain a target feature set; compensating the motion trajectory of dynamic objects based on visual sensor predicted pose to obtain object motion state; determining object detection optimization variables based on object motion state, target feature set and visual sensor motion parameters; fusing object detection optimization variables to detect dynamic objects in real time to obtain dynamic object real-time detection results and generate a static map to which the objects belong.

[0007] This scheme improves mapping accuracy and completeness by acquiring target feature sets to provide sufficient and reliable feature support for subsequent static map construction. Predictive poses from visual sensors can predict the motion state of dynamic objects in the next moment, solving the problem of inaccurate estimation of the motion state of dynamic objects relative to the world coordinate system due to ignoring the influence of camera motion. This improves dynamic object tracking accuracy and further ensures obstacle detection effectiveness and mapping reliability. By fusing various optimization variables, the scheme can effectively and accurately estimate the trajectory of dynamic objects in complex environments, improving positioning accuracy. Simultaneously, the accurate dynamic object detection results are used to inversely affect target feature extraction, thereby improving the accuracy of target features and ensuring the reliability of the static map corresponding to the dynamic object's trajectory.

[0008] Preferably, the step of extracting feature points based on the dynamic detection information of dynamic objects to obtain a target feature set includes: A dynamic detection model is constructed using an improved DETR framework based on historical dynamic detection images. Based on the dynamic detection model, the acquired images of dynamic objects are identified to obtain a set of dynamic object detection images containing the category of the dynamic object and the bounding box of the identifier, which is used as dynamic detection information. Based on the dynamic detection information, feature regions are identified in each image, and feature points are extracted based on the feature regions to obtain the target feature set.

[0009] Preferably, the dynamic detection model constructed using the improved DETR framework based on historical dynamic detection images includes: Object type labeling is performed on historical dynamic detection images. The labeled images are then converted to obtain an image set containing target categories and bounding boxes. This image set is used as input parameters for a convolutional neural network for model pre-training, and a semantic feature layer is selected to output a feature map. A dimensionality-reducing encoder encodes the feature map to generate feature vectors. The dual-branch network of the decoder performs object type prediction and object bounding box labeling on the feature vectors, outputting the target object type and object detection box of the feature map to construct a basic detection model. Based on the target object type and object bounding box, a bipartite graph matching loss function is used to optimize the basic detection model to obtain the dynamic detection model.

[0010] Preferably, the step of identifying feature regions for each image based on the dynamic detection information and extracting feature points based on the feature regions to obtain a target feature set includes: identifying regions for each dynamic detection image in the dynamic detection information, dividing the image region into static feature regions and dynamic feature regions; extracting first feature points based on the static feature regions; identifying latent static sub-regions in the dynamic feature regions based on the temporal synchronous motion characteristics of the dynamic feature regions, extracting second feature points based on the latent static sub-regions; and constructing a target feature set based on the first feature points and the second feature points.

[0011] Preferably, the step of compensating for the motion trajectory of a dynamic object based on the predicted pose of a visual sensor to obtain the object's motion state includes: obtaining the predicted pose of the visual sensor at future moments based on the pose transformation status of the visual sensor; extracting object detection boxes based on the dynamic detection information, compensating the detection boxes of the dynamic object in consecutive frames according to the predicted pose, and thus obtaining the visual residual of the dynamic object in consecutive frames; calculating the spatial motion velocity of the dynamic object based on the visual residual, and then predicting the position of the object detection boxes at future moments according to the spatial motion velocity, so as to obtain a motion state including the object's motion velocity and position.

[0012] Preferably, determining the object detection optimization variables based on the object's motion state, target feature set, and visual sensor motion parameters includes: extracting the object's position information in consecutive frames based on the object's motion state, calculating the motion residual between consecutive frames, and using it as the first object detection optimization variable; extracting the object's motion parameters based on the object's motion state, fusing the observations of the object detection box to construct the object detection error, and determining the second object detection optimization variable; performing feature point reprojection based on the target feature set, obtaining the static reprojection error of each static feature point, and determining the third object detection optimization variable; obtaining the visual sensor's positioning error based on the GPS positioning information and actual displacement information, and using it as the fourth object detection optimization variable; and determining the fifth object detection optimization variable based on the visual sensor's predicted pose error.

[0013] Preferably, the step of fusing object detection optimization variables to perform real-time detection of dynamic objects, obtaining real-time detection results of dynamic objects, and generating a static map to which the objects belong includes: fusing the first object detection optimization variable, the second object detection optimization variable, the third object detection optimization variable, the fourth object detection optimization variable, and the fifth object detection optimization variable to establish an optimization objective function; solving the optimization objective function to obtain the real-time pose of the visual sensor and the real-time detection information of the dynamic objects under the current pose, thereby obtaining the motion trajectory of the dynamic objects and generating the corresponding static map.

[0014] Secondly, embodiments of this application provide a localization and mapping system based on dynamic vision, comprising: a dynamic object detection module for extracting feature points based on the dynamic detection information of dynamic objects to obtain a target feature set; an object motion tracking module for compensating the motion trajectory of dynamic objects based on pose prediction by a visual sensor to obtain the motion state of the objects; an optimization variable acquisition module for determining object detection optimization variables based on the object motion state, the target feature set, and motion parameters of the visual sensor; and a detection mapping optimization module for fusing the object detection optimization variables to perform real-time detection of dynamic objects, obtaining real-time detection results of dynamic objects, and generating a static map to which the objects belong.

[0015] Thirdly, embodiments of this application provide a computer device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to execute the program stored in the memory to implement the steps of the dynamic vision-based localization and mapping method described in the first aspect above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the dynamic vision-based localization and mapping method described in the first aspect above.

[0017] The beneficial effects of this application are: 1. By extracting and supplementing static features, the problem of loss of effective feature points caused by interference from dynamic regions is overcome, providing sufficient and reliable feature support for subsequent static map construction, thereby ensuring the accuracy and completeness of the map construction; 2. The predicted pose of the visual sensor can predict the motion state of the dynamic object in the next moment, which solves the problem of inaccurate estimation of the motion state of the dynamic object relative to the world coordinate system due to ignoring the influence of the camera's own motion. This improves the tracking accuracy of the dynamic object and further ensures the obstacle detection effect and the reliability of the mapping. 3. By integrating various optimization variables, the motion trajectory of dynamic objects in complex environments can be effectively and accurately estimated, improving positioning accuracy. At the same time, based on the accurate dynamic object detection results, the target feature extraction is back-acted, thereby improving the accuracy of target features and ensuring the reliability of the static map corresponding to the motion trajectory of dynamic objects. Attached Figure Description

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0019] Figure 1 A flowchart of a dynamic vision-based localization and mapping method provided in an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of dynamic object detection and static map provided in an embodiment of this application.

[0021] Figure 3 This is a schematic diagram of a system module for a localization and mapping method based on dynamic vision provided in an embodiment of this application.

[0022] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of this application and are only used to explain this application. They do not limit the scope of protection of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Some of the terms or terms that appear in the description of the embodiments of this application shall be interpreted as follows: DETR framework: It is an end-to-end object detection framework based on Transformer. Its structure includes a CNN backbone network for extracting feature maps of the input image, a Transformer encoder, a decoder and a detection head. It integrates CNN feature extraction, Transformer encoder and decoder to directly generate detection results from the input image, simplifying the detection process. MobileNetV3 is a lightweight neural network model that improves upon the MnasNet structure through Neural Architecture Search (NAS) technology. It combines depthwise separable convolutions and linear bottleneck inverse residual structures, achieving a 3.2% improvement in ImageNet classification accuracy compared to MobileNetV2 while maintaining the same computational cost. ResNet (convolutional layer): refers to a residual network used to extract features from input data; Transformer: A deep learning model with self-attention mechanism at its core, enabling efficient parallel computing; ORB feature extraction algorithm (Oriented FAST and Rotated BRIEF) is a feature extraction algorithm that combines FAST keypoint detection and BRIEF descriptor. It is mainly used for feature point detection and description in image processing. IMU integration: The measurement data of the IMU (Inertial Measurement Unit) is converted into physical quantities such as position, velocity, and attitude through mathematical methods. Using the raw data from the integrating gyroscope and accelerometer, combined with the noise model and bias correction, the motion trajectory of the device in three-dimensional space is calculated.

[0025] Example 1: As Figure 1 As shown, a localization and mapping method based on dynamic vision includes steps S1-S4, wherein: S1. Extract feature points based on the dynamic detection information of dynamic objects to obtain the target feature set.

[0026] Specifically, step S1 includes: S11. A dynamic detection model is constructed based on historical dynamic detection images using an improved DETR framework; S12. Based on the dynamic detection model, identify the acquired images of dynamic objects, obtain a set of dynamic object detection images containing the category of dynamic objects and the bounding box of the identification, and use it as dynamic detection information. S13. Based on the dynamic detection information, perform feature region identification on each image, and extract feature points based on the feature regions to obtain the target feature set.

[0027] Specifically, step S11 includes: Object type labeling is performed based on historical dynamic detection images. The labeled images are then converted to obtain an image set containing target categories and bounding boxes. The image set is used as input parameters for the convolutional neural network to pre-train the model, and the semantic feature layer is selected to output the feature map. The encoder performs feature encoding on the feature map based on dimensionality reduction to generate feature vectors; Based on the dual-branch network of the decoder, the object type prediction and object bounding box annotation are performed on the feature vectors respectively, and the target object type and object detection box of the feature map are output to build a basic detection model. Based on the target object type and object bounding box, the basic detection model is optimized using a bipartite graph matching loss function to obtain the dynamic detection model.

[0028] In this embodiment, the convolutional neural network used for model pre-training based on the image set is MobileNetV3, which replaces the original ResNet convolutional layers in the DETR network. This reduces computational cost while accurately extracting multi-scale features of the image, thereby improving the efficiency of dynamic object detection. The dimensionality reduction encoder refers to reducing the Transformer dimension and the number of Transformer layers, specifically reducing the hidden dimension of the original DETR from 256 to 128, the number of attention heads from 8 to 4, and the number of layers in both the encoder and decoder from 6 to 3. This reduces the processing time per frame for dynamic object detection data and improves processing efficiency.

[0029] In some embodiments, within the improved DETR framework, the MobileNetV3 backbone network extracts multi-scale feature maps from image data acquired by the visual sensor, which are then flattened in space to form a one-dimensional feature vector sequence. Since the Transformer itself lacks the ability to perceive sequence order, a learnable positional encoding is superimposed on each feature vector. This encoding contains the two-dimensional spatial coordinates of the feature in the original image. The feature sequence infused with positional information is input into the Transformer encoder. Based on the encoder's internal multi-head self-attention mechanism, the association weight between any two feature vectors in the feature sequence is calculated. This mechanism allows each feature vector to aggregate information from the entire image, resulting in a feature vector containing global image context information. This feature vector is used as the first input to the decoder. Simultaneously, the number of probes used to extract specific information is dynamically set according to the scene detection speed and accuracy requirements, and these probes are used as the second input to the decoder. The probes interact with the feature vector output by the encoder using a cross-attention mechanism to obtain a prediction result for the dynamic object. This prediction result is an embedding vector containing object-related information. The Transformer decoder includes a dual-branch network: a classification branch network and a regression branch network. The dual-branch network decodes the embedding vectors and outputs the type information of the dynamic object and the bounding box of the dynamic object.

[0030] Furthermore, in this embodiment, when training the detection model, a bipartite graph matching loss function is used to optimize the model. The bipartite graph matching loss function includes at least bounding box loss and classification loss. The bounding box loss is calculated by predicting the coordinates of the bounding box and the ground truth box, and the classification loss is calculated by predicting the class and the ground truth class. Then, the model parameters are adjusted and optimized based on the bounding box loss and the classification loss.

[0031] Specifically, step S13 includes: For each dynamic detection image in the dynamic detection information, region recognition is performed to divide the image region into static feature regions and dynamic feature regions; Extracting the first feature point based on the static feature region; Based on the temporal synchronous motion characteristics of the dynamic feature region, the hidden static sub-regions in the dynamic feature region are identified, and the second feature points are extracted based on the hidden static sub-regions. Construct a target feature set based on the first feature point and the second feature point.

[0032] In this embodiment, region identification is performed on each detected image containing an object bounding box. A dynamic mask is set to mark dynamic regions as 1 and the remaining regions in the image as static regions as 0. Morphological dilation is applied to the dynamic mask to improve the feature extraction quality of dynamic and static boundary regions. For static regions, all feature points within the region are extracted based on the ORB feature extraction algorithm, i.e., key points that meet the definition of corner points within the image region and have stable orientations. A rotation-invariant binary descriptor is generated for each key point, which contains position and orientation, and is used as a static feature point (the first feature point in this embodiment).

[0033] Furthermore, the dynamic region is validated to determine whether there is a hidden static sub-region, that is, a static background region in a dynamic object. If it exists, feature points within the hidden static sub-region are extracted, used as second feature points, and merged with the first feature points to obtain the final target feature set.

[0034] In some optional embodiments, the latent static sub-regions in the dynamic feature region are identified based on the temporal synchronous motion characteristics of the dynamic feature region, specifically including: Perform superpixel segmentation on the dynamic region to obtain several dynamic sub-regions; The pixel offset of all pixels in the dynamic sub-region is calculated based on the motion transformation matrix of the visual sensor between consecutive frames. The motion residual of the pixel is calculated based on the actual motion of the pixel and the pixel offset. If the residual is less than a preset residual threshold, the corresponding dynamic sub-region is determined as an initial static sub-region. Based on the initial static sub-region, extract the sub-region texture features and edge features, calculate the texture feature histogram variance based on the regional texture features, and obtain the average edge length based on the edge features. When the texture feature histogram variance is greater than the variance threshold and the average edge length is greater than the edge length threshold, the initial static sub-region is regarded as the latent static sub-region.

[0035] Among them, the motion residual of a pixel is calculated by comparing the actual motion of the pixel with the pixel offset. This residual is the Euclidean distance between the actual motion of the pixel and the pixel offset. The pixel offset refers to the predicted motion position of the pixel.

[0036] It should be noted that by using dynamic masking to divide image regions, accurate segmentation of static regions is achieved, avoiding interference from false feature points in dynamic regions on static mapping. Feature points of static regions in the detected image are extracted. Considering that dynamic objects are not entirely devoid of static parts, effective static feature points in dynamic regions are filtered out. By identifying static sub-regions and extracting features, the loss of static feature points can be reduced, especially in scenes where dynamic objects account for a high proportion. This supplements the completeness of static feature points, overcoming the problem of incomplete mapping caused by discarding dynamic regions in traditional SLAM systems. This provides an effective and sufficient set of feature points for subsequent static map construction, improving the reliability, accuracy, and completeness of the static map.

[0037] S2. Based on the visual sensor, the motion trajectory of the dynamic object is compensated to obtain the motion state of the object.

[0038] Specifically, step S2 includes: Based on the pose transformation situation of the visual sensor, the predicted pose of the visual sensor at future moments is obtained. Based on the dynamic detection information, the object detection box is extracted, and the detection box of the dynamic object in the continuous frames is compensated according to the predicted pose, so as to obtain the visual residual of the dynamic object in the continuous frames. The spatial velocity of the dynamic object is calculated based on the visual residual, and then the position of the object detection box at a future time is predicted based on the spatial velocity, so as to obtain the motion state containing the object's velocity and position.

[0039] In some embodiments, the visual sensor is a device capable of acquiring visual information about dynamic objects, such as a camera.

[0040] In this embodiment, based on IMU integration and the current pose of the visual sensor, the relative motion state of the visual sensor at the current moment is obtained. Then, based on the relative motion state and the current pose, the predicted pose of the visual sensor at the next moment is obtained. For detected dynamic objects, whose two-dimensional bounding box information has been obtained, the two-dimensional bounding box of the object at the next moment can be predicted based on the predicted pose of the visual sensor through the device motion compensation function, thereby obtaining the visual residual of the dynamic object in consecutive frames (i.e., the current frame and the next frame).

[0041] The device motion compensation function is primarily used to calculate new bounding boxes generated solely by the movement of the vision sensor. If the moving object is stationary, its bounding box at the current moment will appear in a new position on the image at the next moment due to the movement of the vision sensor.

[0042] Therefore, in this embodiment, the bounding box (i.e., detection box) of a dynamic object in the future is predicted based on the predicted pose of the visual sensor to compensate for changes in the bounding box. The visual residual of the object in consecutive frames is obtained by calculating the difference between the current bounding box position and the predicted position of the bounding box in the future. The magnitude of the visual residual can then indicate whether the object is moving independently, stationary, or moving in the same direction and at the same speed as the visual sensor. Furthermore, the 3D motion velocity (i.e., spatial motion velocity) of the dynamic object is estimated based on the visual residual. The predicted pose of the visual sensor can be initially estimated using IMU integration.

[0043] Furthermore, by combining the spatial motion speed with the current position of the detection box of the dynamic object, the position of the detection box at the next moment is obtained, thereby compensating for the motion information of the object on the image plane, completing the acquisition of the motion state including the motion speed and position of the dynamic object, and improving the estimation accuracy of the motion state of the dynamic object.

[0044] In this embodiment, by obtaining the position of the bounding box of a dynamic object at the next moment, the position sequence of the object's boundary center point in the image coordinate system as time changes can be obtained. Thus, the size change of the bounding box at each position in the position sequence reflects whether the object is moving closer to or further away from the visual sensor. Combined with the predicted pose of the visual sensor, the motion information of the object in the image plane is compensated, and the true motion state of the object in three-dimensional space, including speed and direction, can be obtained, ultimately achieving continuous tracking of the object.

[0045] It's important to note that by acquiring the object's true motion state in 3D space, we ensure that the bounding box projected onto the 2D image matches the bounding box actually detected by the DETR network. Simultaneously, this ensures the accuracy of continuous object tracking. Specifically, based on the object's spatial velocity and the current position of the detection box, we obtain the box's position at the next moment. This requires that the tracked object be the same dynamic object, and that the object's motion state (e.g., velocity) should not undergo drastic changes over consecutive moments. Acquiring the object's true motion state in 3D space allows us to compare it with historical motion states to determine if changes exceed a preset range. If they do, it indicates a tracking error, allowing for timely adjustments to the visual sensor's pose and ensuring the accuracy of continuous object tracking.

[0046] S3. Determine the object detection optimization variables based on the object's motion state, target feature set, and visual sensor motion parameters.

[0047] Specifically, step S3 includes: Based on the object's motion state, extract the object's position information in consecutive frames, calculate the object's motion residual between consecutive frames, and use it as the first object detection optimization variable; Based on the object's motion state, the motion parameters of the object are extracted, and the observations of the object detection box are fused to construct the object detection error, so as to determine the second object detection optimization variable; Based on the target feature set, feature point reprojection is performed to obtain the static reprojection error of each static feature point, so as to determine the third object detection optimization variable. The positioning error of the visual sensor is obtained by combining GPS positioning information and actual displacement information from the visual sensor, and is used as the fourth object detection optimization variable. The prediction pose error based on the visual sensor determines the optimization variables for the fifth object detection.

[0048] In this embodiment, in traditional SLAM, camera localization, dynamic object detection, and static mapping are often independent modules with no information sharing. For example, camera localization errors are not fed back to dynamic object tracking, and dynamic detection results are not used to optimize static mapping, leading to inconsistent outputs from the three. Inaccurate camera positioning can cause a mismatch between the bounding boxes of dynamic objects and the map, resulting in poor overall performance. Therefore, this embodiment considers object motion residuals in consecutive frames, object detection errors, static reprojection errors, visual sensor localization errors, and predicted pose errors. By fusing these variables, the independent optimization of each submodule of the SLAM system is transformed into global collaborative optimization. This solves the cascading problems of inaccurate dynamic object tracking, visual sensor localization drift, and static map distortion, ensuring the accuracy of dynamic obstacle detection, improving detection efficiency, and simultaneously enhancing the reliability of the static map.

[0049] Among these, each variable also serves as an optimization constraint. The first object detection optimization variable serves as a constraint on the accuracy of continuous object tracking, ensuring that the same object is being tracked. The second object detection optimization variable serves as a consistency constraint for object motion state detection, associating the predicted motion state (velocity, position) of dynamic objects with the visual detection results, ensuring that the 2D bounding box observations of objects detected by DETR on the current frame image are consistent with the 2D bounding boxes converted from the 3D points projected onto the image plane according to the object size. The third object detection optimization variable serves as a static feature point extraction constraint, ensuring that all feature points are static feature points, eliminating interference from dynamic feature points. The fourth object detection optimization variable refers to the IMU pre-integration constraint, used as a relative motion constraint between visual sensor poses, ensuring that the relative motion between visual sensor poses is consistent with the definition of visual inertial odometry, i.e., constraining the position, attitude velocity, and IMU acceleration and angular velocity bias of the visual sensor. The fifth object detection optimization variable serves as a global absolute position constraint for the visual sensor, used to suppress long-term drift of the visual sensor.

[0050] S4. Combine object detection optimization variables to perform real-time detection of dynamic objects, obtain real-time detection results of dynamic objects, and generate a static map to which the object belongs.

[0051] Specifically, step S4 includes: An optimization objective function is established by integrating the first object detection optimization variable, the second object detection optimization variable, the third object detection optimization variable, the fourth object detection optimization variable, and the fifth object detection optimization variable. The optimization objective function is solved to obtain the real-time pose of the visual sensor and the real-time detection information of the dynamic object under the current pose, thereby obtaining the motion trajectory of the dynamic object and generating the corresponding static map.

[0052] In this embodiment, an optimization objective function is established by minimizing the sum of all variables. Based on the objective function, the optimal pose, velocity, IMU bias of the visual sensor, static map points, and motion state of the dynamic object are obtained. At the same time, the motion trajectory of the object is formed according to the motion state of the dynamic object. The dynamic object trajectory is further used for motion prediction and obstacle avoidance. Static map points and static features are used for loop closure detection to avoid mismatches caused by dynamic objects.

[0053] In some examples, the output of a SLAM system includes at least: real-time camera pose, static environment map (e.g., ... Figure 2 As shown), the dynamic object detection results (including category, position, and velocity).

[0054] This application also provides a dynamic vision-based localization and mapping system corresponding to the dynamic vision-based localization and mapping method, such as... Figure 3 As shown, the system includes: The dynamic object detection module is used to extract feature points based on the dynamic detection information of dynamic objects and obtain the target feature set. The object motion tracking module is used to compensate for the motion trajectory of dynamic objects based on the pose prediction of the visual sensor, and to obtain the motion state of the objects. The variable acquisition module is used to determine the object detection optimization variables based on the object's motion state, target feature set, and visual sensor motion parameters. The detection and mapping optimization module is used to integrate object detection optimization variables to perform real-time detection of dynamic objects, obtain real-time detection results of dynamic objects, and generate static maps to which the objects belong.

[0055] As an optional implementation, the dynamic object detection module specifically includes: a first detection unit, used to construct a dynamic detection model based on historical dynamic detection images using an improved DETR framework; a second detection unit, used to identify the acquired images of dynamic objects based on the dynamic detection model, and obtain a set of dynamic object detection images containing the category of the dynamic object and the bounding box of the identifier, which is used as dynamic detection information; and a third detection unit, used to identify feature regions for each image based on the dynamic detection information, and extract feature points based on the feature regions to obtain a target feature set.

[0056] Specifically, the first detection unit includes: labeling object types based on historical dynamic detection images; converting the format of the labeled images to obtain an image set containing target categories and bounding boxes; using the image set as input parameters for a convolutional neural network for model pre-training, and selecting semantic feature layers to output feature maps; performing feature encoding on the feature maps based on a dimensionality reduction encoder to generate feature vectors; performing object type prediction and object bounding box labeling on the feature vectors according to the dual-branch network of the decoder, and outputting the target object type and object detection boxes of the feature maps to construct a basic detection model; and optimizing the basic detection model based on the target object type and object bounding boxes using a bipartite graph matching loss function to obtain the dynamic detection model.

[0057] Specifically, the third detection unit includes: performing region recognition on each dynamic detection image in the dynamic detection information, dividing the image region into static feature regions and dynamic feature regions; extracting a first feature point based on the static feature region; identifying the hidden static sub-regions in the dynamic feature region based on the temporal synchronous motion characteristics of the dynamic feature region, extracting a second feature point based on the hidden static sub-regions; and constructing a target feature set based on the first feature point and the second feature point.

[0058] As an optional implementation, the object motion tracking module specifically includes: obtaining the predicted pose of the visual sensor at future moments based on the pose change status of the visual sensor; extracting object detection boxes based on the dynamic detection information, compensating the detection boxes of the dynamic object in consecutive frames according to the predicted pose, and thus obtaining the visual residual of the dynamic object in consecutive frames; calculating the spatial motion velocity of the dynamic object based on the visual residual, and then predicting the position of the object detection boxes at future moments according to the spatial motion velocity, so as to obtain the motion state including the object's motion velocity and position.

[0059] As an optional implementation, the optimization variable acquisition module specifically includes: extracting the object's position information in consecutive frames based on the object's motion state, calculating the motion residual between consecutive frames, and using it as the first object detection optimization variable; extracting the object's motion parameters based on the object's motion state, fusing the observation values ​​of the object detection box to construct the object detection error, and determining the second object detection optimization variable; performing feature point reprojection based on the target feature set, obtaining the static reprojection error of each static feature point, and determining the third object detection optimization variable; obtaining the positioning error of the visual sensor based on the GPS positioning information and actual displacement information of the visual sensor, and using it as the fourth object detection optimization variable; and determining the fifth object detection optimization variable based on the predicted pose error of the visual sensor.

[0060] As an optional implementation, the detection mapping optimization module specifically includes: establishing an optimization objective function by fusing the first object detection optimization variable, the second object detection optimization variable, the third object detection optimization variable, the fourth object detection optimization variable, and the fifth object detection optimization variable; solving the optimization objective function to obtain the real-time pose of the visual sensor and the real-time detection information of the dynamic object under the current pose, thereby obtaining the motion trajectory of the dynamic object and generating the corresponding static map.

[0061] This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 41, a communication interface 42, a memory 43, and a communication bus, wherein the processor 41, the communication interface 42, and the memory 43 communicate with each other through the communication bus. Memory 43 is used to store computer programs; The processor 41, when executing the program stored in the memory 43, implements a dynamic vision-based localization and mapping method.

[0062] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0063] Communication interface 42 is used for communication between the above-mentioned electronic device and other devices.

[0064] The memory 43 may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory 43 may also be at least one storage device located remotely from the aforementioned processor.

[0065] The processor 41 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0066] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a dynamic vision-based localization and mapping method.

[0067] The above-described embodiments are preferred embodiments of this application and are not intended to limit the specific scope of this application. The scope of this application includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape, structure, and method of this application are within the protection scope of this application.

Claims

1. A localization and mapping method based on dynamic vision, characterized in that: Includes the following steps: Feature points are extracted based on the dynamic detection information of dynamic objects to obtain the target feature set; Compensate for the motion trajectory of dynamic objects by predicting pose using visual sensors, and obtain the motion state of the objects. The object detection optimization variables are determined based on the object's motion state, target feature set, and visual sensor motion parameters. By integrating object detection optimization variables, dynamic objects are detected in real time, and the real-time detection results of dynamic objects are obtained, as well as the static map to which the object belongs is generated.

2. The localization and mapping method based on dynamic vision according to claim 1, characterized in that: The step of extracting feature points based on dynamic detection information of dynamic objects to obtain a target feature set includes: A dynamic detection model is constructed using an improved DETR framework based on historical dynamic detection images; Based on the dynamic detection model, the acquired images of dynamic objects are identified, and a set of dynamic object detection images containing the category of dynamic objects and the bounding box of the identification are obtained, which is used as dynamic detection information. Based on the dynamic detection information, feature regions are identified for each image, and feature points are extracted from the feature regions to obtain the target feature set.

3. The localization and mapping method based on dynamic vision according to claim 2, characterized in that: The dynamic detection model based on historical dynamic detection images employs an improved DETR framework to construct the model, including: Object type labeling is performed based on historical dynamic detection images. The labeled images are then converted to obtain an image set containing target categories and bounding boxes. The image set is used as input parameters for the convolutional neural network to pre-train the model, and the semantic feature layer is selected to output the feature map. The encoder performs feature encoding on the feature map based on dimensionality reduction to generate feature vectors; Based on the dual-branch network of the decoder, the object type prediction and object bounding box annotation are performed on the feature vectors respectively, and the target object type and object detection box of the feature map are output to build a basic detection model. Based on the target object type and object bounding box, the basic detection model is optimized using a bipartite graph matching loss function to obtain the dynamic detection model.

4. The localization and mapping method based on dynamic vision according to claim 2, characterized in that: The step of identifying feature regions for each image based on the dynamic detection information and extracting feature points from the feature regions to obtain a target feature set includes: For each dynamic detection image in the dynamic detection information, region recognition is performed to divide the image region into static feature regions and dynamic feature regions; Extracting the first feature point based on the static feature region; Based on the temporal synchronous motion characteristics of the dynamic feature region, the hidden static sub-regions in the dynamic feature region are identified, and the second feature points are extracted based on the hidden static sub-regions. Construct a target feature set based on the first feature point and the second feature point.

5. The localization and mapping method based on dynamic vision according to claim 1, characterized in that: The method of compensating for the motion trajectory of a dynamic object based on the predicted pose using a visual sensor to obtain the object's motion state includes: Based on the pose transformation situation of the visual sensor, the predicted pose of the visual sensor at future moments is obtained. Based on the dynamic detection information, the object detection box is extracted, and the detection box of the dynamic object in the continuous frames is compensated according to the predicted pose, so as to obtain the visual residual of the dynamic object in the continuous frames. The spatial velocity of the dynamic object is calculated based on the visual residual, and then the position of the object detection box at a future time is predicted based on the spatial velocity, so as to obtain the motion state containing the object's velocity and position.

6. The localization and mapping method based on dynamic vision according to any one of claims 4 or 5, characterized in that: The process of determining the object detection optimization variables based on the object's motion state, target feature set, and visual sensor motion parameters includes: Based on the object's motion state, extract the object's position information in consecutive frames, calculate the object's motion residual between consecutive frames, and use it as the first object detection optimization variable; Based on the object's motion state, the motion parameters of the object are extracted, and the observations of the object detection box are fused to construct the object detection error, so as to determine the second object detection optimization variable; Based on the target feature set, feature point reprojection is performed to obtain the static reprojection error of each static feature point, so as to determine the third object detection optimization variable. The positioning error of the visual sensor is obtained by combining GPS positioning information and actual displacement information from the visual sensor, and is used as the fourth object detection optimization variable. The prediction pose error based on the visual sensor determines the optimization variables for the fifth object detection.

7. The localization and mapping method based on dynamic vision according to claim 6, characterized in that: The fused object detection optimization variables are used to perform real-time detection of dynamic objects, obtain real-time detection results of dynamic objects, and generate a static map to which the objects belong, including: An optimization objective function is established by integrating the first object detection optimization variable, the second object detection optimization variable, the third object detection optimization variable, the fourth object detection optimization variable, and the fifth object detection optimization variable. The optimization objective function is solved to obtain the real-time pose of the visual sensor and the real-time detection information of the dynamic object under the current pose, thereby obtaining the motion trajectory of the dynamic object and generating the corresponding static map.

8. A localization and mapping system based on dynamic vision, characterized in that: The localization and mapping method based on dynamic vision as described in any one of claims 1-7 includes: The dynamic object detection module is used to extract feature points based on the dynamic detection information of dynamic objects and obtain the target feature set. The object motion tracking module is used to compensate for the motion trajectory of dynamic objects based on the pose prediction of the visual sensor and to obtain the motion state of the objects. The variable acquisition module is used to determine the object detection optimization variables based on the object's motion state, target feature set, and visual sensor motion parameters. The detection and mapping optimization module is used to integrate object detection optimization variables to perform real-time detection of dynamic objects, obtain real-time detection results of dynamic objects, and generate static maps to which the objects belong.

9. A computer device, characterized in that: include: The system includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements the steps of the dynamic vision-based localization and mapping method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the dynamic vision-based localization and mapping method as described in any one of claims 1-7.

Images