Map updating method, system and device based on visual SLAM system, medium and product
By using dynamic target culling threads and deep neural networks in the visual SLAM system to identify dynamic targets, combined with feature point motion probability propagation and optimization algorithm, the problem of low positioning and mapping accuracy of visual SLAM system in the dynamic environment is solved, and the system is improved stability and robustness.
Patent Information
- Application Number
- CN202411143076.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-20
AI Technical Summary
Existing visual SLAM systems are difficult to efficiently complete positioning and mapping tasks in unknown dynamic environments, resulting in system tracking failure or crash.
Dynamic target culling thread is used to identify and eliminate dynamic target feature points based on deep neural network, and static feature points are retained through feature point motion probability propagation method, and combined with tracking, local map construction, loop detection and global BA threads, the camera position and map are optimized. structure.
In a dynamic environment, the positioning and mapping accuracy of the SLAM system is improved, the tracking loss problem caused by dynamic objects is avoided, and the system is enhanced.
Smart Images

Figure CN119027686A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of robot vision technology, and in particular to a map updating method, system, device, medium and product based on a visual SLAM system. Background Art
[0002] The simultaneous localization and mapping (SLAM) system uses sensor input data to calculate the relative position (position and posture) transformation of the robot in real time to achieve the robot's positioning, and at the same time incrementally builds a map of the scene seen by the robot during its movement. With the development and progress of science and technology, human demand for robots is constantly increasing, and the goal is to make robots intelligent and automated to complete complex tasks. Since the birth of SLAM, it has attracted the attention of many researchers and is regarded as an important way to solve the problem of robot autonomous navigation.
[0003] According to the different sensors used, it can be divided into visual SLAM (Visual Simultaneous Localization and Mapping, VSLAM) and laser SLAM. LiDAR sensors measure distance accurately with small errors, operate stably outside direct light, and process point clouds relatively simply. However, due to their high price and high energy consumption, they are difficult to achieve widespread commercial application. Cameras have the advantages of low cost and rich image information, so they perform well in environments with similar geometric structures and in dealing with problems such as loop detection. Therefore, visual SLAM with cameras as the main sensor has received widespread attention and made some progress, but it still faces huge challenges.
[0004] Most existing VSLAM algorithms assume that the surrounding environment is static and ignore dynamic objects. However, in real scenarios, there are moving objects such as running vehicles, walking people and animals, and moving objects will cause VSLAM systems to have problems such as feature point mismatching, which ultimately leads to poor positioning accuracy and inability to efficiently complete positioning and mapping tasks in unknown dynamic environments, resulting in system tracking failure or even crash. Summary of the invention
[0005] The purpose of this application is to provide a map updating method, system, device, medium and product based on a visual SLAM system to solve the problem that the existing VSLAM system cannot efficiently complete positioning and mapping tasks in an unknown dynamic environment, resulting in system tracking failure or even crash.
[0006] To achieve the above objectives, this application provides the following solutions:
[0007] In a first aspect, the present application provides a map updating method based on a visual SLAM system, comprising:
[0008] Using a dynamic target elimination thread, a target detection algorithm based on a deep neural network is used to identify dynamic targets in a scene image, and a feature point motion probability propagation method is used to eliminate feature points of the dynamic targets, retaining static feature points in the scene image;
[0009] Using a tracking thread, the static feature points are matched with the static points in the local map, and each frame of the scene image is tracked to estimate the camera pose and the depth of the landmark points in the local map;
[0010] Utilizing the local mapping thread, based on the landmark point depth, optimizing the landmark points and camera pose in the local map, and determining an optimized local map and camera pose graph;
[0011] Using the loop detection thread, loop closure is detected based on the DBoW2 model, and by executing the camera pose graph, the accumulated error of the camera trajectory is corrected to determine the optimized camera pose graph;
[0012] The optimized local map and the optimized camera pose graph are kept globally consistent by using a global BA thread, the optimized local map is updated, and an updated map structure is determined.
[0013] In a second aspect, the present application provides a map updating system based on a visual SLAM system, comprising:
[0014] A dynamic target elimination thread is used to identify dynamic targets in a scene image based on a target detection algorithm of a deep neural network, and to eliminate feature points of the dynamic targets using a feature point motion probability propagation method, while retaining static feature points in the scene image;
[0015] A tracking thread, used for matching the static feature points with the static points in the local map, tracking each frame of the scene image, estimating the camera pose and the depth of the landmark points in the local map;
[0016] A local mapping thread, for optimizing the landmark points and camera pose in the local map based on the landmark point depth, and determining an optimized local map and camera pose graph;
[0017] A loop detection thread is used to detect loops based on the DBoW2 model, and to correct the camera trajectory cumulative error by executing the camera pose graph to determine an optimized camera pose graph;
[0018] The global BA thread is used to keep the optimized local map and the optimized camera pose graph globally consistent, update the optimized local map, and determine the updated map structure.
[0019] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-mentioned map updating methods based on the visual SLAM system.
[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned map updating methods based on a visual SLAM system.
[0021] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned map updating methods based on a visual SLAM system.
[0022] According to the specific embodiments provided by the present application, the present application discloses the following technical effects: the present application utilizes a dynamic target elimination thread, a target detection algorithm based on a deep neural network to identify dynamic targets in a scene image, effectively eliminates the feature points (i.e., dynamic feature points) of dynamic targets through a feature point motion probability propagation method, and designs a tracking thread, a local mapping thread, a loop detection thread, and a global bundle adjustment (BundleAdjustment, BA) thread, based on the matching relationship between the retained static feature points and the static points in the local map, updates the local map and the camera pose graph of the camera, and finally determines the updated map structure. The present application integrates SLAM with a target detection algorithm based on a deep neural network. When performing SLAM tasks, the system will only locate and map according to static objects in the scene, so that it can reliably and efficiently complete the tasks of positioning and mapping in an unknown dynamic environment, avoiding the problem of SLAM system tracking loss when dynamic objects appear in the images collected by the camera, thereby improving the robustness of the SLAM system. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0024] Figure 1This is a flow chart of a map updating method based on a visual SLAM system in an embodiment of the present application;
[0025] Figure 2 A diagram illustrating detailed information on motion probability propagation provided by an embodiment of the present application;
[0026] Figure 3 A schematic diagram of minimizing the reprojection error of two adjacent frames of camera images provided by an embodiment of the present application;
[0027] Figure 4 A schematic diagram of local BA optimization provided in an embodiment of the present application;
[0028] Figure 5 A diagram of an improved ORB-SLAM2 algorithm framework provided in one embodiment of the present application. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0030] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.
[0031] In recent years, the rapid development of deep learning has also greatly accelerated the innovation of target detection algorithms. Since VSLAM systems are usually prone to failure in dynamic environments, in order to avoid the problem of camera tracking loss in dynamic scenes, using target detection algorithms to obtain label information of the surrounding environment for subsequent removal of dynamic targets has become a research direction.
[0032] The present application embodiment provides a map updating method based on a visual SLAM system, such as Figure 1 As shown, the map updating method based on the visual SLAM system includes:
[0033] Step 101: using a dynamic target elimination thread, a target detection algorithm based on a deep neural network is used to identify dynamic targets in a scene image, and a feature point motion probability propagation method is used to eliminate feature points of the dynamic targets, retaining static feature points in the scene image.
[0034] In an exemplary embodiment, step 101 may be replaced by steps 201 to 203, which specifically include the following steps.
[0035] Step 201: Preprocess and forward propagate the scene image using the YOLO v8 network, and propagate the motion probability frame by frame in the tracking thread. When a dynamic target is detected, the image frame corresponding to the dynamic target is inserted into the local map as a key frame, and the motion probability of the landmark point in the local map corresponding to the key point in the key frame is updated.
[0036] Step 202: Estimate the motion probability of each key point in the local map by feature matching and matching point expansion.
[0037] Step 203: When a key point matches a key point of a previous frame, determine that the motion probability of the key point is propagated, remove the key point whose motion probability is propagated, and retain the static feature points in the scene image; wherein the key point is a feature point of a dynamic target.
[0038] In an exemplary embodiment, a target detection algorithm based on a deep neural network is used to detect dynamic objects, and YOLO v8 network detection is used to identify potential dynamic objects such as pedestrians and animals in the images collected by the camera that are dynamic or have a negative effect on map reuse. For example, once a person is detected, whether walking or standing still, it is considered to be a potential dynamic object, and the feature points in the area where the person is detected will be deleted in the image later.
[0039] Furthermore, the input camera image is subjected to oriented Fast and Rotated Brief (ORB) feature point extraction, wherein the ORB feature point is composed of the features from Accelerated Segment Test (FAST) key points and the binary robust independent elementary features (BRIEF) descriptor. After the image captured by the camera is input into the SLAM algorithm framework, the ORB feature points of the two images are extracted in the preprocessing module, and then the input original image data is discarded. All subsequent algorithm operations are based on the extracted ORB feature points, and the image expression is transformed from a set of pixel points to a set of ORB feature points, reducing the amount of data cached during operation. The specific extraction of ORB feature points can be achieved by using the cross-platform computer vision library (OpenBomputerVision Library, OpenCV), first detecting the position of the FAST key point, and then calculating the BRIEF descriptor based on the position of the FAST key point.
[0040] Furthermore, the probability that a feature point belongs to a dynamic object is defined as the motion probability of the feature point. According to the motion probability P, the feature points in the image are divided into four states. When the motion probability P of the feature point is > 0.75, it is called a high-confidence dynamic point; when 0.5 < P < 0.75, it is called a low-confidence dynamic point; when 0.25 < P < 0.5, it is called a low-confidence static point; when P < 0.25, it is called a high-confidence static point.
[0041] Two strategies are proposed to eliminate dynamic targets through the method of feature point motion probability propagation: 1) Detect dynamic objects in the detection frame and update the motion probability of the landmarks in the local map using the detection results; 2) Propagate the motion probability through feature matching and the expansion of matching points to efficiently remove the features extracted on dynamic objects before camera pose estimation.
[0042] First, the input camera image is preprocessed and forward-propagated through the YOLO v8 network, and the motion probability is propagated frame by frame in the tracking thread. Once the detection result is obtained, the frame is inserted into the local map as a key frame, and the motion probability in the local map is updated. The feature points in the key frame are called key points, and the motion probability of the 3D landmark corresponding to the key point in the key frame is updated according to the following formula:
[0043] P t (X i ) = (1 - α)P t-1 (X i ) + αS t (x i )
[0044] where P t-1 (X i ) is the updated motion probability of the 3D landmark X t-1 in the previous key frame I i . If it is a new key point, P t-1 (X i ) = P init = 0.5 will be set. The state of the matching key point x t in the key frame I i is S t (x i ), which depends on the dynamic object detection area output by the YOLO v8 network. If the key point x i is within the bounding box of the dynamic object, it is regarded as a definite dynamic point, and its state value S t (x i ) = 1. Other points are regarded as definite static points, and the state value is S t (x i)=0. α is an influencing factor used to smooth the immediate detection results. A higher value means more sensitivity to the immediate detection results, and a lower value means considering more historical results from multiple perspectives.
[0045] Furthermore, step 202 also includes: based on the propagated key points, extending the motion probability of the key points from high-confidence points to neighboring key points for which corresponding matching points are not found in the previous frame during the feature matching process; the high-confidence points include high-confidence dynamic points and high-confidence static points; the high-confidence dynamic points are feature points with a motion probability greater than 0.75, and the high-confidence static points are feature points with a motion probability less than 0.25; expanding the high-confidence influence area to a circular area of a set radius, and determining the motion probability of neighboring key points in the circular area for which corresponding matching points are not found in the previous frame.
[0046] Reference Figure 2 The figure shows the details of motion probability propagation. The motion probability of each key point is estimated by feature matching and matching point expansion, that is, the motion probability propagation of feature points. Because the motion probability in the current frame is only propagated from the key points of the previous frame.
[0047] In the feature matching process, when a key point The key points of the previous frame When matching, the motion probability Be spread.
[0048] Once a keypoint is matched with any 3D landmark in the local map Match, it is also assigned a motion probability equal to the matching 3D landmark point Note that if a point not only finds a matching point in the previous frame, but also finds a matching 3D landmark point in this map, the motion probability corresponding to the 3D landmark point should be given priority. At the same time, an initial probability P is assigned to other unmatched points in this frame. init = 0.5, since there is no a priori assumption about which state these points belong to.
[0049] In summary, the operation of propagating motion probability using feature point matching is summarized as follows:
[0050]
[0051] in, and Respectively ORB feature points, θ is the threshold for feature matching.
[0052] Furthermore, since the states of key points in a certain neighborhood are consistent in most cases, this is taken as a strong hypothesis, and based on this strong hypothesis, the motion probability is extended from high-confidence points (including high-confidence dynamic points and high-confidence static points) to other neighboring key points that have no corresponding matching points in the feature matching operation.
[0053] After propagation through feature matching, select a high confidence point χt, expand the influence area of the high confidence point to a circular area with a radius of r, and search for unmatched key points in this area. The unmatched key points found (i.e., the key points adjacent to the circular area) )The motion probability is updated according to the following rules:
[0054]
[0055] in, For waypoints The probability of movement; P init is the initial motion probability, set to 0.5. If a point is affected by multiple high confidence points, the sum of all the effects of these neighboring high confidence points will be calculated. When considering the influence of high confidence points, the difference in motion probability needs to be taken into account. If a point is within the influence area of a high confidence point (d≤r), the distance factor λ(d) = Ce -d / r , where C is a constant value; otherwise (d>r), λ(d)=0.
[0056] Step 102: Using the tracking thread, the static feature points are matched with the static points in the local map, and each frame of the scene image is tracked to estimate the camera pose and the depth of the landmark points in the local map.
[0057] In an exemplary embodiment, the camera pose is estimated by minimizing the reprojection error using the perspective-n-Point (PnP) algorithm in two adjacent image frames, the depth of landmark points in the local map is calculated by triangulation, and the method for determining whether the current frame is a key frame remains the same as the rule of ORB-SLAM2.
[0058] In an exemplary embodiment, the pose of the camera is initialized and estimated. The camera pose is initialized using a camera constant velocity model. Assuming that the camera motion is uniform, the relative motion between each frame is unchanged, and based on the relative pose between the t-2 frame and the t-1 frame, the pose of the current frame t is initialized from the t-1 frame. Since the camera is assumed to move at a uniform speed when the pose is initialized, the actual camera does not move at a uniform speed. This assumption is relatively rough, and the pose estimation needs to be further optimized by tracking the local map.
[0059] Further, refer to Figure 3 As shown in the figure, based on the ORB feature point matching between two adjacent frames of images and the retrieval of the ORB feature points in the current frame of the camera in the local map for feature point matching, a PnP problem is constructed, and the reprojection error of the feature points in the current frame is minimized to optimize the current camera posture and obtain a more accurate camera optimized posture T.
[0060] Furthermore, the camera’s optimized pose T and the matched ORB feature points in the two frames are used for triangulation to calculate a large number of new landmark depth values, thereby obtaining more high-quality landmarks. These new landmarks, as a supplement to the local map, greatly improve the accuracy and robustness of the tracking thread.
[0061] Finally, set the key frame selection conditions to determine whether the current frame is set as a key frame. If the current frame meets any of the following conditions, the current frame is set as a key frame.
[0062] 1. The current frame is the first frame after the bionic eye sight line control reaches the target gaze area.
[0063] 2. It has been more than 13 frames since the last global relocation.
[0064] 3. The local mapping thread is in idle state.
[0065] 4. The time between the last key frame and the last key frame is more than 15 frames.
[0066] 5. The translation distance between the current frame and the last set key frame exceeds the threshold t th .
[0067] 6. The number of successfully tracked feature points in the current frame reaches more than 70.
[0068] 7. The number of successfully tracked feature points in the current frame is less than 85% of the reference keyframe.
[0069] Step 103: Utilizing the local mapping thread, based on the landmark point depth, optimizing the landmark points and camera pose in the local map, and determining an optimized local map and camera pose graph.
[0070] In an exemplary instance, step 103 specifically includes: utilizing a local mapping thread to update key frames and landmark points in the local map based on the landmark point depth, and based on a PnP algorithm, utilizing minimization of reprojection errors to perform joint nonlinear optimization on the landmark points and camera poses in the updated local map to determine an optimized local map and camera pose graph.
[0071] In an exemplary embodiment, a local mapping thread is designed to manage a local map and perform local BA optimization, with the purpose of updating and maintaining landmarks and keyframes in the local map and optimizing the poses of keyframes and the coordinates of landmarks through local BA.
[0072] First, the keyframes in the local map are updated. After the tracking thread determines to set the current frame as a keyframe, the new keyframe is associated with the previous keyframe based on the co-view relationship of the landmark points, and the bag-of-words expression of the new keyframe is calculated based on the Dispersed Bag of Words (DBoW) model.
[0073] Furthermore, the landmarks in the local map are updated and maintained. If a landmark is to be retained in the local map, it must meet two conditions in the first three keyframes after it is created: 1. More than 25% of the keyframes in the frames where the landmark is visible based on pose prediction must be successfully tracked; 2. If more than one keyframe is added after the landmark is created, the landmark must be observed in no less than three keyframes. When a landmark meets these two conditions, it will be removed only if the number of keyframes in which the landmark is observed is less than 3.
[0074] Furthermore, local BA optimization is essentially a PnP problem, and the coordinates of the landmark points are also added as parameters in the optimization. Figure 4 As shown in Figure 1, BA is explained from the perspective of graph optimization. Figure 4 Middle C 1 , C 2 , C 3 The circle represents the camera pose node, which represents the camera pose parameters to be optimized; P 1 ,P 2 ...P 7 The circle represents the landmark node, which represents the three-dimensional coordinate parameters of the landmark point to be optimized, and the lines between the nodes represent the error terms defined in the nonlinear optimization. The error terms defined are:
[0075]
[0076] where u 2 represents the observed coordinates of the 3D landmark point P, e represents the error between the projection coordinates and the observed coordinates of the 3D landmark point P, that is, the reprojection error, K represents the camera intrinsic parameter, and T represents the camera extrinsic parameter.
[0077] During iterative optimization, the derivative of the reprojection error e with respect to the camera pose is:
[0078]
[0079] Among them, δξ represents the left perturbation of the camera pose T, X', Y', Z' represent the coordinates of the 3D landmark points transformed to the camera coordinate system, fx represents the focal length of the camera in the x direction, and fy represents the focal length of the camera in the y direction.
[0080] Then, the derivative of the reprojection error e about the 3D landmark point P is:
[0081]
[0082] Among them, R is the rotation matrix in the camera pose.
[0083] After obtaining the derivative of the reprojection error e with respect to the camera pose T and the coordinates of the 3D landmark point P, the Gauss-Newton method or the Levenberg-Marquardt method is used to optimize the objective function (i.e., the reprojection error e), providing a gradient direction to guide the update of the optimization variables camera pose T and the 3D landmark coordinates P in the map. After iterating until the objective function error converges, the optimal key frame camera pose and landmark coordinates in the local map can be obtained.
[0084] Finally, local keyframes are removed. When 90% of the landmarks in a keyframe can be observed in at least three other keyframes, the keyframe is discarded as a redundant keyframe. If the number of keyframes is not limited, the data size in local BA will become larger and larger, slowing down the optimization speed and affecting the real-time performance of local mapping.
[0085] Step 104: Utilize the loop detection thread to detect loops based on the DBoW2 model, and correct the camera trajectory cumulative error by executing the camera pose graph to determine an optimized camera pose graph.
[0086] In an exemplary embodiment, step 104 specifically includes: detecting the loop based on DBoW2 to find the candidate keyframes for the loop, and calculating the relative pose between the current keyframe and the candidate keyframes for the loop; closing the loop based on the common view relationship, and correcting the loop based on the relative pose between the current keyframe and the loop keyframe; constructing a nonlinear optimization problem using the camera pose before the loop correction and the camera pose after the loop correction, performing graph optimization on the camera pose after calculating the optimization gradient, correcting the accumulated error of the camera trajectory, and determining the optimized camera pose graph.
[0087] In an exemplary embodiment, a loop closure thread is used to detect large loops based on DBoW2 and correct the accumulated error of the camera trajectory by performing camera pose graph optimization. First, the loop is detected based on DBoW2 to find the candidate keyframes for the loop, and the relative pose between the current keyframe and the candidate keyframes for the loop is calculated. The loop is further closed based on the common view relationship, and the loop is corrected based on the relative pose between the current keyframe and the loop keyframe calculated in the previous step, and the camera pose graph is optimized.
[0088] The camera pose graph optimization only optimizes the camera pose. The nodes in the camera pose graph are only camera pose nodes, and the edges connecting the nodes are the estimates of the relative pose between the two camera pose nodes. The initial value of the pose node is the camera pose of each key frame before loop correction, and the edge is the relative pose between the camera poses of each key frame calculated after loop correction. Assuming that there are K i , K j The camera poses of the two key frames before loop correction are T Wi and T Wj After correcting the key frame camera pose in the loop fusion, the obtained K i and K j The relative position between them is T ij , then the error e ij for:
[0089]
[0090] Based on this error, a nonlinear optimization is constructed and the error terms are solved with respect to T Wi and T Wj The derivative of is used for optimization, and the optimization of the camera pose graph can be achieved.
[0091] Step 105: Using the global BA thread, the optimized local map and the optimized camera pose graph are kept globally consistent, the optimized local map is updated, and an updated map structure is determined.
[0092] In an exemplary embodiment, step 105 specifically includes: merging the optimized camera pose with the unoptimized camera pose by generating a tree, and correcting the coordinates of the landmark points in the optimized local map according to the update of the reference camera pose to determine the updated map structure.
[0093] In an exemplary embodiment, a global BA thread is started to obtain a globally consistent camera pose and map structure. After the loop closure thread completes the camera pose graph optimization, a global BA optimization is performed in a separate thread to obtain a globally optimal solution. If a new loop is detected during the global BA optimization process, the global BA thread is terminated, and the global BA thread is started again after the loop closure camera pose graph optimization is completed. When the global BA thread optimizes, the updated key frame camera pose is merged with the unupdated key frame camera pose through the spanning tree, and the coordinates of the landmark points are corrected according to the update of their reference key frame camera pose.
[0094] Based on the same inventive concept, the embodiment of the present application also provides a map updating system based on a visual SLAM system, comprising:
[0095] The dynamic target elimination thread is used to identify dynamic targets in the scene image based on the target detection algorithm of the deep neural network, and adopt the feature point motion probability propagation method to eliminate the feature points of the dynamic targets, and retain the static feature points in the scene image.
[0096] The tracking thread is used to match the static feature points with the static points in the local map, track each frame of the scene image, and estimate the camera pose and the depth of the landmark points in the local map.
[0097] The local mapping thread is used to optimize the landmark points and camera pose in the local map based on the landmark point depth, and determine the optimized local map and camera pose graph.
[0098] The loop detection thread is used to detect loops based on the DBoW2 model, and to correct the accumulated error of the camera trajectory by executing the camera pose graph to determine the optimized camera pose graph.
[0099] The global BA thread is used to keep the optimized local map and the optimized camera pose graph globally consistent, update the optimized local map, and determine the updated map structure.
[0100] like Figure 5 As shown, the present invention adopts an improved ORB-SLAM2 algorithm framework, adds a dynamic target removal thread, integrates SLAM with the YOLO v8 target detection network, and uses the method of feature point motion probability propagation to remove dynamic targets in the scene, thereby constructing a visual SLAM system for removing dynamic targets, aiming to improve the robot's positioning and mapping capabilities in unknown dynamic environments. Specifically, the present invention proposes two strategies to remove dynamic targets: 1) Detect dynamic objects in key frames and use the detection results to update the motion probability of landmark points in the local map; 2) Propagate the motion probability through feature matching and expansion of matching points, and efficiently remove the feature points extracted on dynamic objects before camera pose estimation. These innovations avoid the problem of tracking loss caused by a large number of dynamic objects in the images captured by the camera, and improve the stability of the system when performing SLAM tasks. The method of the present invention has important practical application value, can be widely used in fields such as robot navigation and autonomous driving, and provides strong support for technological innovation and progress in related industries.
[0101] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store map update data based on a visual SLAM system. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a map update method based on a visual SLAM system is implemented.
[0102] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.
[0103] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.
[0104] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.
[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0106] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ReadOnlyMemory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (Magnetoresistive RandomAccess Memory, MRAM), ferroelectric random access memory (Ferroelectric RandomAccess Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (RandomAccess Memory, RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0107] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.
[0108] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0109] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.
Claims
1. A map updating method based on a visual SLAM system, characterized in that: The map updating method based on the visual SLAM system comprises: Using a dynamic target elimination thread, a target detection algorithm based on a deep neural network is used to identify dynamic targets in a scene image, and a feature point motion probability propagation method is used to eliminate feature points of the dynamic targets, retaining static feature points in the scene image; Using a tracking thread, the static feature points are matched with the static points in the local map, and each frame of the scene image is tracked to estimate the camera pose and the depth of the landmark points in the local map; Utilizing the local mapping thread, based on the landmark point depth, optimizing the landmark points and camera pose in the local map, and determining an optimized local map and camera pose graph; Using the loop detection thread, loop closure is detected based on the DBoW2 model, and by executing the camera pose graph, the accumulated error of the camera trajectory is corrected to determine the optimized camera pose graph; The optimized local map and the optimized camera pose graph are kept globally consistent by using a global BA thread, the optimized local map is updated, and an updated map structure is determined.
2. The map updating method based on the visual SLAM system according to claim 1, characterized in that: Using the dynamic target elimination thread, the target detection algorithm based on the deep neural network identifies the dynamic target in the scene image, and uses the feature point motion probability propagation method to eliminate the feature points of the dynamic target, and retain the static feature points in the scene image, specifically including: The scene image is preprocessed and forward-propagated using a YOLO v8 network, and motion probabilities are propagated frame by frame in the tracking thread. When a dynamic target is detected, an image frame corresponding to the dynamic target is inserted into the local map as a key frame, and the motion probabilities of landmark points in the local map corresponding to key points in the key frame are updated; estimating the motion probability of each key point in the local map by feature matching and matching point expansion; When a key point matches a key point of a previous frame, it is determined that the motion probability of the key point is propagated, the key point whose motion probability is propagated is eliminated, and the static feature points in the scene image are retained; wherein the key point is a feature point of a dynamic target.
3. The map updating method based on the visual SLAM system according to claim 2, characterized in that: Updating the motion probability of the landmark point in the local map corresponding to the key point in the key frame specifically includes: According to the formula P t (X i )=(1-α)P t-1 (X i )+αS t (x i ) updates the motion probability of the landmark point in the local map corresponding to the key point in the key frame; wherein, P t (X i ) is the motion probability of the landmark point in the local map; P t-1 (X i ) is the previous key frame I t-1 Middle road mark X i Updated motion probability; α is the impact factor; S t (x i ) is the key point x i status.
4. The map updating method based on the visual SLAM system according to claim 1, characterized in that: The motion probability of each key point is estimated in the local map by feature matching and matching point expansion, specifically including: According to the formula Expanding to estimate the motion probability of each key point in the local map; wherein, is the i-th key point at time t The probability of movement; is the i-th key point at time t-1 The probability of movement; is the landmark point corresponding to the i-th key point at time t The probability of movement; P init is the initial probability; is the i-th key point at time t ORB feature points; is the i-th key point at time t-1 ORB feature points; is the landmark point corresponding to the i-th key point at time t ORB feature points; θ is the threshold for feature matching.
5. The map updating method based on the visual SLAM system according to claim 1, characterized in that: The motion probability of each key point is estimated in the local map by feature matching and matching point expansion, and then further includes: Based on the propagated key points, the motion probability of the key points is extended from the high confidence points to the adjacent key points for which no corresponding matching points are found in the previous frame during the feature matching process; the high confidence points include high confidence dynamic points and high confidence static points; the high confidence dynamic points are feature points with a motion probability greater than 0.75, and the high confidence static points are feature points with a motion probability less than 0.25; The high confidence influence area is extended to a circular area of a set radius, and the motion probability of the adjacent key points in the circular area for which no corresponding matching points are found in the previous frame is determined.
6. The map updating method based on the visual SLAM system according to claim 5, characterized in that: The probability of motion of adjacent key points within the circular area for: Among them, P init is the initial probability; is the jth key point at time t; λ(d) is the distance factor; For waypoints The probability of movement; χ t is a high confidence point.
7. The map updating method based on the visual SLAM system according to claim 1, characterized in that: Estimating the camera pose and the depth of the landmarks in the local map, specifically including: In two adjacent frames, the camera pose is estimated by minimizing the reprojection error based on the PnP algorithm, and the depth of the landmark points in the local map is calculated by triangulation.
8. The map updating method based on the visual SLAM system according to claim 1, characterized in that: Utilizing the local mapping thread, based on the landmark point depth, optimizing the landmark points and camera pose in the local map, and determining the optimized local map and camera pose graph, specifically including: Using the local mapping thread, based on the landmark point depth, the key frames and landmark points in the local map are updated, and based on the PnP algorithm, the landmark points in the updated local map and the camera pose are jointly nonlinearly optimized by minimizing the reprojection error to determine the optimized local map and camera pose graph.
9. The map updating method based on the visual SLAM system according to claim 1, characterized in that: Using the loop detection thread, loop detection is performed based on the DBoW2 model, and by executing the camera pose graph, the camera trajectory cumulative error is corrected to determine the optimized camera pose graph, specifically including: Based on DBoW2, loop closure is detected to find loop closure candidate keyframes, and the relative pose between the current keyframe and the loop closure candidate keyframe is calculated; The loop is closed according to the common view relationship, and the loop is corrected based on the relative pose of the current key frame and the loop key frame; The camera pose before and after loop closure correction is used to construct a nonlinear optimization problem. After calculating the optimization gradient, the camera pose is optimized, the accumulated error of the camera trajectory is corrected, and the optimized camera pose graph is determined.
10. The map updating method based on the visual SLAM system according to claim 1, characterized in that: Using the global BA thread, the optimized local map is kept globally consistent with the optimized camera pose graph, the optimized local map is updated, and the updated map structure is determined, specifically including: The optimized camera pose is merged with the unoptimized camera pose by generating a tree, and the coordinates of the landmark points in the optimized local map are corrected according to the update of the reference camera pose to determine the updated map structure.
11. A map updating system based on a visual SLAM system, characterized in that: include: A dynamic target elimination thread is used to identify dynamic targets in a scene image based on a target detection algorithm of a deep neural network, and to eliminate feature points of the dynamic targets using a feature point motion probability propagation method, while retaining static feature points in the scene image; A tracking thread, used for matching the static feature points with the static points in the local map, tracking each frame of the scene image, estimating the camera pose and the depth of the landmark points in the local map; A local mapping thread, for optimizing the landmark points and camera pose in the local map based on the landmark point depth, and determining an optimized local map and camera pose graph; A loop detection thread is used to detect loops based on the DBoW2 model, and to correct the camera trajectory cumulative error by executing the camera pose graph to determine an optimized camera pose graph; The global BA thread is used to keep the optimized local map and the optimized camera pose graph globally consistent, update the optimized local map, and determine the updated map structure.
12. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the map updating method based on the visual SLAM system according to any one of claims 1 to 10.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the map updating method based on the visual SLAM system described in any one of claims 1 to 10 is implemented.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the map updating method based on the visual SLAM system described in any one of claims 1 to 10 is implemented.
Citation Information
Patent Citations
A method and apparatus for setting enhanced interactive content
CN109656363A
Image processing method of augmented reality device and augmented reality device
CN110310373A
SLAM fusion method and system based on multiple types of cameras
CN110517216A
Garbage pickup robot based on visual semantic SLAM (simultaneous localization and mapping)
CN111360780A
Cited By
Migration correction system, method and device for dynamic robot visual action strategy
CN120503199A
Static map and patch combined AGV vehicle end map updating method and device
CN120910182A
