Deep learning-based visual SLAM method, system and storage medium in dynamic scenes

Abstract

The present invention discloses a deep learning-based visual SLAM method, system, and storage medium in dynamic scenes. The method comprises: combining a MASK R-CNN deep learning network with a traditional ORB-SLAM2 algorithm, using an RGB-D camera to capture images of an indoor environment, extracting ORB feature points from the captured RGB image, and simultaneously inputting the RGB image into a MASK R-CNN deep learning network to detect potential moving objects and remove feature points on dynamic objects; adding a semi-static object recognition module to cooperate with a target detection network to further check whether the feature points are potential dynamic feature points; and finally, performing background completion on the positions of the removed dynamic targets based on the positional relationship of adjacent frames and completing the final mapping. The present invention significantly reduces the trajectory error of the SLAM system in a dynamic environment while ensuring the real-time and robustness of the system.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, robotics and computer vision, and in particular to a visual SLAM method, system and storage medium based on deep learning in dynamic scenes. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) relies on installing sensors on drones to determine their location in real time and create a three-dimensional map of their surroundings. SLAM is a key research topic in artificial intelligence, robotics, and computer vision, with widespread applications in autonomous driving and other areas. Localization and mapping in dynamic indoor scenes is a key research area for SLAM.

[0003] Currently, the application scenarios of visual SLAM are all assumed to be static scenes. At the registration level, both point-to-point and point-to-feature registration methods are proposed on a static basis. In theory, dynamic points will inevitably affect the accuracy of registration. If the proportion of dynamic points is large, the accuracy of the trajectory will be greatly reduced. As for how to identify and eliminate dynamic points in real time, it is a major difficulty in the current visual SLAM field. There is currently no optimal solution for identification methods. In terms of mapping, due to the presence of dynamic points, the final reconstructed map will be filled with a large number of "ghosts", which will have a significant impact on real-time positioning and path planning in real-world applications.

[0004] Currently, mainstream methods for identifying dynamic objects primarily focus on identifying the dynamic parts of the input image and omitting the feature points of these parts to minimize their impact on overall system performance. Compared to methods that separate motion features, dynamic object identification methods can better identify the essential properties of moving objects, improving processing efficiency. Some have proposed SLAM methods for separating dynamic objects indoors. This system primarily uses the YOLO semantic segmentation network to identify dynamic objects, obtains corresponding binary masks, and ultimately removes them. It can operate efficiently in real time indoors. Another proposed method, Dynamic-SLAM, builds on convolutional neural networks and constructs a prior probability-based SSD detector to detect dynamic objects in new detection threads at the semantic level. Furthermore, addressing the low recall rate of SSD object detection, a new algorithm was proposed, effectively improving the accuracy of dynamic object detection. However, these methods cannot eliminate unstable feature points of potentially moving objects. Summary of the Invention

[0005] To address the above problems, the present invention provides a deep learning-based visual SLAM method, system and storage medium in dynamic scenes, which can effectively remove the influence of dynamic objects on pose estimation and improve the accuracy and stability of visual SLAM in dynamic environments.

[0006] A first aspect of the present invention provides a deep learning-based visual SLAM method in a dynamic scene, the method comprising the following steps:

[0007] Use an RGB-D camera to acquire RGB images;

[0008] Homogenize the collected RGB image and extract the ORB feature points in the RGB image;

[0009] The homogenized RGB image is input into the MASK R-CNN dynamic object detection network to identify the area where the dynamic object is located, and the ORB feature points contained in the dynamic object are initially removed;

[0010] The frame with ORB feature points initially removed is input into the semi-static object recognition module. The depth calculated based on the mapping relationship between adjacent frames is subtracted from the depth of the actual RGB-D camera, and the dynamic ORB feature points are removed for the second time using a threshold.

[0011] Feature matching is performed on the static feature points remaining after the second removal of dynamic ORB feature points to obtain the positional relationship between adjacent frames, perform pose estimation, and establish the final sparse map.

[0012] A further technical solution of the present invention is: the process of homogenizing the collected RGB image and extracting ORB feature points in the RGB image specifically includes:

[0013] Perform Gaussian blur on the collected RGB image;

[0014] Calculate the descriptors of the Gaussian blurred RGB image feature points and perform RGB image downsampling. Divide the RGB image into several layers, extract FAST corner points in each layer, and build an image pyramid based on the FAST corner points. The specific methods include:

[0015] Calculate the number of FAST corner points N that need to be extracted on each layer of the image pyramid α for:

[0016]

[0017] Where N represents the number of all ORB feature points extracted from the RGB image, s is the scaling factor, n is the total number of pyramid layers, and α represents the number of pyramid layers;

[0018] Each layer of RGB image is gridded, and the OpenCV function is called on each grid to extract FAST corner points. If no FAST corner points can be extracted under the lowest FAST threshold preset by the OpenCV function, no FAST corner points will be extracted in this grid.

[0019] For each grid partition, N is uniformly selected based on the quadtree. a FAST corner points.

[0020] A further technical solution of the present invention is: the MASK R-CNN dynamic target detection network identifies the area where the dynamic object is located, specifically including:

[0021] The dynamic object images to be identified are made into a data set, and the data set is used to train the MASK R-CNN dynamic object detection network;

[0022] Input the RGB image to be detected into the trained MASK R-CNN dynamic target detection network for image preprocessing to obtain the corresponding feature map;

[0023] Calculate the region of interest of the RGB image according to the feature map and obtain multiple candidate frames of the region of interest;

[0024] Multiple ROI candidate boxes are input into the region candidate network for binary classification and regression to filter out some ROI candidate boxes;

[0025] The filtered ROI candidate box is subjected to ROI Align operation to obtain the category of the ROI and generate MASK.

[0026] A further technical solution of the present invention is: the preliminary removal of ORB feature points contained in the dynamic object specifically includes:

[0027] Let the dynamic pixel feature set contained in the Nth frame RGB image input to the MASK R-CNN network be: The static pixel feature set is: in The i-th dynamic pixel feature in the N-th frame image, Represents the i-th static pixel feature in the N-th image frame; the set of feature points extracted from the N-th image frame is: in is the i-th feature point in the N-th frame image; if and but The feature points are dynamic points and are selected from the set T N Eliminate.

[0028] A further technical solution of the present invention is: the frame from which ORB feature points are initially removed is input into a semi-static object recognition module, the depth calculated based on the mapping relationship between adjacent frames is subtracted from the depth of the actual RGB-D camera, and dynamic ORB feature points are removed for the second time using a threshold, specifically including:

[0029] The overlap between the current frame after preliminarily removing ORB feature points and several previous frames is compared. Specifically, the feature points of several previous frames are mapped to the current frame and the angle between the 3D points of the previous frame and the current frame is calculated. For 3D points in the current frame with an angle less than 30 degrees, the error between the projected depth of the 3D point and the RGB-D camera depth is calculated. When the error is greater than a certain threshold, the 3D point in the current frame is determined to be a dynamic ORB feature point.

[0030] A further technical solution of the present invention is: the static feature points remaining after the second removal of the dynamic ORB feature points are matched using the nearest point iterative algorithm to obtain the positional relationship between adjacent frames and perform pose estimation, which specifically includes:

[0031] Call Opencv function to calculate the descriptor of the remaining static feature points, and match the feature points based on the similarity of the descriptors;

[0032] Define the feature points that have been matched between two adjacent frames:

[0033] Define the error term for the i-th point pair matched between the previous and next frames:

[0034] Construct the minimum sum of squares problem and find the solution to minimize the error function.

[0035] in, They represent the rotation matrix and translation vector between two adjacent frames that need to be obtained, and n represents the number of matching feature points.

[0036] Define the centroid of two sets of points:

[0037] The error function is processed as follows:

[0038]

[0039] In the cross-term section, After simplification, it can be eliminated, and the optimization target result is expressed as:

[0040]

[0041] The above equation is solved at three levels:

[0042] ①. Calculate the position of the center of mass of the corresponding point Then calculate the coordinates of each point excluding the centroid:

[0043]

[0044] The rotation matrix R that minimizes the error function is calculated based on the following formula: * :

[0045]

[0046] ②. R obtained based on the error function * , calculate the translation vector that minimizes the error function

[0047] Expand the error term of the error function about R, and we get:

[0048]

[0049] The optimization objective function is simplified to:

[0050] Define the matrix: W is a three-dimensional matrix. Perform singular value decomposition on W and get: W=U∑V T , where ∑ is the eigenvalue matrix, U and V are orthogonal matrices, and when W is full rank, the optimal rotation matrix R * R * =UV T ;

[0051] ③. Solve for R * After that, we can solve

[0052] A second aspect of the present invention is a deep learning-based visual SLAM system in a dynamic scene, comprising:

[0053] An RGB image acquisition unit is used to acquire an RGB image using an RGB-D camera;

[0054] An image homogenization unit is used to homogenize the collected RGB image and extract ORB feature points in the RGB image;

[0055] The dynamic feature point preliminary elimination unit is used to input the homogenized RGB image into the MASK R-CNN dynamic target detection network to identify the area where the dynamic object is located and to perform preliminary elimination of the ORB feature points contained in the dynamic object;

[0056] The second dynamic feature point removal unit inputs the frame with the ORB feature points initially removed into the semi-static object recognition module. The depth calculated based on the mapping relationship between adjacent frames is subtracted from the depth of the actual RGB-D camera, and the dynamic ORB feature points are removed for the second time using a threshold.

[0057] The feature matching and map construction unit is used to perform feature matching on the static feature points remaining after the second removal of dynamic ORB feature points, obtain the position relationship between adjacent frames, perform pose estimation, and establish the final sparse map.

[0058] The third aspect of the present invention provides a visual SLAM system based on deep learning in dynamic scenes, comprising: a processor; and a memory, wherein the memory stores a computer executable program, and when the computer executable program is executed by the processor, the above-mentioned visual SLAM method based on deep learning in dynamic scenes is executed.

[0059] A fourth aspect of the present invention is a storage medium having a program stored thereon, which, when executed by a processor, enables the processor to execute the above-mentioned deep learning-based visual SLAM method in dynamic scenes.

[0060] The present invention provides a deep learning-based visual SLAM method, system, and storage medium for dynamic scenes. The method mainly combines the MASK R-CNN deep learning network with the traditional ORB-SLAM2 algorithm, uses an RGB-D camera to capture images of indoor environments, extracts ORB feature points from the captured RGB images, and simultaneously inputs the RGB images into the MASK R-CNN deep learning network to detect potential moving objects and eliminate feature points on dynamic objects. At the same time, a semi-static object recognition module is added to cooperate with the target detection network to further check whether the feature points are potential dynamic feature points. Finally, the background of the positions of the eliminated dynamic targets is completed based on the positional relationship of adjacent frames, and the final map is built. The present invention significantly reduces the trajectory error of the SLAM system in dynamic environments while ensuring the real-time and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 1 is a flow chart of a deep learning-based visual SLAM method in a dynamic scene according to the first embodiment of the present invention;

[0062] Figure 2 Schematic diagram of the structure of the visual SLAM system based on deep learning in a dynamic scene in the second embodiment of the present invention;

[0063] Figure 3 This is the architecture of a computer device in the third embodiment of the present invention;

[0064] FIG4( a ) is a SLAM trajectory diagram of the present invention under three-dimensional coordinates according to an embodiment of the present invention;

[0065] FIG4( b ) is a trajectory diagram of ORBSLAM2 in three-dimensional coordinates according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, rather than all structures.

[0067] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0068] The present invention provides the following embodiments for a deep learning-based visual SLAM method, system, and storage medium in dynamic scenarios:

[0069] Based on the embodiment 1 of the present invention

[0070] This example is used to illustrate the visual SLAM method based on deep learning in dynamic scenes. Figure 1 , which is a flowchart of the deep learning-based visual SLAM method in dynamic scenes, specifically includes the following steps:

[0071] S110, using an RGB-D camera to acquire an RGB image;

[0072] S120, homogenizing the collected RGB image and extracting ORB feature points in the RGB image;

[0073] S130: Input the homogenized RGB image into the MASK R-CNN dynamic target detection network to identify the area where the dynamic object is located, and preliminarily remove the ORB feature points contained in the dynamic object;

[0074] S140, inputting the frame from which ORB feature points have been preliminarily removed into the semi-static object recognition module, subtracting the depth calculated based on the mapping relationship between adjacent frames from the depth of the actual RGB-D camera, and performing a second removal of dynamic ORB feature points using a threshold;

[0075] S150 , performing feature matching on the static feature points remaining after the second removal of the dynamic ORB feature points using the nearest point iterative algorithm, obtaining the positional relationship between adjacent frames, performing pose estimation, and establishing a final sparse map.

[0076] Among them, in S120, the collected RGB image is homogenized and the ORB feature points in the RGB image are extracted. In the specific implementation process, it is first necessary to establish a pyramid model of the input RGB image, which specifically includes:

[0077] Perform Gaussian blur on the collected RGB image;

[0078] Calculate the descriptors of the Gaussian blurred RGB image feature points and perform RGB image downsampling. Divide the RGB image into several layers, extract FAST corner points in each layer, and build an image pyramid based on the FAST corner points. The specific methods include:

[0079] Calculate the number of FAST corner points N that need to be extracted on each layer of the image pyramid α for:

[0080]

[0081] Where N represents the number of all ORB feature points extracted from the RGB image, s is the scaling factor, n is the total number of pyramid layers, and α is the number of pyramid layers;

[0082] Each layer of RGB image is gridded. In a specific embodiment, the grid size in ORB-SLAM2 is fixed to 30x30 pixels. The OpenCV function is called on each grid to extract FAST corner points. If no FAST corner points can be extracted under the lowest FAST threshold preset by the OpenCV function, no FAST corner points will be extracted in this grid.

[0083] Finally, N is uniformly selected based on the quadtree for each grid division. a FAST corner points.

[0084] In S130, the uniformized RGB image is input into the MASK R-CNN dynamic target detection network to identify the area where the dynamic object is located. The MASK R-CNN dynamic target detection network identifies the area where the dynamic object is located, specifically including:

[0085] The dynamic object images to be identified are made into a data set, and the data set is used to train the MASK R-CNN dynamic object detection network;

[0086] Input the RGB image to be detected into the trained MASK R-CNN dynamic target detection network for image preprocessing to obtain the corresponding feature map;

[0087] Calculate the region of interest of the RGB image according to the feature map and obtain multiple candidate frames of the region of interest;

[0088] Multiple ROI candidate boxes are input into the region candidate network for binary classification and regression to filter out some ROI candidate boxes;

[0089] The filtered ROI candidate box is subjected to ROI Align operation to obtain the category of the ROI and generate MASK.

[0090] During the specific implementation process, a MASK R-CNN dynamic target detection network is built and used to identify dynamic objects in image frames. A data set of dynamic objects to be identified, such as humans and animals, is created to train the MASK R-CNN dynamic target detection network. After the image is input into MASK R-CNN, the image is preprocessed, such as resizing and normalization. The processed image is then passed into the pre-trained neural network to obtain the corresponding feature map. After the coordinates on the feature map are quantized, many ROI candidate frames are obtained through the ROI generation function. That is, the feature map is quantized to a uniform scale, and then thousands of candidate frames are randomly generated on it. The candidate frames are then screened and removed. The generated ROI candidate frames are sent to the trained RPN for binary classification and regression to filter out some ROIs that do not meet the conditions. The remaining ROIs are ROI Aligned, classified, regressed, and MASK generated.

[0091] Furthermore, the ORB feature points contained in the dynamic object are initially removed in S130, specifically including:

[0092] Let the dynamic pixel feature set contained in the Nth frame RGB image input to the MASK R-CNN network be: The static pixel feature set is: in The i-th dynamic pixel feature in the N-th frame image, Represents the i-th static pixel feature in the N-th image frame; the set of feature points extracted from the N-th image frame is: in is the i-th feature point in the N-th frame image; if and but The feature points are dynamic points and are selected from the set T N Eliminate.

[0093] In S140, the frame from which ORB feature points are initially removed is input to the semi-static object recognition module. The depth calculated based on the mapping relationship between adjacent frames is subtracted from the depth of the actual RGB-D camera, and the dynamic ORB feature points are removed for the second time using a threshold. Specifically, the following steps are performed:

[0094] The overlap between the current frame after preliminarily removing ORB feature points and several previous frames is compared. Specifically, the feature points of several previous frames are mapped to the current frame and the angle between the 3D points of the previous frame and the current frame is calculated. For 3D points in the current frame with an angle less than 30 degrees, the error between the projected depth of the 3D point and the RGB-D camera depth is calculated. When the error is greater than a certain threshold, the 3D point in the current frame is determined to be a dynamic ORB feature point.

[0095] In the specific implementation process, a semi-static object recognition module is added to further identify dynamic feature points. The core is to select the previous frame with the largest overlap with the current frame. Considering accuracy and timeliness, the number of previous frames selected in the specific embodiment of the present invention is 5. The feature points of the previous frame are mapped to the current frame, and the angle between it and the corresponding three-dimensional point is calculated. For key points with an angle less than 30 degrees, the error between the projected depth of the key point and the depth of the RGB-D camera is calculated. The calculation formula is as follows: Δz = z proj -z′

[0096] Among them, z proj is the projected depth, and z′ is the depth of the current frame. When the error is greater than a certain threshold, the point is considered a dynamic point, such as a chair a person is sitting on or a book in their hand.

[0097] A further technical solution of the present invention is to use the nearest point iterative algorithm to perform feature matching on the static feature points remaining after the second removal of dynamic ORB feature points, obtain the position relationship between adjacent frames, perform pose estimation, and establish the final sparse map, which specifically includes:

[0098] After calling the Opencv function to calculate the descriptor of the extracted feature points, the feature points are matched according to the similarity of the descriptors, and the 3D feature points that have been matched in two adjacent frames are defined:

[0099]

[0100] Define the error term for the i-th point pair matched between the previous and next frames:

[0101]

[0102] Then construct the minimum sum of squares problem to find the minimum error function.

[0103]

[0104] in, They represent the rotation matrix and translation vector between two adjacent frames that need to be obtained.

[0105] Define the centroid of two sets of points:

[0106]

[0107] Then, in the error function, the following processing can be done:

[0108]

[0109] In the cross-term section, After simplification, it can be eliminated, and the optimization target result can be expressed as:

[0110]

[0111] Observing the two terms on the left and right, we can find that the left side is only related to the rotation matrix R, while the right side has both R and Therefore, ICP can be solved at three levels:

[0112] Calculate the position of the centroid of the corresponding points Then calculate the coordinates of each point excluding the centroid:

[0113]

[0114] The rotation matrix R that minimizes the error function is calculated based on the following formula: * :

[0115]

[0116] According to the above calculation, R * , calculate the translation vector that minimizes the error function

[0117]

[0118] Next, focus on R * Calculate , expand the error term about R, and get:

[0119]

[0120] Note that the first and second terms are independent of R, so the optimization objective function actually becomes:

[0121]

[0122] To understand R, first define the matrix:

[0123]

[0124] W is a three-dimensional matrix. Perform singular value decomposition on W and get:

[0125] W=U∑V T

[0126] Among them, ∑ is the eigenvalue matrix, and U and V are orthogonal matrices. When W is full rank, the optimal rotation matrix R * for:

[0127] R * =UV T

[0128] Solve for R * After that, we can solve

[0129] The algorithm pseudo code of the method of the present invention is shown in Table 1:

[0130] Table 1 Pseudocode of dynamic scene SLAM algorithm

[0131]

[0132]

[0133] Embodiment 2 of the present invention

[0134] A deep learning-based visual SLAM system 200 in a dynamic scene provided in the second embodiment of the present invention can execute the deep learning-based visual SLAM method in a dynamic scene provided in the first embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method. The device can be implemented by software and / or hardware (integrated circuit) and can generally be integrated into a server or terminal device. Figure 2 Schematic diagram of a deep learning-based visual SLAM system 200 in a dynamic scene in Example 2 of the present invention. Figure 2 , the visual SLAM system 200 based on deep learning in a dynamic scene according to an embodiment of the present invention may specifically include:

[0135] An RGB image acquisition unit 210 is configured to acquire an RGB image using an RGB-D camera;

[0136] An image homogenization unit 220 is used to homogenize the collected RGB image and extract ORB feature points in the RGB image;

[0137] The dynamic feature point preliminary elimination unit 230 is used to input the homogenized RGB image into the MASK R-CNN dynamic target detection network to identify the area where the dynamic object is located and to preliminarily eliminate the ORB feature points contained in the dynamic object;

[0138] The dynamic feature point second elimination unit 240 inputs the frame from which the ORB feature points have been initially eliminated into the semi-static object recognition module, subtracts the depth calculated based on the mapping relationship between adjacent frames from the actual RGB-D camera depth, and performs a second elimination of dynamic ORB feature points using a threshold.

[0139] The feature matching and map construction unit 250 is used to perform feature matching on the static feature points remaining after the second removal of dynamic ORB feature points, obtain the position relationship between adjacent frames, perform pose estimation, and build the final sparse map.

[0140] In addition to the above-mentioned units, the deep learning-based visual SLAM system 200 in dynamic scenes may also include other components. However, since these components are irrelevant to the content of the embodiments of the present disclosure, their illustration and description are omitted here.

[0141] The specific working process of a deep learning-based visual SLAM system 200 in a dynamic scene is described in the above-mentioned embodiment 1 of the deep learning-based visual SLAM method in a dynamic scene, and will not be repeated here.

[0142] Based on the third embodiment of the present invention

[0143] The system according to the embodiment of the present invention can also be used by Figure 3 The architecture of the computing device shown is implemented. Figure 3 The architecture of the computing device is shown in FIG. Figure 3 As shown, a computer system 301, a system bus 303, one or more CPUs 304, an input / output 302, a memory 305, etc. The memory 305 can store various data or files used for computer processing and / or communication and program instructions including the method of embodiment 1 executed by the CPU. Figure 3 The architecture shown is only exemplary and may be adjusted based on actual needs when implementing different devices. Figure 3One or more components in. The memory 305, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the deep learning-based visual SLAM method in dynamic scenes in the embodiment of the present invention (for example, the RGB image acquisition unit 210, image homogenization unit 220, dynamic feature point preliminary elimination unit 230, dynamic feature point second elimination unit 240 and feature matching and map construction unit 250 in the deep learning-based visual SLAM system 200 in dynamic scenes). One or more CPUs 304 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 305, that is, to realize the above-mentioned deep learning-based visual SLAM method in dynamic scenes, which includes:

[0144] Use an RGB-D camera to acquire RGB images;

[0145] Homogenize the collected RGB image and extract the ORB feature points in the RGB image;

[0146] The homogenized RGB image is input into the MASK R-CNN dynamic object detection network to identify the area where the dynamic object is located, and the ORB feature points contained in the dynamic object are initially removed;

[0147] The frame with ORB feature points initially removed is input into the semi-static object recognition module. The depth calculated based on the mapping relationship between adjacent frames is subtracted from the depth of the actual RGB-D camera, and the dynamic ORB feature points are removed for the second time using a threshold.

[0148] The static feature points remaining after the second removal of dynamic ORB feature points are matched using the nearest point iterative algorithm to obtain the position relationship between adjacent frames, perform pose estimation, and establish the final sparse map.

[0149] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute relevant operations in the deep learning-based visual SLAM method in dynamic scenes provided in any embodiment of the present invention.

[0150] The memory 305 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal, etc. Furthermore, the memory 305 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 305 may further include memory remotely located relative to one or more CPUs 304, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0151] The input / output 302 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 302 may also include a display device such as a display screen.

[0152] Based on the fourth embodiment of the present invention

[0153] The embodiments of the present invention may also be implemented as a computer-readable storage medium. A computer-readable storage medium according to Embodiment 4 may store a computer program. When the computer program is executed by a processor, the deep learning-based visual SLAM method for dynamic scenes according to Embodiment 1 of the present invention described with reference to the accompanying drawings may be executed.

[0154] Of course, the computer-executable instructions of the storage medium provided in the embodiment of the present invention are not limited to the method operations described above, but can also execute related operations in the deep learning-based visual SLAM method in dynamic scenes provided in any embodiment of the present invention.

[0155] The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0156] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0157] The program code embodied on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0158] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or terminal. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0159] In order to better reflect the effect of the present invention, the operating system used in the embodiment is Ubuntu16.04 and 8GB memory. The programming languages ​​used in SLAM include C++ and Python, and the environments that need to be configured include scipy, Tensorflow, Keras, OpenCV2.4.11, etc. In terms of evaluating visual SLAM errors, the absolute trajectory error (ATE) will be selected as the basis for judgment. The absolute trajectory error represents the direct difference between the actual trajectory coordinates of the selected data set and the trajectory coordinates obtained by running the SLAM algorithm, which can be expressed as the root mean square error:

[0160]

[0161] Among them, N represents the total number of coordinate points, x i is the true trajectory coordinate of the dataset, The estimated trajectory coordinates obtained by running the SLAM algorithm.

[0162] The public dataset freiburg3_walking_xyz is used. This dataset contains a large number of motion scenes. The specific data of the test are shown in Table 2. The trajectory diagram under one-dimensional coordinates is shown in Figure 4(a) 、 4(b) As shown in the above simulation results, it can be seen that the presence of dynamic objects seriously affects the estimation of camera pose. Removing dynamic objects in dynamic scenes can greatly improve the accuracy of pose.

[0163] Table 2 Average ATE error comparison experiment table

[0164]

[0165] Since each frame is interconnected, the positional relationship between adjacent frames can be determined based on pose estimation. When the transformation matrix between adjacent frames is known, the RGB and depth channels in a set of previous key frames can be projected into the dynamic segmentation of the current frame, and the RGB and depth maps of the previous 20 frames can be projected onto the current frame to complete background recovery without dynamic objects.

[0166] In summary, it can be seen from the embodiments that the present invention provides a visual SLAM method, system and storage medium based on deep learning in dynamic scenes, which is mainly achieved by combining the MASK R-CNN deep learning network with the traditional ORB-SLAM2 algorithm, using an RGB-D camera to capture images of the indoor environment, extracting ORB feature points from the collected RGB images, and inputting the RGB images into the MASK R-CNN deep learning network to detect potential moving objects and eliminate feature points on dynamic objects; at the same time, a semi-static object recognition module is added to cooperate with the target detection network to further check whether the feature points are potential dynamic feature points; finally, the background of the position of the eliminated dynamic target is completed according to the position relationship of adjacent frames and the final map is built. The present invention significantly reduces the trajectory error of the SLAM system in a dynamic environment, while ensuring the real-time and robustness of the system.

[0167] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A deep learning-based visual SLAM method in dynamic scenes, characterized by: The method comprises the following steps: Use an RGB-D camera to acquire RGB images; Homogenize the collected RGB image and extract the ORB feature points in the RGB image; The homogenized RGB image is input into the MASK R-CNN dynamic object detection network to identify the area where the dynamic object is located, and the ORB feature points contained in the dynamic object are initially removed; The frame with ORB feature points initially removed is input into the semi-static object recognition module. The depth calculated based on the mapping relationship between adjacent frames is subtracted from the depth of the actual RGB-D camera, and the dynamic ORB feature points are removed for the second time using a threshold. After removing the dynamic ORB feature points for the second time, the remaining static feature points are matched using the nearest point iteration algorithm to obtain the position relationship between adjacent frames, perform pose estimation, and build the final sparse map; The method performs feature matching on the static feature points remaining after the second removal of the dynamic ORB feature points, obtains the position relationship between adjacent frames, and performs pose estimation, specifically including: Call Opencv function to calculate the descriptor of the remaining static feature points, and match the feature points based on the similarity of the descriptors; Define the feature points that have been matched between two adjacent frames: Define the error term for the i-th point pair matched between the previous and next frames: Construct a minimum sum of squares problem and find R that minimizes the error function. Among them, R, They represent the rotation matrix and translation vector between two adjacent frames to be obtained, respectively, and n represents the number of matching feature points; Define the centroid of two sets of points: The error function is processed as follows: Where T represents the matrix transpose, in the cross term part, After simplification, it can be eliminated, and the optimization target result is expressed as: The above equation is solved at three levels: ①. Calculate the position of the center of mass of the corresponding point Then calculate the coordinates of each point excluding the centroid: in Representing feature points Remove the center of mass position The coordinates of Representing feature points Remove the center of mass position The coordinates of the rotation matrix R that minimizes the error function are calculated based on the following formula: * : ②. R obtained based on the error function * , calculate the translation vector that minimizes the error function Expand the error term of the error function about R, and we get: The optimization objective function is simplified to: Where tr() represents the trace of the matrix Define the matrix: W is a three-dimensional matrix. Perform singular value decomposition on W and get: W=U∑V T , where ∑ is the eigenvalue matrix, U and V are orthogonal matrices, and when W is full rank, the optimal rotation matrix is R * R * =UV T ; ③. Solve for R * After that, we can solve 2. The visual SLAM method based on deep learning in dynamic scenes according to claim 1, characterized in that The process of homogenizing the collected RGB image and extracting ORB feature points in the RGB image specifically includes: Perform Gaussian blur on the collected RGB image; Calculate the descriptors of the Gaussian blurred RGB image feature points and perform RGB image downsampling. Divide the RGB image into several layers, extract FAST corner points in each layer, and build an image pyramid based on the FAST corner points. The specific methods include: Calculate the number of FAST corner points N that need to be extracted on each layer of the image pyramid α for: Where N represents the number of all ORB feature points extracted from the RGB image, s is the scaling factor, n is the total number of pyramid layers, and α represents the number of pyramid layers; Each layer of RGB image is gridded, and the OpenCV function is called on each grid to extract the FAST corner points. If no FAST corner points can be extracted under the lowest FAST threshold preset by the OpenCV function, no FAST corner points will be extracted in this grid. For each grid partition, N is uniformly selected based on the quadtree. a FAST corner points.

3. The visual SLAM method based on deep learning in dynamic scenes according to claim 1, characterized in that The MASKR-CNN dynamic target detection network identifies the area where the dynamic object is located, specifically including: The dynamic object images to be identified are made into a data set, and the data set is used to train the MASKR-CNN dynamic object detection network; Input the RGB image to be detected into the trained MASK R-CNN dynamic target detection network for image preprocessing to obtain the corresponding feature map; Calculate the region of interest of the RGB image according to the feature map and obtain multiple candidate frames of the region of interest; Multiple ROI candidate boxes are input into the region candidate network for binary classification and regression to filter out some ROI candidate boxes; The filtered ROI candidate box is subjected to ROIAlign operation to obtain the category of the ROI and generate MASK.

4. The visual SLAM method based on deep learning in dynamic scenes according to claim 1, characterized in that The preliminary elimination of ORB feature points contained in the dynamic object specifically includes: Let the dynamic pixel feature set contained in the Nth frame RGB image input to the MASK R-CNN network be: The static pixel feature set is: in Represents the i-th dynamic pixel feature in the N-th frame image, Represents the i-th static pixel feature in the N-th image frame; the set of feature points extracted from the N-th image frame is: in is the i-th feature point in the N-th frame image; if and but The feature points are dynamic points and are selected from the set T N Eliminate from.

5. The visual SLAM method based on deep learning in dynamic scenes according to claim 1, characterized in that The frame from which ORB feature points are initially removed is input into the semi-static object recognition module, the depth calculated based on the mapping relationship between adjacent frames is subtracted from the depth of the actual RGB-D camera, and the dynamic ORB feature points are removed for the second time using a threshold, specifically including: The overlap between the current frame after preliminarily removing ORB feature points and several previous frames is compared. Specifically, the feature points of several previous frames are mapped to the current frame and the angle between the 3D points of the previous frame and the current frame is calculated. For 3D points in the current frame with an angle less than 30 degrees, the error between the projected depth of the 3D point and the RGB-D camera depth is calculated. When the error is greater than a certain threshold, the 3D point in the current frame is determined to be a dynamic ORB feature point.

6. A deep learning-based visual SLAM system in dynamic scenes, characterized by: include: An RGB image acquisition unit is used to acquire an RGB image using an RGB-D camera; An image homogenization unit is used to homogenize the collected RGB image and extract ORB feature points in the RGB image; The dynamic feature point preliminary elimination unit is used to input the homogenized RGB image into the MASK R-CNN dynamic target detection network to identify the area where the dynamic object is located and to perform preliminary elimination of the ORB feature points contained in the dynamic object; The second dynamic feature point removal unit inputs the frame with the ORB feature points initially removed into the semi-static object recognition module. The depth calculated based on the mapping relationship between adjacent frames is subtracted from the depth of the actual RGB-D camera, and the dynamic ORB feature points are removed for the second time using a threshold. The feature matching and map construction unit is used to perform feature matching on the static feature points remaining after the second removal of dynamic ORB feature points, obtain the position relationship between adjacent frames, perform pose estimation, and build the final sparse map; The method performs feature matching on the static feature points remaining after the second removal of the dynamic ORB feature points, obtains the position relationship between adjacent frames, and performs pose estimation, specifically including: Call Opencv function to calculate the descriptor of the remaining static feature points, and match the feature points based on the similarity of the descriptors; Define the feature points that have been matched between two adjacent frames: Define the error term for the i-th point pair matched between the previous and next frames: Construct a minimum sum of squares problem and find R that minimizes the error function. Among them, R, They represent the rotation matrix and translation vector between two adjacent frames to be obtained, respectively, and n represents the number of matching feature points; Define the centroid of two sets of points: The error function is processed as follows: Where T represents the matrix transpose, in the cross term part, After simplification, it can be eliminated, and the optimization target result is expressed as: The above equation is solved at three levels: ①. Calculate the position of the center of mass of the corresponding point Then calculate the coordinates of each point excluding the centroid: in Representing feature points Remove the center of mass position The coordinates of Representing feature points Remove the center of mass position The coordinates of the rotation matrix R that minimizes the error function are calculated based on the following formula: * : ②. R obtained based on the error function * , calculate the translation vector that minimizes the error function Expand the error term of the error function about R, and we get: The optimization objective function is simplified to: Where tr() represents the trace of the matrix Define the matrix: W is a three-dimensional matrix. Perform singular value decomposition on W and get: W=U∑V T , where ∑ is the eigenvalue matrix, U and V are orthogonal matrices, and when W is full rank, the optimal rotation matrix is R * R * =UV T ; ③. Solve for R * After that, we can solve 7. A deep learning-based visual SLAM system in dynamic scenes, characterized by: include: processor; and a memory, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the deep learning-based visual SLAM method in a dynamic scene according to any one of claims 1 to 5 is executed.

8. A storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the deep learning-based visual SLAM method in a dynamic scene according to any one of claims 1 to 5 is implemented.

Images