Deep learning based visual slam loop closure detection method

Abstract

The application discloses a kind of visual SLAM loop detection methods based on deep learning, by continuously obtaining the image information of the environment where target carrier is located;Image information is sent to loop detection module, the similarity of each frame image and the key frame sequence that has existed in visual SLAM system is calculated based on deep learning neural network model, and the image with the highest similarity is saved;According to the image with the highest similarity, loop information is obtained;According to the feedback loop information, the image information of the target carrier is optimized.The difference of the similarity in adjacent images is compared in the application, the image with the highest similarity is corrected, to ensure that all images in the image information optimization process are closest to live, improve the reliability of image information, further introduce deep learning, eliminate the infection of dynamic object in scene, reduce the probability of false matching, and then the technical effect of improving the accuracy of loop detection is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual detection, and in particular to a visual SLAM loop detection method based on deep learning. BACKGROUND

[0002] Self-positioning of a moving object in an unfamiliar environment and incremental map construction are achieved by using a visual simultaneous localization and mapping (SLAM) technology. In the earliest research, a SLAM system usually uses a laser or radar device as a sensor to collect information. With the improvement of computing performance, a new technology of using a camera as a sensor appears in the SLAM system. This technology is called visual SLAM, and has lower cost, richer environmental information and higher selectivity of implementation mode than the corresponding traditional laser or radar SLAM system.

[0003] Loop detection is an important measure to improve positioning accuracy and reduce errors in a visual SLAM system. With the rise and rapid development of artificial intelligence and deep learning algorithms, it is a feasible direction to apply a deep learning method to a loop detection module. How to design a visual SLAM loop detection method based on deep learning to reduce the cumulative error of system calculation has become a problem to be solved.

[0004] Therefore, a visual SLAM loop detection method based on deep learning is needed. SUMMARY

[0005] Embodiments of the present application provide a visual SLAM loop detection method based on deep learning, which at least solves the technical problems of high cost and low accuracy of loop detection of a traditional SLAM system in the related art.

[0006] According to an aspect of an embodiment of the present application, a visual SLAM loop detection method based on deep learning is provided, which includes:

[0007] Step 1: continuously acquiring image information of an environment in which a target carrier is located;

[0008] Step 2: sending the image information to a loop detection module, calculating the similarity of each frame of image and an existing key frame sequence in a visual SLAM system based on a deep learning neural network model, and saving an image with the highest similarity;

[0009] Step 3: acquiring loop information according to the image with the highest similarity;

[0010] Step 4: optimizing the image information of the target carrier according to the feedback loop information.

[0011] Optionally, the step one comprises: continuously acquiring images of the environment where the target carrier is located by using a camera, and sending the images to a visual SLAM system module; the visual SLAM system module calculates the images to obtain the pose of the target carrier and an environment map, i.e., image information.

[0012] Optionally, the step two specifically comprises:

[0013] Based on the deep learning neural network model, the similarity between each frame of image and any two adjacent frames of images a and b in the visual SLAM system is calculated, and the images with the highest similarity are images amax and bmax.

[0014] The similarity s1 between the image a and the image amax is calculated, and the similarity s2 between the image b and the image bmax is calculated.

[0015] It is judged whether the difference between the similarity s1 and the similarity s2 exceeds a set threshold c, if yes, a loop is occurred, and the loop information is fed back to the visual SLAM system; if not, the image b is taken as the image c, the image bmax is taken as the image amax, and the step two is re-entered.

[0016] Optionally, the deep learning neural network model is optimized.

[0017] Optionally, the optimization process comprises: inputting a training data set in a ternary format into the trained deep learning network for optimization to obtain an optimized network model comprising image feature descriptors.

[0018] According to another aspect of the embodiment of the present application, a visual SLAM loop detection system based on deep learning is also provided, comprising:

[0019] An acquisition module is configured to continuously acquire image information of the environment where the target carrier is located;

[0020] A loop detection module is configured to receive the image information from the acquisition module, calculate the similarity between each frame of image and a key frame sequence already existing in the visual SLAM system based on a deep learning neural network model, save the image with the highest similarity, and acquire loop information based on the image with the highest similarity.

[0021] An optimization module is configured to optimize the image information of the target carrier based on the loop information fed back by the loop detection module.

[0022] According to another aspect of the embodiment of the present application, a computer readable storage medium is also provided, which comprises a stored program, wherein the program controls the device where the computer readable storage medium is located to execute the visual SLAM loop detection method based on deep learning according to any one of the above embodiments when the program is running.

[0023] According to another aspect of the embodiments of the present application, a processor is also provided, which is used to run a program, wherein the program performs the deep learning based visual SLAM loop detection method according to any one of the above aspects when running.

[0024] Compared with the prior art, the present application has the following beneficial effects:

[0025] The deep learning based visual SLAM loop detection method provided by the present application continuously acquires image information of an environment where a target carrier is located, sends the image information to a loop detection module, calculates the similarity of each frame of image and an existing key frame sequence in a visual SLAM system based on a deep learning neural network model, and saves the image with the highest similarity. The loop information is acquired according to the image with the highest similarity, and the image information of the target carrier is optimized according to the feedback loop information. That is, the present application compares the difference in similarity between adjacent images, corrects the image with the highest similarity, and ensures that all images in the image information optimization process are closest to the live image, thereby improving the reliability of the image information, further introducing deep learning, eliminating the infection of dynamic object in the scene, reducing the probability of false matching, and thereby improving the accuracy of loop detection. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only one embodiment of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 is a flow chart of a deep learning based visual SLAM loop detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0029] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] According to an embodiment of the present application, an embodiment of a deep learning-based visual SLAM loop detection method is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0033] As Figure 1 is a flowchart of a deep learning-based visual SLAM loop detection method according to an embodiment of the present application, as Figure 1 shown, the method comprises the following steps:

[0034] Step one, continuously acquiring image information of the environment where the target carrier is located.

[0035] As an optional embodiment, step one specifically comprises: using a camera to continuously acquire images of the environment where the target carrier is located, and sending them to a visual SLAM system module. The visual SLAM system module calculates the images to obtain the pose of the target carrier and the environment map, i.e. image information.

[0036] Step two, sending the image information to a loop detection module, calculating the similarity of each frame of image with the existing key frame sequence in the visual SLAM system based on a deep learning neural network model, and saving the image with the highest similarity;

[0037] As an optional embodiment, step two specifically comprises:

[0038] Based on the deep learning neural network model, the similarity of each frame of image with any two adjacent images a and b in the visual SLAM system is calculated, and the corresponding image with the highest similarity is image amax and image bma;

[0039] The similarity s1 of image a and image amax is calculated, and the similarity s2 of image b and image bmax is calculated.

[0040] determining whether the difference between the similarity s1 and the similarity s2 exceeds a set threshold value c, if yes, a loop is generated, and the loop information is fed back to the visual SLAM system, if no, the image b is taken as the image c, the image bmax is taken as the image amax, and the step two is re-entered.

[0041] As an optional embodiment, the deep learning neural network model is optimized.

[0042] Specifically, the optimization process includes: inputting the training data set in the form of triplets into the trained deep learning network for optimization to obtain an optimized network model including image feature descriptors.

[0043] Step three, obtaining loop information according to the image with the highest similarity;

[0044] Step four, optimizing the image information of the target carrier according to the feedback loop information.

[0045] Embodiment 3

[0046] According to another aspect of the embodiment of the present application, a visual SLAM loop detection system based on deep learning is also provided, which comprises:

[0047] The acquisition module is used to continuously acquire image information of an environment where the target carrier is located;

[0048] The loop detection module is used to accept image information from the acquisition module, calculate the similarity between each frame of image and the existing key frame sequence in the visual SLAM system based on a deep learning neural network model, save the image with the highest similarity, and obtain loop information based on the image with the highest similarity;

[0049] The optimization module is used to optimize the image information of the target carrier according to the loop information fed back by the loop detection module.

[0050] The present application is not limited to the above specific embodiments, and the above is only a preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

[0051] Embodiment 3

[0052] According to another aspect of the embodiment of the present application, a computer readable storage medium is also provided, which comprises a stored program, wherein the program controls the device where the computer readable storage medium is located to execute the visual SLAM loop detection method based on deep learning of any one of the above when the program is running.

[0053] Optionally, in the embodiment, the computer readable storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group, and the computer readable storage medium comprises a stored program.

[0054] Optionally, the device in which the computer readable storage medium is located performs the following functions when the program is running: continuously acquiring image information of an environment in which the target carrier is located; sending the image information to a loop detection module, calculating a similarity between each frame of image and an existing key frame sequence in a visual SLAM system based on a deep learning neural network model, and saving an image with the highest similarity; obtaining loop information according to the image with the highest similarity; and optimizing image information of the target carrier according to the feedback loop information.

[0055] Embodiment 4

[0056] According to another aspect of the embodiments of the present application, a processor is also provided, which is used to run a program, wherein the program performs the deep learning-based visual SLAM loop detection method of any one of the above embodiments when running.

[0057] The embodiments of the present application provide a device, which comprises a processor, a memory, and a program stored in the memory and capable of running on the processor, and the processor implements the steps of the deep learning-based visual SLAM loop detection method when executing the program.

[0058] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0059] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0060] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the system embodiments described above are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules can be electrical or other forms.

[0061] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0062] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0063] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various program code storage media.

[0064] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for deep learning-based visual SLAM loop detection, characterized in that, The method comprises the following steps: Step 1: continuously acquiring image information of an environment in which a target carrier is located; a camera is used to continuously acquire images of the environment in which the target carrier is located, and the images are sent to a visual SLAM system module; the visual SLAM system module calculates the images to obtain a pose of the target carrier and an environment map, i.e., image information; Step 2: sending the image information to a loop detection module; based on a deep learning neural network model, the similarity of each frame of image to a key frame sequence already existing in the visual SLAM system is calculated, and the image with the highest similarity is saved; based on the deep learning neural network model, the similarity of each frame of image to any two adjacent images a and b in the visual SLAM system is calculated, and the images with the highest corresponding similarity are images amax and bmax; The similarity s1 of the image a to the image amax is calculated, and the similarity s2 of the image b to the image bmax is calculated; It is judged whether the difference between the similarity s1 and the similarity s2 exceeds a set threshold c; if yes, a loop occurs, and the loop information is fed back to the visual SLAM system; if no, the image b is taken as an image c, the image bmax is taken as the image amax, and step 2 is re-entered; Step 3: obtaining loop information according to the image with the highest similarity; Step 4: optimizing image information of the target carrier according to the feedback loop information. 2.The deep learning-based visual SLAM loop detection method of claim 1, wherein, The deep learning neural network model is optimized. 3.The deep learning-based visual SLAM loop detection method of claim 2, wherein, The optimization process comprises: inputting a training data set in a ternary format into a trained deep learning network for optimization to obtain an optimized network model comprising image feature descriptors. 4.A deep learning based visual SLAM loop detection device, characterized in that, The method of any one of claims 1-3 comprises: a collection module for continuously acquiring image information of an environment in which a target carrier is located; a loop detection module for receiving image information from the collection module, calculating the similarity of each frame of image to a key frame sequence already existing in the visual SLAM system based on a deep learning neural network model, saving the image with the highest similarity, and obtaining loop information based on the image with the highest similarity; an optimization module for optimizing image information of the target carrier according to loop information fed back by the loop detection module.

5. A computer readable storage medium, characterized in that, The computer-readable storage medium comprises a stored program, wherein the program controls a device in which the computer-readable storage medium is located to execute the deep learning-based visual SLAM loop detection method of any one of claims 1-3 when the program is running.

6. A processor, comprising: The processor is configured to run a program, wherein the program executes the deep learning-based visual SLAM loop detection method of any one of claims 1-3 when the program is running.

Images