Loopback detection method and apparatus, device, and storage medium

Abstract

The application discloses a loop detection method and device, equipment and storage medium, and relates to the technical field of computer vision, and comprises the following steps: determining a candidate key frame sub-set according to an original image and a loop detection key frame database; when the candidate key frame sub-set is not empty, respectively determining a target image set for constructing a key frame and an original visual image; determining a matching point cloud pair based on a dense reconstruction thread according to the target image set and the original visual image, calculating a similarity error of the matching point cloud pair; and performing loop detection according to the similarity error. In the foregoing manner, after the candidate key frame sub-set is determined, it is judged whether the candidate key frame sub-set is empty. If yes, it is indicated that there is no loop. If no, a dense reconstruction thread is opened, and an effectively matched matching point cloud pair is determined based on the dense reconstruction thread. Then, an image pair corresponding to the similarity error is used for loop detection, so that the accuracy of loop detection can be effectively improved, and the SLAM tracking precision is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to methods, apparatus, devices and storage media for loop closure detection. Background Technology

[0002] Loop closure detection is crucial for Simultaneous Localization and Mapping (SLAM) systems. On one hand, it determines the accuracy of trajectories and maps over long periods; on the other hand, it enables relocalization. Currently, common methods for loop closure detection rely on the personal experience of software engineers, followed by evaluation using loop closure detection rules. However, individual experience varies, and human detection inevitably introduces errors, resulting in relatively low accuracy for these methods.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a loop closure detection method, apparatus, device, and storage medium, aiming to solve the technical problem of low accuracy in existing loop closure detection techniques.

[0005] To achieve the above objectives, this application proposes a loop closure detection method, the method comprising:

[0006] A subset of candidate keyframes is determined based on the original image and the loop closure detection keyframe database;

[0007] When the candidate keyframe subset is not empty, the target image set and the original visual image for constructing the keyframe are determined respectively;

[0008] Based on the dense reconstruction thread, matching point cloud pairs are determined according to the target image set and the original visual image, and the similarity error of the matching point cloud pairs is calculated.

[0009] Loop closure detection is performed based on the similarity error of the matched point cloud pairs.

[0010] In one embodiment, the step of determining a subset of candidate keyframes based on the original image and the loop closure detection keyframe database includes:

[0011] Determine the current keyframe based on the original image;

[0012] Obtain the words in the current keyframe and count the number of words;

[0013] Traverse the target keyframes in the loop closure detection keyframe database to the same number as the stated quantity;

[0014] Calculate the number of words similar to each of the target keyframes and the current keyframe;

[0015] A subset of candidate keyframes is determined based on the number of similarities between each of the aforementioned words and the loop closure detection keyframe database.

[0016] In one embodiment, the step of determining a subset of candidate keyframes based on the number of similarities of each of the words and the loop closure detection keyframe database includes:

[0017] Select keyframes from the loop closure detection keyframe database whose number of word similarities is greater than a preset threshold.

[0018] A candidate keyframe set is generated based on keyframes whose word similarity is greater than a preset threshold.

[0019] Select candidate keyframes that are connected to the current keyframe from the set of candidate keyframes;

[0020] Generate a subset of candidate keyframes based on each candidate keyframe that has a connection relationship with the current keyframe.

[0021] In one embodiment, the step of determining the target image set for constructing keyframes and the original visual image when the candidate keyframe subset is not empty includes:

[0022] When the candidate keyframe subset is not empty, obtain image identification information;

[0023] Based on the image identification information, the original image used to construct keyframes is extracted by the dense reconstruction thread.

[0024] Generate a set of target images based on the original images of each of the constructed keyframes;

[0025] The original visual image is extracted based on the dense reconstruction thread.

[0026] In one embodiment, the step of determining matching point cloud pairs based on the target image set and the original visual image using a dense reconstruction thread includes:

[0027] The similarity between each image in the target image set and the original visual image is calculated based on the dense reconstruction thread.

[0028] Extract images from the target image set whose similarity is greater than a preset similarity threshold;

[0029] Based on the dense reconstruction thread, a first dense 3D point cloud is reconstructed from each image whose similarity is greater than a preset similarity threshold;

[0030] Based on the dense reconstruction thread, a second dense 3D point cloud is reconstructed according to the original visual image and the target's intrinsic and extrinsic parameters;

[0031] Matching point cloud pairs are determined based on the first dense 3D point cloud and the second dense 3D point cloud.

[0032] In one embodiment, the step of determining a matching point cloud pair based on the first dense 3D point cloud and the second dense 3D point cloud includes:

[0033] Traverse the current near-3D point cloud in the first dense 3D point cloud for each 3D point cloud in the second dense 3D point cloud;

[0034] Calculate the distances between the current near-3D point cloud and each 3D point cloud in the second dense 3D point cloud;

[0035] A matching point cloud pair is generated based on the 3D point cloud with a distance greater than a preset distance threshold and the current near-3D point cloud.

[0036] In one embodiment, the step of performing loop closure detection based on the similarity error of the matched point cloud pairs includes:

[0037] Select the minimum similarity error from the similarity errors of the multiple matching point cloud pairs;

[0038] Identify the image pair corresponding to the minimum similarity error;

[0039] Obtain the target keyframes corresponding to the image pair;

[0040] Loop closure detection is performed based on the target keyframe.

[0041] Furthermore, to achieve the above objectives, this application also proposes a loop closure detection device, which includes:

[0042] The determination module is used to determine a subset of candidate keyframes based on the original image and the loop closure detection keyframe database;

[0043] The determining module is further configured to determine the target image set and the original visual image for constructing the keyframes when the candidate keyframe subset is not empty;

[0044] The calculation module is used to determine matching point cloud pairs based on the target image set and the original visual image according to the dense reconstruction thread, and to calculate the similarity error of the matching point cloud pairs;

[0045] The detection module is used to perform loop closure detection based on the similarity error of the matched point cloud pairs.

[0046] In addition, to achieve the above objectives, this application also proposes a loop closure detection device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the loop closure detection method as described above.

[0047] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the loop closure detection method described above.

[0048] One or more technical solutions proposed in this application have at least the following technical effects: A subset of candidate keyframes is determined based on the original image and a loop closure detection keyframe database; when the subset of candidate keyframes is not empty, a set of target images and the original visual image for constructing keyframes are determined respectively; matching point cloud pairs are determined based on the target image set and the original visual image using a dense reconstruction thread, and the similarity error of the matching point cloud pairs is calculated; loop closure detection is performed based on the similarity error of the matching point cloud pairs; through the above method, after determining the subset of candidate keyframes, it is determined whether the subset of candidate keyframes is empty. If it is empty, it indicates that there is no loop closure; if not, a dense reconstruction thread is opened, and a valid matching point cloud pair is determined based on the dense reconstruction thread. Then, loop closure detection is performed using the image pairs corresponding to the similarity error, thereby effectively improving the accuracy of loop closure detection and thus improving the SLAM tracking accuracy. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating Embodiment 1 of the loop closure detection method of this application;

[0052] Figure 2 This is a flowchart illustrating Embodiment 2 of the loop closure detection method of this application;

[0053] Figure 3 This is a schematic diagram of the module structure of the loop closure detection device according to an embodiment of this application;

[0054] Figure 4This is a schematic diagram of the device structure of the hardware operating environment involved in the loop closure detection method in this application embodiment.

[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or loop closure detection device capable of performing the above functions. The following description uses a loop closure detection device as an example to illustrate this embodiment and the subsequent embodiments.

[0057] Based on this, embodiments of this application provide a loop closure detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the loop closure detection method of this application.

[0058] In this embodiment, the loop closure detection method includes steps S10 to S40:

[0059] Step S10: Determine a subset of candidate keyframes based on the original image and the loop closure detection keyframe database.

[0060] It should be noted that the candidate keyframe subset refers to the set of keyframes that are not connected to the current keyframe, the loop closure detection keyframe database refers to the database used to store loop closure detection keyframes with different single-pass counts, and the original image refers to the image used to construct the keyframes. After obtaining the original image, the feature points of the image are identified and the keyframe bag of words is constructed.

[0061] Step S20: When the candidate keyframe subset is not empty, determine the target image set and the original visual image for constructing the keyframes.

[0062] Understandably, after determining the candidate keyframe subset, it is necessary to determine whether the candidate keyframe subset is empty. If it is, it indicates that there is no loop closure. If not, further loop closure detection is required. At this time, the target image set and the original visual image for constructing the keyframe are determined respectively. The target image set refers to the set generated by each original image for constructing the keyframe, and the original visual image refers to the image tracked by the SLAM system at the current moment.

[0063] Further, step S20 includes: when the candidate keyframe subset is not empty, obtaining image identification information; extracting the original images for constructing keyframes based on the image identification information using a dense reconstruction thread; generating a target image set based on the original images of each constructed keyframe; and extracting the original visual images based on the dense reconstruction thread.

[0064] It should be understood that image identification information refers to the identification information used to identify different images. This image identification information can be an ID. The dense reconstruction thread is a real-time thread that is added to the entire process during loop closure detection. This dense reconstruction thread can be controlled by the loop closure detection thread in the loop closure detection device. This dense reconstruction thread not only needs to retain the original images of the keyframes, but also needs to perform three-dimensional dense reconstruction of the keyframes.

[0065] It is understandable that after adding a dense reconstruction thread, the original images for constructing keyframes are extracted based on the image identification information by the dense reconstruction thread, and they are also marked to correspond one-to-one with the keyframes. While generating the target image set, the original visual images are extracted using the dense reconstruction thread.

[0066] Step S30: Based on the dense reconstruction thread, determine matching point cloud pairs according to the target image set and the original visual image, and calculate the similarity error of the matching point cloud pairs.

[0067] It should be understood that a matching point cloud pair refers to a dense 3D point cloud pair that is effectively matched, and the similarity error refers to the similarity error between two matching point clouds in the matching point cloud pair. After determining the matching point cloud pair using the dense reconstruction thread, the similarity error of the matching point cloud pair is calculated.

[0068] Further, the step of determining matching point cloud pairs based on the dense reconstruction thread according to the target image set and the original visual image includes: calculating the similarity between each image in the target image set and the original visual image based on the dense reconstruction thread; extracting images from the target image set whose similarity is greater than a preset similarity threshold; reconstructing a first dense three-dimensional point cloud based on the dense reconstruction thread according to each of the images whose similarity is greater than the preset similarity threshold; reconstructing a second dense three-dimensional point cloud based on the dense reconstruction thread according to the original visual image and target intrinsic and extrinsic parameters; and determining matching point cloud pairs based on the first dense three-dimensional point cloud and the second dense three-dimensional point cloud.

[0069] It is understandable that similarity represents the degree of similarity between each image in the target image set and the original visual image. The larger the similarity value, the more similar the two images are. After calculating the similarity between each image in the target image set and the original visual image, they are compared with a preset similarity threshold in turn, and images with a similarity greater than the preset similarity threshold are extracted from the target image set based on the comparison results.

[0070] It should be understood that the first dense 3D point cloud refers to the 3D point cloud reconstructed from images with a similarity greater than a preset similarity threshold, while the second dense 3D point cloud refers to the 3D point cloud reconstructed from the original visual image. In the event of loop closure, since the original visual image and the images in the target image set have a large degree of overlap, the 3D dense point clouds reconstructed by the dense reconstruction thread using the aforementioned images also have a large degree of overlap in the world coordinate system.

[0071] Further, the step of determining matching point cloud pairs based on the first dense 3D point cloud and the second dense 3D point cloud includes: traversing the current near 3D point cloud in the first dense 3D point cloud for each 3D point cloud in the second dense 3D point cloud; calculating the distance between the current near 3D point cloud and each 3D point cloud in the second dense 3D point cloud respectively; and generating matching point cloud pairs based on the 3D point cloud whose distance is greater than a preset distance threshold and the current near 3D point cloud.

[0072] It should be understood that the current near-3D point cloud refers to the closest point cloud in the first dense 3D point cloud among all 3D point clouds in the second dense 3D point cloud. When it is determined that the current near-3D point cloud is not empty, the distance between the current near-3D point cloud and each 3D point cloud in the second dense 3D point cloud is calculated using a point cloud distance algorithm. It is then determined whether this distance is less than a preset distance threshold. If so, it indicates that the two 3D point clouds are a valid match, and all valid matches constitute a pair of matched point clouds. Furthermore, the algorithm used to calculate the similarity error of the matched point cloud pair can be the Iterative Closest Point (ICP) algorithm.

[0073] Step S40: Perform loop closure detection based on the similarity error of the matched point cloud pairs.

[0074] Further, step S40 includes: selecting the minimum similarity error from the similarity errors of multiple matching point cloud pairs; determining the image pair corresponding to the minimum similarity error; acquiring the target keyframe corresponding to the image pair; and performing loop closure detection based on the target keyframe.

[0075] Understandably, for each image in the target image set, it is necessary to calculate the similarity error of the matching point cloud pairs. At this time, there are multiple sets of matching point cloud pairs and multiple similarity errors. Then, the minimum similarity error is selected from the multiple similarity errors of the matching point cloud pairs. After determining the image pair corresponding to the minimum similarity error, the target keyframe corresponding to the image pair is further determined. The target keyframe at this time is the loop closure frame, and the loop closure detection has been completed.

[0076] It should be understood that after obtaining the target keyframe, the target keyframe is used as the loop closure frame for global loop closure correction, thereby optimizing the global sparse map and further improving SLAM tracking accuracy.

[0077] This embodiment determines a subset of candidate keyframes based on the original image and a loop closure detection keyframe database. When the subset of candidate keyframes is not empty, a set of target images and the original visual image for constructing keyframes are determined. A dense reconstruction thread determines matching point cloud pairs based on the target image set and the original visual image, and calculates the similarity error of the matching point cloud pairs. Loop closure detection is performed based on the similarity error of the matching point cloud pairs. Through this method, after determining the subset of candidate keyframes, it is determined whether the subset is empty. If it is, it indicates that there are no loop closures. If not, a dense reconstruction thread is opened, and valid matching point cloud pairs are determined based on the dense reconstruction thread. Then, loop closure detection is performed using the image pairs corresponding to the similarity error, thereby effectively improving the accuracy of loop closure detection and thus improving SLAM tracking accuracy.

[0078] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S10 includes steps S101 to S105:

[0079] Step S101: Determine the current keyframe based on the original image.

[0080] It should be noted that the current keyframe refers to the keyframe at the current moment when loop closure detection is performed, and this current keyframe can be determined from the original image.

[0081] Step S102: Obtain the words in the current keyframe and count the number of words.

[0082] It is understandable that each keyframe has its own words. Since this embodiment traverses the target keyframes by counting the number of words with the same words, it is necessary to count the number of words in the current keyframe.

[0083] Step S103: Traverse the target keyframes in the loop closure detection keyframe database, matching the stated number.

[0084] It should be understood that the target keyframe refers to a keyframe whose number of words is the same as the number of words in the current keyframe. After counting the words in the current keyframe, the target keyframe is traversed from the loop closure detection keyframe database based on the number of words.

[0085] Step S104: Calculate the number of words similar to each of the target keyframes and the current keyframe.

[0086] Understandably, word similarity refers to the number of words that are similar between each target keyframe and the current keyframe. The more words that are similar, the more similar the two keyframes are.

[0087] Step S105: Determine a subset of candidate keyframes based on the number of similarities of each word and the loop closure detection keyframe database.

[0088] It should be understood that the candidate keyframe subset refers to the set of keyframes that are not connected to the current keyframe. After obtaining the number of word similarities, the candidate keyframe subset is determined by combining the loop closure detection keyframe database.

[0089] Further, step S105 includes: selecting keyframes from the loop closure detection keyframe database whose word similarity is greater than a preset threshold; generating a candidate keyframe set based on each keyframe whose word similarity is greater than the preset threshold; selecting candidate keyframes from the candidate keyframe set that have a connection relationship with the current keyframe; and generating a candidate keyframe subset based on each candidate keyframe that has a connection relationship with the current keyframe.

[0090] It is understandable that the candidate keyframe set refers to the set of keyframes consisting of keyframes whose word similarity is greater than a preset threshold. This candidate keyframe set includes candidate keyframes that are connected to the current keyframe and candidate keyframes that are not connected to the current keyframe. At this time, a subset of candidate keyframes is generated from the candidate keyframes that are connected to the current keyframe.

[0091] This embodiment determines the current keyframe based on the original image; obtains the words in the current keyframe and counts the number of words; traverses the target keyframes with the same number of words from the loop closure detection keyframe database; calculates the word similarity between each target keyframe and the current keyframe; and determines a candidate keyframe subset based on the word similarity and the loop closure detection keyframe database. By doing so, after determining the current keyframe, counting the number of words in the current keyframe, and after traversing the target keyframes, calculating the word similarity between each target keyframe and the current keyframe, and then determining the candidate keyframe subset based on the word similarity and the comparison result of the word similarity with a preset threshold and the loop closure detection keyframe database, the accuracy of determining the candidate keyframe subset can be effectively improved.

[0092] This application also provides a loop closure detection device, please refer to... Figure 3 The loop closure detection device includes:

[0093] The determination module 10 is used to determine a subset of candidate keyframes based on the original image and the loop closure detection keyframe database.

[0094] The determining module 10 is further configured to determine the target image set and the original visual image for constructing the keyframes when the candidate keyframe subset is not empty.

[0095] The calculation module 20 is used to determine matching point cloud pairs based on the target image set and the original visual image according to the dense reconstruction thread, and to calculate the similarity error of the matching point cloud pairs.

[0096] The detection module 30 is used to perform loop closure detection based on the similarity error of the matched point cloud pairs.

[0097] This embodiment determines a subset of candidate keyframes based on the original image and a loop closure detection keyframe database. When the subset of candidate keyframes is not empty, a set of target images and the original visual image for constructing keyframes are determined. A dense reconstruction thread determines matching point cloud pairs based on the target image set and the original visual image, and calculates the similarity error of the matching point cloud pairs. Loop closure detection is performed based on the similarity error of the matching point cloud pairs. Through this method, after determining the subset of candidate keyframes, it is determined whether the subset is empty. If it is, it indicates that there are no loop closures. If not, a dense reconstruction thread is opened, and valid matching point cloud pairs are determined based on the dense reconstruction thread. Then, loop closure detection is performed using the image pairs corresponding to the similarity error, thereby effectively improving the accuracy of loop closure detection and thus improving SLAM tracking accuracy.

[0098] The loop closure detection device provided in this application, employing the loop closure detection method in the above embodiments, can solve the technical problem of low accuracy in loop closure detection in the prior art. Compared with the prior art, the beneficial effects of the loop closure detection device provided in this application are the same as those of the loop closure detection method provided in the above embodiments, and other technical features in the loop closure detection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0099] In one embodiment, the determining module 10 is further configured to: determine the current keyframe based on the original image; obtain the words in the current keyframe and count the number of words; traverse the target keyframes with the same number from the loop closure detection keyframe database; calculate the number of words similar to each of the target keyframes and the current keyframe; and determine a subset of candidate keyframes based on the number of words similar to each of the target keyframes and the loop closure detection keyframe database.

[0100] In one embodiment, the determining module 10 is further configured to: select keyframes from the loop closure detection keyframe database whose word similarity is greater than a preset threshold; generate a candidate keyframe set based on each keyframe whose word similarity is greater than the preset threshold; select candidate keyframes from the candidate keyframe set that are connected to the current keyframe; and generate a candidate keyframe subset based on each candidate keyframe that is connected to the current keyframe.

[0101] In one embodiment, the determining module 10 is further configured to: obtain image identification information when the candidate keyframe subset is not empty; extract the original image for constructing the keyframe based on the image identification information using a dense reconstruction thread; generate a target image set based on the original image of each constructed keyframe; and extract the original visual image based on the dense reconstruction thread.

[0102] In one embodiment, the calculation module 20 is further configured to: calculate the similarity between each image in the target image set and the original visual image based on a dense reconstruction thread; extract images from the target image set whose similarity is greater than a preset similarity threshold; reconstruct a first dense three-dimensional point cloud based on the dense reconstruction thread according to each of the images whose similarity is greater than the preset similarity threshold; reconstruct a second dense three-dimensional point cloud based on the dense reconstruction thread according to the original visual image and the target intrinsic and extrinsic parameters; and determine matching point cloud pairs based on the first dense three-dimensional point cloud and the second dense three-dimensional point cloud.

[0103] In one embodiment, the calculation module 20 is further configured to traverse the current near-3D point cloud in the first dense 3D point cloud for each 3D point cloud in the second dense 3D point cloud; calculate the distance between the current near-3D point cloud and each 3D point cloud in the second dense 3D point cloud respectively; and generate matching point cloud pairs based on the 3D point cloud whose distance is greater than a preset distance threshold and the current near-3D point cloud.

[0104] In one embodiment, the detection module 30 is further configured to select the minimum similarity error from the similarity errors of the plurality of matching point cloud pairs; determine the image pair corresponding to the minimum similarity error; acquire the target keyframe corresponding to the image pair; and perform loop closure detection based on the target keyframe.

[0105] This application provides a loop closure detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the loop closure detection method in Embodiment 1 above.

[0106] The following is for reference. Figure 4 The diagram illustrates a structural schematic suitable for implementing the loop closure detection device in the embodiments of this application. The loop closure detection device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The loop closure detection device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0107] like Figure 4As shown, the loop closure detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the loop closure detection device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the loopback detection device to communicate wirelessly or wiredly with other devices to exchange data. Although loopback detection devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0108] Specifically, according to the embodiments disclosed in this application, the process described above with reference to the flowcharts can be implemented as a computer software program. This computer program includes program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0109] The loop closure detection device provided in this application, employing the loop closure detection method described in the above embodiments, can solve the technical problem of low accuracy in loop closure detection in the prior art. Compared with the prior art, the beneficial effects of the loop closure detection device provided in this application are the same as those of the loop closure detection method provided in the above embodiments, and other technical features of the loop closure detection device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0110] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0112] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the loop closure detection method in the above embodiments.

[0113] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0114] The aforementioned computer-readable storage medium may be included in the loop closure detection device; or it may exist independently and not be assembled into the loop closure detection device.

[0115] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0118] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described loop closure detection method, which can solve the technical problem of low accuracy in loop closure detection in the prior art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the loop closure detection method provided in the above embodiments, and will not be repeated here.

[0119] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A loop closure detection method, characterized in that, The method includes: A subset of candidate keyframes is determined based on the original image and the loop closure detection keyframe database; When the candidate keyframe subset is not empty, the target image set and the original visual image for constructing the keyframe are determined respectively; Based on the dense reconstruction thread, matching point cloud pairs are determined according to the target image set and the original visual image, and the similarity error of the matching point cloud pairs is calculated. Loop closure detection is performed based on the similarity error of the matched point cloud pairs.

2. The method as described in claim 1, characterized in that, The step of determining a subset of candidate keyframes based on the original image and the loop closure detection keyframe database includes: Determine the current keyframe based on the original image; Obtain the words in the current keyframe and count the number of words; Traverse the target keyframes in the loop closure detection keyframe database to the same number as the stated quantity; Calculate the number of words similar to each of the target keyframes and the current keyframe; A subset of candidate keyframes is determined based on the number of similarities between each of the aforementioned words and the loop closure detection keyframe database.

3. The method as described in claim 2, characterized in that, The step of determining a subset of candidate keyframes based on the number of similarities between each of the words and the loop closure detection keyframe database includes: Select keyframes from the loop closure detection keyframe database whose number of word similarities is greater than a preset threshold. A candidate keyframe set is generated based on keyframes whose word similarity is greater than a preset threshold. Select candidate keyframes that are connected to the current keyframe from the set of candidate keyframes; Generate a subset of candidate keyframes based on each candidate keyframe that has a connection relationship with the current keyframe.

4. The method as described in claim 1, characterized in that, The step of determining the target image set and the original visual image for constructing keyframes when the candidate keyframe subset is not empty includes: When the candidate keyframe subset is not empty, obtain image identification information; Based on the image identification information, the original image used to construct keyframes is extracted by the dense reconstruction thread. Generate a set of target images based on the original images of each of the constructed keyframes; The original visual image is extracted based on the dense reconstruction thread.

5. The method as described in claim 1, characterized in that, The step of determining matching point cloud pairs based on the target image set and the original visual image using the dense reconstruction thread includes: The similarity between each image in the target image set and the original visual image is calculated based on the dense reconstruction thread. Extract images from the target image set whose similarity is greater than a preset similarity threshold; Based on the dense reconstruction thread, a first dense 3D point cloud is reconstructed from each image whose similarity is greater than a preset similarity threshold; Based on the dense reconstruction thread, a second dense 3D point cloud is reconstructed according to the original visual image and the target's intrinsic and extrinsic parameters; Matching point cloud pairs are determined based on the first dense 3D point cloud and the second dense 3D point cloud.

6. The method as described in claim 5, characterized in that, The step of determining matching point cloud pairs based on the first dense 3D point cloud and the second dense 3D point cloud includes: Traverse the current near-3D point cloud in the first dense 3D point cloud for each 3D point cloud in the second dense 3D point cloud; Calculate the distances between the current near-3D point cloud and each 3D point cloud in the second dense 3D point cloud; A matching point cloud pair is generated based on the 3D point cloud with a distance greater than a preset distance threshold and the current near-3D point cloud.

7. The method according to any one of claims 1 to 6, characterized in that, The step of performing loop closure detection based on the similarity error of the matched point cloud pairs includes: Select the minimum similarity error from the similarity errors of the multiple matching point cloud pairs; Identify the image pair corresponding to the minimum similarity error; Obtain the target keyframes corresponding to the image pair; Loop closure detection is performed based on the target keyframe.

8. A loop closure detection device, characterized in that, The device includes: The determination module is used to determine a subset of candidate keyframes based on the original image and the loop closure detection keyframe database; The determining module is further configured to determine the target image set and the original visual image for constructing the keyframes when the candidate keyframe subset is not empty; The calculation module is used to determine matching point cloud pairs based on the target image set and the original visual image according to the dense reconstruction thread, and to calculate the similarity error of the matching point cloud pairs; The detection module is used to perform loop closure detection based on the similarity error of the matched point cloud pairs.

9. A loop closure detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the loop closure detection method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the loop closure detection method as described in any one of claims 1 to 7.

Images