Underwater visual closed-loop detection method, device, equipment and medium

Abstract

This invention relates to the field of computer technology and discloses an underwater visual loop closure detection method, apparatus, device, and medium. It utilizes a convolutional neural network to simultaneously extract global and local feature information from acquired keyframe images. Based on the global feature information of the current keyframe, it quantifies the global similarity and temporal correlation between the current keyframe and other keyframes to construct a loop closure database. Simultaneously, based on the global feature information and inter-frame adjacency characteristics of the current keyframe, it searches for loop closure candidate frames from the database. The loop closure candidate frames are then verified based on the local feature information of the current keyframe and the local feature information of the loop closure candidate frames. If the verification is successful, the loop closure candidate frame is used as the target loop closure frame for the current keyframe. By constraining the loop closure detection process through image similarity, temporal correlation, and inter-frame adjacency, the accuracy and recall rate of underwater visual loop closure are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an underwater visual closed-loop detection method, apparatus, equipment, and medium. Background Technology

[0002] With the continuous development of positioning technology, visual simultaneous localization and mapping (SLAM) technology has been widely used in underwater research.

[0003] Loop closure detection is an important part of SLAM technology, which can effectively reduce positioning errors caused by trajectory drift. However, the simple structure and insufficient texture of the underwater environment pose a great challenge to loop closure detection, resulting in low recall rate and poor positioning accuracy.

[0004] Therefore, how to solve the problem of low positioning accuracy and recall rate in underwater visual loop closure detection has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This invention provides an underwater visual loop closure detection method, apparatus, equipment, and medium, addressing the problem of improving the positioning accuracy and recall rate of underwater visual loop closure detection.

[0006] To address the aforementioned technical problems, the first aspect of this invention provides an underwater visual loop closure detection method, comprising:

[0007] Multiple keyframes and their corresponding keyframe images are acquired sequentially. Global and local feature information of the keyframe images are extracted using a convolutional neural network model, and the keyframes are sequentially placed into a queue of frames waiting to be inserted.

[0008] Obtain the current keyframe in the queue of frames waiting to be inserted, and calculate the global similarity and temporal correlation between the current keyframe and other keyframes in the queue of frames waiting to be inserted based on the global feature information of the current keyframe, so as to construct a closed-loop database.

[0009] Based on the global feature information and inter-frame adjacency of the current keyframe, search for closed-loop candidate frames from the closed-loop database;

[0010] The closed-loop candidate frame is verified based on the local feature information of the current key frame and the local feature information of the closed-loop candidate frame. When the verification is successful, the closed-loop candidate frame is used as the target closed-loop frame of the current key frame to achieve underwater visual closed-loop detection.

[0011] As one preferred embodiment, the step of sequentially acquiring multiple keyframes and their corresponding keyframe images, and extracting global and local feature information of the keyframe images using a convolutional neural network model, includes:

[0012] Multiple keyframes and their corresponding keyframe images are sequentially acquired from the front end of the SLAM system.

[0013] The local features of the keyframe image are extracted by the convolutional layer of the convolutional neural network model, and the high discriminative information and low-dimensional representation information of the local features are extracted by the attention layer and the autoencoder of the convolutional neural network model, respectively, to obtain the local feature information of the keyframe image.

[0014] The global features of the keyframe image are extracted by the convolutional layer of the convolutional neural network model, and the contribution of the global features is weighted by the generalized average pooling layer of the convolutional neural network model to obtain the global feature information of the keyframe image.

[0015] As one preferred embodiment, the step of calculating the global similarity and temporal correlation between the current keyframe and other keyframes in the queue of frames waiting to be inserted, based on the global feature information of the current keyframe, to construct a closed-loop database, includes:

[0016] Based on the global feature information of the current keyframe and the global feature information of the first keyframe in the queue of waiting frames, calculate the global similarity and temporal correlation between the current keyframe and the first keyframe.

[0017] In the determination phase, if the global similarity and the temporal correlation satisfy the first requirement determination or the second requirement determination, the first keyframe is inserted into the closed-loop database and removed from the queue of frames waiting to be inserted, so as to update the first keyframe in the queue of frames waiting to be inserted; the first requirement determination is that the global similarity meets the similarity requirement, and the second requirement determination is that the global similarity does not meet the similarity requirement but the temporal correlation meets the correlation requirement.

[0018] Based on the global feature information of the current key frame, the determination process is repeated until the number of frames in the waiting-to-insert-frame queue reaches a preset number, and then the closed-loop database is output.

[0019] As one preferred embodiment, the step of calculating the global similarity and temporal correlation between the current keyframe and the first keyframe in the queue of waiting-to-be-inserted frames, based on their global feature information, includes:

[0020] When the global similarity and the temporal correlation meet the third requirement, the current keyframe is updated, and the global similarity and temporal correlation between the two are recalculated based on the global feature information of the updated current keyframe and the global feature information of the first keyframe in the queue of frames waiting to be inserted; the third requirement is that the global similarity does not meet the similarity requirement and the temporal correlation does not meet the correlation requirement.

[0021] The requirements are determined based on the recalculated global similarity and temporal correlation. The key frame at the head of the queue or the updated current key frame is updated based on the determination result. The determination process is repeated based on the update result until all current key frames in the queue of frames waiting to be inserted are traversed, and the closed-loop database is output.

[0022] As one preferred embodiment, the step of searching for loop-closing candidate frames from the loop-closing database based on the global feature information and inter-frame adjacency of the current keyframe includes:

[0023] The similarity between the global feature information of each closed-loop frame in the closed-loop database and the global feature information of the current key frame is calculated respectively, and the frames are sorted according to the similarity to construct a closed-loop frame queue.

[0024] A closed-loop frame sequence is constructed based on the current closed-loop frame, the first closed-loop frame, and the second closed-loop frame in the closed-loop frame queue, and the closed-loop frame sequence is taken as the nearest neighbor of the current closed-loop frame; the first closed-loop frame and the second closed-loop frame are the previous closed-loop frame and the next closed-loop frame of the current closed-loop frame, respectively.

[0025] Each closed-loop frame in the closed-loop frame queue, excluding the first closed-loop frame, is taken as a closed-loop frame to be verified and verified sequentially. Each closed-loop frame to be verified that satisfies either the first verification condition or the second verification condition is taken as a preferred closed-loop frame, and together with the first closed-loop frame in the closed-loop frame sequence, is taken as a candidate closed-loop frame for the current keyframe. The first verification condition is that when the first closed-loop frame of the closed-loop frame to be verified is the nearest neighbor of the first closed-loop frame, the closed-loop frame to be verified is not the nearest neighbor of the first closed-loop frame. The second verification condition is that when the first closed-loop frame of the closed-loop frame to be verified is not the nearest neighbor of the first closed-loop frame, the closed-loop frame to be verified is neither the nearest neighbor of the first closed-loop frame nor the nearest neighbor of the first closed-loop frame of the closed-loop frame to be verified.

[0026] As a preferred embodiment, the step of verifying the loop closure candidate frame based on the local feature information of the current keyframe and the local feature information of the loop closure candidate frame, and when the verification is successful, using the loop closure candidate frame as the target loop closure frame of the current keyframe to achieve underwater visual loop closure detection, includes:

[0027] Based on the local feature information of the current keyframe and the local feature information of the closed-loop candidate frames, the closed-loop candidate frames are geometrically verified using a random sampling consensus algorithm to obtain the number of interior points of each closed-loop candidate frame.

[0028] The candidate frame with the most inliers is selected as the initial candidate frame for time-series verification, and the initial candidate frame that passes the time-series verification is selected as the target candidate frame for the current keyframe to achieve underwater visual loop closure detection.

[0029] As one preferred embodiment, selecting the closed-loop candidate frame with the highest number of internal points as the initial closed-loop candidate frame for timing verification includes:

[0030] The closed-loop candidate frame with the most internal points is selected as the first preliminary closed-loop candidate frame, and the nearest neighbor of the first preliminary closed-loop candidate frame is constructed.

[0031] The current keyframe is updated, and the search and geometric verification steps for the closed-loop candidate frames of the updated current keyframe are repeated to obtain the initial closed-loop candidate frames of the updated current keyframe as the second initial closed-loop candidate frames.

[0032] When the second preliminary closed-loop candidate frame is the nearest neighbor of the first preliminary closed-loop candidate frame, the timing verification of the first preliminary closed-loop candidate frame passes; otherwise, the timing verification of the first preliminary closed-loop candidate frame fails, and the timing verification step is performed on the second preliminary closed-loop candidate frame until all key frames in the waiting-to-insert-frame queue are iteratively verified.

[0033] A second aspect of the present invention provides an underwater visual closed-loop detection device, comprising:

[0034] The feature calculation module is used to sequentially acquire multiple keyframes and their corresponding keyframe images, extract global and local feature information of the keyframe images through a convolutional neural network model, and sequentially place the keyframes into a queue of frames waiting to be inserted.

[0035] The database construction module is used to obtain the current key frame in the queue of frames waiting to be inserted, and calculate the global similarity and temporal correlation between the current key frame and other key frames in the queue of frames waiting to be inserted based on the global feature information of the current key frame, so as to construct a closed-loop database.

[0036] The candidate frame search module is used to search for closed-loop candidate frames from the closed-loop database based on the global feature information and inter-frame adjacency of the current key frame.

[0037] The candidate frame verification module is used to verify the closed-loop candidate frame based on the local feature information of the current key frame and the local feature information of the closed-loop candidate frame, and when the verification is successful, the closed-loop candidate frame is used as the target closed-loop frame of the current key frame to realize underwater visual closed-loop detection.

[0038] A third aspect of the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the underwater visual loop closure detection method as described above.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the underwater visual loop closure detection method as described above.

[0040] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following:

[0041] (1) By extracting global and local feature information from keyframe images, important features in the images can be captured comprehensively and accurately, providing a solid foundation for subsequent analysis and matching;

[0042] (2) Based on the global feature information of the current key frame, calculate its global similarity and temporal correlation with other key frames. This can build a closed-loop database with strong correlation. It not only contains the visual features of the key frame, but also incorporates temporal information, which helps to identify the closed loop more accurately.

[0043] (3) Considering the difficulties in constructing the closed-loop database and selecting closed-loop frames, we propose to use image similarity, temporal correlation and inter-frame adjacency to constrain the closed-loop detection process, which strengthens the effectiveness of global and local features of the image and improves the accuracy and recall of underwater visual closed-loop detection. Attached Figure Description

[0044] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of an underwater visual loop closure detection method provided in a certain embodiment of the present invention;

[0046] Figure 2This is a flowchart of image feature extraction provided in a certain embodiment of the present invention;

[0047] Figure 3 This is a flowchart of a closed-loop database construction process provided in a certain embodiment of the present invention;

[0048] Figure 4 This is a structural diagram of an underwater visual closed-loop detection device provided in a certain embodiment of the present invention;

[0049] Figure 5 This is a structural diagram of an electronic device provided in a certain embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings and examples. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0051] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0052] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0053] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0054] Currently, simultaneous localization and mapping (SLAM) technology utilizes cameras to perceive the surrounding environment, enabling self-localization without prior maps. Due to its low cost and high visibility, it is increasingly widely used in underwater research. The loop closure detection module is a crucial part of SLAM, used to identify previously visited locations and perform loop closure correction, effectively reducing localization errors caused by trajectory drift. However, the simple structure and insufficient texture of the underwater environment pose significant challenges to loop closure detection.

[0055] To address this issue, patents such as the one authorized on January 25, 2022, with authorization announcement number CN113112565B, entitled "A Map Construction Method, Storage Medium, and Device Based on Robot Vision Loop Closure Detection Algorithm," utilize Gaussian HOG descriptors to extract global features and perform secondary detection on the position matching of strong feature regions to achieve efficient loop closure detection. However, this scheme does not consider the impact of loop closure database construction on loop closure recall during loop closure frame search. Another example is the patent authorized on December 23, 2022, with authorization announcement number CN113033555B, entitled "Visual SLAM Loop Closure Detection Method Based on Metric Learning," which uses a triplet loss function to train metric learning in the SLAM system and performs similarity judgment on the keyframe feature vectors obtained by the learned model. However, this scheme does not involve the verification process of loop closure candidate frames, resulting in a loss of loop closure accuracy. Furthermore, it does not consider the construction of a loop closure database, leading to a low loop closure recall rate.

[0056] Based on this, this invention leverages the advantage that convolutional neural network features extracted from specific layers can demonstrate higher recognition capabilities compared to manually created image features. Convolutional neural networks are commonly used in loop closure detection. This invention employs a convolutional neural network model to extract global features from frame images for rapid searching of similar images, and simultaneously extracts local features from frame images for geometric verification. Considering the impact of loop closure database construction and HNSW graph connection frames on loop closure detection, loop closure detection is constrained by image similarity, temporal correlation, and inter-frame adjacency to improve loop closure recall and accuracy.

[0057] In one embodiment, such as Figure 1 As shown, the first aspect of the present invention provides an underwater visual loop closure detection method, comprising:

[0058] S1. Sequentially acquire multiple keyframes and their corresponding keyframe images, extract global and local feature information of the keyframe images through a convolutional neural network model, and sequentially place the keyframes into the queue of frames waiting to be inserted.

[0059] In one embodiment, the step of sequentially acquiring multiple keyframes and their corresponding keyframe images, and extracting global and local feature information of the keyframe images using a convolutional neural network model, includes:

[0060] Multiple keyframes and their corresponding keyframe images are sequentially acquired from the front end of the SLAM system.

[0061] The local features of the keyframe image are extracted by the convolutional layer of the convolutional neural network model, and the high discriminative information and low-dimensional representation information of the local features are extracted by the attention layer and the autoencoder of the convolutional neural network model, respectively, to obtain the local feature information of the keyframe image.

[0062] The global features of the keyframe image are extracted by the convolutional layer of the convolutional neural network model, and the contribution of the global features is weighted by the generalized average pooling layer of the convolutional neural network model to obtain the global feature information of the keyframe image.

[0063] Specifically, considering the structural uniformity of the underwater environment, this invention employs a ResNet-50 network as the backbone convolutional neural network for feature extraction of keyframe images: After obtaining keyframe images from the keyframe queue at the front end of the SLAM system, local features are extracted using the Conv2 convolutional layer of the ResNet-50 network. The attention layer of the ResNet-50 network is then used to extract highly discriminative features from these local features. A convolutional autoencoder of the ResNet-50 network is then used to learn low-dimensional representations of these local features. Combining these two methods yields the local feature information of the keyframe image. Simultaneously, the Conv2 convolutional layer of the ResNet-50 network is used to extract global features of the keyframe image, and a generalized average pooling layer of the ResNet-50 network is used to weight the contribution of each feature to obtain the global feature description information of the keyframe image. The feature extraction process of the keyframe image is as follows: Figure 2 As shown.

[0064] In the initial stage of the SLAM system, loop closure does not occur. Therefore, keyframes can be directly inserted into the loop closure database after feature extraction. When the number of frames in the loop closure database reaches a certain threshold, new keyframes are sequentially added to the queue of frames waiting to be inserted. Considering the motion characteristics of underwater vehicles and the elements required to construct the loop closure database, this example sets the keyframe threshold to 50 frames. This invention takes into account the impact of low-quality underwater images on visual loop closure detection. It utilizes different layers of a convolutional neural network to simultaneously learn the global and local features of keyframe images, thereby improving the search efficiency of loop closure candidate frames and the topological information of data association.

[0065] S2. Obtain the current keyframe in the queue of frames waiting to be inserted, and calculate the global similarity and temporal correlation between the current keyframe and other keyframes in the queue of frames waiting to be inserted based on the global feature information of the current keyframe, so as to construct a closed-loop database.

[0066] In one embodiment, the step of calculating the global similarity and temporal correlation between the current keyframe and other keyframes in the queue of frames waiting to be inserted, based on the global feature information of the current keyframe, to construct a closed-loop database, includes:

[0067] Based on the global feature information of the current keyframe and the global feature information of the first keyframe in the queue of waiting frames, calculate the global similarity and temporal correlation between the current keyframe and the first keyframe.

[0068] In the determination phase, if the global similarity and the temporal correlation satisfy the first requirement determination or the second requirement determination, the first keyframe is inserted into the closed-loop database and removed from the queue of frames waiting to be inserted, so as to update the first keyframe in the queue of frames waiting to be inserted; the first requirement determination is that the global similarity meets the similarity requirement, and the second requirement determination is that the global similarity does not meet the similarity requirement but the temporal correlation meets the correlation requirement.

[0069] Based on the global feature information of the current key frame, the determination process is repeated until the number of frames in the waiting-to-insert-frame queue reaches a preset number, and then the closed-loop database is output.

[0070] Specifically, after determining the current keyframe in the queue of frames waiting to be inserted, the global similarity and temporal correlation between the two are calculated based on the current keyframe's global feature information and the global feature information of the first keyframe in the queue. The global similarity, also known as image similarity, characterizes the global similarity between the current keyframe and the first keyframe in the queue, and is expressed by the following formula:

[0071]

[0072] In the formula, gf i This provides global feature information for the current keyframe; gf q Global feature information for the first keyframe of the queue; α is the L2 norm; α is the global feature similarity threshold, obtained through similarity comparison experiments on underwater images.

[0073] Temporal correlation is used to characterize the temporal similarity between the current keyframe and the first keyframe in the queue of frames waiting to be inserted, and is expressed by the following formula:

[0074] |t i -t q |·f≥β

[0075] In the formula, t i The timestamp corresponding to the global feature information of the current keyframe; t q is the timestamp corresponding to the global feature information of the first keyframe; f is the camera frequency; β is the temporal correlation threshold.

[0076] This invention utilizes image similarity and temporal correlation to determine whether the first keyframe in the waiting-to-be-inserted frame queue can be inserted into the closed-loop database. Specifically, in the determination phase, if the global similarity is not less than the global feature similarity threshold (i.e., the global similarity meets the similarity requirement), the global similarity calculation formula holds true; or if the global similarity is less than the global feature similarity threshold, and the temporal correlation is not less than the temporal correlation threshold (i.e., the temporal correlation meets the correlation requirement), the temporal correlation calculation formula holds true. In this case, the first keyframe in the insertion frame queue can be inserted into the closed-loop database as a closed-loop frame and can be removed from the waiting-to-be-inserted frame queue to obtain a new first keyframe. Then, the determination phase is repeated based on the global feature information of the new first keyframe and the global feature information of the current keyframe until the number of frames in the waiting-to-be-inserted frame queue reaches a preset number, preferably zero, to obtain the closed-loop database.

[0077] This invention considers the impact of the closed-loop database construction method of the SLAM system on the closed-loop detection effect. It proposes to utilize the image similarity and temporal correlation between the current keyframe and the first keyframe in the queue of waiting frames to be inserted to reasonably determine whether the first keyframe can be inserted as a closed-loop frame into the closed-loop database. This effectively avoids unnecessary duplicate data addition, reduces database redundancy, and improves system resource utilization. The dual judgment conditions (first requirement judgment and second requirement judgment) adopted ensure that only those keyframes that are both similar in features and temporally reasonable (i.e., not too far apart and not too close, to avoid interference from outdated information) can be selected to be added to the closed-loop database, thereby improving the accuracy and efficiency of closed-loop detection.

[0078] In one embodiment, after calculating the global similarity and temporal correlation between the current keyframe and the head keyframe in the queue of waiting-to-be-inserted frames based on their global feature information, the process includes:

[0079] When the global similarity and the temporal correlation meet the third requirement, the current keyframe is updated, and the global similarity and temporal correlation between the two are recalculated based on the global feature information of the updated current keyframe and the global feature information of the first keyframe in the queue of frames waiting to be inserted; the third requirement is that the global similarity does not meet the similarity requirement and the temporal correlation does not meet the correlation requirement.

[0080] The requirements are determined based on the recalculated global similarity and temporal correlation. The key frame at the head of the queue or the updated current key frame is updated based on the determination result. The determination process is repeated based on the update result until all current key frames in the queue of frames waiting to be inserted are traversed, and the closed-loop database is output.

[0081] Specifically, when the global similarity between the current keyframe and the first keyframe is less than the global feature similarity threshold, and the temporal correlation between them is less than the temporal correlation threshold (i.e., the global similarity does not meet the similarity requirement and the temporal correlation does not meet the correlation requirement), the current keyframe is updated. The keyframe of the next moment can be used as the current keyframe, or other update methods can be used; the specific method used is not limited here. Next, the global similarity and temporal correlation between the new current keyframe and the first keyframe are calculated, and the calculated results are used to determine the requirements. Based on the determination results, the first keyframe is updated and the closed-loop database is enriched, or the current keyframe is updated. The determination process is repeated using the updated data until all keyframes in the waiting-to-insert-frame queue have been traversed, i.e., only keyframes that do not meet the similarity and correlation requirements remain in the waiting-to-insert-frame queue. Finally, the closed-loop database is output for subsequent searches.

[0082] The closed-loop database construction process is as follows Figure 3As shown, this invention calculates the global similarity and temporal correlation between the current keyframe and the first keyframe in the queue of frames waiting to be inserted, and uses these calculation results to determine when and how to update keyframes. This helps ensure that only keyframes with potential loop closure value are selected, thereby improving the accuracy and efficiency of loop closure detection. When the global similarity and temporal correlation do not meet the third requirement, the dynamic update and re-evaluation mechanism allows the system to more flexibly cope with complex and changing scenarios, improving the system's adaptability and robustness. By constructing the loop closure database through iterative updates and recalculations, unnecessary redundant calculations and data processing are avoided, thereby saving system resources and improving overall performance.

[0083] S3. Based on the global feature information and inter-frame adjacency of the current key frame, search for closed-loop candidate frames from the closed-loop database;

[0084] In one embodiment, step S3 includes:

[0085] The similarity between the global feature information of each closed-loop frame in the closed-loop database and the global feature information of the current key frame is calculated respectively, and the frames are sorted according to the similarity to construct a closed-loop frame queue.

[0086] A closed-loop frame sequence is constructed based on the current closed-loop frame, the first closed-loop frame, and the second closed-loop frame in the closed-loop frame queue, and the closed-loop frame sequence is taken as the nearest neighbor of the current closed-loop frame; the first closed-loop frame and the second closed-loop frame are the previous closed-loop frame and the next closed-loop frame of the current closed-loop frame, respectively.

[0087] Each closed-loop frame in the closed-loop frame queue, excluding the first closed-loop frame, is taken as a closed-loop frame to be verified and verified sequentially. Each closed-loop frame to be verified that satisfies either the first verification condition or the second verification condition is taken as a preferred closed-loop frame, and together with the first closed-loop frame in the closed-loop frame sequence, is taken as a candidate closed-loop frame for the current keyframe. The first verification condition is that when the first closed-loop frame of the closed-loop frame to be verified is the nearest neighbor of the first closed-loop frame, the closed-loop frame to be verified is not the nearest neighbor of the first closed-loop frame. The second verification condition is that when the first closed-loop frame of the closed-loop frame to be verified is not the nearest neighbor of the first closed-loop frame, the closed-loop frame to be verified is neither the nearest neighbor of the first closed-loop frame nor the nearest neighbor of the first closed-loop frame of the closed-loop frame to be verified.

[0088] Specifically, using the current keyframe Kf iBased on global feature information, similarity algorithms such as color histogram, texture feature, and GIST feature are used to search for the top k similar closed-loop candidate frames in the closed-loop database, and these frames are sorted by similarity to form a closed-loop frame queue. A closed-loop frame is randomly selected from the queue and combined with its preceding and following closed-loop frames to obtain the closed-loop frame sequence, which is the nearest neighbor of that closed-loop frame. Then, all closed-loop frames in the queue except the first closed-loop frame are used as closed-loop frames to be verified and verified sequentially. This represents the first closed-loop frame to be verified in the closed-loop frame sequence, i.e., the closed-loop frame most similar to the current keyframe. If satisfied If the first verification condition is met, then it can be used as the preferred closed-loop frame. For the closed-loop frame to be verified The nearest neighbor; if not satisfied, then the closed-loop frame needs to be verified. This is an invalid closed-loop frame; while for the closed-loop frame to be verified If it satisfies the following formula:

[0089]

[0090] Then the closed-loop frame to be verified To optimize the closed-loop frame, the aforementioned method is iteratively applied to verify all closed-loop frames in the queue sequentially. The verified optimized closed-loop frames and the first closed-loop frame in the queue are then used as candidate closed-loop frames for the current keyframe. It should be noted that during verification, since the first closed-loop frame has the highest similarity to the current keyframe, it does not need to participate in the verification process and can be used as a candidate closed-loop frame for the current keyframe.

[0091] This invention efficiently constructs a closed-loop frame queue based on the similarity between the global feature information of the closed-loop frame and the current keyframe, effectively reducing interference from irrelevant frames and improving the efficiency and accuracy of closed-loop detection. By constructing the nearest neighbors of the current closed-loop frame, it helps maintain the temporal or spatial coherence between closed-loop frames, which helps to more accurately identify and verify closed-loop events in subsequent steps. By verifying the closed-loop frames to be verified and selecting the preferred closed-loop frames based on the verification results, the impact of mismatches and noise can be further reduced, ensuring that the correlation between the finally selected closed-loop candidate frames and the current keyframe is more reliable. At the same time, the range of candidate frames is gradually narrowed, thereby reducing unnecessary computation and avoiding multiple closed-loop candidate keyframes being adjacent frames. By filtering closed-loop candidate frames through nearest neighbors, the selection of closed-loop candidate frames for adjacent frames is suppressed, thereby improving the closed-loop recall rate.

[0092] S4. Verify the closed-loop candidate frame based on the local feature information of the current key frame and the local feature information of the closed-loop candidate frame, and when the verification is successful, use the closed-loop candidate frame as the target closed-loop frame of the current key frame to achieve underwater visual closed-loop detection.

[0093] In one embodiment, step S4 includes:

[0094] Based on the local feature information of the current keyframe and the local feature information of the closed-loop candidate frames, the closed-loop candidate frames are geometrically verified using a random sampling consensus algorithm to obtain the number of interior points of each closed-loop candidate frame.

[0095] The candidate frame with the most inliers is selected as the initial candidate frame for time-series verification, and the initial candidate frame that passes the time-series verification is selected as the target candidate frame for the current keyframe to achieve underwater visual loop closure detection.

[0096] Specifically, this invention first uses a feature matching algorithm (such as brute-force matching, FLANN, etc.) to match the local feature information of the current keyframe with the local feature information of each closed-loop candidate frame; then, it initializes the RANSAC algorithm to determine the parameters required by the RANSAC algorithm, including the maximum number of iterations, the inlier decision threshold, and the minimum number of samples required for model estimation (such as the four point pairs required for point cloud alignment), and randomly selects a set of samples from the matched features of the current keyframe and the closed-loop candidate frames to estimate the geometric transformation model (such as rigid body transformation, affine transformation, projection transformation, etc.); the remaining feature matching pairs are substituted into the estimated model, and the inlier decision threshold (such as distance error, angle error, etc.) is used to determine the inlier decision. Determine which matching pairs are inliers and count the number of inliers that conform to the model; repeat the above process of random sampling, model estimation and inlier determination until the maximum number of iterations is reached or other stopping conditions are met (such as the number of inliers has reached a preset number). After each iteration, select the model with the most inliers as the current optimal model. For each closed-loop candidate frame, record its number of inliers under the optimal model. Finally, output the number of inliers of each closed-loop candidate frame under the optimal model as an important indicator to evaluate the consistency of the geometric relationship between the closed-loop candidate frame and the current keyframe. The closed-loop candidate frame with the most inliers is selected as the initial closed-loop candidate frame for time-series verification. The verified initial closed-loop candidate frame can be used as the target closed-loop frame of the current keyframe.

[0097] This invention can more accurately capture the geometric relationship between the current keyframe and the loop closure candidate frame by matching the local feature information of the current keyframe and the loop closure candidate frame, thereby reducing the mismatch that may be caused by global feature information. By performing geometric verification through the RANSAC algorithm, loop closure candidate frames that are consistent with the geometric relationship of the current keyframe can be effectively screened, thereby improving the accuracy of loop closure detection. In complex or dynamic environments, loop closure candidate frames can be accurately identified, thus enhancing the robustness of the system.

[0098] In one embodiment, selecting the closed-loop candidate frame with the highest number of interior points as the initial closed-loop candidate frame for timing verification includes:

[0099] The closed-loop candidate frame with the most internal points is selected as the first preliminary closed-loop candidate frame, and the nearest neighbor of the first preliminary closed-loop candidate frame is constructed.

[0100] The current keyframe is updated, and the search and geometric verification steps for the closed-loop candidate frames of the updated current keyframe are repeated to obtain the initial closed-loop candidate frames of the updated current keyframe as the second initial closed-loop candidate frames.

[0101] When the second preliminary closed-loop candidate frame is the nearest neighbor of the first preliminary closed-loop candidate frame, the timing verification of the first preliminary closed-loop candidate frame passes; otherwise, the timing verification of the first preliminary closed-loop candidate frame fails, and the timing verification step is performed on the second preliminary closed-loop candidate frame until all key frames in the waiting-to-insert-frame queue are iteratively verified.

[0102] Specifically, this invention uses the closed-loop candidate frame with the most interior points as the first preliminary closed-loop candidate frame, and determines the nearest neighbor of the first preliminary closed-loop candidate frame by referring to the method for constructing the closed-loop frame sequence, for timing verification, also known as adjacency verification: the current keyframe is updated and the updated preliminary closed-loop candidate frame of the current keyframe is obtained as the second preliminary closed-loop candidate frame. If the second preliminary closed-loop candidate frame is the nearest neighbor of the first preliminary closed-loop candidate frame, it means that the first preliminary closed-loop candidate frame of the current keyframe has passed the timing verification, and it is used as the target closed-loop frame of the current keyframe; if it does not meet the requirement, it means that its timing verification has failed, and the second preliminary closed-loop candidate frame is subjected to timing verification until all keyframes in the waiting-to-be-inserted frame queue are obtained after iterative verification.

[0103] This invention uses the candidate frame with the most interior points as the initial candidate frame for loop closure, ensuring the geometric relationship between the selected candidate frame and the current keyframe is most consistent, thereby improving the accuracy of loop closure detection. Constructing the nearest neighbors of the initial candidate frames helps to quickly filter out the candidate frames most similar to the current keyframe during subsequent verification, further improving the efficiency of loop closure detection. The addition of a temporal verification step ensures that the selected loop frames are temporally consistent with the current keyframe, avoiding loop closure detection errors caused by mismatches or abnormal data. The use of nearest neighbor construction and iterative updates avoids the huge computational load of global search, reducing computational resource consumption. Dual temporal verification of the candidate frames and search frames using the nearest neighbor sequence guarantees the accuracy of loop closure detection.

[0104] This application proposes an underwater visual loop closure detection method to improve the positioning accuracy and recall rate of underwater visual loop closure detection. The method involves sequentially acquiring multiple keyframes and their corresponding keyframe images, extracting global and local feature information from the keyframe images using a convolutional neural network model, and sequentially placing the keyframes into a queue of waiting-to-insert frames. The method then acquires the current keyframe in the queue and, based on its global feature information, calculates the global similarity and temporal correlation between the current keyframe and other keyframes in the queue. A closed-loop database is constructed. Based on the global feature information and inter-frame adjacency of the current keyframe, closed-loop candidate frames are searched from the closed-loop database. The closed-loop candidate frames are verified based on the local feature information of the current keyframe and the local feature information of the closed-loop candidate frames. When the verification is successful, the closed-loop candidate frame is used as the target closed-loop frame of the current keyframe to realize the technical solution of underwater visual closed-loop detection. By constraining the closed-loop detection through image similarity, temporal correlation and inter-frame adjacency, the effectiveness of global and local image features is enhanced, and the accuracy and recall rate of underwater visual closed-loop detection are improved.

[0105] It should be noted that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders.

[0106] In another embodiment, such as Figure 4 As shown, a second aspect of the present invention provides an underwater visual closed-loop detection device, comprising:

[0107] The feature calculation module 10 is used to sequentially acquire multiple key frames and their corresponding key frame images, extract global and local feature information of the key frame images through a convolutional neural network model, and sequentially put the key frames into the queue of frames waiting to be inserted.

[0108] The database construction module 20 is used to obtain the current key frame in the queue of frames waiting to be inserted, and calculate the global similarity and temporal correlation between the current key frame and other key frames in the queue of frames waiting to be inserted based on the global feature information of the current key frame, so as to construct a closed-loop database.

[0109] The candidate frame search module 30 is used to search for closed-loop candidate frames from the closed-loop database based on the global feature information and inter-frame adjacency of the current key frame.

[0110] The candidate frame verification module 40 is used to verify the closed-loop candidate frame based on the local feature information of the current key frame and the local feature information of the closed-loop candidate frame, and when the verification is successful, the closed-loop candidate frame is used as the target closed-loop frame of the current key frame to realize underwater visual closed-loop detection.

[0111] It should be noted that the various modules in the aforementioned underwater vision-based loop closure detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module. For specific limitations regarding the underwater vision-based loop closure detection device, please refer to the limitations regarding the underwater vision-based loop closure detection method above; both have the same function and role, and will not be repeated here.

[0112] A third aspect of the present invention provides an electronic device comprising:

[0113] Processor, memory, and bus;

[0114] The bus is used to connect the processor and the memory;

[0115] The memory is used to store operation instructions;

[0116] The processor is configured to execute instructions by calling the operation instructions, causing the processor to perform operations corresponding to the underwater visual loop closure detection method shown in the first aspect of this application.

[0117] In one alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5The illustrated electronic device 5000 includes a processor 5001 and a memory 5003. The processor 5001 and the memory 5003 are connected, for example, via a bus 5002. Optionally, the electronic device 5000 may also include a transceiver 5004. It should be noted that in practical applications, the transceiver 5004 is not limited to one type, and the structure of this electronic device 5000 does not constitute a limitation on the embodiments of this application.

[0118] Processor 5001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 5001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0119] Bus 5002 may include a path for transmitting information between the aforementioned components. Bus 5002 may be a PCI bus or an EISA bus, etc. Bus 5002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0120] The memory 5003 may be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0121] The memory 5003 is used to store application code that executes the scheme of this application, and its execution is controlled by the processor 5001. The processor 5001 is used to execute the application code stored in the memory 5003 to implement the content shown in any of the foregoing method embodiments.

[0122] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers.

[0123] The fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements an underwater visual loop closure detection method as shown in the first aspect of the present application.

[0124] Another embodiment of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0125] Furthermore, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0126] In summary, this invention relates to the field of computer technology and discloses an underwater visual loop closure detection method, apparatus, device, and medium. The method includes sequentially acquiring multiple keyframes and their corresponding keyframe images; extracting global and local feature information of the keyframe images using a convolutional neural network model; and sequentially placing the keyframes into a queue of waiting-to-insert frames. The method then acquires the current keyframe in the queue and, based on the global feature information of the current keyframe, calculates the global similarity and temporal correlation between the current keyframe and other keyframes in the queue to construct a loop closure database. Based on the current keyframe's global feature information... The invention utilizes global feature information and inter-frame adjacency of keyframes to search for loop closure candidate frames in the loop closure database. It then verifies the loop closure candidate frames based on local feature information of the current keyframe and the local feature information of the loop closure candidate frames. Upon successful verification, the loop closure candidate frame is designated as the target loop closure frame for the current keyframe, thereby achieving underwater visual loop closure detection. This invention addresses the difficulties in constructing the loop closure database and selecting loop closure frames by constraining loop closure detection through image similarity, temporal correlation, and inter-frame adjacency, thereby enhancing the effectiveness of global and local image features and improving the accuracy and recall of underwater visual loop closure.

[0127] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts among the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; refer to the description of the method embodiments for relevant details. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0128] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. An underwater visual loop closure detection method, characterized in that, include: Multiple keyframes and their corresponding keyframe images are acquired sequentially. Global and local feature information of the keyframe images are extracted using a convolutional neural network model, and the keyframes are sequentially placed into a queue of frames waiting to be inserted. Obtain the current keyframe in the queue of frames waiting to be inserted, and calculate the global similarity and temporal correlation between the current keyframe and other keyframes in the queue of frames waiting to be inserted based on the global feature information of the current keyframe, so as to construct a closed-loop database. Based on the global feature information and inter-frame adjacency of the current keyframe, search for closed-loop candidate frames from the closed-loop database; The closed-loop candidate frame is verified based on the local feature information of the current key frame and the local feature information of the closed-loop candidate frame. When the verification is successful, the closed-loop candidate frame is used as the target closed-loop frame of the current key frame to achieve underwater visual closed-loop detection.

2. The underwater visual loop closure detection method according to claim 1, characterized in that, The process of sequentially acquiring multiple keyframes and their corresponding keyframe images, and extracting global and local feature information from the keyframe images using a convolutional neural network model, includes: Multiple keyframes and their corresponding keyframe images are sequentially acquired from the front end of the SLAM system. The local features of the keyframe image are extracted by the convolutional layer of the convolutional neural network model, and the high discriminative information and low-dimensional representation information of the local features are extracted by the attention layer and the autoencoder of the convolutional neural network model, respectively, to obtain the local feature information of the keyframe image. The global features of the keyframe image are extracted by the convolutional layer of the convolutional neural network model, and the contribution of the global features is weighted by the generalized average pooling layer of the convolutional neural network model to obtain the global feature information of the keyframe image.

3. The underwater visual loop closure detection method according to claim 1, characterized in that, The step of calculating the global similarity and temporal correlation between the current keyframe and other keyframes in the queue of frames waiting to be inserted, based on the global feature information of the current keyframe, to construct a closed-loop database, includes: Based on the global feature information of the current keyframe and the global feature information of the first keyframe in the queue of waiting frames, calculate the global similarity and temporal correlation between the current keyframe and the first keyframe. In the determination phase, if the global similarity and the temporal correlation satisfy the first requirement determination or the second requirement determination, the first keyframe is inserted into the closed-loop database and removed from the queue of frames waiting to be inserted, so as to update the first keyframe in the queue of frames waiting to be inserted; the first requirement determination is that the global similarity meets the similarity requirement, and the second requirement determination is that the global similarity does not meet the similarity requirement but the temporal correlation meets the correlation requirement. Based on the global feature information of the current key frame, the determination process is repeated until the number of frames in the waiting-to-insert-frame queue reaches a preset number, and then the closed-loop database is output.

4. The underwater visual loop closure detection method according to claim 3, characterized in that, The step of calculating the global similarity and temporal correlation between the current keyframe and the first keyframe in the queue of frames waiting to be inserted, based on their global feature information, includes: When the global similarity and the temporal correlation meet the third requirement, the current keyframe is updated, and the global similarity and temporal correlation between the two are recalculated based on the global feature information of the updated current keyframe and the global feature information of the first keyframe in the queue of frames waiting to be inserted; the third requirement is that the global similarity does not meet the similarity requirement and the temporal correlation does not meet the correlation requirement. The requirements are determined based on the recalculated global similarity and temporal correlation. The key frame at the head of the queue or the updated current key frame is updated based on the determination result. The determination process is repeated based on the update result until all current key frames in the queue of frames waiting to be inserted are traversed, and the closed-loop database is output.

5. The underwater visual loop closure detection method according to claim 1, characterized in that, The step of searching for loop-closing candidate frames from the loop-closing database based on the global feature information and inter-frame adjacency of the current keyframe includes: The similarity between the global feature information of each closed-loop frame in the closed-loop database and the global feature information of the current key frame is calculated respectively, and the frames are sorted according to the similarity to construct a closed-loop frame queue. A closed-loop frame sequence is constructed based on the current closed-loop frame, the first closed-loop frame, and the second closed-loop frame in the closed-loop frame queue, and the closed-loop frame sequence is taken as the nearest neighbor of the current closed-loop frame; the first closed-loop frame and the second closed-loop frame are the previous closed-loop frame and the next closed-loop frame of the current closed-loop frame, respectively. Each closed-loop frame in the closed-loop frame queue, excluding the first closed-loop frame, is taken as a closed-loop frame to be verified and verified sequentially. Each closed-loop frame to be verified that satisfies either the first verification condition or the second verification condition is taken as a preferred closed-loop frame, and together with the first closed-loop frame in the closed-loop frame sequence, is taken as a candidate closed-loop frame for the current keyframe. The first verification condition is that when the first closed-loop frame of the closed-loop frame to be verified is the nearest neighbor of the first closed-loop frame, the closed-loop frame to be verified is not the nearest neighbor of the first closed-loop frame. The second verification condition is that when the first closed-loop frame of the closed-loop frame to be verified is not the nearest neighbor of the first closed-loop frame, the closed-loop frame to be verified is neither the nearest neighbor of the first closed-loop frame nor the nearest neighbor of the first closed-loop frame of the closed-loop frame to be verified.

6. The underwater visual loop closure detection method according to claim 5, characterized in that, The step of verifying the loop closure candidate frame based on the local feature information of the current keyframe and the local feature information of the loop closure candidate frame, and when the verification is successful, using the loop closure candidate frame as the target loop closure frame of the current keyframe to achieve underwater visual loop closure detection, includes: Based on the local feature information of the current keyframe and the local feature information of the closed-loop candidate frames, the closed-loop candidate frames are geometrically verified using a random sampling consensus algorithm to obtain the number of interior points of each closed-loop candidate frame. The candidate frame with the most inliers is selected as the initial candidate frame for time-series verification, and the initial candidate frame that passes the time-series verification is selected as the target candidate frame for the current keyframe to achieve underwater visual loop closure detection.

7. The underwater visual loop closure detection method according to claim 6, characterized in that, The step of selecting the closed-loop candidate frame with the most internal points as the initial closed-loop candidate frame for timing verification includes: The closed-loop candidate frame with the most internal points is selected as the first preliminary closed-loop candidate frame, and the nearest neighbor of the first preliminary closed-loop candidate frame is constructed. The current keyframe is updated, and the search and geometric verification steps for the closed-loop candidate frames of the updated current keyframe are repeated to obtain the initial closed-loop candidate frames of the updated current keyframe as the second initial closed-loop candidate frames. When the second preliminary closed-loop candidate frame is the nearest neighbor of the first preliminary closed-loop candidate frame, the timing verification of the first preliminary closed-loop candidate frame passes; otherwise, the timing verification of the first preliminary closed-loop candidate frame fails, and the timing verification step is performed on the second preliminary closed-loop candidate frame until all key frames in the waiting-to-insert-frame queue are iteratively verified.

8. An underwater visual closed-loop detection device, characterized in that, include: The feature calculation module is used to sequentially acquire multiple keyframes and their corresponding keyframe images, extract global and local feature information of the keyframe images through a convolutional neural network model, and sequentially place the keyframes into a queue of frames waiting to be inserted. The database construction module is used to obtain the current key frame in the queue of frames waiting to be inserted, and calculate the global similarity and temporal correlation between the current key frame and other key frames in the queue of frames waiting to be inserted based on the global feature information of the current key frame, so as to construct a closed-loop database. The candidate frame search module is used to search for closed-loop candidate frames from the closed-loop database based on the global feature information and inter-frame adjacency of the current key frame. The candidate frame verification module is used to verify the closed-loop candidate frame based on the local feature information of the current key frame and the local feature information of the closed-loop candidate frame, and when the verification is successful, the closed-loop candidate frame is used as the target closed-loop frame of the current key frame to realize underwater visual closed-loop detection.

9. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the underwater visual loop closure detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the underwater visual loop closure detection method as described in any one of claims 1 to 7.

Images