Loop detection method, electronic device, and readable storage medium

Abstract

The present application discloses a loop detection method, electronic device and readable storage medium. The method determines the previous key image frame corresponding to the current image frame and the reference image frame corresponding to the previous key image frame; compares the first feature change of the reference image frame relative to the previous key image frame with an adaptive threshold, wherein the adaptive threshold is determined based on the feature matching amount and feature change amount between the current image frame, the previous key image frame and the reference image frame; if the first feature change amount is greater than the adaptive threshold, the current image frame is used as the target key image frame. The above method determines high-quality key image frames, simplifies the number of key image frames, speeds up the efficiency of loop detection, improves the real-time performance of loop detection, and ensures the robustness of the visual SLAM system in high-dynamic scenes.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a loop detection method, an electronic device, and a readable storage medium. Background Art

[0002] In recent years, Simultaneous Localization and Mapping (SLAM) technology has been widely studied because it allows for the instant creation of maps and sensor pose estimation in unknown environments. Vision-based SLAM technology has played an important role in this field. Traditional visual SLAM is based on the assumption of a static environment, which limits its application scenarios. For example, it is only applicable to low-dynamic scenes with few dynamic elements. In high-dynamic scenes, when dynamic objects occupy a large area of ​​the image, the feature points extracted by visual SLAM may be distributed on the dynamic objects in large quantities, resulting in a significant decrease in its accuracy. The estimated trajectory will no longer be usable. Therefore, visual SLAM systems generally suffer from poor robustness in high-dynamic environments.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a loop detection method, an electronic device and a readable storage medium, aiming to solve the problem that current visual SLAM systems generally have poor robustness in high dynamic environments.

[0005] To achieve the above object, a loop detection method is provided, the loop detection method comprising the following steps:

[0006] Determining a previous key image frame corresponding to the current image frame and a reference image frame corresponding to the previous key image frame;

[0007] comparing a first feature change of the reference image frame relative to the previous key image frame with an adaptive threshold, wherein the adaptive threshold is determined based on a feature matching amount and a feature change amount between the current image frame, the previous key image frame, and the reference image frame, the feature matching amount being the number of identical feature points between the two image frames, and the feature change amount being the number of different feature points between the two image frames;

[0008] If the first feature variation is greater than the adaptive threshold, the current image frame is used as a target key image frame, and loop detection is performed based on the target key image frame.

[0009] Optionally, before the step of comparing the first feature change of the reference image frame relative to the previous key image frame with an adaptive threshold, the method includes:

[0010] Determining a first feature matching amount between the current image frame and the previous key image frame;

[0011] determining a second feature matching amount between the reference image frame and the previous key image frame;

[0012] Detecting the number of all feature points in the current image frame to obtain a first full feature quantity;

[0013] Detecting the number of all feature points in the reference image frame to obtain a second full feature quantity;

[0014] A basic adaptive threshold is constructed based on the first feature matching amount, the second feature matching amount, the first full feature amount, the second full feature amount, and the first feature change amount, and an adaptive threshold is determined based on the basic adaptive threshold.

[0015] Optionally, the step of determining an adaptive threshold based on the basic adaptive threshold includes:

[0016] determining a first difference between the second feature matching amount and the first feature matching amount, and constructing a first correction coefficient according to a first ratio between the first difference and the second feature matching amount;

[0017] constructing a second correction coefficient according to the number of corresponding image frames between the current image frame and the previous key image frame;

[0018] Determining a second ratio between the first feature change and the first feature matching amount, determining a third ratio between the second feature change of the current image frame relative to the previous key image frame and the second feature matching amount, and constructing a third correction coefficient based on a second difference between the second ratio and the third ratio;

[0019] constructing a fourth correction coefficient according to a third ratio between the first feature matching amount and the first feature change amount;

[0020] determining a third difference between the second feature matching amount and the first feature matching amount, determining a total of the second feature matching amount and the first feature matching amount, and constructing a fifth correction coefficient based on a fourth ratio between the third difference and the total;

[0021] Modifying the basic adaptive threshold based on the first correction coefficient, the second correction coefficient, the third correction coefficient, the fourth correction coefficient, and the fifth correction coefficient to obtain a modified adaptive threshold;

[0022] The modified adaptive threshold is used as the adaptive threshold.

[0023] Optionally, the step of correcting the basic adaptive threshold based on the first correction coefficient, the second correction coefficient, the third correction coefficient, the fourth correction coefficient, and the fifth correction coefficient to obtain a corrected adaptive threshold includes:

[0024] When the first correction coefficient is less than or equal to a preset first coefficient threshold, constructing a modified adaptive threshold according to the basic adaptive threshold and a preset constant parameter;

[0025] When the first correction coefficient is greater than the preset first coefficient threshold, a total correction coefficient is generated based on the first correction coefficient, the second correction coefficient, the third correction coefficient, the fourth correction coefficient and the fifth correction coefficient, and the basic adaptive threshold is corrected according to the total correction coefficient to obtain the corrected adaptive threshold.

[0026] Optionally, before the step of using the current image frame as a target key image frame, the method includes:

[0027] determining a uniformity of feature distribution in the current image frame based on position information between a preset first pixel point and a preset second pixel point in the current image frame, wherein a pixel value of the preset first pixel point and a pixel value of the preset second pixel point are adjacent in a pixel value sequence, the pixel value sequence being obtained by arranging the pixels in the current image frame according to pixel value magnitude, and the position information including a positional correspondence between a pixel position corresponding to the preset first pixel point and a pixel position corresponding to the preset second pixel point;

[0028] When the feature distribution uniformity is greater than an adaptive distribution threshold, the step of using the current image frame as a target key image frame is performed, wherein the adaptive distribution threshold is determined based on each pixel value in the current image frame and the position distribution of each pixel value.

[0029] Optionally, the step of performing loop detection based on the target key image frame includes:

[0030] forming a full key image frame sequence based on the target key image frame and historical key image frames, wherein the historical key image frames are the key image frames determined before the target key image frame is determined;

[0031] determining different local key image frame sequences based on the degree of change in viewing angle between adjacent key image frames in the full key image frame sequence;

[0032] Calculating the similarity between the local key image frame sequences;

[0033] Two local key image frame sequences whose similarity is greater than a preset similarity threshold are determined as target key image frame sequences that form a loop.

[0034] Optionally, the viewing angle variation includes a first viewing angle variation and a second viewing angle variation;

[0035] The step of determining different local key image frame sequences based on the degree of change in viewing angle between adjacent key image frames in the full key image frame sequence comprises:

[0036] Determining a first viewing angle change between a reference key image frame and a previous key image frame located before the reference key image frame, wherein the reference key image frame is any key image frame in the full key image frame sequence except a first key image frame and an end key image frame;

[0037] determining a second viewing angle change between the reference key image frame and a subsequent key image frame located after the reference key image frame;

[0038] If the first viewing angle change degree is less than a preset viewing angle change threshold and the second viewing angle change degree is greater than the preset viewing angle change threshold, using the reference key image frame as a division frame;

[0039] The full key image frame sequence is divided according to the division frames to obtain the local key image frame sequences.

[0040] Optionally, before the step of determining a previous key image frame corresponding to the current image frame and a reference image frame corresponding to the previous key image frame, the method includes:

[0041] Detect each target object in the currently acquired image based on a preset target detection model;

[0042] Determining a dynamic object among the target objects based on prior information, wherein the prior information includes a target object marked as dynamic, and the target object marked as dynamic is the dynamic object;

[0043] The dynamic features on the dynamic object in the image are deleted to obtain the current image frame.

[0044] In addition, to achieve the above-mentioned purpose, the present application also provides an electronic device, which includes: a memory, a processor, and a loop detection program stored on the memory and runnable on the processor, and when the loop detection program is executed by the processor, the steps of the loop detection method as described above are implemented.

[0045] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, on which a loop detection program is stored. When the loop detection program is executed by a processor, the steps of the loop detection method as described above are implemented.

[0046] The embodiment of the present application proposes a loop detection method, electronic device and readable storage medium. In this embodiment, the previous key image frame and the reference image frame of the previous key image frame are determined based on the current image frame, and then the first feature change of the reference image frame relative to the previous key image frame is compared with the adaptive threshold. The adaptive threshold is determined by combining the feature matching amount and feature change amount between the current image frame, the previous key image frame and the reference image frame, wherein the feature matching amount is the number of identical feature points between the two image frames, and the feature change amount is the number of different feature points between the two image frames, so that the adaptive threshold can be used to measure the uniformity of the distribution of feature points in the current image frame. When the current image frame has good uniformity, it means that the current image frame has fewer or no dynamic features, so as to reduce the influence of dynamic features on the loop detection results and improve the quality of the key frame. Then, loop detection is performed based on the key image frame determined in the above manner to ensure the robustness of the visual SLAM system in high dynamic scenes. In addition, it should be noted that by screening key image frames through the above-mentioned adaptive threshold, in addition to ensuring the quality of key image frames, the number of key image frames can be reduced, the efficiency of loop detection can be accelerated, and the real-time performance of loop detection can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application;

[0048] Figure 2 This is a flowchart of the first embodiment of the loop detection method of the present application;

[0049] Figure 3 This is a flow chart of the second embodiment of the loop detection method of the present application;

[0050] Figure 4 This is a flowchart of the third embodiment of the loop detection method of the present application;

[0051] Figure 5 This is a flowchart of the fourth embodiment of the loop detection method of the present application;

[0052] Figure 6 Schematic diagram of feature clustering in the loop detection method of this application;

[0053] Figure 7 Schematic diagram of the dynamic feature deletion effect in the loop detection method of this application;

[0054] Figure 8This is a schematic diagram of the structure of the target detection model in the loop detection method of this application;

[0055] Figure 9 This is a comparison of the absolute trajectory error between the loop closure detection method in this application and ORB-SLAM3;

[0056] Figure 10 This is the ablation experiment result diagram of the loop detection method in this application;

[0057] Figure 11 This is a comparison diagram of the loop detection method in this application and other SLAM systems in dynamic environments;

[0058] Figure 12 This is a comparison of the time required to process each frame of the loop detection method in this application and the DynaSLAM algorithm on the tracking thread;

[0059] Figure 13 This is a comparison result of the test time of ORB-SLAM3 in loop closure detection after combining the local key image frame sequence in ORB-SLAM3 in this application;

[0060] Figure 14 This is a schematic diagram of the trajectory and error distribution of the ORB-SLAM3 estimated loop detection method in this application;

[0061] Figure 15 This is the parameter diagram of Astra Pro in the loop detection method of this application;

[0062] Figure 16 Schematic diagram of the experimental scenario in the loop detection method of this application;

[0063] Figure 17 Schematic diagram comparing the loop detection method of this application with the ORB-SLAM3 system and the actual trajectory under indoor dynamic conditions;

[0064] Figure 18 This is a flowchart of the fifth embodiment of the loop detection method of the present application;

[0065] Figure 19 The loop detection method of this application is applied to the error distribution diagram of RPE value in different scenarios.

[0066] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] like Figure 1As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0069] The device in the embodiment of the present application can be a robot, or it can be an electronic terminal device such as a server, a smart phone, a PC, a tablet computer, a portable computer, etc.

[0070] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0071] Optionally, the electronic device may also include a camera, RF (Radio Frequency) circuit, sensor, audio circuit, WiFi module, etc. The terminal may also be configured with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here. Those skilled in the art will understand that Figure 1 The electronic device structure shown in the figure does not constitute a limitation to the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0072] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a loop detection program.

[0073] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the loop detection program stored in the memory 1005. The processor performs the following operations:

[0074] Determining a previous key image frame corresponding to the current image frame and a reference image frame corresponding to the previous key image frame;

[0075] comparing a first feature change of the reference image frame relative to the previous key image frame with an adaptive threshold, wherein the adaptive threshold is determined based on a feature matching amount and a feature change amount between the current image frame, the previous key image frame, and the reference image frame, the feature matching amount being the number of identical feature points between the two image frames, and the feature change amount being the number of different feature points between the two image frames;

[0076] If the first feature variation is greater than the adaptive threshold, the current image frame is used as a target key image frame, and loop detection is performed based on the target key image frame.

[0077] It should be noted that the SLAM problem typically involves a camera-equipped robot moving from an unknown location in an unknown environment. During this movement, it localizes itself based on its location and a map, while simultaneously building an incremental map based on its localization, enabling autonomous positioning and navigation. In visual SLAM, pose estimation is often a recursive process, where the pose of the current image frame is calculated from the pose of the previous frame. Therefore, errors are propagated frame by frame, resulting in cumulative errors. The pose can be used to determine the robot's trajectory, and these accumulated errors can affect the accuracy of the trajectory estimate and, ultimately, the accuracy of the map or model. To mitigate these accumulated errors, loop closure detection is typically performed. This involves detecting the locations where the camera used to acquire the image passes through, and then re-localizing the image based on these locations to eliminate the accumulated errors. The key to loop closure detection is determining image similarity to identify the locations where the cameras passed through the same location. It should be noted that the determination of image similarity is essentially based on the comparison of features in the image. However, if there are dynamic features in the actual scene, such as people, animals or vehicles, it will affect the determination of similarity between images, thereby reducing the accuracy of determining the results of cameras passing through the same position, resulting in the general problem of poor robustness of visual SLAM systems in highly dynamic environments.

[0078] Therefore, in order to address the above problems, this application proposes a loop detection method to improve the robustness of visual SLAM systems in high dynamic environments.

[0079] Reference Figure 2 , the first embodiment of the loop detection method of the present application, the loop detection method includes:

[0080] Step S10, determining a previous key image frame corresponding to the current image frame and a reference image frame corresponding to the previous key image frame;

[0081] It should be noted that determining image similarity actually involves comparing image frames. However, if the current image frame is compared with every previous frame, a significant amount of computational processing time would be required. Therefore, to reduce computational processing time, key image frames are typically selected and only these key frames are compared for similarity determination, thus reducing the number of computational processing steps. However, in dynamic scenes, if the selected key image frames contain a large number of dynamic features, this can affect the loop detection results. To address this issue, in this embodiment, frames with fewer dynamic features are selected as key image frames.

[0082] Exemplarily, a previous key image frame of the current image frame is determined, where the previous key image frame of the current image frame is a key image frame determined before the current image frame is acquired. The reference image frame is the frame with the highest similarity to the previous key image frame. Typically, the reference image frame is an image frame near the previous key image frame, such as a frame adjacent to the previous key image frame.

[0083] Step S20: comparing a first feature change of the reference image frame relative to the previous key image frame with an adaptive threshold, wherein the adaptive threshold is determined based on a feature matching amount and a feature change amount between the current image frame, the previous key image frame, and the reference image frame, wherein the feature matching amount is the number of identical feature points between the two image frames, and the feature change amount is the number of different feature points between the two image frames;

[0084] Exemplarily, a first feature change in a reference image frame relative to a previous key image frame is compared to an adaptive threshold. The first feature change refers to the number of feature points in the reference image frame that differ from the previous key image frame. The feature points may be ORB feature points. The adaptive threshold is determined based on the feature matching and feature change between the current image frame, the previous key image frame, and the reference image frame. The feature matching between two frames is the number of identical feature points, while the feature change between two frames is the number of different feature points.

[0085] In a feasible implementation manner, before the step of comparing the first feature change of the reference image frame relative to the previous key image frame with an adaptive threshold, the method includes:

[0086] Step S211, determining a first feature matching amount between the current image frame and the previous key image frame;

[0087] Step S212, determining a second feature matching amount between the reference image frame and the previous key image frame;

[0088] Step S213, detecting the number of all feature points in the current image frame to obtain a first full feature value;

[0089] Step S214, detecting the number of all feature points in the reference image frame to obtain a second full feature value;

[0090] Step S215, constructing a basic adaptive threshold based on the first feature matching amount, the second feature matching amount, the first full feature amount, the second full feature amount, and the first feature change amount, and determining an adaptive threshold based on the basic adaptive threshold;

[0091] Step S216: Using the basic adaptive threshold as the adaptive threshold.

[0092] Exemplarily, the first feature matching amount may refer to the number of feature point matches between the current image frame and the previous key image frame; the second feature matching amount may refer to the number of feature point matches between the reference image frame and the previous key image frame; the first full feature amount may refer to the number of all feature points in the current image frame; the second full feature amount may refer to the number of all feature points in the reference image frame. A basic adaptive threshold is generated based on the first feature matching amount, the second feature matching amount, the first feature change amount, the first full feature amount, and the second full feature amount. The basic adaptive threshold is defined as follows:

[0093]

[0094] Where N1 is the first feature matching amount, N2 is the second feature matching amount, N3 is the first feature change amount, N4 is the first full feature amount, N5 is the second full feature amount, T initial is the basic adaptive threshold, N1 / N2 is the ratio between the first feature matching amount and the second feature matching amount, and N4 / N5 is the ratio between the first full feature amount and the second full feature amount.

[0095] The basic adaptive threshold can be used as an adaptive threshold for selecting key image frames.

[0096] In a feasible embodiment, after the steps of constructing a basic adaptive threshold based on the first feature matching amount, the second feature matching amount, the first full feature amount, the second full feature amount, and the first feature change amount, and determining an adaptive threshold based on the basic adaptive threshold, the method includes:

[0097] Step S221, determining a first difference between the second feature matching amount and the first feature matching amount, and constructing a first correction coefficient according to a first ratio between the first difference and the second feature matching amount;

[0098] Step S222, constructing a second correction coefficient according to the number of corresponding image frames between the current image frame and the previous key image frame;

[0099] Step S221, determining a second ratio between the first feature change and the first feature matching amount, determining a third ratio between the second feature change of the current image frame relative to the previous key image frame and the second feature matching amount, and constructing a third correction coefficient based on a second difference between the second ratio and the third ratio.

[0100] Step S223: constructing a fourth correction coefficient according to a third ratio between the first feature matching amount and the first feature change amount;

[0101] Step S224, determining a third difference between the second feature matching amount and the first feature matching amount, determining the total of the second feature matching amount and the first feature matching amount, and constructing a fifth correction coefficient based on a fourth ratio between the third difference and the total;

[0102] Step S225, correcting the basic adaptive threshold based on the first correction coefficient, the second correction coefficient, the third correction coefficient, the fourth correction coefficient, and the fifth correction coefficient to obtain a corrected adaptive threshold;

[0103] Step S226: Using the modified adaptive threshold as the adaptive threshold.

[0104] In a feasible implementation manner, the step of correcting the basic adaptive threshold based on the first correction coefficient, the second correction coefficient, the third correction coefficient, the fourth correction coefficient, and the fifth correction coefficient to obtain a corrected adaptive threshold includes:

[0105] Step S001: when the first correction coefficient is less than or equal to a preset first coefficient threshold, constructing a modified adaptive threshold according to the basic adaptive threshold and a preset constant parameter;

[0106] Step S002, when the first correction coefficient is greater than the preset first coefficient threshold, a total correction coefficient is generated according to the first correction coefficient, the second correction coefficient, the third correction coefficient, the fourth correction coefficient and the fifth correction coefficient, and the basic adaptive threshold is corrected according to the total correction coefficient to obtain the corrected adaptive threshold.

[0107] It should be noted that the basic adaptive threshold is designed to accommodate the influence of all feature points and feature matching points, employing an average method. However, this method is generally more suitable when the current image frame and the reference image frame are sufficiently similar. However, in practical applications, the current image frame is farther from the last key image frame than the reference image frame, resulting in a significant difference between the current and reference image frames. To further improve the scene adaptability of the adaptive threshold, this embodiment modifies the basic adaptive threshold to obtain a modified adaptive threshold. Accordingly, the modified adaptive threshold is used as the basis for selecting key image frames.

[0108] Exemplarily, the first correction coefficient is defined as follows:

[0109]

[0110] Where N1 is the first feature matching value, N2 is the second feature matching value, θ is the first correction coefficient, and N2-N1 is the first difference between the second feature matching value and the first feature matching value. The first correction coefficient is used to reduce the number of feature point matches from the reference image frame to the current image frame.

[0111] The second correction factor is defined as follows:

[0112]

[0113] Wherein, Δid is the number of corresponding image frames between the current image frame and the previous key image frame (the number of image frames is the number of image frames obtained in the time span between the time when the current image frame is obtained and the time when the previous key image frame is obtained, which is usually reflected by the frame id (Identity document, ID card identification number). For example, the frame id can indicate the number when the frame is obtained), r is the second correction coefficient, and 6 is a constant. The second correction coefficient is used to reduce the probability of two frames being selected as key image frames after the previous key image frame, and to ensure that another key image frame is selected after six frames from the previous key image frame. At the same time, the third correction coefficient is also used as an adaptive coefficient to multiply the second correction coefficient θ to ensure the correct spatial distribution of the key image frames.

[0114] The third correction factor is defined as follows:

[0115]

[0116] Where N1 is the first feature match, N2 is the second feature match, N3 is the first feature change, N6 is the second feature change, λ is the third correction coefficient, N3 / N1 is the second ratio, and N6 / N2 is the third ratio. The third correction coefficient is used to reduce the adaptive threshold, addressing the problem of the cone area changing but the first feature change not exceeding the adaptive threshold.

[0117] The fourth correction factor is defined as follows:

[0118]

[0119] Where N1 is the first feature matching amount, N3 is the first feature change amount, ε is the fourth correction coefficient, 0.5 is a constant, and N3 / N1 is the third ratio between the first feature matching amount and the first feature change amount. The fourth correction coefficient is used to reduce the threshold to prevent the adaptive threshold from failing if the number of matching points in the cone exceeds half.

[0120] The fifth correction factor is defined as follows:

[0121]

[0122] Where N1 is the first feature matching value, N2 is the second feature matching value, δ is the fifth correction coefficient, and 2 is a constant. The fifth correction coefficient is used to reduce the threshold to prevent the adaptive threshold from failing when the number of matching points between the reference image frame and the last key image frame is half greater than the number of matching points between the current image frame and the last key image frame.

[0123] The modified adaptive threshold is defined as follows:

[0124]

[0125] Among them, T adaptive To modify the adaptive threshold, T initial is the basic adaptive threshold, θ is the first correction coefficient, r is the second correction coefficient, λ is the third correction coefficient, ε is the fourth correction coefficient, δ is the fifth correction coefficient, (1+rθ-λ-ε-δ) is the total correction coefficient, and Others is θ>01.

[0126] The modified adaptive threshold is used as the adaptive threshold for selecting key image frames.

[0127] It should be noted that when there are angular dynamic features in the frame, it will affect the uniformity of the feature points in the frame. The above-mentioned basic adaptive threshold and modified adaptive threshold are determined by combining multiple factors (such as the current image frame, the previous key image frame, and the reference frame, etc.), and can be used to measure the uniformity of the feature points of the current image frame.

[0128] Step S30: If the first feature variation is greater than the adaptive threshold, the current image frame is used as a target key image frame, and loop detection is performed based on the target key image frame.

[0129] For example, if the first feature variation is greater than the adaptive threshold, it indicates that the uniformity of the current image frame is good and the quality of the current image frame is high, and the current image frame can be used as a target key image frame.

[0130] On the contrary, if the first feature transformation amount is less than or equal to the adaptive threshold, the current image frame can be discarded, and the next frame is obtained to determine whether it can be used as the target key image frame through the above method.

[0131] For example, loop detection is then performed based on the key image frames selected in the above manner. It is understandable that, once the key image frames have been screened, those skilled in the art will be aware of the specific process of loop detection based on the key image frames. For example, based on a bag-of-words model, the similarity between the key image frames can be determined to determine whether the camera that captured the image passed through the same location, that is, to detect whether a loop occurs. After the loop is detected, the path is corrected based on the loop. The detailed process is not repeated here.

[0132] In this embodiment, the previous key image frame and the reference image frame of the previous key image frame are determined based on the current image frame, and then the first feature change amount of the reference image frame relative to the previous key image frame is compared with the adaptive threshold, and the adaptive threshold is determined by combining the feature matching amount and feature change amount between the current image frame, the previous key image frame and the reference image frame, wherein the feature matching amount is the number of identical feature points between the two image frames, and the feature change amount is the number of different feature points between the two image frames, so that the adaptive threshold can be used to measure the uniformity of the distribution of feature points in the current image frame. When the current image frame has good uniformity, it means that the current image frame has fewer or no dynamic features to reduce the impact of dynamic features on the loop detection results. Then, loop detection is performed based on the key image frame determined by the above method to ensure the robustness of the visual SLAM system in high dynamic scenes. In addition, it should be noted that by screening the key image frames through the above adaptive threshold, in addition to ensuring the quality of the key image frames, the number of key image frames can also be simplified, the efficiency of loop detection can be accelerated, and the real-time performance of loop detection can be improved.

[0133] Reference Figure 3 Based on the first embodiment of the loop detection method of the present application, a second embodiment of the loop detection method of the present application is proposed. Parts of this embodiment similar to the above embodiment can refer to the above content and will not be repeated here. Before the step of using the current image frame as the target key image frame, the method includes:

[0134] Step A10: determining a uniformity of feature distribution in the current image frame based on position information between a preset first pixel point and a preset second pixel point in the current image frame, wherein the pixel value of the preset first pixel point and the pixel value of the preset second pixel point are adjacent in a pixel value sequence, the pixel value sequence being obtained by arranging the pixels in the current image frame according to pixel value magnitude, and the position information including a positional correspondence between a pixel position corresponding to the preset first pixel point and a pixel position corresponding to the preset second pixel point;

[0135] Step A20, when the feature distribution uniformity is greater than the adaptive distribution threshold, execute the step of using the current image frame as the target key image frame, wherein the adaptive distribution threshold is determined based on each pixel value in the current image frame and the position distribution of each pixel value.

[0136] It should be noted that the adaptive threshold in the above embodiment can be used as one of the bases for selecting key image frames. To further ensure the quality of the selected key image frames, this embodiment determines the uniformity of the distribution of feature points in the image from another perspective and screens the key image frames again.

[0137] Exemplarily, in this embodiment, a preset first pixel point and a preset second pixel point in the current image frame are used as a basis for measuring the uniformity of feature distribution in the current image frame. In the pixel value sequence of each pixel point in the current image frame arranged according to the size of the pixel value, the pixel value of the preset first pixel point and the pixel value of the second pixel point are adjacent. For example, assuming that the pixel values ​​of each pixel point in the current image frame are arranged in order of size as 1, 4, 6, 7, 11, ..., the pixel values ​​that are adjacent in size can be 1 and 4, 4 and 6, 6 and 7, etc. The position information includes the positional correspondence between the pixel position corresponding to the preset first pixel point and the pixel position corresponding to the preset second pixel point, for example, the distance between the preset first pixel point and the preset second pixel point. To simplify the process of determining the uniformity of feature distribution, a 3×3 grid is generated after the current image frame enters the SLAM system, forming a corresponding 3×3 matrix. The preset first pixel point may be the pixel with the largest value in the matrix, and the preset second pixel point may be the pixel with the second largest value in the matrix. The feature distribution uniformity in the current image frame may also be determined based on the position information between the preset first pixel point and the preset second pixel point, the pixel value of the preset first pixel point, and the pixel value of the preset second pixel point. The formula for generating the feature distribution uniformity is as follows:

[0138]

[0139] Where T2 is the uniformity of feature distribution in the current image frame, M1 is the pixel value of the preset first pixel point, M2 is the pixel value of the preset second pixel point, and cmax1 is the row where the first pixel point is located in the matrix, c max2 is the row where the second pixel point is located in the matrix, r max1 is the column where the first pixel point is located in the matrix, r max2 is the column where the second pixel point is preset in the matrix. It can be understood that In fact, it can reflect the position information between the preset first pixel point and the preset second pixel point in the matrix.

[0140] The above-mentioned adaptive distribution threshold is determined based on each pixel value and the distribution of each pixel value. The feature center of gravity of the current image frame is calculated based on each pixel value and the distribution of each pixel value. The adaptive distribution threshold is determined based on the position of the feature center of gravity in the current image frame. The calculation formula is as follows:

[0141]

[0142] Where p i is the pixel value of pixel i, x i is the horizontal position of pixel i in the current image frame, and x is the horizontal position of the feature center of gravity in the current image frame.

[0143]

[0144] Where p i is the pixel value of pixel i, y i is the position of pixel i in the longitudinal direction of the current image frame, and y is the position of the feature center of gravity in the longitudinal direction of the current image frame.

[0145]

[0146] Where x is the position of the feature center of gravity in the horizontal direction of the current image frame, y is the position of the feature center of gravity in the vertical direction of the current image frame, and T1 is the adaptive distribution threshold.

[0147] If the feature distribution uniformity of the current image frame is greater than the adaptive distribution threshold, the step of selecting the current image frame as the target key image frame is performed. That is, when T2>T1, the feature point distribution of the current image frame is determined to be relatively appropriate and the frame has sufficient stability. Furthermore, if the above two conditions are met (the first feature change is greater than the adaptive threshold and the feature distribution uniformity is greater than the adaptive distribution threshold), the current image frame is selected as the key image frame.

[0148] Reference Figure 4Based on the second embodiment of the loop detection method of the present application, a third embodiment of the loop detection method of the present application is proposed. Parts of this embodiment similar to the above embodiments can refer to the above content and will not be repeated here. The step of performing loop detection based on the target key image frame includes:

[0149] Step B10, forming a full key image frame sequence based on the target key image frame and historical key image frames, wherein the historical key image frames are the key image frames determined before the target key image frame is determined;

[0150] Step B20, determining different local key image frame sequences based on the degree of change in viewing angle between adjacent key image frames in the full key image frame sequence;

[0151] Step B30, calculating the similarity between the local key image frame sequences;

[0152] Step B40 : determining two local key image frame sequences whose similarity is greater than a preset similarity threshold as target key image frame sequences forming a loop.

[0153] In a feasible implementation manner, the viewing angle variation includes a first viewing angle variation and a second viewing angle variation; and the step of determining different local key image frame sequences based on the viewing angle variation between adjacent key image frames in the full key image frame sequence includes:

[0154] Step B110, determining a first viewing angle change between a reference key image frame and a previous key image frame located before the reference key image frame, wherein the reference key image frame is any key image frame in the full key image frame sequence except the first key image frame and the last key image frame;

[0155] Step B120, determining a second viewing angle change between the reference key image frame and a subsequent key image frame located after the reference key image frame;

[0156] Step B130: If the first viewing angle change degree is less than a preset viewing angle change threshold and the second viewing angle change degree is greater than the preset viewing angle change threshold, the reference key image frame is used as a division frame.

[0157] It should be noted that traditional visual SLAM systems typically compare single key image frames to determine similarity. To reduce the number of comparisons, this embodiment proposes the concept of key image frame sequences. Key image frames generated by the local mapping thread are identified as sequences with common features, based on the degree of viewpoint change between frames.

[0158] Exemplarily, the currently determined target key image frame and the historical key image frames determined before the target key image frame is determined constitute a full key image frame sequence, and the key image frames in the full key image frame sequence can be arranged in the order of the acquisition time of each key image frame. Determine a reference key image frame, and the reference key image frame can be a key image frame in the full key image frame sequence (considering that all reference key image frames can be performed, usually the reference key image frame is any key image frame in the full key image frame sequence except the head key image frame and the end key image frame). Determine a first angle of view change between the reference key image frame and a previous key image frame located before the reference key image frame, and determine a second angle of view change between the reference key image frame and a subsequent key image frame located after the reference key image frame. For example, suppose frames 1, 2, and 3 are arranged in chronological order, with frame 2 being the base key image frame. Frame 1, which precedes frame 2, is the preceding key image frame. A first perspective change is calculated between the base key image frame and the preceding key image frame. The first perspective change refers to the difference between the viewing angle when the base key image frame was acquired and the viewing angle when the preceding key image frame was acquired. Similarly, frame 3, which follows frame 2, is the following key image frame. A second perspective change is calculated between the base key image frame and the following key image frame. If the first perspective change is less than a preset perspective change threshold and the second perspective change is greater than the preset perspective change threshold, the base key image frame is used as a splitting frame, which is used to determine each of the local key image frame sequences. Again based on the above example, assuming frame 2 is the splitting frame, the key image frames before frame 2 (including frame 2 itself) and up to the previous splitting frame are considered one local key image frame sequence, and the key image frames after frame 2 and up to the next splitting frame are considered another local key image frame sequence.

[0159] In addition, it should be noted that any degree of change in viewing angle can be determined based on the rotation matrix between two key image frames. For example, let key image frame k1 and key image frame k2, R cw is the rotation matrix from the world coordinate system to the camera coordinate system. The rotation matrix of the key image frame k1 is R cw1 , the rotation matrix of key image frame k2 is R cw2 , the rotation matrix between key image frame k1 and key image frame k2 is

[0160] The calculation formula is as follows:

[0161]

[0162] The rotation angle θ1 between the two can be calculated through the rotation matrix:

[0163]

[0164] Where tr(X) is the sum of the diagonal elements of the matrix X.

[0165] The first viewing angle change degree and the second viewing angle change degree are respectively compared with a preset viewing angle change threshold value (set to 0.1). If the first viewing angle change degree is less than the preset viewing angle change threshold value and the second viewing angle change degree is greater than the preset viewing angle change threshold value, the reference key image frame can be used as a dividing frame. Otherwise, it cannot be used as a dividing frame. In addition, in order to prevent the sequence from being too long or too short, a length threshold value (such as the longest threshold value (15) and the shortest threshold value (2) of the local key image frame sequence) and the sequence length of the local key image frame sequence are set. For example, when the sequence length is less than 2, it is prohibited to generate a new local key image frame sequence. When the sequence length is greater than 15, the current local key image frame sequence is terminated and a new local key image frame sequence is generated.

[0166] Through the above method, each local key image frame sequence can be divided and then the similarity of each local key image frame sequence can be compared. The details are as follows:

[0167] At present, the Bag of Words (BoW) model is widely used in loop detection tasks due to its simple operation and good versatility. For example, the loop detection module in the classic ORB-SLAM3 system also adopts the BoW model.

[0168] In typical SLAM systems, most use BoW for loop detection. First, a dictionary is generated offline. The dictionary is usually generated using feature clustering. Feature clustering can be done using a k-ary tree. The first step is to set a root node. At the root node, all features are clustered into k categories to obtain the second-level child nodes of the k-ary tree. This is repeated in sequence until the preset number of layers is reached. At this time, each leaf node is added to the dictionary as a word, and the description vector of the cluster center is used as the representation vector of the word. After this operation, k branches can be established with a depth of d and a total word count of k. d Whenever a new feature point of an image is obtained, a word needs to be mapped to the feature point, such as Figure 6 As shown in the figure, it is a schematic diagram of feature clustering in the loop detection method of this application. For a new feature point, by comparing k times at the first-layer nodes, the node with the closest distance is found, and then the comparison is carried out downward along the branch of the node, and finally the word represented by the leaf node is obtained.

[0169] In the field of text detection, different words have varying degrees of importance within a text, and this importance can be measured by assigning different weights to each word. Term Frequency-Inverse Document Frequency (TF-IDF) is a key method used in text detection to measure text similarity. In the field of visual SLAM, this method can also be used to calculate the similarity between two images. The idea behind IDF is that if a feature appears very infrequently in a dictionary, it indicates high discriminative power and should be assigned a higher weight.

[0170] It should be noted that the step of calculating the similarity between the local key image frame sequences includes:

[0171] For any local key image frame sequence, a word weight vector of the local key image frame sequence is generated based on TFIDF, where the words of the local key image frame sequence are obtained by mapping the feature points in the local key image frame sequence based on the bag-of-words model. The similarity between each local key image frame sequence is calculated based on the weight vector and L2 norm of each local key image frame sequence.

[0172] For example, if there are M images in the sequence S, m∈[1,M] represents the mth image. represents the number of times the i-th word appears in the sequence S, Indicates the number of times the i-th word appears in the m-th image, n m represents the total number of words in the mth image, n S Represents the total number of words in sequence S.

[0173]

[0174]

[0175] For the TF part:

[0176]

[0177] Among them, TF s i is the weight of the TF part of word i in sequence S, n S =|n S |, n d =|n d |, |X| represents the cardinality of the multiset X.

[0178] For the IDF part, two different multisets need to be defined, namely and nd They correspond to the number of occurrences of the i-th word in the dictionary and the number of occurrences of the total visual words in the dictionary, respectively, where the dictionary contains the visual words in all key image frames.

[0179]

[0180] Where, IDF i is the weight of word i for the IDF part.

[0181] The total weight of word i is expressed as the product of IDF and TF:

[0182]

[0183] Where V i (S) is the total weight of word i.

[0184] The vector description of the calculated sequence is:

[0185]

[0186] Where, is the weight vector, V i (S) is the total weight of word i in sequence s.

[0187] To limit matching and searches to images containing identical visual information, an inverted index is used. The inverted index table is word-based, storing word weights and the image in which the word appears. This is very convenient for querying the database, as it allows easy comparison of images that share a word. Sequence similarity scores are calculated using the inverted index table, reducing computational complexity.

[0188] This embodiment uses a similarity metric based on the L2 norm to match a single sequence. The vector description of other sequences for the dictionary can be obtained by calculating the local key image frame sequence Database sequence indicated by inverted index The similarity matrix between them is used to represent the similarity between sequences:

[0189]

[0190] It is understood that the similarity between local key image frames can be determined through the above method. When the similarity between two local key image frame sequences exceeds a preset similarity threshold, the two local key image frame sequences are used as target key image frame sequences for determining loop formation. The resulting target key image frame sequence can be used to eliminate the accumulated error in the loop corresponding path. In this embodiment, the similarity comparison is based on the comparison of key image frame sequences with key image frame sequences, which can significantly reduce the number of comparisons, thereby accelerating the efficiency of loop detection and improving the real-time performance of loop detection.

[0191] Reference Figure 5 Based on the third embodiment of the loop detection method of the present application, a fourth embodiment of the loop detection method of the present application is proposed. Parts of this embodiment similar to the above embodiments can refer to the above content and are not repeated here. Before the step of determining the previous key image frame corresponding to the current image frame and the reference image frame corresponding to the previous key image frame, the method includes:

[0192] Step S110, detecting each target object in the currently acquired image based on a preset target detection model;

[0193] Step S120, determining a dynamic object among the target objects based on prior information, wherein the prior information includes a target object marked as dynamic, and the target object marked as dynamic is the dynamic object;

[0194] Step S130: Deleting the dynamic features of the dynamic object in the image to obtain the current image frame.

[0195] Exemplarily, the above-mentioned preset target detection model can be YOLOv7-tiny (a target detection algorithm). Based on actual needs, technicians can train the target detection model so that the target detection model can identify the target object. For example, the COCO (Microsoft Common Objects in Context) dataset can be used to train the target detection model, wherein the target object can include static or dynamic objects such as people, vehicles, animals, chairs, tables and computers. Through the preset target detection model, each target object on the acquired image can be identified. Then, based on the prior information, the dynamic object is determined from each target object. The prior information can be a marked target object. For example, people, animals and vehicles can be marked as dynamic. Accordingly, it can be determined which target objects in the image are dynamic objects based on the prior information. The feature points on the dynamic objects in the image are then deleted as dynamic features to obtain the current image frame. The feature points can be ORB (Oriented FAST and Rotated BRIEF) feature points. As Figure 7 As shown, Figure 7 This is a schematic diagram of the effect of dynamic feature deletion in the loop detection method of this application. It can be seen that after the dynamic feature is deleted, the feature points on the person (dynamic object) are all deleted. It is understandable that in this embodiment, before performing key image frame screening, the feature points (dynamic features) of the dynamic object in the image under a high dynamic environment can be deleted, which can effectively improve the accuracy of image similarity determination in loop detection, thereby ensuring the effect of loop detection and improving the robustness of the visual SLAM system in a high dynamic environment.

[0196] In addition, it should be noted that YOLOv7 is a regression-based target detection algorithm and is currently one of the most widely used target detection algorithms. In terms of the performance of the tiny model, YOLOv7-tiny has 39% fewer parameters and 49% less computation than YOLOv4-tiny, but the AP (accuracy Precision) remains unchanged. This paper uses YOLOv7-tiny as the detection model. Figure 8 As shown in FIG, a schematic diagram of the structure of the target detection model in this application is shown, which mainly consists of four parts: input module (Input), backbone feature extraction module (Backbone), feature enhancement module (Neck), and multi-scale prediction module (prediction).

[0197] First, after undergoing a series of preprocessing operations, including data augmentation, the image is fed into the backbone network, which extracts features from the processed image. These extracted features are then fused in the Neck module to produce features of three sizes: large, medium, and small. Finally, these fused features are fed into the detection head, which then performs detection and outputs the results.

[0198] Using YOLOv7-tiny to eliminate dynamic feature points. In the dynamic object prediction frame generated based on object detection, dynamic feature points are determined based on prior knowledge. The specific elimination process is described as follows:

[0199] When the kth frame image is input, all the feature points of the image extracted by the visual odometry are represented as F k . F k It can be expressed as:

[0200] F k ={f1,f2,f3,…,f n}

[0201] After the image passes through the target detection network, all dynamic feature points can be used as D k According to the semantic information prediction box, D k It can be expressed as:

[0202] D k ={d1,d2,d3,…,d n}

[0203] If f i ∈D k (i=1,2,3,…,n), then the feature point f i is considered as a dynamic feature point and then removed from the F k The remaining feature points are quasi-static feature points, and the set of these points is represented by P K , so P K ∪D k =F k .

[0204] Reference Figure 18 Based on the fourth embodiment of the loop detection method of the present application, the fifth embodiment of the loop detection method of the present application is proposed. Parts of this embodiment similar to the above embodiments can refer to the above content and will not be repeated here. The method includes:

[0205] Step C10: detecting each target object in the currently acquired image based on a preset target detection model;

[0206] Step C11, determining a dynamic object among the target objects based on prior information;

[0207] Step C12, deleting the dynamic features of the dynamic object in the image to obtain the current image frame;

[0208] Step C13, determining a previous key image frame corresponding to the current image frame and a reference image frame corresponding to the previous key image frame;

[0209] Step C14: comparing a first feature change of the reference image frame relative to the previous key image frame with an adaptive threshold, wherein the adaptive threshold is determined based on a feature matching amount and a feature change amount between the current image frame, the previous key image frame, and the reference image frame, wherein the feature matching amount is the number of identical feature points between the two image frames, and the feature change amount is the number of different feature points between the two image frames;

[0210] Step C15: If the first feature variation is greater than the adaptive threshold, determining the feature distribution uniformity in the current image frame based on position information between a preset first pixel point and a preset second pixel point in the current image frame, wherein the pixel value of the preset first pixel point and the pixel value of the preset second pixel point are adjacent in a pixel value sequence, the pixel value sequence is obtained by arranging the pixels in the current image frame according to pixel value size, and the position information includes a positional correspondence between a pixel position corresponding to the preset first pixel point and a pixel position corresponding to the preset second pixel point;

[0211] Step C16: when the feature distribution uniformity is greater than an adaptive distribution threshold, using the current image frame as a target key image frame, wherein the adaptive distribution threshold is determined based on each pixel value in the current image frame and the distribution of each pixel value;

[0212] Step C17, forming a full key image frame sequence based on the target key image frame and historical key image frames, wherein the historical key image frames are the key image frames determined before the target key image frame is determined;

[0213] Step C18, determining different local key image frame sequences based on the degree of change in viewing angle between adjacent key image frames in the full key image frame sequence;

[0214] Step C19, calculating the similarity between the local key image frame sequences;

[0215] In step C20 , two local key image frame sequences whose similarity is greater than a preset similarity threshold are determined as target key image frame sequences forming a loop.

[0216] It is understood that this embodiment is an overall process after combining the above embodiments. Its implementation process and beneficial effects can be referred to the above embodiments.

[0217] In addition, to clearly illustrate the advantages of this application, the degree of optimization of the system by the present invention is verified by evaluating the trajectory accuracy. The pose estimation error analysis experiment uses the evo tool and the evaluate_ate and evaluate rpe tools of TUM (TUM is a dataset for indoor dynamic environments) to test and compare the camera pose CameraTrajectory.txt estimated by the ORB-SLAM3 system with the real pose groundtruth.txt given in the dataset. The evaluation parameters mainly use relative pose error (RPE) and absolute trajectory error (ATE).

[0218] Relative Pose Error: Relative pose error (RPE) mainly calculates the error in the relative motion between each pair of trajectory points. At the same timestamp, the relative motion of the trajectory point in the real motion and the relative motion of the trajectory point in the estimated motion are calculated, and then the difference between the two results is calculated, which can effectively measure the trajectory drift. The estimated pose is defined as P1,…,P n ∈SE(3), the true pose Q1,…,Q n ∈SE(3), and △ represents the time interval. The RPE of the i-th frame is defined as E i :

[0219]

[0220] For the trajectory drift within the entire motion timestamp, the root mean square error RMSE is used to calculate the error results:

[0221]

[0222] Absolute Trajectory Error: Absolute trajectory error (ATE) is the difference between the actual pose and the trajectory estimated by the system. It can clearly reflect the accuracy of the algorithm and the consistency of the global trajectory. The transformation matrix from the estimated pose to the actual pose is defined as S∈SE(3). The ATE of the i-th frame is calculated as F i :

[0223]

[0224] The root mean square error RMSE of the absolute trajectory error is further calculated as:

[0225]

[0226] Comparing the error of the system in the embodiment with the error of ORB-SLAM3, the calculation formula of the relative improvement rate η is:

[0227]

[0228] TUM is a dataset for indoor dynamic environments. This embodiment experiment is tested on multiple TUM datasets. The experimental results of the embodiment of the present invention will be compared with ORB-SLAM3, DynaSLAM (a visual SLAM system based on ORB-SLAM2), DS-SLAM (a semantic visual SLAM robust in dynamic environments), SOF-SLAM (a semantic visual SLAM for dynamic environments), and OFM-SLAM (visual semantic SLAM for dynamic indoor environments). Figure 9: It is a comparison result diagram of absolute trajectory error between the present application method (Ours) and ORB-SLAM3. The present embodiment provides the values ​​of root mean square error RMSE, mean error Mean, median error Median and standard deviation STD. Wherein RMSE and STD are more important because they can better illustrate the robustness and stability of the system. In dynamic scenarios, the algorithm in the present invention has a certain improvement compared to most algorithms. However, in low dynamic environments, compared with the ORB-SLAM3 algorithm, the improvement of the algorithm in the present invention to the root mean square error estimation is not too high. This shows that traditional ORB-SLAM3 algorithm can achieve better results when not disturbed by dynamic objects. Therefore, the algorithm in the embodiment of the present invention can overcome the problem of low pose estimation accuracy caused by the interference of moving objects in dynamic environments.

[0229] In order to ensure the effectiveness of each module, this embodiment conducts ablation experiments on YOLOv7-tiny and adaptive threshold modules. The experimental results are as follows: Figure 10 As shown in the figure (1 represents the existence of this module, 0 represents the deletion of this module), it is the ablation experiment result diagram of this application, and the evaluation standard is the RMSE value. Figure 10 The impact of each module is clearly shown. It can be concluded from the data results that the root mean square error of the algorithm is the smallest when both modules are used.

[0230] Figure 11 The comparison results of the algorithm in this application and other SLAM systems in dynamic environments are shown in Figure 2. Figure 11 It can be seen that the DynaSLAM algorithm and the algorithm of the embodiment of the present invention have the highest positioning accuracy. Since the MaskRCNN semantic segmentation network used by DynaSLAM is executed pixel by pixel, the detection accuracy will be higher than the target detection model used in the embodiment of the present invention. However, the lightweight target detection network YOLOv7-tiny used in the present invention is superior to the above two algorithms in terms of algorithm execution speed. The embodiment of the present application performs a time test on the algorithm of the embodiment of the present invention and the DynaSLAM algorithm on the w_xyz dataset. Figure 12 This figure compares the time required to process each frame of an image on the tracking thread for the algorithm in this application and the DynaSLAM algorithm. It can be seen that DynaSLAM has poor real-time performance, while the algorithm in this application achieves a good balance between accuracy and real-time performance, and can effectively handle the impact of moving objects in dynamic environments on the stability of the SLAM system.

[0231] Figure 13This figure compares the test time of ORB-SLAM3 loop closure detection after combining local key image frame sequences with ORB-SLAM3. It can be seen that in the indoor loop closure datasets of fr1_room and fr2_desk, the system's total loop closure detection time is relatively reduced.

[0232] Figure 14 The figure shows the trajectory diagram and error distribution of the algorithm in this application and the ORB-SLAM3 algorithm. The figure uses different types of line segments to represent the true trajectory of the dataset, the estimated trajectory, and the error between the true trajectory and the estimated trajectory. It can be clearly observed from the figure that the pose estimated by the algorithm in this application is very close to the true trajectory. Compared with the ORB-SLAM3 algorithm, the various error values ​​are reduced by an order of magnitude. Figure 19 , which is the error distribution diagram of the RPE value of the method of this application under different scenarios.

[0233] In order to further verify the effectiveness of the algorithm in this paper, this part of the experiment uses the Jetson Nano mobile robot as the mobile platform, and the visual sensor on it is the Astra Pro model RGB-D camera produced by Orbbec, and the experiment is carried out in an actual real scene. Its parameters are as follows Figure 15 Astra Pro uses structured light for depth measurement and triangulation, and performs well in indoor environments.

[0234] The above experimental site is a rectangular conference room. Figure 16 The figure shows the experimental scenario of this application. The mobile robot starts from point A and moves in the clockwise order of A, B, C, D, A. At the same time, there will be people walking back and forth at points B and C during the movement. Figure 7 As shown in , dynamic objects in indoor scenes will be recognized. Finally, after multiple evaluations of real environments, the actual running trajectory of indoor dynamic scenes is as follows Figure 17 shown. Figure 17 In order to utilize the real trajectory of the ORB-SLAM3 system and the method of this application under indoor dynamic conditions, it can be seen that the actual trajectory of the ORB-SLAM3 system has a large deviation under the interference of dynamic objects, that is, the real trajectory of the method of this application (ours) under indoor dynamic conditions is more accurate than the real trajectory of the ORB-SLAM3 system, and can form a closed loop better.

[0235] In addition, to achieve the above-mentioned purpose, the present application also provides a loop detection device, which includes: a memory, a processor, and a loop detection program stored on the memory and executable on the processor. When the loop detection program is executed by the processor, the steps of the loop detection method described above are implemented.

[0236] The specific implementation of the loop detection device of the present application is basically the same as the embodiments of the loop detection method described above, and will not be repeated here.

[0237] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, on which a loop detection program is stored. When the loop detection program is executed by a processor, the steps of the loop detection method as described above are implemented.

[0238] The specific implementation of the computer medium of the present application is basically the same as the embodiments of the loop detection method described above, and will not be repeated here.

[0239] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0240] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0241] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a robot, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0242] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A loop detection method, characterized in that: The loop detection method comprises the following steps: Determining a previous key image frame corresponding to the current image frame and a reference image frame corresponding to the previous key image frame; comparing a first feature change of the reference image frame relative to the previous key image frame with an adaptive threshold, wherein the adaptive threshold is determined based on a feature matching amount and a feature change amount between the current image frame, the previous key image frame, and the reference image frame, the feature matching amount being the number of identical feature points between the two image frames, and the feature change amount being the number of different feature points between the two image frames; If the first feature variation is greater than the adaptive threshold, the current image frame is used as a target key image frame, and loop detection is performed based on the target key image frame.

2. The loop detection method according to claim 1, wherein: Before the step of comparing the first feature change of the reference image frame relative to the previous key image frame with an adaptive threshold, the method includes: Determining a first feature matching amount between the current image frame and the previous key image frame; determining a second feature matching amount between the reference image frame and the previous key image frame; Detecting the number of all feature points in the current image frame to obtain a first full feature quantity; Detecting the number of all feature points in the reference image frame to obtain a second full feature quantity; A basic adaptive threshold is constructed based on the first feature matching amount, the second feature matching amount, the first full feature amount, the second full feature amount, and the first feature change amount, and an adaptive threshold is determined based on the basic adaptive threshold.

3. The loop detection method according to claim 2, wherein: After the steps of constructing a basic adaptive threshold based on the first feature matching amount, the second feature matching amount, the first full feature amount, the second full feature amount, and the first feature change amount, and determining an adaptive threshold based on the basic adaptive threshold, the method includes: determining a first difference between the second feature matching amount and the first feature matching amount, and constructing a first correction coefficient according to a first ratio between the first difference and the second feature matching amount; constructing a second correction coefficient according to the number of corresponding image frames between the current image frame and the previous key image frame; Determining a second ratio between the first feature change and the first feature matching amount, determining a third ratio between the second feature change of the current image frame relative to the previous key image frame and the second feature matching amount, and constructing a third correction coefficient based on a second difference between the second ratio and the third ratio; constructing a fourth correction coefficient according to a third ratio between the first feature matching amount and the first feature change amount; determining a third difference between the second feature matching amount and the first feature matching amount, determining a total of the second feature matching amount and the first feature matching amount, and constructing a fifth correction coefficient based on a fourth ratio between the third difference and the total; Modifying the basic adaptive threshold based on the first correction coefficient, the second correction coefficient, the third correction coefficient, the fourth correction coefficient, and the fifth correction coefficient to obtain a modified adaptive threshold; The modified adaptive threshold is used as the adaptive threshold.

4. The loop detection method according to claim 3, wherein: The step of correcting the basic adaptive threshold based on the first correction coefficient, the second correction coefficient, the third correction coefficient, the fourth correction coefficient, and the fifth correction coefficient to obtain a corrected adaptive threshold comprises: When the first correction coefficient is less than or equal to a preset first coefficient threshold, constructing a modified adaptive threshold according to the basic adaptive threshold and a preset constant parameter; When the first correction coefficient is greater than the preset first coefficient threshold, a total correction coefficient is generated based on the first correction coefficient, the second correction coefficient, the third correction coefficient, the fourth correction coefficient and the fifth correction coefficient, and the basic adaptive threshold is corrected according to the total correction coefficient to obtain the corrected adaptive threshold.

5. The loop detection method according to claim 1, wherein: Before the step of using the current image frame as a target key image frame, the method includes: determining a uniformity of feature distribution in the current image frame based on position information between a preset first pixel point and a preset second pixel point in the current image frame, wherein a pixel value of the preset first pixel point and a pixel value of the preset second pixel point are adjacent in a pixel value sequence, the pixel value sequence being obtained by arranging the pixels in the current image frame according to pixel value magnitude, and the position information including a positional correspondence between a pixel position corresponding to the preset first pixel point and a pixel position corresponding to the preset second pixel point; When the feature distribution uniformity is greater than an adaptive distribution threshold, the step of using the current image frame as a target key image frame is performed, wherein the adaptive distribution threshold is determined based on each pixel value in the current image frame and the position distribution of each pixel value.

6. The loop detection method according to claim 1, wherein: The step of performing loop detection based on the target key image frame comprises: forming a full key image frame sequence based on the target key image frame and historical key image frames, wherein the historical key image frames are the key image frames determined before the target key image frame is determined; determining different local key image frame sequences based on the degree of change in viewing angle between adjacent key image frames in the full key image frame sequence; Calculating the similarity between the local key image frame sequences; Two local key image frame sequences whose similarity is greater than a preset similarity threshold are determined as target key image frame sequences that form a loop.

7. The loop detection method according to claim 6, wherein: The viewing angle variation includes a first viewing angle variation and a second viewing angle variation; The step of determining different local key image frame sequences based on the degree of change in viewing angle between adjacent key image frames in the full key image frame sequence comprises: Determining a first viewing angle change between a reference key image frame and a previous key image frame located before the reference key image frame, wherein the reference key image frame is any key image frame in the full key image frame sequence except a first key image frame and an end key image frame; determining a second viewing angle change between the reference key image frame and a subsequent key image frame located after the reference key image frame; If the first viewing angle change degree is less than a preset viewing angle change threshold and the second viewing angle change degree is greater than the preset viewing angle change threshold, using the reference key image frame as a division frame; The full key image frame sequence is divided according to the division frames to obtain the local key image frame sequences.

8. The loop detection method according to claim 1, wherein: Before the step of determining the previous key image frame corresponding to the current image frame and the reference image frame corresponding to the previous key image frame, the method includes: Detect each target object in the currently acquired image based on a preset target detection model; Determining a dynamic object among the target objects based on prior information, wherein the prior information includes a target object marked as dynamic, and the target object marked as dynamic is the dynamic object; The dynamic features on the dynamic object in the image are deleted to obtain the current image frame.

9. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a loop detection program stored in the memory and executable on the processor. When the loop detection program is executed by the processor, the steps of the loop detection method according to any one of claims 1 to 8 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a loop detection program, which, when executed by a processor, implements the steps of the loop detection method according to any one of claims 1 to 8.

Images