Loopback detection method, self-moving device and storage medium

Abstract

The invention provides a loopback detection method, a self-moving device and a computer readable storage medium. The loopback detection method comprises the following steps: acquiring a key frame set acquired before a to-be-detected image frame; acquiring a first disparity map of the to-be-detected image frame; traversing each frame of image in the key frame set to obtain a second disparity map of the currently traversed key frame; and if the parallax matching degree between the second parallax image and the first parallax image is greater than a preset parallax matching degree threshold, taking the current traversal key frame as a target loopback frame of the to-be-detected image. According to the method and the device, the performance overhead required by loopback detection can be reduced, and the influence of ambient light on loopback detection precision is avoided.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a loop detection method, a self-mobile device, and a computer-readable storage medium. Background Art

[0002] Simultaneous Localization and Mapping (SLAM), a key technology for enabling autonomous robot operation in unknown environments, is becoming a growing research hotspot in the field of robotics. However, over extended periods of operation, visual SLAM systems experience increasing cumulative errors, leading to a growing deviation between the robot's localized position and its actual location. Therefore, loop closure detection is necessary to correct the robot's position.

[0003] Related techniques use a method called bag-of-words (DBOW) to generate image descriptors and compare the current frame's descriptor with the keyframe's descriptor to determine if a loop has occurred. However, during actual research and development, the inventors discovered that loop detection requires significant performance overhead due to the generation and comparison of descriptors and is easily affected by ambient lighting, resulting in low loop detection accuracy. Summary of the Invention

[0004] The present application provides a loop detection method, a mobile device, and a computer-readable storage medium, which can reduce the performance overhead required for loop detection and avoid the influence of ambient light on loop detection accuracy.

[0005] In a first aspect, the present application provides a loop detection method, the method comprising:

[0006] Obtain a set of key frames collected before the image frame to be detected;

[0007] Acquire a first disparity map of the image frame to be detected;

[0008] Traversing each frame image in the key frame set to obtain a second disparity map of the currently traversed key frame;

[0009] If the disparity matching degree between the second disparity map and the first disparity map is greater than a preset disparity matching degree threshold, the current traversal key frame is used as a target loop frame of the image to be detected.

[0010] In a second aspect, the present application further provides a self-mobile device, which includes a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, any loop detection method provided in the present application is executed.

[0011] In a third aspect, the present application further provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the loop detection method.

[0012] In the present application, a first disparity map of the image frame to be detected is obtained; each frame image in the key frame set is traversed to obtain a second disparity map of the currently traversed key frame; if the disparity matching degree between the second disparity map and the first disparity map is greater than a preset disparity matching degree threshold, the currently traversed key frame is used as the target loop frame of the image to be detected; thereby, images with higher disparity matching degrees are screened from the key frames by disparity map comparison as the target loop frame of the image to be detected. On the one hand, since there is no need to generate and compare frame descriptors, the performance overhead required for loop detection can be reduced to a certain extent. On the other hand, since there is no need to generate and compare frame descriptors for loop detection, the problem that the descriptor cannot be accurately expressed because the color and texture changes of the image are easily affected by the ambient light can be avoided, thereby avoiding the problem of inaccurate matching caused by relying on the descriptor to match the image frame to be detected with the loop frame, avoiding the influence of ambient light on the accuracy of loop detection, and thus improving the matching accuracy of the loop frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 This is a schematic block diagram of the structure of a self-moving device provided in an embodiment of the present application;

[0015] Figure 2 This is a flow chart of a loop detection method provided in an embodiment of the present application;

[0016] Figure 3 is a schematic diagram illustrating the conversion relationship between various coordinate systems in the embodiment of the present application;

[0017] Figure 4 is a schematic diagram illustrating re-projection error in an embodiment of the present application;

[0018] Figure 5 This is a schematic diagram illustrating the construction of loop constraints for pose graph optimization in an embodiment of the present application;

[0019] Figure 6 This is a schematic diagram illustrating an application scenario process of loop detection in an embodiment of the present application;

[0020] Figure 7This is a schematic diagram illustrating the end-to-end feature point matching provided in the embodiments of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0022] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0023] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0024] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that one of ordinary skill in the art will recognize that the present application can be implemented without using these specific details. In other examples, well-known processes will not be elaborated in detail to avoid obscuring the description of the embodiments of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.

[0025] The present invention provides a loop detection method, a self-moving device, and a computer-readable storage medium. The self-moving device may be a cleaning robot, a food delivery robot, or the like.

[0026] The following embodiments of the present application are described in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0027] Figure 1 This is a schematic structural diagram of a self-moving device provided in an embodiment of the present application.

[0028] like Figure 1As shown, the mobile device 100 includes a processor 101 and a memory 102 , and the processor 101 and the memory 102 are connected via a bus 103 , such as an I 2 C (Inter-integrated Circuit) bus.

[0029] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire mobile device 100. The processor 101 can be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0030] Specifically, the memory 102 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0031] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the embodiment scheme of the present application, and does not constitute a limitation on the self-moving device to which the embodiment scheme of the present application is applied. The specific self-moving device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0032] The processor 101 is configured to run a computer program stored in the memory 102 and implement any one of the loop detection methods provided in the embodiments of the present application when executing the computer program. For example, the processor 101 is configured to run a computer program stored in the memory 102 and implement the following steps when executing the computer program:

[0033] Obtain a set of key frames collected before the image frame to be detected; obtain a first disparity map of the image frame to be detected; traverse each frame image in the key frame set to obtain a second disparity map of the currently traversed key frame; if the disparity match between the second disparity map and the first disparity map is greater than a preset disparity match threshold, use the currently traversed key frame as the target loop frame of the image to be detected.

[0034] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the mobile device described above can refer to the corresponding process in the following loop detection method embodiment, and will not be repeated here.

[0035] In the following, Figure 1 The self-mobile device shown in is used as the execution subject of the loop detection method as an example to introduce the loop detection method provided by the embodiment of the present application in detail. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments. Figure 1 The scenarios are only used to explain the loop detection method provided in the embodiments of the present application, but do not constitute a limitation on the application scenarios of the loop detection method provided in the embodiments of the present application.

[0036] See also Figure 2 , Figure 2 This is a flow chart of a loop detection method provided by an embodiment of the present application. Figure 2 The flowcharts shown in the accompanying drawings or other figures show a logical order, but in some cases, the steps shown or described may be performed in a different order than here; the step numbers shown in the flowcharts or the following text are only for illustrative purposes and are not intended to limit the specific order of the steps. The loop detection method includes steps 201 to 204, wherein:

[0037] 201. Obtain a key frame set of an image frame to be detected.

[0038] In order to better understand the embodiments of the present application, the following first introduces some of the names involved in this embodiment:

[0039] 1. Feature Point Pair: When a feature point in one image frame (referred to as the reference frame) matches a feature point in another image frame (referred to as the comparison frame), the feature point in the reference frame and the corresponding matching feature point in the comparison frame constitute a feature point pair. That is, each feature point pair includes the feature points mapped to the 3D spatial point in both image frames (e.g., the feature point mapped to the 3D spatial point in image 1 and the feature point mapped to the 3D spatial point in image 2).

[0040] 2. Spatial three-dimensional coordinate system: A coordinate system used to describe the position of an object (such as a camera) in three-dimensional space, for example Figure 3 As shown, the three-dimensional coordinate system can be O w X w Y w Z w .

[0041] 3. Camera coordinate system: A three-dimensional coordinate system originally defined as the light point of the camera, for example Figure 3 As shown, the camera coordinate system can be O c Xc Y c Z c .

[0042] The conversion relationship between the spatial three-dimensional coordinate system and the camera coordinate system is shown in the following formula (1):

[0043]

[0044] In formula (1), T represents the offset, R=R1R2R3 represents the rotation matrix, and R1, R2, and R3 respectively represent the matrices of rotation of the camera coordinate system relative to the three-dimensional spatial coordinate system.

[0045] 4. Image coordinate system: It is a two-dimensional coordinate system. The origin is the corner point between the camera optical axis and the imaging plane. The image coordinate system and the camera coordinate system are in a perspective projection relationship. The image coordinate system is a two-dimensional coordinate system obtained by projecting the three-dimensional camera coordinate system. For example, Figure 3 As shown, the image coordinate system can be oxy.

[0046] The conversion relationship between the image coordinate system and the camera coordinate system is shown in the following formula (2):

[0047]

[0048] In formula (2), f represents the focal length of the camera.

[0049] 5. Pixel coordinate system: It is a two-dimensional coordinate system. Both the pixel coordinate system and the image coordinate system are on the imaging plane. The difference is that the pixel coordinate system uses row and column values ​​to represent the coordinate values ​​of the pixel points. For example, Figure 3 As shown, the pixel coordinate system can be uv.

[0050] The conversion relationship between the pixel coordinate system and the image coordinate system is shown in the following formula (3):

[0051]

[0052] In formula (3), dx and dy represent the width and height of each pixel in the pixel coordinate system, respectively.

[0053] Therefore, for the three-dimensional space point P in the three-dimensional coordinate system w (X w ,Y w ,Z w ) is mapped (also called projection in this article) to the pixel coordinate system. The coordinate transformation relationship is shown in the following formula (4):

[0054]

[0055] In formula (4), Indicates the camera's intrinsic parameters, which are usually fixed after leaving the factory and can be calculated using the camera calibration method; Represents the camera extrinsics.

[0056] 6. Pose (ie camera pose): refers to the camera coordinate system (such as coordinate system O c X c Y c Z c ) relative to the three-dimensional coordinate system (such as O w X w Y w Z w ) position (X w ,Y w ,Z w ) and posture (rX w ,rY w ,rZ w ), that is, (X w ,Y w ,Z w ,rX w ,rY w ,rZ w ), X w 、Y w , Z w 、rX w 、rY w 、rZ w Represents the X coordinates of the camera in the three-dimensional coordinate system w Coordinates, camera coordinate system in the three-dimensional coordinate system Y w Coordinates, camera coordinate system Z in the three-dimensional coordinate system of space w Coordinates, camera coordinate system and OX w Axis angle, camera coordinate system and OY w Axis angle, camera coordinate system and OZ w The angle between the axes.

[0057] 7. Observation coordinates: refers to the three-dimensional space point P in the three-dimensional coordinate system obtained by camera observation w (X w ,Y w ,Z w ) is mapped to the camera's image coordinate system, and then converted to the coordinates p(u,v) of the mapping point p in the pixel coordinate system.

[0058] like Figure 4 As shown, in the camera coordinate system O c X c Y c Z c Relative space three-dimensional coordinate system O w X w Yw Z w In the case of posture a, the camera captures the following Figure 4 Image 1 shown by the left quadrilateral frame, for any three-dimensional space point P w The pixel coordinate system is projected as pixel point p1. At this time, the pixel coordinate system coordinate of pixel point p1 is point P w Observation coordinates at pose a.

[0059] Similarly, in the camera coordinate system O c X c Y c Z c Relative space three-dimensional coordinate system O w X w Y w Z w In the case of posture b, the camera captures the following Figure 4 Image 2 shown by the right quadrilateral frame, for any three-dimensional space point P w The pixel coordinate system is projected as pixel point p2. At this time, the pixel coordinate system coordinate of pixel point p2 is point P. w Observation coordinates at pose b.

[0060] 8. Reprojection coordinates: The projected position of a known 3D space point in the image obtained by mapping the known 3D space point to the known camera pose. For example, taking the 3D coordinates of a 3D space point based on epipolar geometry as an example, first, the observation coordinates of the same 3D space point in two different poses (for example, reference Figure 4 As shown, the three-dimensional space point P w The observation coordinates p1(u,v) under the left camera and p2(u,v) under the right camera of the binocular camera are respectively used to construct a triangle based on geometric information (such as epipolar geometry) to determine the three-dimensional space point P w The three-dimensional coordinate system coordinate P w (X w ,Y w ,Z w ); Then, use the calculated spatial three-dimensional coordinate system coordinate P w (X w ,Y w ,Z w ), and the camera pose [R, T] at image 2 are reprojected to obtain the pixel coordinate system coordinates of the mapping point p2'.

[0061] 9. Reprojection error: Reprojection error refers to the difference between the reprojected coordinates of a 3D point and the actual observed coordinates. For example, you can refer to Figure 4In the part shown by e in , the reprojection error is the error between the reprojected coordinates (p2') and the observed coordinates (p2) under the right camera.

[0062] 10. Autonomous mobile device: A mobile robot that uses SLAM technology to achieve autonomous operation in an unknown environment (such as a cleaning robot, a food delivery robot, etc.). In this embodiment, the loop detection method is applied to an autonomous mobile device. The camera set on the autonomous mobile device captures scene images (such as the image frame to be detected and each image frame i before the image frame to be detected) as an example for explanation, where the camera pose is regarded as the pose of the autonomous mobile device.

[0063] There are many ways to obtain the key frame set in step 201, illustratively including:

[0064] (1) For each image frame i acquired in real time in the current scene, determine whether image frame i is a key frame; when the current scene acquires the image frame n to be detected, the set of all key frames acquired before the image frame n to be detected is used as the key frame set of the image frame n to be detected. Specifically, in actual business scenarios, whether image frame i is used as a key frame can be determined based on the selection interval, the number of feature points, etc. For example, the process of determining whether image frame i is a key frame can be as follows: ① The robot acquires image frame i and extracts the feature points in image frame i; ② The feature points in image frame i are matched with the feature points of the previous image frame i-1 to obtain a matched feature point pair; ③ Based on the matched feature point pairs between the previous and next image frames obtained by historical matching, and the last key frame before image frame i, the number of feature points that differ between image frame i and the last key frame is compared; ④ If the number of feature points that differ between image frame i and the last key frame is greater than the preset feature point number threshold, image frame i is used as a key frame. And so on. For another example, referring to steps J1 to J3, it can be determined whether image frame i is a key frame.

[0065] (2) Real-time acquisition. When the image frame n to be detected is acquired in the current scene, refer to the method in (1) to acquire all key frames collected before the image frame n to be detected, and obtain the key frame set of the image frame n to be detected.

[0066] 202. Obtain a first disparity map of the image frame to be detected.

[0067] The first disparity map refers to the disparity map of the image frame to be detected.

[0068] For example, when a mobile device uses a binocular camera to capture images, the first disparity map can be obtained by using the feature point pairs matched between the left camera view and the right camera view of the image frame to be detected. In this case, step 202 can specifically include the following steps 2021A to 2023A:

[0069] 2021A. Obtain a left camera view of the image frame to be detected and a right camera view of the image frame to be detected.

[0070] 2022A. Obtain a first matching pixel pair between the left camera view of the image frame to be detected and the right camera view of the image frame to be detected.

[0071] The first matching pixel pair includes a first pixel in the left camera view of the image frame to be detected and a second pixel in the right camera view of the image frame to be detected that matches the first pixel.

[0072] 2023A. Store the disparity value between the first pixel point and the second pixel point in a corresponding position of the disparity map to obtain a first disparity map of the image frame to be detected.

[0073] Specifically, the process of calculating the first disparity map in steps 2021A to 2023A may be as follows:

[0074] 1. Epipolar correction: The purpose of correction is to make the optical centers of the left camera view and the right camera view of the image frame to be detected on the same horizontal line.

[0075] Among them, two cameras (called left camera and right camera) capture two images of the same scene. For any point in the left image, its corresponding matching point in the right image must be located on a straight line, and this straight line is the epipolar line.

[0076] The pole is the projection point of the left camera's optical center onto the right image plane, and vice versa. Every image has one pole.

[0077] Among them, the baseline is a straight line connecting the optical centers of two cameras. In this embodiment, it is specifically a straight line connecting the camera optical centers corresponding to two images (such as the left camera view and the right camera view of the image frame to be detected, the image frame to be detected and the previous frame of the image frame to be detected, the left camera view of the current traversal image frame and the right camera view of the current traversal image frame, the current traversal image frame and the previous frame of the current traversal image frame, etc.).

[0078] 2. Feature Matching: Normalized cross correlation (NCC) is used to match the left and right camera views of the rectified image frame to be tested. The optimal match is the one with the highest correlation between the pixels to be tested on the same horizontal line in the left and right camera views of the image frame to be tested. The matching principle is as follows: For example, for the left camera view I1 and the right camera view I2 of the image frame to be tested, the NCC algorithm constructs an n*n matching window for each pixel to be matched in image I1. The matching window is then constructed for each pixel on the epipolar line of image I2 and the correlation is calculated with the matching window of the pixel to be matched. The one with the highest correlation is considered the optimal match.

[0079] 3. Record the disparity d of the first matching pixel pair. After completing NCC matching, record the difference (xr-xl) between the horizontal direction xl of the pixel to be tested (i.e., the first pixel) and the horizontal direction xr of the matching pixel (i.e., the second pixel). According to the disparity relationship (the disparity relationship is: Where f is the focal length of the rectified image, T is the distance between the two camera centers, and xr and xl are the x-coordinates of the corresponding points in the left camera view I1 and the right camera view I2, respectively. This serves as the disparity d of the first matching pixel pair (i.e., the disparity d between the first pixel and the second pixel). Similarly, the disparity d of each first matching pixel pair between the left camera view and the right camera view can be calculated, ultimately yielding a disparity map D of the same size as the original image, which serves as the first disparity map.

[0080] 203. Traverse each frame image in the key frame set to obtain a second disparity map of the currently traversed key frame.

[0081] The second disparity map refers to the disparity map of the currently traversed key frame.

[0082] The currently traversed key frame refers to the currently traversed key frame when traversing each frame image in the key frame set.

[0083] For example, when a mobile device uses a binocular camera to capture images, a second disparity map can be obtained using the feature point pairs matched between the left camera view and the right camera view of the currently traversed image frame. In this case, "obtaining the second disparity map of the currently traversed key frame" can specifically include the following steps 2031A to 2033A:

[0084] 2031A. Obtain the left camera view of the current traversal image frame and the right camera view of the current traversal image frame.

[0085] 2032A. Obtain a third matching pixel pair between the left camera view of the current traversal image frame and the right camera view of the current traversal image frame.

[0086] The third matching pixel pair includes a fifth pixel in the left camera view of the current traversal image frame and a sixth pixel in the right camera view of the current traversal image frame that matches the fifth pixel.

[0087] 2033A. Store the disparity value between the fifth pixel point and the sixth pixel point in a corresponding position of the disparity map to obtain a second disparity map of the currently traversed image frame.

[0088] The implementation method of “obtaining the second disparity map of the currently traversed key frame” is similar to that of step 202 . For details, please refer to the relevant description of step 202 . For simplicity of description, it will not be repeated here.

[0089] 204 : If the disparity matching degree between the second disparity map and the first disparity map is greater than a preset disparity matching degree threshold, use the current traversal key frame as a target loop frame of the image to be detected.

[0090] There are multiple ways to determine the target loop frame in step 204, illustratively including:

[0091] (1) When the disparity matching degree is greater than the preset disparity matching degree threshold, the currently traversed key frame is used as the target loop frame. In this case, step 204 includes step 205 of obtaining the disparity matching degree. Step 204 may specifically include the following step 2041A:

[0092] 2041A: If the disparity matching degree between the second disparity map and the first disparity map is greater than a preset disparity matching degree threshold, use the current traversal key frame as a target loop frame of the image to be detected.

[0093] Step 205 (obtaining the disparity matching degree) is specifically as follows:

[0094] 205. Obtain a disparity matching degree between the second disparity map and the first disparity map.

[0095] For example, step 205 may specifically include the following steps 2051A to 2053A:

[0096] 2051. Obtain a target feature point pair between the image frame to be detected and the currently traversed key frame.

[0097] Each of the target feature point pairs includes a first feature point mapped by the target three-dimensional space point in the image frame to be detected, and a second feature point mapped in the currently traversed key frame.

[0098] The first feature point is a feature point mapped to the target three-dimensional space point in the image frame to be detected, specifically a feature point in the image frame to be detected that matches the second feature point in the currently traversed key frame.

[0099] The second feature point is a feature point mapped to the target three-dimensional space point in the current traversal key frame, specifically a feature point in the current traversal key frame that matches the first feature point in the image frame to be detected.

[0100] The target three-dimensional space point is the location point of the physical object corresponding to the target feature point pair in the three-dimensional space.

[0101] In some embodiments, referring to steps H1 to H2, an end-to-end model can be used to perform feature matching between the image frame to be detected and the currently traversed key frame, thereby obtaining a target feature point pair between the image frame to be detected and the currently traversed key frame.

[0102] In other embodiments, a Brute-Force Matcher algorithm or a Fast Library for Approximate Nearest Neighbors (FLANN) algorithm may be used to perform feature matching between the image frame to be detected and the currently traversed key frame, thereby obtaining a target feature point pair between the image frame to be detected and the currently traversed key frame.

[0103] 2052. Obtain a first disparity value of a position corresponding to the first feature point in the first disparity map, and a second disparity value of a position corresponding to the first feature point in the second disparity map.

[0104] 2053. Determine a disparity matching degree between the second disparity map and the first disparity map based on a distance between the first disparity value and the second disparity value.

[0105] The greater the distance between the first disparity value and the second disparity value, the smaller the disparity matching degree; conversely, the smaller the distance between the first disparity value and the second disparity value, the greater the disparity matching degree. For example, the first disparity value and the second disparity value can be substituted into a preset relationship between the disparity matching degree and the disparity value distance, and the calculated result can be used as the disparity matching degree between the second disparity map and the first disparity map.

[0106] (2) When the disparity matching degree is greater than a preset disparity matching degree threshold, the currently traversed key frame is used as a candidate loop frame; the positioning matching result between the candidate loop frame and the image frame to be detected is further detected. When the candidate loop frame and the image frame to be detected are positioned and matched, the currently traversed key frame (i.e., the candidate loop frame) is used as the target loop frame to improve the loop detection accuracy. In this case, step 204 also includes step 205 of obtaining the disparity matching degree and step 206 of obtaining the positioning matching result. Step 204 may specifically include the following step 2041B:

[0107] 2041B: If the disparity matching degree between the second disparity map and the first disparity map is greater than a preset disparity matching degree threshold, and the positioning matching result is a positioning match, use the current traversal key frame as the target loop frame of the image to be detected.

[0108] Exemplarily, first, refer to step 205 to obtain the disparity matching degree. When the disparity matching degree is detected to be greater than the preset disparity matching degree threshold, the current traversal key frame is used as an alternative loop frame. Refer to step 206 to obtain the positioning matching result between the image frame to be detected and the current traversal key frame. If the image frame to be detected is positioned matched with the current traversal key frame, the current traversal key frame is used as the target loop frame; otherwise, if the image frame to be detected is not positioned matched with the current traversal key frame, the current traversal key frame is not used as the target loop frame, and the loop judgment is continued on the next frame in the key frame set. Refer to step 2041B to determine whether the next key frame is used as the target loop frame.

[0109] Step 205 and step 206 (obtaining disparity matching and positioning matching results) are specifically as follows:

[0110] 205. Obtain a disparity matching degree between the second disparity map and the first disparity map.

[0111] 206. Obtain a positioning matching result between the image frame to be detected and the currently traversed key frame.

[0112] There are many ways to obtain the positioning matching result in step 206, illustratively including:

[0113] (1) Two sets of 2D points are known, namely, the observed coordinates of the known target matching feature point pair in the image frame to be detected and the observed coordinates in the currently traversed key frame. Using the two-dimensional information (i.e., the observed coordinates) of the image frame to be detected and the currently traversed key frame, an essential matrix constraint is constructed for verification. In this case, step 206 may specifically include the following steps 2061A to 2063A:

[0114] 2061A. Obtain a target feature point pair between the image frame to be detected and the currently traversed key frame.

[0115] The implementation of step 2061A is similar to that of step 2051. For details, please refer to the relevant instructions above and will not be repeated here.

[0116] 2062A. Obtain the essential matrix between the image frame to be detected and the currently traversed key frame.

[0117] For example, the epipolar constraint can be used to solve the essential matrix E according to the pixel positions of the target matching point pairs. The epipolar constraint is shown in the following formula (5):

[0118]

[0119] In formula (5), p2 represents the pixel coordinate system coordinate of the image frame to be detected (i.e., the observation coordinate of the first feature point), p1 represents the pixel coordinate system coordinate of the current traversed key frame (i.e., the observation coordinate of the second feature point), K is the camera intrinsic parameter matrix, and the essential matrix E = t^R.

[0120] 2063A. Determine a positioning matching result between the image frame to be detected and the currently traversed key frame based on the essential matrix, the observed coordinates of the first feature point, and the observed coordinates of the second feature point.

[0121] Exemplarily, step 2063A may specifically include: substituting the essential matrix, the observation coordinates of the first feature point and the observation coordinates of the second feature point into the preset epipolar constraint relationship to obtain the constraint result value; if the constraint result value is within the preset value range, then taking the positioning match as the positioning matching result; or, if the constraint result value is outside the preset value range, then taking the positioning mismatch as the positioning matching result.

[0122] For example, the epipolar constraint relation is Substitute the observation coordinates of the first feature point and the second feature point of all target feature point pairs into the epipolar constraint relationship , the calculated result is the constraint result value. If the constraint result value is close to 0 (less than or equal to 0.001), the positioning match is taken as the positioning match result between the image frame to be detected and the current traversal key frame; otherwise, if the constraint result value is not close to 0 (such as greater than 0.001), the positioning mismatch is taken as the positioning match result between the image frame to be detected and the current traversal key frame.

[0123] Since the projection of the same spatial point on the two imaging planes satisfies the epipolar constraint relationship, the epipolar constraint relationship reflects the spatial position relationship between the two matching points (i.e., the essential matrix). Through steps 2061A to 2063A, it is determined whether the positioning between the image frame to be detected and the currently traversed key frame matches. The essential matrix relationship can be used to detect whether the feature points of the image frame to be detected and the currently traversed key frame conform to a certain transformation relationship, thereby reflecting whether the positioning between the image frame to be detected and the currently traversed key frame matches. It can be seen that through steps 2061A to 2063A, the epipolar constraint can be used to verify whether the positioning corresponding to the image frame to be detected and the alternative loop frame is close, thereby verifying whether the alternative loop frame is a true loop frame, thereby improving the matching accuracy of the target loop frame.

[0124] (2) Two sets of 3D points are known, namely: the three-dimensional coordinates of the target feature point pair in the current traversal key frame and the three-dimensional coordinates in the image frame to be detected. For example, when the camera is a binocular camera, an RGB-D camera, or when the distance information of the feature points is known by some method, the three-dimensional coordinates of the target feature point pair in the current traversal key frame and the three-dimensional coordinates in the image frame to be detected can be directly obtained; the three-dimensional coordinates of the target feature point pair in the current traversal key frame and the three-dimensional coordinates in the image frame to be detected can be directly used to construct the three-dimensional transformation relationship between the feature points in the image frame to be detected and the current traversal key frame. At this time, step 206 can specifically include the following steps 2061B to 2064B:

[0125] 2061B. Obtain a target feature point pair between the image frame to be detected and the currently traversed key frame.

[0126] The implementation of step 2061B is similar to that of step 2051. For details, please refer to the relevant instructions above and will not be repeated here.

[0127] 2062B. Construct a three-dimensional transformation relationship between the feature points of the image frame to be detected and the currently traversed key frame based on the three-dimensional coordinates of the first feature point in the camera coordinate system and the three-dimensional coordinates of the second feature point in the camera coordinate system.

[0128] For example, first, the observed coordinates of the first feature point in the image frame to be detected and the distance information to the first feature point can be combined to obtain the three-dimensional coordinates of the first feature point in the camera coordinate system. For example, assuming that the observed coordinates of the first feature point are (x1, y1) and the distance to the first feature point is z1, (x1, y1, z1) can be used as the three-dimensional coordinates of the first feature point in the camera coordinate system.

[0129] Similarly, the observed coordinates of the second feature point in the currently traversed keyframe and the distance information to the second feature point can be combined to obtain the three-dimensional coordinates of the second feature point in the camera coordinate system. For example, if the observed coordinates of the second feature point are (x2, y2) and the distance to the second feature point is z2, (x2, y2, z2) can be used as the three-dimensional coordinates of the second feature point in the camera coordinate system.

[0130] Then, using the conversion relationship between the three-dimensional coordinates of the first feature point in the camera coordinate system and the three-dimensional coordinates of the second feature point in the camera coordinate system as the unknown variable, a conversion relationship between the three-dimensional coordinates of the first feature point in the camera coordinate system and the three-dimensional coordinates of the second feature point in the camera coordinate system is constructed as the three-dimensional conversion relationship of the feature points. For example, the three-dimensional conversion relationship of the feature points can be constructed as h1=f*h2, where h1 represents the three-dimensional coordinates of the first feature point in the camera coordinate system, h2 represents the three-dimensional coordinates of the second feature point in the camera coordinate system, and f represents the conversion relationship between the two.

[0131] 2063B. If the solution of the three-dimensional transformation relationship of the feature points converges, the positioning matching is used as the positioning matching result.

[0132] 2064B. If the solution of the three-dimensional transformation relationship of the feature points diverges, a positioning mismatch is taken as the positioning matching result.

[0133] For example, the solution method of the iterative closest point (ICP) can be referred to, and a linear algebra solution method such as singular value decomposition (SVD) or a nonlinear optimization method can be used to solve the three-dimensional transformation relationship of the feature points. If the solution result of the three-dimensional transformation relationship of the feature points converges, the positioning match is used as the positioning matching result; otherwise, if the solution result of the three-dimensional transformation relationship of the feature points diverges, the positioning mismatch is used as the positioning matching result.

[0134] Since the corresponding three-dimensional coordinates of the feature points between two images with similar positioning satisfy a certain conversion relationship, through steps 2061B to 2064B, when two groups of 3D points are known (i.e., the three-dimensional coordinates of the known target feature point pair in the current traversal key frame and the three-dimensional coordinates in the image frame to be detected), a three-dimensional conversion relationship between the feature points in the image frame to be detected and the feature points in the current traversal key frame can be established, and whether the positioning between the image frame to be detected and the current traversal key frame matches can be determined by determining whether the solution results diverge or converge. It can be seen that through steps 2061B to 2064B, the three-dimensional conversion relationship of the feature points can be used to verify whether the positioning corresponding to the image frame to be detected and the alternative loop frame is close, thereby verifying whether the alternative loop frame is a true loop frame, thereby improving the matching accuracy of the target loop frame.

[0135] (3) A set of 3D points and a set of 2D points are known, that is, the spatial three-dimensional coordinates of the feature points in the currently traversed key frame and their projected positions in the image frame to be detected are known (the projected positions in the image frame to be detected are the observed coordinates of the feature points in the image to be detected). At this time, the 2D points can be converted into 3D points first. If the solution of the three-dimensional transformation relationship of the feature points diverges, the positioning mismatch is regarded as the positioning matching result. At this time, step 206 can specifically include the following steps 2061C to 2065C:

[0136] 2061C. Obtain a target feature point pair between the image frame to be detected and the currently traversed key frame.

[0137] The implementation of step 2061C is similar to that of step 2051. For details, please refer to the relevant instructions above and will not be repeated here.

[0138] 2062C. Based on the observation coordinates of the first feature point and the three-dimensional spatial coordinates corresponding to the second feature point, obtain the three-dimensional coordinates of the first feature point in the camera coordinate system.

[0139] Exemplarily, the PnP principle can be referred to, and the input data is a 3D-2D matching point (i.e., the observation coordinates of the first feature point and the spatial three-dimensional coordinates corresponding to the second feature point), wherein the spatial three-dimensional coordinates corresponding to the second feature point can be regarded as a 3D point, and the observation coordinates of the first feature point can be regarded as a 2D point (i.e., the projection point of the 3D point in the camera coordinate system). By using the similarity property of triangles, the 3D coordinates of the projection point in the camera coordinate system are calculated as the three-dimensional coordinates of the first feature point in the camera coordinate system.

[0140] Thus, a set of 3D-2D matching points can be converted into a set of 3D-3D matching points. Subsequently, similar methods to 2061B to 2064B can be referred to to construct a three-dimensional transformation relationship of feature points using a set of 3D-3D matching points.

[0141] 2063C. Based on the three-dimensional coordinates of the first feature point in the camera coordinate system and the spatial three-dimensional coordinates corresponding to the second feature point, construct a three-dimensional transformation relationship between the feature points of the image frame to be detected and the currently traversed key frame.

[0142] Exemplarily, the conversion relationship between the three-dimensional coordinates of the first feature point in the camera coordinate system and the three-dimensional coordinates of the second feature point is used as an unknown variable, and a conversion relationship between the three-dimensional coordinates of the first feature point in the camera coordinate system and the three-dimensional coordinates of the second feature point is constructed as the three-dimensional conversion relationship of the feature points. For example, a three-dimensional conversion relationship of the feature points can be constructed as h1=f*h2, where h1 represents the three-dimensional coordinates of the first feature point in the camera coordinate system, h2 represents the three-dimensional coordinates of the second feature point, and f represents the conversion relationship between the two.

[0143] 2064C. If the solution of the three-dimensional transformation relationship of the feature points converges, taking the positioning matching as the positioning matching result;

[0144] 2065C. If the solution of the three-dimensional transformation relationship of the feature points diverges, a positioning mismatch is taken as the positioning matching result.

[0145] The implementation of steps 2064C to 2065C is similar to that of steps 2063B to 2064B. For details, please refer to the relevant description above and will not be repeated here.

[0146] Since the corresponding three-dimensional coordinates of the feature points between two images with close positioning satisfy a certain transformation relationship, through steps 2061C to 2065C, when a group of 3D points and a group of 2D points are known (that is, the spatial three-dimensional coordinates of the feature points in the current traversal key frame and their projection positions in the image frame to be detected are known), the three-dimensional coordinates of the first feature point in the camera coordinate system are first obtained based on the observation coordinates of the first feature point and the spatial three-dimensional coordinates corresponding to the second feature point; then, the three-dimensional coordinates of the first feature point in the camera coordinate system and the spatial three-dimensional coordinates corresponding to the second feature point are used to establish a three-dimensional transformation relationship between the feature points in the image frame to be detected and the feature points in the current traversal key frame; finally, whether the positioning between the image frame to be detected and the current traversal key frame matches is determined by calculating whether the solution results diverge or converge. This can improve the matching accuracy of the positioning matching results, thereby improving the loop detection accuracy. It can be seen that through steps 2061C to 2065C, the three-dimensional transformation relationship of the feature points can be used to verify whether the positioning corresponding to the image frame to be detected and the alternative loop frame is close, thereby verifying whether the alternative loop frame is a true loop frame and improving the matching accuracy of the target loop frame.

[0147] Thus, loop detection is performed by way of step 2041B. On the one hand, since the disparity map is used to compare and preliminarily screen images with higher disparity matching from key frames as candidate loop frames (that is, when the disparity matching between the second disparity map and the first disparity map is greater than the preset disparity matching threshold, the currently traversed key frame is used as the candidate loop frame), and then the positioning matching result is used to determine whether it is a true loop. Since there is no need to generate and compare frame descriptors for loop detection, the performance overhead required for loop detection due to the generation and comparison of frame descriptors is reduced. On the other hand, since there is no need to generate and compare frame descriptors, the problem of the inability of the descriptor to accurately express the image due to the color and texture changes being easily affected by the ambient light can be avoided, thereby avoiding the problem of inaccurate matching caused by relying on the descriptor to match the loop frame of the image frame to be detected, thereby improving the matching accuracy of the loop frame.

[0148] Furthermore, to reduce the cumulative error in pose estimation and improve the positioning accuracy and mapping accuracy of the self-mobile device, when a loop is detected in the image frame to be detected, loop constraints can be constructed using the image frame to be detected and the target loop frame to optimize the camera pose and map points. In this case, step 204 can further include the following steps G1 to G3:

[0149] G1. Construct a loop constraint relationship based on the target loop frame and the image frame to be detected.

[0150] Please refer to Figure 5 , Figure 5In the pose graph shown in (a), each circular node (x1, x2, ..., xt) represents the camera pose when capturing the frame image (image 1, 2, ..., t), and the edge between two circular nodes represents the spatial constraint between the two camera poses (such as the spatial constraint between camera pose xi and camera pose xj, that is, the camera motion of image j relative to image i). Figure 5 The spatial constraints shown include loop edge constraints (such as Figure 5 (a) dashed arrow), inter-frame edge constraints (such as Figure 5 (a) as indicated by the solid arrow), as follows:

[0151] 1. Inter-frame edge constraint: used to indicate the pose transformation relationship between adjacent nodes, specifically used to indicate the pose transformation relationship x between two adjacent frame images (such as image i and image j) ij .

[0152] 2. Loop closure edge constraint: used to indicate the pose transformation relationship between the image frame where the loop is detected (such as the image frame to be detected) and the loop closure frame (such as the target loop closure frame).

[0153] like Figure 5 As shown in (b), the camera motion of image j relative to image i (denoted by x ij In essence, it is the pose transformation relationship between image i and image j, that is, the camera pose at image i (expressed by x i denoted by) and the camera pose at image j (denoted by x j Represents the transformation relationship between them.

[0154] Among them, when the matching point pairs between image i and image j are known, the camera motion of image j relative to image i can be calculated based on the matching point pairs between image i and image j through epipolar geometry, ICP, PnP and other algorithms.

[0155] Therefore, after the target loop frame is determined in step 204, the target loop frame (with Figure 5 2 in the image as an example) and the image frame to be detected (with Figure 5 The camera motion of the image frame to be detected (i.e., image t) relative to the target loop closure frame (i.e., image 2) is calculated using algorithms such as epipolar geometry, ICP, and PnP. The camera motion of the image frame to be detected (i.e., image t) relative to the target loop closure frame (i.e., image 2) is the loop closure constraint. For example, the camera motion of the image frame to be detected relative to the target loop closure frame can be expressed as: [R, T].

[0156] G2. Construct a target cost function based on the motion equation of the mobile device and the reprojection error constraint.

[0157] To better understand this embodiment, the motion equation and observation equation of the self-moving device are first introduced below, as shown in the following formula (6):

[0158]

[0159] In formula (6), x k is the camera pose at time k (i.e. the camera pose at image k), u k is the sensor input, y j is the coordinate of the landmark point in the three-dimensional space coordinate system, z k,j is the pixel coordinate system coordinate corresponding to the landmark point, w k 、v k,j For noise.

[0160] Please refer to Figure 4 , for image k-1 (reference Figure 4 Image 1) and image k (reference Figure 4 Image 2 in ), for the matching feature point pairs of image k-1 and image k (reference Figure 4 Points p1 and p2 in the image can be used to construct the reprojection error constraint as shown in formula (7):

[0161] e=z k,j '-z k,j Formula (7)

[0162] In formula (7), z k,j Represents the landmark point y j (refer to Figure 4 A three-dimensional point P in w ) is the reprojected coordinate in image k, z k,j 'Indicates landmark point y j The actual observation coordinates in image k.

[0163] Among them, the landmark point y j The reprojected coordinate z in image k is k,j It can be expressed by the motion equation in formula (6). Therefore, the reprojection error constraint formula of formula (7) and the motion equation of formula (6) can be used to construct the target cost function. For example, the target cost function can be shown in the following formula (8):

[0164]

[0165] G3. Using the loop constraint relationship, solve the target cost function to update the camera pose in the pose graph and the coordinates of the landmark points in the map.

[0166] Taking the target cost function as formula (8) as an example, solving the target cost function is equivalent to solving the camera pose xk , landmark point y j At the same time, adjustments are made. Specifically, the camera pose x k , landmark point y j is the variable to be optimized. Under the condition of satisfying the loop constraint between the target loop frame and the image frame to be detected, the target cost function is minimized (i.e., the reprojection error is minimized). The camera pose x is obtained at this time. k (i.e., optimized camera pose), landmark point y j (i.e., optimized landmark points), use the optimized camera pose to update the original camera pose in the pose graph, and use the optimized landmark points to update the landmark point coordinates in the map, thereby updating the camera pose and map points.

[0167] Please refer to Figure 6 To better understand this embodiment, the following uses the current image frame captured by a mobile device (hereinafter referred to as the "current frame") as an example of an image frame to be detected to illustrate the data processing flow after the image frame is captured in this embodiment, as follows:

[0168] 1> Collect images from mobile devices and input the collected images into the end-to-end model (refer to Figure 6 )

[0169] 2> Use the end-to-end model to match the feature points of the current frame collected from the mobile device with the historical image frame, and obtain the preliminary feature point pairs between the current frame and any historical image frame (reference Figure 6 )

[0170] 3> Perform dynamic obstacle filtering on the initial feature point pairs (refer to Figure 6 )

[0171] 4> Determine whether the current frame is a key frame (refer to Figure 6 Build Keyframes in ).

[0172] 5> Send the key frames and the feature points matched between any two frames to the backend for optimization (refer to Figure 6 (as shown in "Send to backend" and "Backend optimization" in the .

[0173] Among them, the following steps H1 to H3 can be referred to to detect the feature point pairs matching between the current frame and any historical image frame, and the feature point pairs matching between the current frame and the historical image frame are sent to the SLAM backend for backend optimization.

[0174] 6> Start the loop detection thread, use the current frame collected from the mobile device as the image frame to be detected, detect whether the current frame has a loop based on the key frame, and send the loop result to the backend for optimization when a loop is detected in the current frame, so that the backend can use the current frame and the target loop frame to build a loop constraint relationship for optimization. The process of detecting whether the current frame has a loop is as follows:

[0175] 6.1 Get the first disparity map of the current frame.

[0176] 6.2 Traverse each frame image in the key frame set and obtain the second disparity map of the currently traversed key frame.

[0177] 6.3 Detect whether the disparity matching degree between the second disparity map and the first disparity map is greater than a preset disparity matching degree threshold.

[0178] 6.4 If the disparity matching degree between the second disparity map and the first disparity map is greater than the preset disparity matching degree threshold, the current traversal key frame is used as a candidate loop frame. (6.1-6.4 process can refer to Figure 6 (As shown in "Traversing all key frames and performing loop detection" and "Detecting whether loops are generated through disparity maps")

[0179] 6.5 If the disparity matching degree is greater than the preset disparity matching degree threshold, the positioning matching result between the current frame and the current traversal key is obtained.

[0180] 6.6 If the current frame matches the current traversal key position, the current traversal key frame is used as the target loop frame of the image to be detected (the process of 6.5-6.6 can refer to Figure 6 (as shown in "Whether the current frame and loop frame pose match" in the figure).

[0181] By using the disparity map method to detect whether the current frame has a loop, only the feature points of the key frame and the feature point information of the current frame need to be used. There is no need to compare the descriptors of the image frame to detect whether a loop has occurred, thus reducing the process of generating and comparing descriptors.

[0182] 7> Backend optimization. During optimization, first construct the reprojection error constraints and the SLAM-based motion equations, and finally update the current pose and map points through nonlinear optimization methods.

[0183] Back-end optimization includes BA (Bundle Adjustment) optimization and PoseGraph optimization. When performing BA optimization, you can refer to steps G1 to G3 to construct a target cost function for optimization. During the optimization process, the camera extrinsic parameters (i.e., camera pose) and the spatial three-dimensional coordinate system coordinates of three-dimensional space points (i.e., feature points) are used as optimization objects. The camera extrinsic parameters (i.e., camera pose) and the spatial three-dimensional coordinate system coordinates of three-dimensional space points (i.e., feature points) are adjusted through an iterative optimization algorithm to minimize the reprojection error, thereby optimizing the camera pose and optimizing the coordinates of landmark points in the map.

[0184] 8> Use the optimized pose for positioning and mapping. This can improve positioning accuracy and mapping accuracy.

[0185] For example, the following steps H1 to H3 may be used to detect feature point pairs that match between the current frame and any historical image frame, and the feature point pairs that match between the image frame to be detected and the historical image frame are sent to the SLAM backend for backend optimization. Steps H1 to H3 are as follows:

[0186] H1. Through the end-to-end model, feature point matching is performed on the image frame to be detected collected from the mobile device and the historical image frames before the image frame to be detected to obtain preliminary feature point pairs that match the image frame to be detected and the historical image frames.

[0187] Among them, the preliminary feature point pair refers to the feature point pair obtained by matching the feature points of the image frame to be detected with the historical image frame through the end-to-end model.

[0188] The historical image frame refers to the image frame captured before the image frame to be detected.

[0189] An end-to-end model, that is, the input is two images, and the output is a pair of matched feature points on the two images. There are no other intermediate products in the process, that is, no descriptors are generated, and the feature point matching results are generated directly. End-to-end feature point matching uses a single deep learning model to complete the entire process from the original image input to the final feature point matching output. This method is different from the traditional step-by-step method (such as first detecting feature points, then extracting descriptors, and finally matching). End-to-end feature point matching attempts to directly learn the correspondence between feature points between images by training a unified model. In this embodiment, image feature point matching is performed through an end-to-end model to improve the front-end operation efficiency, and based on this, the loop detection problem of the end-to-end model without descriptors is solved.

[0190] For example, taking the historical image frame as the previous image frame of the image frame to be detected, as Figure 7 As shown, the image frame to be detected (reference Figure 7) and the previous image frame to be detected (reference Figure 7 Then, at the SLAM front end, feature matching is performed through the end-to-end model, and the matched feature points on the image frame to be detected are output (refer to Figure 7 The feature points in the image 2 are shown in the figure) and the feature points matched on the previous image frame of the image frame to be detected (reference Figure 7 (as shown in “Feature points in image 1” in the figure).

[0191] H2. Perform dynamic obstacle elimination processing on the preliminary feature point pairs to obtain a first feature point pair that matches the image frame to be detected and the historical image frame.

[0192] The first feature point pair is a feature point pair obtained by removing the dynamic feature point pair from the preliminary feature points.

[0193] Furthermore, in order to avoid the influence of dynamic obstacles, the dynamic feature point pairs corresponding to dynamic objects can be removed from the preliminary feature point pairs through the dynamic obstacle elimination algorithm; thereby, the dynamic feature point pairs corresponding to dynamic objects existing in the feature point pairs matched in the previous and next frames can be eliminated, thereby improving the matching accuracy of the feature point pairs.

[0194] H3. Send the first feature point pair to the synchronous positioning and map construction backend.

[0195] The first feature point pair matched between the image frame to be detected and the historical image frame is used for back-end optimization.

[0196] For example, the following steps J1 to J3 may be referred to determine whether the image frame to be detected is a key frame. Steps J1 to J3 are as follows:

[0197] J1. Obtain a second feature point pair that matches the image frame to be detected and the nearest key frame.

[0198] The most recent key frame refers to a key frame that was acquired before the image frame to be detected and was acquired most recently to the image frame to be detected.

[0199] Specifically, first, the second feature point pairs matching between the image frame to be detected and the nearest key frame can be determined by referring to steps H1 to H2, which will not be repeated here for simplicity. Then, the number of second feature point pairs matching between the image frame to be detected and the nearest key frame is obtained by counting.

[0200] J2. Obtain the number of feature points of the image frame to be detected.

[0201] For example, feature point detection algorithms such as Harris corner detection, FAST corner detection, SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) can be used to detect feature points in the image frame to be detected, and the number of feature points in the image frame to be detected can be counted.

[0202] J3. If the number of the second feature point pairs and the number of the feature points meet a preset number condition, the image frame to be detected is used as a key frame.

[0203] Among them, the preset quantity condition can be set according to the actual business scenario requirements. There is no restriction on the specific setting of the preset quantity condition here. For example, the preset quantity condition can be that the ratio of the number of second feature point pairs N1 matching between the image frame to be detected and the nearest key frame of the image frame to be detected to the number of feature points N2 of the image frame to be detected is less than the preset ratio threshold R (that is: N1 / N2<R).

[0204] From the above content, it can be seen that by obtaining the first disparity map of the image frame to be detected; traversing each frame image in the key frame set to obtain the second disparity map of the currently traversed key frame; if the disparity matching degree between the second disparity map and the first disparity map is greater than the preset disparity matching degree threshold, the currently traversed key frame is used as the target loop frame of the image to be detected; thereby, by comparing the disparity maps, images with higher disparity matching degrees are selected from the key frames as the target loop frame of the image to be detected. On the one hand, since there is no need to generate and compare frame descriptors, the performance overhead required for loop detection can be reduced to a certain extent, and the load required for the front-end to generate descriptors can be reduced. On the other hand, since there is no need to generate and compare frame descriptors for loop detection, the problem of the inability of the descriptor to accurately express the image due to the color and texture changes of the image being easily affected by the ambient light can be avoided, thereby avoiding the problem of inaccurate matching caused by relying on the descriptor to match the image frame to be detected with the loop frame, avoiding the influence of ambient light on the accuracy of loop detection, and thus improving the matching accuracy of the loop frame. Third, the loop detection solution based on this embodiment enables complete front-end independence, making it easier to separate the front and back ends. Many current development boards use a motherboard + external NPU solution, making such boards more feasible for current SLAM system deployment. Fourth, the loop detection and pose optimization solution of this embodiment does not affect processes such as dynamic feature point culling and noise removal, making the solution highly scalable.

[0205] Those skilled in the art will appreciate that all or part of the steps in the above loop detection method can be completed by instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0206] To this end, an embodiment of the present application provides a computer-readable storage medium storing a plurality of computer programs, which can be loaded by a processor to execute any of the loop detection methods provided in the embodiments of the present application. For example, the computer program can be loaded by a processor to execute the following steps:

[0207] Obtain a set of key frames collected before the image frame to be detected; obtain a first disparity map of the image frame to be detected; traverse each frame image in the key frame set to obtain a second disparity map of the currently traversed key frame; if the disparity match between the second disparity map and the first disparity map is greater than a preset disparity match threshold, use the currently traversed key frame as the target loop frame of the image to be detected.

[0208] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0209] In the above-mentioned loop detection method, computer-readable storage medium, and self-mobile device embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a particular embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, the specific working processes and beneficial effects of the computer-readable storage medium, self-mobile device, and their corresponding units described above can be referred to the description of the loop detection method in the above embodiments, and the details will not be repeated here.

[0210] The above is a detailed introduction to a loop detection method, a mobile device, and a computer-readable storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. At the same time, for those skilled in the art, based on the ideas of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A loop detection method, characterized in that: The method comprises: Obtain a set of key frames collected before the image frame to be detected; Acquire a first disparity map of the image frame to be detected; Traversing each frame image in the key frame set to obtain a second disparity map of the currently traversed key frame; If the disparity matching degree between the second disparity map and the first disparity map is greater than a preset disparity matching degree threshold, the current traversal key frame is used as a target loop frame of the image to be detected.

2. The loop detection method according to claim 1, wherein: The obtaining of the first disparity map of the image frame to be detected includes: Obtaining a left camera view and a right camera view of the image frame to be detected; Obtaining a first matching pixel pair between the left camera view and the right camera view, wherein the first matching pixel pair includes a first pixel in the left camera view and a second pixel in the right camera view that matches the first pixel; The disparity value between the first pixel point and the second pixel point is stored in a corresponding position of the disparity map to obtain a first disparity map of the image frame to be detected.

3. The loop detection method according to claim 1, wherein: Before using the current traversal key frame as the target loop frame of the image to be detected, the method further includes: Obtaining a positioning matching result between the image frame to be detected and the currently traversed key frame; If the disparity matching degree between the second disparity map and the first disparity map is greater than a preset disparity matching degree threshold, the currently traversed key frame is used as the target loop frame of the image to be detected, including: If the disparity matching degree between the second disparity map and the first disparity map is greater than a preset disparity matching degree threshold, and the positioning matching result is a positioning match, the current traversal key frame is used as the target loop frame of the image to be detected.

4. The loop detection method according to claim 3, wherein: The obtaining of the positioning matching result between the image frame to be detected and the currently traversed key frame includes: Acquire a pair of target feature points between the image frame to be detected and the currently traversed key frame, wherein each pair of target feature points includes a first feature point mapped by a target three-dimensional space point in the image frame to be detected and a second feature point mapped in the currently traversed key frame; Obtaining the essential matrix between the image frame to be detected and the currently traversed key frame; Based on the essential matrix, the observed coordinates of the first feature point, and the observed coordinates of the second feature point, a positioning matching result between the image frame to be detected and the currently traversed key frame is determined.

5. The loop detection method according to claim 4, wherein: The determining, based on the essential matrix, the observed coordinates of the first feature point, and the observed coordinates of the second feature point, a positioning matching result between the image frame to be detected and the currently traversed key frame includes: Substituting the essential matrix, the observation coordinates of the first feature point, and the observation coordinates of the second feature point into a preset epipolar constraint relationship to obtain a constraint result value; If the constraint result value is within the preset value range, the positioning match is taken as the positioning match result; Alternatively, if the constraint result value is outside the preset value range, a positioning mismatch is taken as the positioning matching result.

6. The loop detection method according to claim 3, wherein: The obtaining of the positioning matching result between the image frame to be detected and the currently traversed key frame includes: Acquire a pair of target feature points between the image frame to be detected and the currently traversed key frame, wherein each pair of target feature points includes a first feature point mapped by a target three-dimensional space point in the image frame to be detected and a second feature point mapped in the currently traversed key frame; Based on the target feature point pair, construct a three-dimensional transformation relationship between the feature points of the image frame to be detected and the currently traversed key frame; If the solution of the three-dimensional transformation relationship of the feature points converges, the positioning matching is taken as the positioning matching result; Alternatively, if the solution of the three-dimensional transformation relationship of the feature points diverges, a positioning mismatch is taken as the positioning matching result.

7. The loop detection method according to claim 6, wherein: The constructing of a three-dimensional transformation relationship between the feature points of the image frame to be detected and the currently traversed key frame based on the target feature point pair includes: Based on the observed coordinates of the first feature point and the three-dimensional coordinates corresponding to the second feature point, obtaining the three-dimensional coordinates of the first feature point in the camera coordinate system; Based on the three-dimensional coordinates of the first feature point in the camera coordinate system and the spatial three-dimensional coordinates corresponding to the second feature point, a three-dimensional transformation relationship between the feature points of the image frame to be detected and the currently traversed key frame is constructed.

8. The loop detection method according to claim 6, wherein: The constructing of a three-dimensional transformation relationship between the feature points of the image frame to be detected and the currently traversed key frame based on the target feature point pair includes: Based on the three-dimensional coordinates of the first feature point in the camera coordinate system and the three-dimensional coordinates of the second feature point in the camera coordinate system, a three-dimensional transformation relationship between the feature points of the image frame to be detected and the currently traversed key frame is constructed.

9. The loop detection method according to claim 1, wherein: The method further comprises: Using an end-to-end model, feature point matching is performed on the image frame to be detected collected from the mobile device and the historical image frames before the image frame to be detected, thereby obtaining preliminary feature point pairs that match between the image frame to be detected and the historical image frames; Performing dynamic obstacle elimination processing on the preliminary feature point pairs to obtain a first feature point pair that matches the image frame to be detected and the historical image frame; The first feature point pair is sent to a synchronous positioning and mapping backend, wherein the first feature point pair is used for backend optimization.

10. The loop detection method according to any one of claims 1 to 9, characterized in that: The image frame to be detected is collected by a mobile device, and the method further includes: Constructing a loop constraint relationship based on the target loop frame and the image frame to be detected; Constructing a target cost function based on the motion equation of the self-moving device and the reprojection error constraint; The target cost function is solved using the loop constraint relationship to update the camera pose in the pose graph and the coordinates of the landmark points in the map.

11. A self-propelled device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the loop detection method according to any one of claims 1 to 10 is executed.

12. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is loaded by a processor to execute the loop detection method according to any one of claims 1 to 10.

Images