Loopback detection method and computer device

Abstract

The application relates to a loop detection method and a computer device. The method comprises the following steps: acquiring a plurality of image frames collected by a robot, determining a last image frame and a first preset number of first preceding image frames of the last image frame in the plurality of image frames; determining a feature matching degree between each first preceding image frame and the last image frame, and determining a first target image frame corresponding to the last image frame in the first preset number of first preceding image frames; determining a second target image frame corresponding to the first target image frame corresponding to the last image frame in the plurality of image frames; and performing loop detection in the plurality of image frames based on a feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame to determine at least one loop frame of the first target image frame corresponding to the last image frame. The method can improve the accuracy of loop detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to a loop detection method and a computer device. BACKGROUND

[0002] With the rapid development of robot technology, visual simultaneous localization and mapping (VSLAM) technology is increasingly applied in commercial service robots, and consequently, the challenges encountered by VSLAM technology in the application process are also increasing. In order to enable the robot to construct an accurate and low-error visual map, visual loop detection is essential.

[0003] In an indoor running scene, the visual camera equipped by the commercial service robot is generally a monocular camera with a vertical field of view upward, for obtaining a ceiling image above the robot. In some complex environments, such as the ceiling environment of a large factory or an office building, the ceiling environment usually has high similarity. Therefore, if the loop pose cannot be accurately detected or a wrong loop pose is detected in these complex environments, the robot will construct a map with considerable error, which will undoubtedly cause serious adverse effects on the subsequent business of the commercial service robot. SUMMARY

[0004] Therefore, it is necessary to provide a loop detection method, device, computer device, computer readable storage medium and computer program product capable of improving the accuracy of loop detection, in view of the above technical problems.

[0005] In a first aspect, the present application provides a loop detection method, comprising:

[0006] obtaining a plurality of image frames collected by a robot, and determining a last image frame and a first preset number of first preceding image frames of the last image frame in the plurality of image frames;

[0007] determining a feature matching degree between each first preceding image frame and the last image frame, and determining a first target image frame corresponding to the last image frame in the first preset number of first preceding image frames; the first target image frame corresponding to the last image frame is an image frame having the highest feature matching degree between the first preset number of first preceding image frames and the last image frame;

[0008] In the plurality of image frames, a first target image frame corresponding to the last image frame and a second target image frame corresponding to the first target image frame are determined; the second target image frame corresponding to the last image frame is an image frame in the plurality of image frames, a distance between which and the first target image frame corresponding to the last image frame satisfies a first distance condition, and between which and the first target image frame corresponding to the last image frame has the highest feature matching degree;

[0009] Based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed in the plurality of image frames to determine at least one loop frame corresponding to the first target image frame corresponding to the last image frame.

[0010] In a second aspect, the present application further provides a loop detection device, comprising:

[0011] An acquisition module is configured to acquire a plurality of image frames collected by a robot, and determine a last image frame and a first preset number of first preceding image frames of the last image frame in the plurality of image frames;

[0012] A first determination module is configured to determine a feature matching degree between each first preceding image frame and the last image frame, and determine a first target image frame corresponding to the last image frame in the first preset number of first preceding image frames; the first target image frame corresponding to the last image frame is an image frame in the first preset number of first preceding image frames, between which and the last image frame has the highest feature matching degree;

[0013] A second determination module is configured to determine, in the plurality of image frames, a second target image frame corresponding to the first target image frame corresponding to the last image frame; the second target image frame corresponding to the last image frame is an image frame in the plurality of image frames, a distance between which and the first target image frame corresponding to the last image frame satisfies a first distance condition, and between which and the first target image frame corresponding to the last image frame has the highest feature matching degree;

[0014] A detection module is configured to perform loop detection in the plurality of image frames through the second target image frame based on the feature matching degree between the first target image frame and the second target image frame, to obtain a loop pose; based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed in the plurality of image frames to determine at least one loop frame corresponding to the first target image frame corresponding to the last image frame.

[0015] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize part or all steps described in any method of the first aspect of the present application.

[0016] In a fourth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements some or all of the steps described in any method of the first aspect of the embodiments of the present application.

[0017] In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program, and the computer program, when executed by a processor, implements some or all of the steps described in any method of the first aspect of the embodiments of the present application.

[0018] The above loop detection method, device, computer device, computer readable storage medium and computer program product acquire a plurality of image frames collected by a robot, and determine a last image frame and a first preset number of first preceding image frames of the last image frame in the plurality of image frames. The feature matching degrees between each first preceding image frame and the last image frame are determined, and a first target image frame corresponding to the last image frame is determined in the first preset number of first preceding image frames. The first target image frame corresponding to the last image frame is an image frame having the highest feature matching degree between the first preset number of first preceding image frames and the last image frame. In the plurality of image frames, a second target image frame corresponding to the first target image frame corresponding to the last image frame is determined. The second target image frame corresponding to the last image frame is an image frame having the highest feature matching degree between the first target image frame corresponding to the last image frame and the last image frame, and satisfying a first distance condition with the first target image frame corresponding to the last image frame. Based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed in the plurality of image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame. By determining the first target image frame corresponding to the last image frame having the highest feature matching degree between the first preceding image frames and the last image frame, and then determining the second target image frame corresponding to the last image frame having the highest feature matching degree between the first target image frame and the last image frame and satisfying the first distance condition with the first target image frame in the plurality of image frames, the loop detection method provided by the present application performs accurate loop detection on the first target image frame corresponding to the last image frame in the plurality of image frames to obtain a loop frame with high accuracy, which can significantly improve the accuracy of loop detection, and further ensure the accuracy of map construction when the loop detection has high accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0020] Figure 1 A flowchart of a loopback detection method in an embodiment;

[0021] Figure 2 A block diagram of a loopback detection device in an embodiment;

[0022] Figure 3 An internal structure diagram of a computer device in an embodiment;

[0023] Figure 4 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0025] An embodiment of the present application provides a loopback detection method, which is used in a computer device for example, and the computer device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is realized through the interaction of the terminal and the server. The terminal can be, but is not limited to, various robots, unmanned vehicles, automatic food delivery vehicles, personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The robot can be a cleaning robot, a delivery robot, a logistics robot, an inspection robot and a disinfection robot, etc. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0026] In an exemplary embodiment, as Figure 1As shown, a loop detection method is provided, which is described by taking a robot as an example, and includes the following steps 102 to 108. Among them:

[0027] Step 102, obtaining a plurality of image frames collected by the robot, and determining a last image frame and a first preset number of first preceding image frames of the last image frame in the plurality of image frames.

[0028] Among them, the robot is provided with an image acquisition device, and a map and positioning are constructed by images obtained by the image acquisition device. The image acquisition device can acquire images of at least any one position above, beside, in front of, behind, and on the ground of the robot.

[0029] An image frame is a single image in a video stream or image sequence collected by the robot. In VSLAM technology, an image frame is a key data unit for constructing an environment map and a robot path. As can be easily understood, each image frame corresponds to a collection time node and an initial pose. The initial pose of each image frame can be obtained by a wheeled odometer, or by other sensors capable of being installed on the robot and obtaining the pose of the robot, or by fusing multiple sensors.

[0030] In an exemplary embodiment, the above-mentioned obtaining a plurality of image frames collected by the robot includes: obtaining a plurality of image frames collected by the robot through an image acquisition device; wherein the lens of the image acquisition device faces upward, and the image frame is an image above the environment in which the robot is located; when the robot is in an indoor scene, the image frame can be a ceiling image; when the robot is in an outdoor scene, the image frame can be an image of the sky, buildings or trees above the robot.

[0031] Optionally, the number of the plurality of image frames can be 5, 10, 20 or other numbers, which is set according to actual algorithm calculation requirements and is not specifically limited here.

[0032] Optionally, the image acquisition device can be a monocular camera, a binocular camera or a depth camera, etc.

[0033] Optionally, the robot can be, but is not limited to, various industrial robots (such as carrying robots, palletizing robots, spraying robots, etc.) that need to move autonomously for map construction, service robots (such as cleaning robots, delivery robots, mowing robots, guiding robots, building robots, etc.) or special robots (firefighting robots, underwater robots, security robots, etc.).

[0034] The last image frame refers to one image frame in the plurality of image frames, the collection time node of which is located at the end. In other words, the last image frame is the last image frame collected by the robot in the plurality of image frames. The first presequence image frame refers to the first preset number of image frames in front of the last image frame in the plurality of image frames. As can be easily understood, the collection time node of the first presequence image frame is located at the end of the last image frame in the collection time node. For example, the robot collects 100 image frames, the last image frame is the 100th image frame collected by the robot, and assuming that the first preset number is 10, the first presequence image frame is the 90th image frame, the 91st image frame,..., and the 99th image frame collected by the robot.

[0035] Optionally, the first preset number can be 5, 10, 15, or other numbers.

[0036] In step 104, the feature matching degrees between the first presequence image frames and the last image frame are determined, and the first target image frame corresponding to the last image frame is determined in the first preset number of first presequence image frames. The first target image frame corresponding to the last image frame is the image frame with the highest feature matching degree between the first preset number of first presequence image frames and the last image frame.

[0037] The first target image frame corresponding to the last image frame has the highest feature matching degree between the first preset number of first presequence image frames and the last image frame, that is, the first target image frame is the image frame with the highest similarity in visual features in the first preset number of first presequence image frames and the last image frame, which means that the first target image frame corresponding to the last image frame and the last image frame can be collected by the robot at the same or similar position.

[0038] Optionally, in order to ensure the accuracy of the loop detection of the robot, while ensuring that the first target image frame corresponding to the last image frame is the image frame with the highest feature matching degree between the first preset number of first presequence image frames and the last image frame, it can be further ensured that the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame satisfies the first matching degree condition. The first matching degree condition can be that the feature matching degree is greater than or equal to 50%, or greater than 60%, etc., which can be set according to actual needs.

[0039] In one exemplary embodiment, the above determination of the feature matching degrees between the first presequence image frames and the last image frame includes: extracting feature points of each image frame, and matching each image frame based on the feature points of each image frame to determine the feature matching relationship between each image frame; based on the feature matching relationship between each image frame, the feature matching degrees between the first presequence image frames and the last image frame are determined.

[0040] The feature matching relationship between each image frame includes a feature matching relationship between any two image frames in the plurality of image frames, and represents a feature point correspondence relationship between each image frame. Thus, the feature matching relationship between any two image frames in the plurality of image frames can be determined based on the feature matching relationship between each image frame.

[0041] Alternatively, the feature points of each image frame can be Oriented FAST and Rotated BRIEF (ORB) feature points, Scale-Invariant Feature Transform (SIFT) feature points, or other deep learning feature points.

[0042] In step 106, among the plurality of image frames, a first target image frame corresponding to the last image frame and a second target image frame corresponding to the first target image frame are determined. The second target image frame corresponding to the last image frame is an image frame that satisfies a first distance condition with the first target image frame corresponding to the last image frame and has the highest feature matching degree with the first target image frame corresponding to the last image frame.

[0043] The first distance condition can be a distance within a square circle of 3 m, 4 m, 5 m, 6 m, or other distances. For example, when the first distance condition is a distance within a square circle of 5 m, the second target image frame corresponding to the last image frame is a part of the image frames in the plurality of image frames that are within a square circle of 5 m from the first target image frame corresponding to the last image frame.

[0044] The second target image frame corresponding to the last image frame satisfies the first distance condition with the first target image frame corresponding to the last image frame and has the highest feature matching degree with the first target image frame corresponding to the last image frame, that is, the second target image frame corresponding to the last image frame is an image frame in the plurality of image frames that is close in distance to the first target image frame corresponding to the last image frame and has the highest similarity in visual features with the first target image frame corresponding to the last image frame, which means that the second target image frame corresponding to the last image frame and the first target image frame corresponding to the last image frame are likely to be obtained by the robot at the same or similar position. Further, since the first target image frame corresponding to the last image frame has the highest feature matching degree with the last image frame representing the current position of the robot, the second target image frame corresponding to the last image frame is an image frame of a historical position that is likely to have the highest similarity with the current position of the robot.

[0045] Specifically, the feature matching degree between the second target image frame corresponding to the last image frame and the first target image frame corresponding to the last image frame can also be determined based on the feature matching relationship between the image frames.

[0046] At step 108, based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed in the plurality of image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame.

[0047] Wherein, loop detection, also known as closed loop detection, refers to the ability of the robot to recognize that it has reached a certain scene, so as to close the loop of the map. In simple terms, it means that the robot can realize that a certain place is "I" once visited when building a map by turning left and right. Then match the map generated at this moment with the map generated just now. Loop detection is a difficult point because: if loop detection is successful, it can significantly reduce cumulative error and help the robot to more accurately and quickly avoid obstacles and navigate. False detection results may make the map very bad, affecting the subsequent obstacle avoidance and navigation of the robot. Therefore, loop detection is very necessary for large-area and large-scene map construction.

[0048] Optionally, the at least one loop frame of the first target image frame corresponding to the last image frame obtained by performing loop detection can be an image frame that meets the loop detection condition, or a plurality of image frames that meet the loop detection condition.

[0049] Specifically, in the case of obtaining a loop frame, it indicates that the robot has returned to the position visited before, at this time, pose optimization can be performed on the plurality of image frames, so that the robot can construct a map with higher accuracy. It is easy to understand that if there is actually no loop frame in the plurality of image frames, but it is mistakenly thought that there is a loop frame, in this false case, if pose optimization is performed on the plurality of image frames, it will cause the map constructed by the robot to have a large error, therefore, the robot needs to ensure that the loop detection has accuracy to avoid such undesirable situation that the constructed map does not have accuracy.

[0050] It is easy to understand that in the case of finding at least one loop frame in the plurality of image frames, the pose of the first target image frame corresponding to the last image frame at this time is a loop pose.

[0051] In the loop detection method, a plurality of image frames collected by the robot are obtained, and a last image frame and a first preset number of first preceding image frames of the last image frame are determined in the plurality of image frames; a feature matching degree between each first preceding image frame and the last image frame is determined, and a first target image frame corresponding to the last image frame is determined in the first preset number of first preceding image frames; the first target image frame corresponding to the last image frame is an image frame having the highest feature matching degree between the first preset number of first preceding image frames and the last image frame; a second target image frame corresponding to the first target image frame corresponding to the last image frame is determined in the plurality of image frames; the second target image frame corresponding to the last image frame is an image frame having the highest feature matching degree between the first target image frame corresponding to the last image frame and the last image frame in the plurality of image frames, and satisfying a first distance condition; based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed in the plurality of image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame. By determining the first target image frame corresponding to the last image frame having the highest feature matching degree in the first preceding image frames, and then determining the second target image frame corresponding to the last image frame having the highest feature matching degree and satisfying the first distance condition between the first target image frame in the plurality of image frames, the loop detection is performed in the plurality of image frames based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame to determine at least one loop frame of the first target image frame corresponding to the last image frame. Obviously, the loop detection method provided in the embodiment can accurately detect the loop in the plurality of image frames based on the first target image frame corresponding to the last image frame to obtain a loop frame with high accuracy, which can significantly improve the accuracy of loop detection, and further ensure the accuracy of map construction when the loop detection has high accuracy.

[0052] In one exemplary embodiment, in a case where at least one loop frame of the first target image frame corresponding to the last image frame is determined, the method further comprises:

[0053] Based on the initial pose of each loop frame, the calculated pose of each loop frame, and the first target function, the initial pose of each image frame is optimized to determine a first optimized pose of each image frame.

[0054] In an example embodiment, the method further comprises: determining the calculated poses of the image frames based on the feature matching relationships between the image frames and the initial poses of the image frames; and determining the calculated poses of the looped image frames among the calculated poses of the image frames after determining the at least one looped image frame corresponding to the first target image frame of the last image frame.

[0055] The calculated poses of the image frames can be used for pose optimization of the image frames to enable the robot to construct a map with higher accuracy. By performing pose optimization of the image frames, the accumulated error in the map construction process can be reduced, and thus the consistency between the entire motion trajectory of the robot and the actual map can be improved.

[0056] The first optimized poses of the image frames can be used for optimization of the constructed map to reduce the accumulated error in the map construction process, and thus to construct a map with higher accuracy and higher consistency with the actual map.

[0057] In an example embodiment, the determining of the calculated poses of the image frames based on the feature matching relationships between the image frames and the initial poses of the image frames comprises: determining three-dimensional coordinates of the image frames based on the feature matching relationships between the image frames and the initial poses of the image frames; and determining the calculated poses of the image frames based on the feature matching relationships between the image frames and the three-dimensional coordinates of the image frames.

[0058] For example, the determination of the calculated pose of the last image frame is described. First, the three-dimensional coordinates of the first target image frame of the last image frame can be calculated based on the feature matching relationships between the first target image frame of the last image frame and other image frames and the initial poses of the image frames. Then, the calculated pose of the last image frame can be calculated based on the feature matching relationships between the last image frame and the first target image frame of the last image frame and the three-dimensional coordinates of the first target image frame of the last image frame. The calculation principle of the calculated poses of other image frames is the same, and thus is not described herein.

[0059] Optionally, the calculated poses of the image frames can be determined based on the feature matching relationships between the image frames and the three-dimensional coordinates of the image frames by using a Perspective-n-Point (PnP) algorithm. Specifically, in the VSLAM technology, the PnP algorithm is usually used to recover the pose of a robot in a two-dimensional image frame from the correspondence between a set of matched feature points of the two-dimensional image frame and the corresponding feature points in a three-dimensional space.

[0060] Specifically, the first optimized pose of each image frame is a pose that makes the first objective function reach a minimum value, and is solved based on the initial pose of each loop frame, the calculated pose of each loop frame and the first objective function.

[0061] For example, the number of the plurality of image frames is denoted as n, and the first optimized pose of each image frame is denoted as T i The initial pose of each loop frame is denoted as T k The calculated pose of each loop frame based on the initial pose is denoted as T cal_k The first objective function can be wherein, , .

[0062] In an embodiment, the initial pose of each loop frame is input into the first objective function, and the initial pose that makes the first objective function reach a minimum value is the first optimized pose of each image frame.

[0063] In this embodiment, in the case that at least one loop frame corresponding to the first target image frame of the last image frame is obtained, the first optimized pose of each image frame is determined based on the initial pose of each loop frame, the calculated pose of each loop frame and the first objective function. Therefore, based on the accurate loop frame, the accuracy of the first optimized pose of each image frame determined can be improved, and further, the accuracy of the map construction can be further ensured.

[0064] In an exemplary embodiment, after the first target image frame corresponding to the last image frame is determined in the first preset number of first preceding image frames, the method further comprises:

[0065] The initial pose of each image frame is optimized based on the initial pose of the last image frame, the calculated pose of the last image frame and the second objective function, to determine a second optimized pose of each image frame.

[0066] In an exemplary embodiment, the determination of the calculated pose of each image frame based on the feature matching relationship between each image frame and the initial pose corresponding to each image frame comprises: determining the calculated pose of each image frame based on the feature matching relationship between each image frame and the second optimized pose of each image frame.

[0067] The second optimized pose of each image frame is an optimized pose of each image frame obtained by performing trajectory drift correction on each image frame by the last image frame, so that the loop pose can be accurately determined in the subsequent loop detection process.

[0068] Specifically, the second optimized pose of each image frame is respectively a pose of each image frame that can be solved based on the initial pose of the last image frame, the calculated pose of the last image frame and the second target function, and makes the second target function reach a minimum value.

[0069] Exemplarily, the second optimized pose of each image frame is represented as T k The initial pose of the last image frame is represented as T end The calculated pose of the last image frame is represented as T cal_end The second target function can be Wherein, , .

[0070] In an exemplary embodiment, the initial pose of each image frame is input into the second target function, and the initial pose that makes the second target function reach a minimum value is the second optimized pose of each image frame.

[0071] In an exemplary embodiment, in a case that the second optimized pose of each image frame is optimized based on the second optimized pose of each loop-back frame, the calculated pose of each loop-back frame and the first target function, the first optimized pose of each image frame is determined, and the calculated pose of each loop-back frame is calculated based on the second optimized pose of each image frame, the calculated pose of each image frame calculated based on the second optimized pose is represented as T cal_k The first target function is Wherein, , .

[0072] In the embodiment, the second optimized pose of each image frame is determined based on the initial pose of the last image frame, the calculated pose of the last image frame and the second target function, and the initial correction of the trajectory drift of each image frame is performed by the last image frame, so that the calculated pose of each image frame determined subsequently and the first optimized pose of each image frame determined subsequently are obtained based on the second optimized pose of each image frame that has a higher accuracy after the initial correction of the trajectory drift, and further, the accuracy of the loop-back detection can be further improved.

[0073] In an exemplary embodiment, the above method further comprises:

[0074] In a case that the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame satisfies a first matching degree condition, the three-dimensional coordinates of the first target image frame corresponding to the last image frame are determined based on the feature matching relationship between the first target image frame corresponding to the last image frame and each image frame and the initial pose of each image frame.

[0075] The calculated pose of the last image frame is determined based on a feature matching relationship between the first target image frame corresponding to the last image frame and each image frame, and a three-dimensional coordinate of the first target image frame corresponding to the last image frame.

[0076] The three-dimensional coordinate of the first target image frame corresponding to the last image frame can be determined by a triangulation method or other three-dimensional reconstruction technology based on the feature matching relationship between the first target image frame corresponding to the last image frame and each image frame, and the initial pose of each image frame.

[0077] The calculated pose of the last image frame can be determined by a PnP algorithm based on the feature matching relationship between the first target image frame corresponding to the last image frame and each image frame, and the three-dimensional coordinate of the first target image frame corresponding to the last image frame.

[0078] In this embodiment, the calculated pose of the last image frame is determined based on the corresponding relationship between the two-dimensional feature matching relationship and the three-dimensional coordinate, so as to ensure the accuracy of the calculated pose of the last image frame, and further ensure that the second optimized pose of each image frame also has high accuracy, thereby further improving the accuracy of loop detection.

[0079] In an exemplary embodiment, in a case where the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not satisfy the first matching degree condition, the initial pose of the first target image frame corresponding to the last image frame is determined as the calculated pose of the last image frame.

[0080] In this embodiment, in a case where the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not satisfy the first matching degree condition, it is difficult to accurately determine the calculated pose of the last image frame by calculation at this time, and since the initial pose and the final pose of the robot are usually ensured to be the same during manual intervention or manual operation, the initial pose of the first target image frame corresponding to the last image frame is directly determined as the calculated pose of the last image frame at this time, so as to avoid calculating an incorrect calculated pose and further causing the map constructed by the robot to have a large error.

[0081] In an exemplary embodiment, the above determining, in the plurality of image frames, the second target image frame corresponding to the first target image frame corresponding to the last image frame comprises:

[0082] In the plurality of image frames, at least one peripheral image frame is determined, which satisfies a first distance condition with the first target image frame corresponding to the last image frame.

[0083] determining a second target image frame corresponding to the last image frame in the at least one peripheral image frame with the highest feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame;

[0084] wherein the collection time node of each peripheral image frame is earlier than the collection time node of the first target image frame corresponding to the last image frame, and in the case that the direction of the peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame, the time difference between the collection time node of the peripheral image frame and the collection time node of the first target image frame corresponding to the last image frame satisfies a preset time condition.

[0085] Specifically, the collection time node of each peripheral image frame is earlier than the collection time node of the first target image frame corresponding to the last image frame, that is, each peripheral image frame has been collected before the first target image frame corresponding to the last image frame is collected.

[0086] Optionally, the preset time condition can be that the time difference is greater than or equal to 5 seconds, 10 seconds, 15 seconds or other time difference.

[0087] Optionally, whether the direction of each peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame can be determined by respectively performing pose matching between the calculated pose of each peripheral image frame and the calculated pose of the first target image frame corresponding to the last image frame.

[0088] Specifically, in order to perform loop detection in multiple image frames, therefore, the collection time node of the at least one peripheral image frame used to determine the second target image frame corresponding to the last image frame should be earlier than the collection time node of the first target image frame corresponding to the last image frame, and in order to avoid consuming excessive computing power when performing loop detection, there is a certain collection time difference between the peripheral image frame and the first target image frame corresponding to the last image frame in the case that the direction of the peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame. Obviously, through the limitation of the collection time node of the image frame, while ensuring that the second target image frame corresponding to the last image frame determined by the at least one peripheral image frame has accuracy, it also ensures that the subsequent loop detection has sufficient efficiency.

[0089] In an example embodiment, the second target image frame corresponding to the last image frame and having the highest feature matching degree between the first target image frame corresponding to the last image frame and the at least one peripheral image frame is determined by: rotating the first target image frame corresponding to the last image frame to the same direction of each peripheral image frame respectively to obtain a plurality of rotated first target image frames, the rotated first target image frames corresponding to the peripheral image frames one by one; determining the feature matching relationship between each rotated first target image frame and each peripheral image frame respectively; and determining the second target image frame corresponding to the last image frame and having the highest feature matching degree between the first target image frame corresponding to the last image frame and the at least one peripheral image frame based on the feature matching relationship between each rotated first target image frame and each peripheral image frame.

[0090] In the embodiment, the at least one peripheral image frame having a smaller distance between the first target image frame corresponding to the last image frame is determined in the plurality of image frames, so that the at least one peripheral image frame is an image frame in which the candidate loop position may exist, then the second target image frame corresponding to the last image frame and having the highest feature matching degree between the first target image frame corresponding to the last image frame and the at least one peripheral image frame is determined in the at least one peripheral image frame, so that the second target image frame corresponding to the last image frame is an image frame in which the historical position having the highest similarity with the current position of the robot may exist, further, the loop position is determined by performing loop detection on the plurality of image frames through the second target image frame based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, obviously, the loop detection method provided by the embodiment can accurately perform loop detection on the plurality of image frames through the first target image frame to obtain a loop position with higher accuracy, and further, the accuracy of loop detection can be significantly improved; at the same time, since there is a limitation condition on the time node of collecting the peripheral image frame, the accuracy of loop detection can be improved while the efficiency of loop detection can also be improved.

[0091] In an example embodiment, the loop position is determined by performing loop detection on the plurality of image frames through the second target image frame based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, and the at least one loop frame of the first target image frame corresponding to the last image frame is determined.

[0092] In a case where the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the second matching degree condition,

[0093] the first feature matching degree between the last image frame of the first target image frame and the last image frame of the second target image frame is determined,

[0094] determining a first relative position difference between the first target image frame corresponding to the last image frame and a previous image frame of the first target image frame corresponding to the last image frame,

[0095] determining a first pose difference between the second target image frame corresponding to the last image frame and a previous image frame of the second target image frame corresponding to the last image frame,

[0096] and determining a second pose difference between the previous image frame of the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame;

[0097] When at least one of the following conditions is met: the first feature matching degree meets the third matching degree condition, the distance difference value between the first relative position difference and the first pose difference meets the second distance condition, and the distance difference value between the first relative position difference and the second pose difference meets the third distance condition, it is determined that the loop frame of the first target image frame corresponding to the last image frame is the second target image frame corresponding to the last image frame.

[0098] The second matching degree condition can be that the feature matching degree is greater than or equal to 50%, 60%, 80%, or another feature matching degree.

[0099] Optionally, the third matching degree condition can be that the feature matching degree is greater than or equal to 50%, 60%, 80%, or another feature matching degree. Optionally, the second matching degree condition can be the same as the third matching degree condition.

[0100] Optionally, the second distance condition can be that the distance difference value is less than 15 cm, 20 cm, 25 cm, or another distance difference value.

[0101] Optionally, the third distance condition can be that the distance difference value is less than 15 cm, 20 cm, 25 cm, or another distance difference value. Optionally, the second distance condition can be the same as the third distance condition.

[0102] The previous image frame of the first target image frame corresponding to the last image frame is an image frame whose acquisition time node is the previous acquisition time node of the first target image frame corresponding to the last image frame. That is, among the multiple image frames with consecutive acquisition time nodes, the previous image frame of the first target image frame corresponding to the last image frame is an image frame adjacent to the first target image frame corresponding to the last image frame and whose acquisition time node is earlier than that of the first target image frame corresponding to the last image frame.

[0103] The last image frame corresponds to the last image frame of the second target image frame, that is, the image frame of the last image frame of the second target image frame at the last image frame of the second target image frame. That is, in the plurality of image frames of the acquisition time node, the last image frame of the second target image frame of the last image frame of the second target image frame is adjacent to the last image frame of the second target image frame and the acquisition time node is earlier than the last image frame of the second target image frame.

[0104] Specifically, the first relative position difference between the last image frame corresponding to the first target image frame and the last image frame corresponding to the first target image frame of the last image frame is determined by the straight line distance between the position of the last image frame corresponding to the first target image frame and the position of the last image frame corresponding to the first target image frame of the last image frame.

[0105] Specifically, the first pose difference between the last image frame corresponding to the second target image frame and the last image frame corresponding to the second target image frame of the last image frame is determined by the relative change between the pose of the last image frame corresponding to the second target image frame and the pose of the last image frame corresponding to the second target image frame of the last image frame. The second pose difference between the last image frame corresponding to the first target image frame and the last image frame corresponding to the second target image frame is the same as the first pose difference, so it is not described here.

[0106] Specifically, in the VSLAM technology, the position refers to a specific point of the robot in the three-dimensional space, and the position does not involve the direction information of the robot but only involves the translation information of the robot, that is, the position describes the linear movement of the robot in the three-dimensional space, and does not involve any rotation or direction change. In the VSLAM technology, the pose refers to the combination of the position of the robot and the attitude of the robot, and the attitude refers to the direction or orientation of the robot, so the pose not only describes the specific point of the robot in the three-dimensional space, but also describes the direction or orientation of the robot, that is, the pose involves the direction information and translation information of the robot.

[0107] Specifically, at least one of the following conditions is satisfied: the first feature matching degree satisfies the third matching condition, the distance difference between the first relative position difference and the first pose difference satisfies the second distance condition, and the distance difference between the first relative position difference and the second pose difference satisfies the third distance condition. It can mean any one of the three conditions, it can mean any two of the three conditions, and it can mean all of the three conditions.

[0108] It can be understood that the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the second matching degree condition, which indicates that the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame are likely to be collected at similar or same positions. In order to more accurately perform loopback judgment, it can be further judged whether the first feature matching degree satisfies a third matching degree condition, and / or whether the distance difference between the first relative position difference and the first pose difference satisfies a second distance condition, and / or whether the distance difference between the first relative position difference and the second pose difference satisfies a third distance condition.

[0109] In a specific embodiment, the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the second matching degree condition, which indicates that the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame are likely to be collected at similar or same positions. In order to further confirm whether the second target image frame corresponding to the last image frame forms a loopback with the first target image frame corresponding to the last image frame, that is, in order to further confirm whether the loopback frame of the first target image frame corresponding to the last image frame is the second target image frame corresponding to the last image frame, it is necessary to confirm whether the last image frame corresponding to the first target image frame and the last image frame corresponding to the second target image frame are also collected at similar or same positions, and to confirm whether the first target image frame corresponding to the last image frame and the last image frame corresponding to the first target image frame, the second target image frame corresponding to the last image frame and the last image frame corresponding to the second target image frame, and the last image frame corresponding to the first target image frame and the last image frame corresponding to the second target image frame also have certain motion continuity and motion similarity. Therefore, when the first feature matching degree satisfies the third matching degree condition, the distance difference between the first relative position difference and the first pose difference satisfies the second distance condition, and the distance difference between the first relative position difference and the second pose difference satisfies the third distance condition, it can be considered that the robot returns to the previously visited position, and it is determined that the pose of the second target image frame corresponding to the last image frame forms a loopback with the first target image frame corresponding to the last image frame, that is, it is determined that the loopback frame of the first target image frame corresponding to the last image frame is the second target image frame corresponding to the last image frame.

[0110] It can be seen that, in the case that the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame have high similarity, the loop pose is found by setting the specific loop condition, which can ensure that the second target image frame corresponding to the last image frame is the loop frame of the first target image frame corresponding to the last image frame, thereby improving the accuracy of loop detection.

[0111] In an exemplary embodiment, the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame is used to perform loop detection in the plurality of image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame, including:

[0112] In the case that the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the fourth matching degree condition,

[0113] A second preset number of continuous candidate image frames and a third preset number of second previous image frames of each candidate image frame are determined in the plurality of image frames;

[0114] The feature matching degree between each second previous image frame and each candidate image frame is determined, and the first target image frame corresponding to each candidate image frame is determined in the third preset number of second previous image frames of each candidate image frame, and the second target image frame corresponding to each candidate image frame is determined in the plurality of image frames;

[0115] Loop detection is performed on the relationship between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame to determine at least one loop frame of the first target image frame corresponding to the last image frame;

[0116] The first target image frame corresponding to the candidate image frame is the image frame having the highest feature matching degree among the third preset number of second previous image frames and the candidate image frame, and the second target image frame corresponding to the candidate image frame is the image frame having the highest feature matching degree between the first target image frame corresponding to the candidate image frame and the distance between the first target image frame corresponding to the candidate image frame in the plurality of image frames satisfying the first distance condition.

[0117] The second preset number of continuous candidate image frames refers to the second preset number of candidate image frames having continuity and continuity in the acquisition time node. The second previous image frame refers to the third preset number of image frames in the plurality of image frames whose acquisition time node is in front of each candidate image frame. As easily understood, the acquisition time node of the last image frame in the second previous image frame is adjacent to the acquisition time node of the candidate image frame corresponding to the second previous image frame in the acquisition time node.

[0118] In an example embodiment, the above-mentioned loop detection between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame is performed to determine at least one loop frame of the first target image frame corresponding to the last image frame, comprising:

[0119] determining a second feature matching degree between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame,

[0120] determining a second relative position difference between the first target image frame corresponding to each candidate image frame and the previous image frame of the first target image frame corresponding to each candidate image frame,

[0121] and determining a second pose difference between the second target image frame corresponding to each candidate image frame and the previous image frame of the second target image frame corresponding to each candidate image frame;

[0122] When at least one of the following conditions is met: the second feature matching degree meets a fifth matching degree condition, and the distance difference between the second relative position difference and the second pose difference meets a fourth distance condition, then the loop frame of the first target image frame corresponding to the last image frame is determined to be the second preset number of candidate image frames.

[0123] Wherein, the fourth matching degree condition can be that the feature matching degree is greater than 20% and less than 50%, greater than 10% and less than 50%, greater than 10 and less than 60%, or other feature matching degrees.

[0124] Optionally, the fifth matching degree condition can be that the feature matching degree is greater than 20% and less than 50%, greater than 10% and less than 50%, greater than 10 and less than 60%, or other feature matching degrees. Optionally, the fourth matching degree condition can be the same as the fifth matching degree condition.

[0125] Optionally, the fourth distance condition can be that the distance difference is less than 15 cm, 20 cm, 25 cm, or other distance difference. Optionally, the second distance condition, the third distance condition, and the fourth distance condition can be the same.

[0126] Optionally, the second preset number can be 5, 10, 15, or other number. Optionally, the first preset number can be the same as the second preset number.

[0127] Optionally, the third preset number can be 5, 10, 15, or other number. Optionally, the second preset number can be the same as the third preset number.

[0128] Specifically, the at least one of the following conditions that the second feature matching degree meets the fifth matching degree condition and the distance difference value between the second relative position difference and the second pose difference meets the fourth distance condition can mean that any one of the two conditions is met, or all of the two conditions are met.

[0129] It can be understood that the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame meeting the fourth matching degree condition means that the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame have a certain similarity but it cannot be determined whether they are collected at similar or same positions. In order to confirm whether the similarity between the two is sufficient to constitute a loop for more accurate loop judgment, more image frames are needed to assist in loop detection. Therefore, it can be further judged whether the second feature matching degree meets the fifth matching degree condition and / or whether the distance difference value between the second relative position difference and the second pose difference meets the fourth distance condition.

[0130] In a specific embodiment, the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame meeting the fourth matching degree condition means that the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame are likely to be collected at similar or same positions. However, since the feature matching degree is not sufficient to completely determine this, in order to further confirm whether the robot has returned to the position it has visited before, it is necessary to confirm whether the continuous second preset number of candidate image frames collected by the robot at the historical collection time node meet the motion continuity and motion similarity of the loop frame judgment. Therefore, when both the second feature matching degree meeting the fifth matching degree condition and the distance difference value between the second relative position difference and the second pose difference meeting the fourth distance condition are met, it can be considered that the robot has returned to the position it has visited before, and the loop frame of the first target image frame corresponding to the last image frame is determined to be the second preset number of candidate image frames.

[0131] It can be seen that in the case where the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame have a certain similarity but the similarity is not high enough, the loop frame is found by setting a specific loop condition, which can ensure that the second preset number of candidate image frames is the loop frame of the first target image frame corresponding to the last image frame, and can avoid missing the loop frame, thereby improving the accuracy of loop detection.

[0132] The application process of the above loop detection method will be described below in conjunction with a detailed embodiment as follows:

[0133] (1) Acquisition process of multiple image frames

[0134] obtaining a plurality of image frames collected by the robot, and determining a last image frame and a first preset number of first preceding image frames of the last image frame in the plurality of image frames.

[0135] (2) Determination process of the first target image frame

[0136] determining a feature matching degree between each first preceding image frame and the last image frame, and determining a first target image frame corresponding to the last image frame in the first preset number of first preceding image frames; the first target image frame corresponding to the last image frame is an image frame having the highest feature matching degree between the first preset number of first preceding image frames and the last image frame.

[0137] (3) Determination process of the second optimized pose of each image frame

[0138] in a case where the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame satisfies a first matching degree condition, determining a three-dimensional coordinate of the first target image frame corresponding to the last image frame based on a feature matching relationship between the first target image frame corresponding to the last image frame and each image frame and an initial pose of each image frame, and determining a calculated pose of the last image frame based on the feature matching relationship between the first target image frame corresponding to the last image frame and the last image frame and the three-dimensional coordinate of the first target image frame corresponding to the last image frame;

[0139] in a case where the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not satisfy the first matching degree condition, determining the initial pose of the first target image frame corresponding to the last image frame as the calculated pose of the last image frame;

[0140] determining a second optimized pose of each image frame based on the initial pose of the last image frame, the calculated pose of the last image frame and a second target function.

[0141] (4) Determination process of the loop closure pose

[0142] In the plurality of image frames, at least one peripheral image frame between the first target image frame corresponding to the last image frame and satisfying a first distance condition is determined; a second target image frame corresponding to the last image frame with the highest feature matching degree between the at least one peripheral image frame and the first target image frame corresponding to the last image frame is determined; in the case that the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies a second matching degree condition, a first feature matching degree between the last image frame and the first target image frame corresponding to the last image frame and the last image frame and the second target image frame corresponding to the last image frame is determined, a first relative position difference between the first target image frame corresponding to the last image frame and the last image frame and the last image frame and the second target image frame corresponding to the last image frame is determined, a first pose difference between the second target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame and the last image frame and the last image frame and the second target image frame corresponding to the last image frame is determined, and a second pose difference between the last image frame and the first target image frame corresponding to the last image frame and the last image frame and the second target image frame corresponding to the last image frame is determined; in the case that at least one of the first feature matching degree satisfying a third matching degree condition, the distance difference between the first relative position difference and the first pose difference satisfying a second distance condition, and the distance difference between the first relative position difference and the second pose difference satisfying a third distance condition is satisfied, the loop frame of the first target image frame corresponding to the last image frame is determined as the second target image frame corresponding to the last image frame.

[0143] In the case that the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies a fourth matching degree condition, a second preset number of candidate image frames and a third preset number of second previous image frames of each candidate image frame are determined in the plurality of image frames, the feature matching degree between each second previous image frame and each candidate image frame is determined, the first target image frame corresponding to each candidate image frame is determined in the third preset number of second previous image frames of each candidate image frame, and the second target image frame corresponding to each candidate image frame is determined in the plurality of image frames, thereby determining the second feature matching degree between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame, the second relative position difference between the first target image frame corresponding to each candidate image frame and the last image frame of the first target image frame corresponding to each candidate image frame, and the second pose difference between the second target image frame corresponding to each candidate image frame and the last image frame of the second target image frame corresponding to each candidate image frame, in the case that at least one of the second feature matching degree satisfying a fifth matching degree condition, the distance difference between the second relative position difference and the second pose difference satisfying a fourth distance condition is satisfied, the loop frame of the first target image frame corresponding to the last image frame is determined as the second preset number of candidate image frames.

[0144] The collection time node of each peripheral image frame is earlier than the collection time node of the first target image frame corresponding to the last image frame, and in the case that the direction of the peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame, the time difference between the collection time node of the peripheral image frame and the collection time node of the first target image frame corresponding to the last image frame satisfies a preset time condition.

[0145] (5) Determination process of the calculated pose of each image frame

[0146] In the case that at least one loop frame of the first target image frame corresponding to the last image frame is determined, the calculated pose of each image frame is determined based on the feature matching relationship between each image frame and the second optimized pose of each image frame.

[0147] (6) Determination process of the first optimized pose of each image frame

[0148] The first optimized pose of each image frame is determined based on the second optimized pose of each image frame, the calculated pose of each image frame, and the first target function.

[0149] In the embodiment, the first target image frame corresponding to the last image frame with the highest feature matching degree between the last image frame is determined, so that the calculated pose of the last image frame can be determined through the first target image frame corresponding to the last image frame, the second optimized pose of each image frame can be determined based on the initial pose of the last image frame, the calculated pose of the last image frame, and the second target function, the preliminary correction of the trajectory drift of each image frame is realized through the last image frame, further, the second target image frame corresponding to the last image frame with close distance and high similarity is determined through the first target image frame corresponding to the last image frame, and then the loop detection is performed in multiple image frames based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, at least one loop frame of the first target image frame corresponding to the last image frame is obtained, and the accuracy of the loop detection can be improved by ensuring the accuracy of the loop frame. It can be seen that the loop detection method provided in the embodiment enables the robot to accurately detect loops in a complex environment, so that the robot can distinguish correct loops and false loops in a complex environment, and further, in a complex environment with similar ceiling features, the robot can avoid the situation that the map construction error is large due to the detection of false loops, and in a non-similar feature environment, the accuracy of the loop detection can be improved to improve the accuracy of the map construction.

[0150] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time but can be executed at different times, and the execution of the steps or stages is not necessarily sequential but can be executed alternately or alternately with at least some of the other steps or steps or stages in other steps.

[0151] Based on the same inventive concept, the embodiments of the present application also provide a loop detection device for implementing the loop detection method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more loop detection device embodiments provided below can refer to the limitations of the loop detection method described above, which will not be repeated here.

[0152] In one exemplary embodiment, as shown in Figure 2 A loop detection device is provided, comprising: an acquisition module 202, a first determination module 204, a second determination module 206, and a detection module 208, wherein:

[0153] The acquisition module 202 is configured to acquire a plurality of image frames collected by a robot, and determine a last image frame and a first preset number of first preceding image frames of the last image frame in the plurality of image frames.

[0154] The first determination module 204 is configured to determine a feature matching degree between each first preceding image frame and the last image frame, and determine a first target image frame corresponding to the last image frame in the first preset number of first preceding image frames; the first target image frame corresponding to the last image frame is an image frame in the first preset number of first preceding image frames that has the highest feature matching degree with the last image frame.

[0155] The second determination module 206 is configured to determine, in the plurality of image frames, a second target image frame corresponding to the first target image frame corresponding to the last image frame; the second target image frame corresponding to the last image frame is an image frame in the plurality of image frames that has the highest feature matching degree with the first target image frame corresponding to the last image frame and satisfies a first distance condition with the first target image frame corresponding to the last image frame.

[0156] The detection module 208 is configured to perform loop detection in the plurality of image frames based on the feature matching degree between the first target image frame and the second target image frame, and obtain a loop pose based on the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame.

[0157] In an exemplary embodiment, the detection module 208 is further configured to optimize the initial pose of each image frame based on the initial pose of each loop frame, the calculated pose of each loop frame, and the first target function, and determine a first optimized pose of each image frame.

[0158] In an exemplary embodiment, the first determination module 204 is further configured to optimize the initial pose of each image frame based on the initial pose of the last image frame, the calculated pose of the last image frame, and the second target function, and determine a second optimized pose of each image frame.

[0159] In an exemplary embodiment, the first determination module 204 is further configured to, in a case where the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame satisfies a first matching degree condition, determine a three-dimensional coordinate of the first target image frame corresponding to the last image frame based on the feature matching relationship between the first target image frame corresponding to the last image frame and each image frame and the initial pose of each image frame, and determine the calculated pose of the last image frame based on the feature matching relationship between the first target image frame corresponding to the last image frame and the last image frame and the three-dimensional coordinate of the first target image frame corresponding to the last image frame.

[0160] In an exemplary embodiment, the first determination module 204 is further configured to, in a case where the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not satisfy the first matching degree condition, determine the initial pose of the first target image frame corresponding to the last image frame as the calculated pose of the last image frame.

[0161] In an example embodiment, the second determining module 206 is further configured to determine at least one peripheral image frame between the last image frame and the first target image frame corresponding to the last image frame in the plurality of image frames, wherein a distance between the last image frame and the first target image frame corresponding to the last image frame in the at least one peripheral image frame satisfies a first distance condition; determine a second target image frame corresponding to the last image frame with the highest feature matching degree between the last image frame and the first target image frame corresponding to the last image frame in the at least one peripheral image frame; and wherein a capture time node of each peripheral image frame is earlier than a capture time node of the first target image frame corresponding to the last image frame, and in a case where a direction of the peripheral image frame is the same as or opposite to a direction of the first target image frame corresponding to the last image frame, a time difference between the capture time node of the peripheral image frame and the capture time node of the first target image frame corresponding to the last image frame satisfies a preset time condition.

[0162] In an example embodiment, the detecting module 208 is further configured to, in a case where a feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies a second matching degree condition, determine a first feature matching degree between a previous image frame of the first target image frame corresponding to the last image frame and a previous image frame of the second target image frame corresponding to the last image frame, determine a first relative position difference between the first target image frame corresponding to the last image frame and the previous image frame of the first target image frame corresponding to the last image frame, determine a first pose difference between the second target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame, and determine a second pose difference between the previous image frame of the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame; and in a case where at least one of the first feature matching degree satisfies a third matching degree condition, a distance difference between the first relative position difference and the first pose difference satisfies a second distance condition, and a distance difference between the first relative position difference and the second pose difference satisfies a third distance condition, determine the loop frame of the first target image frame corresponding to the last image frame as the second target image frame corresponding to the last image frame.

[0163] In an example embodiment, the detection module 208 is further configured to determine, in the case that the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies a fourth matching degree condition, a second preset number of candidate image frames and a third preset number of second preceding image frames of each candidate image frame in the plurality of image frames; determine the feature matching degree between each second preceding image frame and each candidate image frame, and determine the first target image frame corresponding to each candidate image frame in the third preset number of second preceding image frames of each candidate image frame, and determine the second target image frame corresponding to the first target image frame of each candidate image frame in the plurality of image frames; perform loop detection through the relationship between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame to determine at least one loop frame of the first target image frame corresponding to the last image frame; wherein the first target image frame corresponding to each candidate image frame is the image frame having the highest feature matching degree between the second preset number of second preceding image frames and each candidate image frame; and the second target image frame corresponding to each candidate image frame is the image frame having the highest feature matching degree between the first target image frame corresponding to each candidate image frame and the distance between the first target image frame corresponding to each candidate image frame in the plurality of image frames satisfying a first distance condition.

[0164] In an example embodiment, the detection module 208 is further configured to determine the second feature matching degree between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame, determine the second relative position difference between the first target image frame corresponding to each candidate image frame and the previous image frame of the first target image frame corresponding to each candidate image frame, and determine the second pose difference between the second target image frame corresponding to each candidate image frame and the previous image frame of the second target image frame corresponding to each candidate image frame; in the case that at least one of the second feature matching degree satisfying a fifth matching degree condition and the distance difference between the second relative position difference and the second pose difference satisfying a fourth distance condition, the loop frame of the first target image frame corresponding to the last image frame is determined to be the second preset number of candidate image frames.

[0165] Each module in the loop detection device can be realized by software, hardware, and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0166] In an example embodiment, a computer device is provided, which can be a server, and the internal structure diagram thereof can be as shown in FIG. 1. Figure 3As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image frame data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a loop detection method.

[0167] In an exemplary embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in the figure. Figure 4 As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be realized through WIFI, mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. The computer program is executed by the processor to implement a loop detection method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0168] Those skilled in the art can understand that, Figure 4The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0169] In an exemplary embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0170] In an exemplary embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0171] In an exemplary embodiment, a computer program product is provided, including a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0172] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0173] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0174] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0175] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A loop closure detection method, characterized in that, The method includes: Acquire multiple image frames collected by the robot, and determine the last image frame and a first preset number of first preceding image frames among the multiple image frames; The feature matching degree between each of the first preceding image frames and the last image frame is determined, and the first target image frame corresponding to the last image frame is determined among a first preset number of first preceding image frames; the first target image frame corresponding to the last image frame is the image frame with the highest feature matching degree between it and the last image frame among the first preset number of first preceding image frames. Among multiple image frames, a second target image frame corresponding to the first target image frame corresponding to the last image frame is determined; the second target image frame corresponding to the last image frame is an image frame among multiple image frames whose distance to the first target image frame corresponding to the last image frame satisfies a first distance condition and has the highest feature matching degree with the first target image frame corresponding to the last image frame; Based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop closure detection is performed in multiple image frames to determine at least one loop closure frame of the first target image frame corresponding to the last image frame.

2. The method according to claim 1, characterized in that, If at least one loopback frame is determined to be connected to the first target image frame corresponding to the last image frame, the method further includes: Based on the initial pose of each loopback frame, the calculated pose of each loopback frame, and the first objective function, the initial pose of each image frame is optimized to determine the first optimized pose of each image frame.

3. The method according to claim 1, characterized in that, After determining the first target image frame corresponding to the last image frame in a first preset number of first preceding image frames, the above method further includes: Based on the initial pose of the last image frame, the calculated pose of the last image frame, and the second objective function, the initial pose of each image frame is optimized to determine the second optimized pose of each image frame.

4. The method according to claim 3, characterized in that, The method further includes: When the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame satisfies the first matching degree condition, the three-dimensional coordinates of the first target image frame corresponding to the last image frame are determined based on the feature matching relationship between the first target image frame corresponding to the last image frame and each of the image frames and the initial pose of each of the image frames. Based on the feature matching relationship between the first target image frame corresponding to the last image frame and the last image frame, as well as the three-dimensional coordinates of the first target image frame corresponding to the last image frame, the calculated pose of the last image frame is determined.

5. The method according to claim 4, characterized in that, If the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not meet the first matching degree condition, the initial pose of the first target image frame corresponding to the last image frame is determined as the calculated pose of the last image frame.

6. The method according to claim 1, characterized in that, Determining the second target image frame corresponding to the first target image frame corresponding to the last image frame among multiple image frames includes: Among multiple image frames, at least one surrounding image frame whose distance to the first target image frame corresponding to the last image frame satisfies the first distance condition is determined. In at least one surrounding image frame, a second target image frame is identified that has the highest feature matching degree with the first target image frame corresponding to the last image frame. Wherein, the acquisition time node of each of the surrounding image frames is earlier than the acquisition time node of the first target image frame corresponding to the last image frame, and when the direction of the surrounding image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame, the time difference between the acquisition time node of the surrounding image frame and the acquisition time node of the first target image frame corresponding to the last image frame satisfies a preset time condition.

7. The method according to claim 1, characterized in that, The step of performing loop closure detection in multiple image frames to determine at least one loop closure frame of the first target image frame corresponding to the last image frame, based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, includes: When the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the second matching degree condition. Determine the first feature matching degree between the image frame preceding the first target image frame corresponding to the last image frame and the image frame preceding the second target image frame corresponding to the last image frame. Determine the first relative position difference between the first target image frame corresponding to the last image frame and the image frame preceding the first target image frame corresponding to the last image frame. Determine the first pose difference between the second target image frame corresponding to the last image frame and the image frame preceding the second target image frame corresponding to the last image frame. And, determine the second pose difference between the previous image frame of the first target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame; If at least one of the following conditions is met: the first feature matching degree satisfies the third matching degree condition; the distance difference between the first relative position difference and the first pose difference satisfies the second distance condition; and the distance difference between the first relative position difference and the second pose difference satisfies the third distance condition, then the loopback frame of the first target image frame corresponding to the last image frame is determined to be the second target image frame corresponding to the last image frame.

8. The method according to claim 1, characterized in that, The step of performing loop closure detection in multiple image frames to determine at least one loop closure frame of the first target image frame corresponding to the last image frame, based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, includes: When the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the fourth matching degree condition. In a plurality of image frames, a second preset number of consecutive candidate image frames and a third preset number of second preceding image frames for each of the candidate image frames are determined; The feature matching degree between each second preceding image frame and each candidate image frame is determined respectively, and the first target image frame corresponding to each candidate image frame is determined in the third preset number of second preceding image frames of each candidate image frame, and the second target image frame corresponding to the first target image frame corresponding to each candidate image frame is determined in the multiple image frames respectively. Loop closure detection is performed by considering the relationship between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame, so as to determine at least one loop closure frame of the first target image frame corresponding to the last image frame. Wherein, the first target image frame corresponding to the candidate image frame is the image frame with the highest feature matching degree among the third preset number of second preceding image frames; the second target image frame corresponding to the candidate image frame is the image frame with the highest feature matching degree among multiple image frames whose distance to the first target image frame corresponding to the candidate image frame satisfies the first distance condition and whose distance to the first target image frame corresponding to the candidate image frame satisfies the first distance condition.

9. The method according to claim 8, characterized in that, The step of performing loop closure detection based on the relationship between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame, to determine at least one loop closure frame of the first target image frame corresponding to the last image frame, includes: Determine the second feature matching degree between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame. Determine the second relative position difference between the first target image frame corresponding to each candidate image frame and the previous image frame corresponding to the first target image frame. In addition, the second pose difference between the second target image frame corresponding to each candidate image frame and the previous image frame of the second target image frame corresponding to each candidate image frame is determined. When the second feature matching degree satisfies the fifth matching degree condition, and the distance difference between the second relative position difference and the second pose difference satisfies the fourth distance condition, the loopback frame of the first target image frame corresponding to the last image frame is determined to be the second preset number of candidate image frames.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images