Loopback detection method and computer equipment

Abstract

The invention relates to a loopback detection method and computer equipment. The method comprises the following steps: acquiring a plurality of image frames acquired by a robot, and determining a last image frame and a first preset number of first preorder image frames of the last image frame in the plurality of image frames; determining a feature matching degree between each first preorder image frame and the last image frame, and determining a first target image frame corresponding to the last image frame in the first preset number of first preorder image frames; in the plurality of image frames, determining a second target image frame corresponding to the first target image frame corresponding to the last image frame; and based on a feature matching degree between a first target image frame corresponding to the last image frame and a second target image frame corresponding to the last image frame, performing loopback detection in the plurality of image frames to determine at least one loopback frame of the first target image frame corresponding to the last image frame. By adopting the method, the accuracy of loopback detection can be improved.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular, to a loop detection method and a computer device. Background Art

[0002] With the rapid development of robotics, Visual Simultaneous Localization and Mapping (VSLAM) technology has been increasingly widely used in commercial service robots. Accordingly, more and more challenges have emerged in the application process of VSLAM technology. In order for a robot to construct an accurate and low-error visual map, visual loop detection is an essential task.

[0003] In an indoor operating scenario, the visual camera equipped on a commercial service robot is generally a monocular camera with a vertically upward field of view for obtaining an image of the ceiling above the robot. In some complex environments, such as the ceiling environments of large factories and office buildings, usually have a high degree of similarity. Therefore, if the loop pose cannot be accurately detected or an incorrect loop pose is detected in these complex environments, it will result in a significant error in the map constructed by the robot, which will undoubtedly have a serious adverse impact on the subsequent operations of the commercial service robot. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a loop detection method, device, computer device, computer-readable storage medium, and computer program product that can improve the accuracy of loop detection.

[0005] In a first aspect, this application provides a loop detection method, including:

[0006] Obtain a plurality of image frames collected by the robot, and determine the last image frame and the first preset number of first pre-order image frames of the last image frame among the plurality of image frames;

[0007] Determine the feature matching degree between each first pre-order image frame and the last image frame, and determine the first target image frame corresponding to the last image frame among the first preset number of first pre-order image frames; the first target image frame corresponding to the last image frame is the image frame with the highest feature matching degree with the last image frame among the first preset number of first pre-order image frames;

[0008] Among multiple image frames, determine a second target image frame corresponding to a first target image frame corresponding to the last image frame; the second target image frame corresponding to the last image frame is the image frame among the multiple image frames whose distance from the first target image frame corresponding to the last image frame satisfies a first distance condition and has the highest feature matching degree with the first target image frame corresponding to the last image frame;

[0009] Based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, perform loop detection among the multiple image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame.

[0010] In a second aspect, the present application also provides a loop detection device, including:

[0011] An acquisition module, configured to acquire multiple image frames collected by a robot, and determine a last image frame and a first preset number of first pre-order image frames of the last image frame among the multiple image frames;

[0012] A first determination module, configured to determine the feature matching degree between each first pre-order image frame and the last image frame, and determine a first target image frame corresponding to the last image frame among the first preset number of first pre-order image frames; the first target image frame corresponding to the last image frame is the image frame among the first preset number of first pre-order image frames that has the highest feature matching degree with the last image frame;

[0013] A second determination module, configured to determine a second target image frame corresponding to the first target image frame corresponding to the last image frame among the multiple image frames; the second target image frame corresponding to the last image frame is the image frame among the multiple image frames whose distance from the first target image frame corresponding to the last image frame satisfies a first distance condition and has the highest feature matching degree with the first target image frame corresponding to the last image frame;

[0014] A detection module, configured to perform loop detection among the multiple image frames through the second target image frame based on the feature matching degree between the first target image frame and the second target image frame, and obtain a loop pose. Based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, perform loop detection among the multiple image frames to determine at least one loop frame corresponding to the first target image frame corresponding to the last image frame.

[0015] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements some or all of the steps described in any method of the first aspect of the embodiments of the present application.

[0016] Fourthly, the present application also provides a computer-readable storage medium. On the computer-readable storage medium, there is a computer program stored, and when the computer program is executed by a processor, some or all of the steps described in any of the methods in the first aspect of the embodiments of the present application are implemented.

[0017] Fifthly, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, some or all of the steps described in any of the methods in the first aspect of the embodiments of the present application are implemented.

[0018] For the above loop detection method, device, computer device, computer-readable storage medium and computer program product, a plurality of image frames collected by a robot are obtained, and a last image frame and a first preset number of first pre-order image frames of the last image frame are determined among the plurality of image frames; the feature matching degree between each first pre-order image frame and the last image frame is determined, and a first target image frame corresponding to the last image frame is determined among the first preset number of first pre-order image frames; the first target image frame corresponding to the last image frame is the image frame with the highest feature matching degree between the last image frame and the first preset number of first pre-order image frames; among the plurality of image frames, a second target image frame corresponding to the first target image frame corresponding to the last image frame is determined; the second target image frame corresponding to the last image frame is the image frame whose distance from the first target image frame corresponding to the last image frame satisfies a first distance condition and has the highest feature matching degree with the first target image frame corresponding to the last image frame; based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed among the plurality of image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame. By using the loop detection method provided by the present application, by determining the first target image frame corresponding to the last image frame with the highest feature matching degree between the last image frame and the first pre-order image frames, and then determining the second target image frame corresponding to the last image frame whose distance from the first target image frame satisfies the first distance condition and has the highest feature matching degree with the first target image frame among the plurality of image frames, thus, based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed among the plurality of image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame. Obviously, the loop detection method provided in this embodiment can accurately perform loop detection among the plurality of image frames through the first target image frame corresponding to the last image frame to obtain loop frames with relatively high accuracy, can significantly improve the accuracy of loop detection, and further, when the loop detection has relatively high accuracy, it can further ensure the accuracy of map construction. Description of the Drawings

[0019] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the accompanying drawings required for the description of the embodiments of the present application or related technologies. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related accompanying drawings can also be obtained based on these drawings.

[0020] Figure 1 It is a schematic flowchart of the loop detection method in an embodiment;

[0021] Figure 2 It is a structural block diagram of the loop detection device in an embodiment;

[0022] Figure 3 It is an internal structure diagram of a computer device in an embodiment;

[0023] Figure 4 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0024] In order to make the purpose, technical solutions and advantages of the present application clearer, the following further details the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] The embodiments of the present application provide a loop detection method. Taking the application of this method to a computer device as an example, the computer device can be a terminal or a server. It can be understood that this method can also be applied to a system including a terminal and a server and realized through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various robots, driverless vehicles, automatic food delivery vehicles, personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The robot can be a cleaning robot, a delivery robot, a logistics robot, a patrol robot and a disinfection robot, etc. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, a smart glasses, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0026] In an exemplary embodiment, as Figure 1As shown, a loop detection method is provided. Taking the application of this method to a robot as an example, it includes the following steps 102 to 108. Among them:

[0027] Step 102: Obtain multiple image frames collected by the robot, and determine the last image frame and the first preset number of first pre-order image frames of the last image frame among the multiple image frames.

[0028] Among them, an image acquisition device is set on the robot, and maps and positioning are constructed through the images acquired by the image acquisition device. The image acquisition device can acquire images at at least any one position above, on the side, in front, behind, or on the ground of the robot.

[0029] An image frame is an individual image in the video stream or image sequence collected by the robot. In VSLAM technology, an image frame is a key data unit for constructing the environmental map and the robot path. Understandably, each image frame corresponds to a collection time node and an initial pose. The initial pose of each image frame can be obtained through a wheel odometer, or through other sensors that can be installed on the robot and obtain the robot's pose, or through the fusion of multiple sensors.

[0030] In an exemplary embodiment, the above-mentioned obtaining of multiple image frames collected by the robot includes: obtaining multiple image frames collected by the robot through the image acquisition device; among them, the lens of the image acquisition device faces upward, and the image frame is an image above the environment where the robot is located; when the robot is in an indoor scene, the image frame can be a ceiling image; when the robot is in an outdoor scene, the image frame can be an image of the sky, buildings, or trees above the robot, etc.

[0031] Optionally, the number of multiple image frames can be 5, 10, 20 or other numbers, which is set according to the actual algorithm calculation requirements and is not specifically limited here.

[0032] Optionally, the image acquisition device can be a monocular camera, a binocular camera, or a depth camera, etc.

[0033] Optionally, the robot can be, but is not limited to, various industrial robots that need to move autonomously for map construction (such as handling robots, palletizing robots, spraying robots, etc.), service robots (such as cleaning robots, delivery robots, lawn mowing robots, guiding robots, building robots, etc.), or special robots (fire fighting robots, underwater robots, security robots, etc.).

[0034] The last image frame refers to an image frame among multiple image frames whose acquisition time node is at the end. That is to say, the last image frame is the last image frame collected by the robot among multiple image frames; the first preceding image frames refer to the first preset number of image frames among multiple image frames whose acquisition time nodes are before the last image frame. Understandably, the image frame with the last acquisition time node among the first preceding image frames is adjacent to the acquisition time node of the last image frame in terms of the acquisition time node. Exemplarily, if the robot collects 100 image frames, then the last image frame is the 100th image frame collected by the robot. Assuming the first preset number is 10, then the first preceding image frames are the 90th image frame, the 91st image frame... the 99th image frame collected by the robot.

[0035] Optionally, the first preset number can be 5, 10, 15 or other numbers.

[0036] Step 104, determine the feature matching degrees between each of the first preceding image frames and the last image frame, and determine the first target image frame corresponding to the last image frame among the first preset number of first preceding image frames; the first target image frame corresponding to the last image frame is the image frame among the first preset number of first preceding image frames that has the highest feature matching degree with the last image frame.

[0037] Among them, the first target image frame corresponding to the last image frame has the highest feature matching degree with the last image frame among the first preset number of first preceding image frames. That is to say, the first target image frame is an image frame among the first preset number of first preceding image frames that has the highest similarity with the last image frame in terms of visual features, which means that the first target image frame corresponding to the last image frame and the last image frame may be collected by the robot at the same or similar positions.

[0038] Optionally, in order to ensure the accuracy of the loop detection of the robot, while ensuring that the first target image frame corresponding to the last image frame is an image frame with the highest feature matching degree with the last image frame, it can also be further ensured that the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame meets the first matching condition. The first matching condition can be that the feature matching degree is greater than or equal to 50%, or greater than 60%, etc., and can be specifically set according to actual needs.

[0039] In an exemplary embodiment, the above determination of the feature matching degrees between each of the first preceding image frames and the last image frame includes: extracting the feature points of each image frame, and pairwise matching each image frame based on the feature points of each image frame to determine the feature matching relationships between each image frame; based on the feature matching relationships between each image frame, determine the feature matching degrees between each of the first preceding image frames and the last image frame.

[0040] Among them, the feature matching relationship between each image frame, including the feature matching relationship between any two image frames among multiple image frames, represents the corresponding relationship of feature points between each image frame. Thus, based on the feature matching relationship between each image frame, the feature matching relationship between any two image frames among multiple image frames can be determined.

[0041] Optionally, the feature points of each image frame can be Oriented FAST and Rotated BRIEF (ORB) feature points, Scale-Invariant Feature Transform (SIFT) feature points, or other deep learning feature points.

[0042] Step 106: In multiple image frames, determine the second target image frame corresponding to the first target image frame corresponding to the last image frame; the second target image frame corresponding to the last image frame is the image frame among multiple image frames that satisfies the first distance condition with respect to the first target image frame corresponding to the last image frame and has the highest feature matching degree with the first target image frame corresponding to the last image frame.

[0043] Among them, the first distance condition can be that the distance is within a radius of 3m, 4m, 5m, 6m, or other distances; for example, when the first distance condition is that the distance is within a radius of 5m, the second target image frame corresponding to the last image frame is some of the image frames among multiple image frames that are within a radius of 5m from the first target image frame corresponding to the last image frame.

[0044] The second target image frame corresponding to the last image frame satisfies the first distance condition with respect to the first target image frame corresponding to the last image frame and has the highest feature matching degree with the first target image frame corresponding to the last image frame. That is to say, the second target image frame corresponding to the last image frame is an image frame among multiple image frames that is relatively close to the first target image frame corresponding to the last image frame and has the highest similarity in visual features with the first target image frame corresponding to the last image frame, which means that the second target image frame corresponding to the last image frame and the first target image frame corresponding to the last image frame may be obtained by the robot at the same or similar positions. Further, since the first target image frame corresponding to the last image frame has the highest feature matching degree with the last image frame representing the current position of the robot, the second target image frame corresponding to the last image frame is an image frame of a historical position that may have the highest similarity with the current position of the robot.

[0045] Specifically, the feature matching degree between the second target image frame corresponding to the last image frame and the first target image frame corresponding to the last image frame can also be determined based on the feature matching relationship between each image frame.

[0046] Step 108: Based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, perform loop detection among multiple image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame.

[0047] Among them, loop detection, also known as closed-loop detection, refers to the ability of a robot to recognize that it has reached a certain scene, enabling the map to form a closed loop. Simply put, it means that when the robot is building a map by turning left and then right, it can realize that a certain place is where it has been before, and then match the map generated at this moment with the map just generated. The reason why loop detection is a difficult point is that: if the loop detection is successful, it can significantly reduce the cumulative error and help the robot perform obstacle avoidance and navigation work more accurately and quickly. However, incorrect detection results may make the map very bad and affect the subsequent obstacle avoidance and navigation work of the robot. Therefore, loop detection is very necessary in the construction of large-area and large-scene maps.

[0048] Optionally, at least one loop frame of the first target image frame corresponding to the last image frame obtained by performing loop detection can be a single image frame that meets the loop detection conditions, or multiple image frames that meet the loop detection conditions.

[0049] Specifically, in the case of obtaining a loop frame, it indicates that the robot has returned to a previously visited position. At this time, the poses of multiple image frames can be optimized so that the robot can construct a map with higher accuracy. It is easy to understand that if there is actually no loop frame among multiple image frames but it is misjudged as having a loop frame, and the poses of multiple image frames are optimized in this error situation, it will cause a particularly large error in the map constructed by the robot. Therefore, the robot needs to ensure the accuracy of loop detection to avoid this bad situation that leads to inaccurate map construction.

[0050] It is easy to understand that in the case of finding at least one loop frame among multiple image frames, the pose of the first target image frame corresponding to the last image frame at this time is the loop pose.

[0051] In the above loop detection method, multiple image frames collected by the robot are obtained, and the last image frame and the first preset number of first preceding image frames of the last image frame are determined among the multiple image frames; the feature matching degrees between each first preceding image frame and the last image frame are determined, and the first target image frame corresponding to the last image frame is determined among the first preset number of first preceding image frames; the first target image frame corresponding to the last image frame is the image frame with the highest feature matching degree with the last image frame among the first preset number of first preceding image frames; among the multiple image frames, the second target image frame corresponding to the first target image frame corresponding to the last image frame is determined; the second target image frame corresponding to the last image frame is the image frame whose distance from the first target image frame corresponding to the last image frame satisfies the first distance condition and has the highest feature matching degree with the first target image frame corresponding to the last image frame; based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed among the multiple image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame. By using the loop detection method provided by this application, by determining the first target image frame corresponding to the last image frame with the highest feature matching degree with the last image frame among the first preceding image frames, and then determining the second target image frame corresponding to the last image frame whose distance from the first target image frame satisfies the first distance condition and has the highest feature matching degree with the first target image frame among the multiple image frames, thus, based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed among the multiple image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame. Obviously, the loop detection method provided in this embodiment can perform accurate loop detection through the first target image frame corresponding to the last image frame among the multiple image frames to obtain loop frames with relatively high accuracy, can significantly improve the accuracy of loop detection, and further, in the case of relatively high accuracy of loop detection, can further ensure the accuracy of map construction.

[0052] In an exemplary embodiment, in the case of determining at least one loop frame of the first target image frame corresponding to the last image frame, the above method further includes:

[0053] Based on the initial poses of each loop frame, the calculated poses of each loop frame, and the first objective function, the initial poses of each image frame are optimized to determine the first optimized poses of each image frame.

[0054] In an exemplary embodiment, the above method further includes: determining the calculated pose of each image frame based on the feature matching relationship between each image frame and the initial pose corresponding to each image frame; after determining at least one loop closure frame of the first target image frame corresponding to the last image frame, the above method further includes: determining the calculated pose of each loop closure frame among the calculated poses of each image frame.

[0055] Among them, the calculated pose of each image frame can be used to optimize the pose of each image frame so that the robot constructs a more accurate map. By optimizing the pose of each image frame, the cumulative error of the robot during the map construction process can be reduced, thereby improving the consistency between the entire motion trajectory of the robot and the actual map.

[0056] The first optimized pose of each image frame is used to enable the robot to optimize the constructed map, reduce the cumulative error during the map construction process, and construct a map with higher accuracy and higher consistency with the actual map.

[0057] In an exemplary embodiment, the above determining the calculated pose of each image frame based on the feature matching relationship between each image frame and the initial pose corresponding to each image frame includes: determining the three-dimensional coordinates of each image frame based on the feature matching relationship between each image frame and the initial pose corresponding to each image frame; determining the calculated pose of each image frame based on the feature matching relationship between each image frame and the three-dimensional coordinates of each image frame.

[0058] Exemplarily, taking the determination of the calculated pose of the last image frame as an example, first, based on the feature matching relationship between the first target image frame of the last image frame and other image frames and the initial pose of each image frame, the three-dimensional coordinates of the first target image frame of the last image frame can be calculated. Then, based on the feature matching relationship between the last image frame and the first target image frame of the last image frame and the three-dimensional coordinates of the first target image frame of the last image frame, the calculated pose of the last image frame can be calculated. The calculation principle of the calculated poses of other image frames is the same, so it will not be elaborated here.

[0059] Optionally, based on the feature matching relationship between each image frame and the three-dimensional coordinates of each image frame, the calculated pose of each image frame can be determined by the Perspective-n-Point (PnP) algorithm. Specifically, in the VSLAM technology, the PnP algorithm is usually used to calculate and recover the pose of the robot in the two-dimensional image frame from the corresponding relationship between a set of feature points of the matched two-dimensional image frames and their corresponding feature points in the three-dimensional space.

[0060] Specifically, the first optimized pose of each image frame is the pose that can be obtained by each image frame based on the initial pose of each loop closure frame, the calculated pose of each loop closure frame, and the first objective function, which minimizes the first objective function.

[0061] Exemplarily, the number of multiple image frames is represented as n, the first optimized pose of each image frame is represented as T i , the initial pose of each loop closure frame is represented as T' k , the calculated pose of each loop closure frame obtained based on the initial pose is represented as T' cal_k , then the first objective function can be , where , .

[0062] In one embodiment, the initial pose of each loop closure frame is input into the first objective function, and the initial pose that minimizes the first objective function is the first optimized pose of each image frame.

[0063] In this embodiment, in the case of obtaining at least one loop closure frame of the first target image frame corresponding to the last image frame, based on the initial pose of each loop closure frame, the calculated pose of each loop closure frame, and the first objective function, the first optimized pose of each image frame is determined. Thus, based on accurate loop closure frames, the accuracy of the determined first optimized pose of each image frame can be improved, and further, the accuracy of map construction can be further ensured.

[0064] In an exemplary embodiment, after determining the first target image frame corresponding to the last image frame among the first preset number of first pre-order image frames, the method further includes:

[0065] Based on the initial pose of the last image frame, the calculated pose of the last image frame, and the second objective function, the initial pose of each image frame is optimized to determine the second optimized pose of each image frame.

[0066] In an exemplary embodiment, the above-mentioned determination of the calculated pose of each image frame based on the feature matching relationship between each image frame and the initial pose corresponding to each image frame includes: determining the calculated pose of each image frame based on the feature matching relationship between each image frame and the second optimized pose of each image frame.

[0067] Among them, the second optimized pose of each image frame is the optimized pose of each image frame obtained by correcting the trajectory drift of each image frame by the last image frame, so as to accurately determine the loop closure pose in the subsequent loop closure detection process.

[0068] Specifically, the second optimized pose of each image frame is the pose that can minimize the second objective function obtained by each image frame based on the initial pose of the last image frame, the calculated pose of the last image frame, and the second objective function.

[0069] Exemplarily, represent the second optimized pose of each image frame as T k , represent the initial pose of the last image frame as T end , represent the calculated pose of the last image frame as T cal_end , then the second objective function can be , where , .

[0070] In an exemplary embodiment, input the initial pose of each image frame into the second objective function, and the initial pose that minimizes the second objective function is the second optimized pose of each image frame.

[0071] In an exemplary embodiment, when optimizing the second optimized pose of each image frame based on the second optimized pose of each loop closure frame, the calculated pose of each loop closure frame, and the first objective function to determine the first optimized pose of each image frame, and the calculated pose of each loop closure frame is calculated based on the second optimized pose of each image frame, represent the calculated pose of each image frame calculated based on the second optimized pose as T cal_k , then the first objective function is , where , .

[0072] In this embodiment, based on the initial pose of the last image frame, the calculated pose of the last image frame, and the second objective function, determine the second optimized pose of each image frame, and preliminarily correct the trajectory drift of each image frame through the last image frame. Thus, the calculated pose of each image frame determined subsequently and the first optimized pose of each image frame determined are obtained based on the second optimized pose of each image frame that has been preliminarily corrected for trajectory drift and has higher accuracy. Furthermore, the accuracy of loop closure detection can be further improved.

[0073] In an exemplary embodiment, the above method further includes:

[0074] When the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame satisfies the first matching degree condition, determine the three-dimensional coordinates of the first target image frame corresponding to the last image frame based on the feature matching relationship between the first target image frame corresponding to the last image frame and each image frame and the initial pose of each image frame;

[0075] Determine the calculated pose of the last image frame based on the feature matching relationship between the first target image frame corresponding to the last image frame and the last image frame, and the three-dimensional coordinates of the first target image frame corresponding to the last image frame.

[0076] Among them, based on the feature matching relationship between the first target image frame corresponding to the last image frame and each image frame, and the initial pose of each image frame, the three-dimensional coordinates of the first target image frame corresponding to the last image frame can be determined by triangulation or other 3D reconstruction techniques.

[0077] Optionally, based on the feature matching relationship between the first target image frame corresponding to the last image frame and the last image frame, and the three-dimensional coordinates of the first target image frame corresponding to the last image frame, the calculated pose of the last image frame can be determined by the PnP algorithm.

[0078] In this embodiment, based on the feature matching relationship between the first target image frame corresponding to the last image frame and the last image frame, and the three-dimensional coordinates of the first target image frame corresponding to the last image frame, the calculated pose of the last image frame can be determined based on the correspondence between the two-dimensional feature matching relationship and the three-dimensional coordinates. Thus, by ensuring the accuracy of the calculated pose of the last image frame obtained, it can further ensure that the second optimized poses of the obtained image frames also have relatively high accuracy. Furthermore, the accuracy of loop detection can be further improved.

[0079] In an exemplary embodiment, when the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not meet the first matching degree condition, the initial pose of the first target image frame corresponding to the last image frame is determined as the calculated pose of the last image frame.

[0080] In this embodiment, when the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not meet the first matching degree condition, it indicates that it is difficult to accurately determine the calculated pose of the last image frame through calculation at this time. Since during manual intervention or manual operation, the initial pose and the last pose of the robot are usually ensured to be the same, therefore, at this time, the initial pose of the first target image frame corresponding to the last image frame will be directly determined as the calculated pose of the last image frame to avoid calculating an incorrect calculated pose and thus causing a large error in the map constructed by the robot.

[0081] In an exemplary embodiment, in the above-mentioned multiple image frames, determining the second target image frame corresponding to the first target image frame corresponding to the last image frame includes:

[0082] In the multiple image frames, determine at least one surrounding image frame whose distance from the first target image frame corresponding to the last image frame satisfies the first distance condition;

[0083] Determine a second target image frame corresponding to the last image frame that has the highest feature matching degree with the first target image frame corresponding to the last image frame in at least one peripheral image frame;

[0084] Wherein, the acquisition time node of each peripheral image frame is earlier than the acquisition time node of the first target image frame corresponding to the last image frame, and when the direction of the peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame, the time difference between the acquisition time node of the peripheral image frame and the acquisition time node of the first target image frame corresponding to the last image frame meets a preset time condition.

[0085] Specifically, the acquisition time node of each peripheral image frame is earlier than the acquisition time node of the first target image frame corresponding to the last image frame, that is to say, each peripheral image frame has been acquired before the first target image frame corresponding to the last image frame is acquired.

[0086] Optionally, the preset time condition can be that the time difference is greater than or equal to 5 seconds, 10 seconds, 15 seconds or other time differences.

[0087] Optionally, it is possible to determine whether the direction of each peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame by performing pose matching between the calculated poses of each peripheral image frame and the calculated pose of the first target image frame corresponding to the last image frame respectively.

[0088] Specifically, in order to perform loop detection among multiple image frames, therefore, the acquisition time node of at least one peripheral image frame used to determine the second target image frame corresponding to the last image frame should be earlier than the acquisition time node of the first target image frame corresponding to the last image frame. At the same time, in order to avoid excessive computing power consumption during loop detection, there is a certain acquisition time difference between the peripheral image frame and the first target image frame corresponding to the last image frame when the direction of the peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame. Obviously, through the limiting condition of the acquisition time node of the image frame, while ensuring the accuracy of the second target image frame corresponding to the last image frame determined by at least one peripheral image frame, it also ensures sufficient efficiency for subsequent loop detection.

[0089] In an exemplary embodiment, the second target image frame corresponding to the last image frame, which has the highest feature matching degree among the first target image frames corresponding to the last image frame determined in at least one peripheral image frame, includes: rotating the first target image frame corresponding to the last image frame to the same direction as each peripheral image frame respectively to obtain a plurality of rotated first target image frames, and there is a one-to-one correspondence between the rotated first target image frames and the peripheral image frames; respectively determining the feature matching relationships between each rotated first target image frame and each peripheral image frame; and based on the feature matching relationships between each rotated first target image frame and each peripheral image frame, determining the second target image frame corresponding to the last image frame, which has the highest feature matching degree among the first target image frames corresponding to the last image frame, in at least one peripheral image frame.

[0090] In this embodiment, by determining at least one peripheral image frame with a relatively small distance between the first target image frame corresponding to the last image frame in a plurality of image frames, accordingly, at least one peripheral image frame is an image frame where a candidate loop pose may exist. Then, the second target image frame corresponding to the last image frame, which has the highest feature matching degree among the first target image frames corresponding to the last image frame, is determined in at least one peripheral image frame. Accordingly, the second target image frame corresponding to the last image frame is an image frame of a historical position that may have the highest similarity to the current position of the robot. Further, based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed in a plurality of image frames through the second target image frame to obtain a loop pose. Obviously, the loop detection method provided in this embodiment can perform accurate loop detection in a plurality of image frames through the first target image frame to obtain a loop pose with relatively high accuracy. Furthermore, it can significantly improve the accuracy of loop detection; at the same time, due to the limited conditions on the acquisition time nodes of peripheral image frames, therefore, while improving the accuracy of loop detection, the efficiency of loop detection can also be improved.

[0091] In an exemplary embodiment, the above-mentioned loop detection is performed in a plurality of image frames based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame to determine at least one loop frame of the first target image frame corresponding to the last image frame, including:

[0092] When the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame meets the second matching degree condition,

[0093] determining the first feature matching degree between the previous image frame of the first target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame,

[0094] Determine the first relative position difference between the first target image frame corresponding to the last image frame and the previous image frame of the first target image frame corresponding to the last image frame.

[0095] Determine the first pose difference between the second target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame.

[0096] Moreover, determine the second pose difference between the previous image frame of the first target image frame corresponding to the last image frame and the previous image frame between the second target image frame corresponding to the last image frame.

[0097] When at least one of the following conditions is met: the first feature matching degree meets the third matching degree condition, the distance difference between the first relative position difference and the first pose difference meets the second distance condition, and the distance difference between the first relative position difference and the second pose difference meets the third distance condition, then determine that the loopback frame of the first target image frame corresponding to the last image frame is the second target image frame corresponding to the last image frame.

[0098] Wherein, the second matching degree condition can be that the feature matching degree is greater than or equal to 50%, 60%, 80% or other feature matching degrees.

[0099] Optionally, the third matching degree condition can be that the feature matching degree is greater than or equal to 50%, 60%, 80% or other feature matching degrees. Optionally, the second matching degree condition can be the same as the third matching degree condition.

[0100] Optionally, the second distance condition can be that the distance difference is less than 15 cm, 20 cm, 25 cm or other distance differences.

[0101] Optionally, the third distance condition can be that the distance difference is less than 15 cm, 20 cm, 25 cm or other distance differences. Optionally, the second distance condition can be the same as the third distance condition.

[0102] The previous image frame of the first target image frame corresponding to the last image frame, that is, the image frame at the acquisition time node that is the previous acquisition time node of the first target image frame corresponding to the last image frame. That is to say, among multiple image frames with consecutive acquisition time nodes, the previous image frame of the first target image frame corresponding to the last image frame is an image frame adjacent to the first target image frame corresponding to the last image frame and with an acquisition time node earlier than the first target image frame corresponding to the last image frame.

[0103] The previous image frame of the second target image frame corresponding to the last image frame, that is, the image frame with the acquisition time node being the previous acquisition time node of the second target image frame corresponding to the last image frame. That is to say, among multiple image frames with consecutive acquisition time nodes, the previous image frame of the second target image frame corresponding to the last image frame is an image frame adjacent to the second target image frame corresponding to the last image frame and with an acquisition time node earlier than that of the second target image frame corresponding to the last image frame.

[0104] Specifically, the first relative position difference between the first target image frame corresponding to the last image frame and the previous image frame of the first target image frame corresponding to the last image frame is determined by the straight-line distance between the position of the first target image frame corresponding to the last image frame and the position of the previous image frame of the first target image frame corresponding to the last image frame.

[0105] Specifically, the first pose difference between the second target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame is determined by the relative change situation between the pose of the second target image frame corresponding to the last image frame and the pose of the previous image frame of the second target image frame corresponding to the last image frame. The second pose difference between the previous image frame of the first target image frame corresponding to the last image frame and the previous image frame between the second target image frame corresponding to the last image frame is the same as the first pose difference, so it will not be elaborated here.

[0106] Specifically, in VSLAM technology, the position refers to a specific point of the robot in three-dimensional space. The position does not involve the direction information of the robot but only involves the translation information of the robot. That is to say, the position describes the linear movement of the robot in three-dimensional space without involving any rotation or direction change. In VSLAM technology, the pose refers to the combination of the position of the robot and the attitude of the robot. The attitude refers to the direction or orientation of the robot. Therefore, the pose not only describes the specific point of the robot in three-dimensional space but also describes the direction or orientation of the robot. That is to say, the pose involves the direction information and translation information of the robot.

[0107] Specifically, that the first feature matching degree satisfies the third matching degree condition, that the distance difference between the first relative position difference and the first pose difference satisfies the second distance condition, and that the distance difference between the first relative position difference and the second pose difference satisfies the third distance condition, at least one of these situations can refer to any one of the three situations, or any two of the three situations, or all of the three situations.

[0108] It can be understood that the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the second matching degree condition, indicating that the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame may be acquired at similar or the same positions. To perform loop closure judgment more accurately, it is possible to further determine whether the first feature matching degree satisfies the third matching degree condition, and / or whether the distance difference between the first relative position difference and the first pose difference satisfies the second distance condition, and / or whether the distance difference between the first relative position difference and the second pose difference satisfies the third distance condition.

[0109] In a specific embodiment, the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the second matching degree condition, indicating that the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame may be acquired at similar or the same positions. To further confirm whether the second target image frame corresponding to the last image frame forms a loop with the first target image frame corresponding to the last image frame, that is, to further confirm whether the loop frame of the first target image frame corresponding to the last image frame is the second target image frame corresponding to the last image frame, it is necessary to confirm whether the previous image frame of the first target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame are also acquired at similar or the same positions, and to confirm whether there is also a certain degree of motion continuity and motion similarity between the first target image frame corresponding to the last image frame and the previous image frame of the first target image frame corresponding to the last image frame, between the second target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame, and between the previous image frame of the first target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame. Therefore, when all three conditions are met: the first feature matching degree satisfies the third matching degree condition, the distance difference between the first relative position difference and the first pose difference satisfies the second distance condition, and the distance difference between the first relative position difference and the second pose difference satisfies the third distance condition, it can be considered that the robot has returned to a previously visited position. At this time, it is determined that the pose of the second target image frame corresponding to the last image frame forms a loop with the first target image frame corresponding to the last image frame, that is, at this time, it is determined that the loop frame of the first target image frame corresponding to the last image frame is the second target image frame corresponding to the last image frame.

[0110] It can be seen that in the case where there is a high similarity between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, by setting specific loop conditions to find the loop pose, it can be ensured that the second target image frame corresponding to the last image frame is the loop frame of the first target image frame corresponding to the last image frame, thereby, the accuracy of loop detection can be improved.

[0111] In an exemplary embodiment, the above-mentioned loop detection is performed among multiple image frames based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame to determine at least one loop frame of the first target image frame corresponding to the last image frame, including:

[0112] In the case where the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the fourth matching degree condition,

[0113] Determine a continuous second preset number of candidate image frames and a third preset number of second previous image frames of each candidate image frame among multiple image frames;

[0114] Respectively determine the feature matching degree between each second previous image frame and each candidate image frame, and determine the first target image frame corresponding to each candidate image frame among the third preset number of second previous image frames of each candidate image frame, and respectively determine the second target image frame corresponding to the first target image frame corresponding to each candidate image frame among multiple image frames;

[0115] Perform loop detection through the relationship between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame to determine at least one loop frame of the first target image frame corresponding to the last image frame;

[0116] Among them, the first target image frame corresponding to the candidate image frame is the image frame with the highest feature matching degree with the candidate image frame among the third preset number of second previous image frames; the second target image frame corresponding to the candidate image frame is the image frame whose distance from the first target image frame corresponding to the candidate image frame satisfies the first distance condition and has the highest feature matching degree with the first target image frame corresponding to the candidate image frame among multiple image frames.

[0117] The continuous second preset number of candidate image frames refers to the second preset number of candidate image frames that are continuous and uninterrupted at the acquisition time node. The second previous image frame refers to the third preset number of image frames among multiple image frames whose acquisition time nodes are in front of each candidate image frame. It is easy to understand that the image frame with the last acquisition time node among the second previous image frames is adjacent to the acquisition time node of the candidate image frame corresponding to this second previous image frame at the acquisition time node.

[0118] In an exemplary embodiment, the loop detection is performed based on the relationship between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame to determine at least one loop frame of the first target image frame corresponding to the last image frame, including:

[0119] Determine the second feature matching degree between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame,

[0120] Determine the second relative position difference between the first target image frame corresponding to each candidate image frame and the previous image frame of the first target image frame corresponding to each candidate image frame,

[0121] And determine the second pose difference between the second target image frame corresponding to each candidate image frame and the previous image frame of the second target image frame corresponding to each candidate image frame;

[0122] When at least one of the conditions that the second feature matching degree satisfies the fifth matching degree condition and the distance difference between the second relative position difference and the second pose difference satisfies the fourth distance condition is met, then determine that the loop frame of the first target image frame corresponding to the last image frame is the second preset number of candidate image frames.

[0123] Wherein, the fourth matching degree condition may be that the feature matching degree is greater than 20% and less than 50%, greater than 10% and less than 50%, greater than 10 and less than 60% or other feature matching degrees.

[0124] Optionally, the fifth matching degree condition may be that the feature matching degree is greater than 20% and less than 50%, greater than 10% and less than 50%, greater than 10 and less than 60% or other feature matching degrees. Optionally, the fourth matching degree condition may be the same as the fifth matching degree condition.

[0125] Optionally, the fourth distance condition may be that the distance difference is less than 15 cm, 20 cm, 25 cm or other distance differences. Optionally, the second distance condition, the third distance condition and the fourth distance condition may be the same.

[0126] Optionally, the second preset number may be 5, 10, 15 or other numbers. Optionally, the first preset number may be the same as the second preset number.

[0127] Optionally, the third preset number may be 5, 10, 15 or other numbers. Optionally, the second preset number may be the same as the third preset number.

[0128] Specifically, satisfying that multiple second feature matching degrees satisfy the fifth matching degree condition, and the distance difference between the first relative position difference and the first pose difference satisfies at least one of the fourth distance conditions may refer to satisfying any one of the two conditions or satisfying all of the two conditions.

[0129] It can be understood that the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the fourth matching degree condition, indicating that there is a certain similarity between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, but it cannot be completely determined whether they are collected at similar or the same positions. In order to confirm whether the similarity between the two is sufficient to form a loop for more accurate loop judgment, more image frames are needed to assist in loop detection. Therefore, it can be further determined whether the second feature matching degree satisfies the fifth matching degree condition, and / or whether the distance difference between the second relative position difference and the second pose difference satisfies the fourth distance condition.

[0130] In a specific embodiment, the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the fourth matching degree condition, indicating that the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame may be collected at similar or the same positions. However, since the feature matching degree is not sufficient to completely determine this, in order to further confirm whether the robot has returned to a position it has visited before, it is necessary to confirm whether there is motion continuity and motion similarity that satisfy the loop frame judgment among the consecutive second preset number of candidate image frames collected by the robot at historical acquisition time nodes. Therefore, when both of the two conditions that the second feature matching degree satisfies the fifth matching degree condition and the distance difference between the second relative position difference and the second pose difference satisfies the fourth distance condition are met, it can be considered that the robot has returned to a previously visited position, and at this time, the loop frame of the first target image frame corresponding to the last image frame is determined to be the second preset number of candidate image frames.

[0131] It can be seen that in the case where there is a certain similarity but not a high enough similarity between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, by setting specific loop conditions to find the loop frame, it can be ensured that the second preset number of candidate image frames is the loop frame of the first target image frame corresponding to the last image frame, which can avoid missing the detection of the loop frame, and thus can improve the accuracy of loop detection.

[0132] The application process of the above loop detection method is elaborated below in combination with a detailed embodiment, as follows:

[0133] (1) Acquisition process of multiple image frames

[0134] Obtain multiple image frames collected by the robot, and determine the last image frame and the first preset number of first pre-order image frames of the last image frame among the multiple image frames.

[0135] (2) Determination process of the first target image frame

[0136] Determine the feature matching degree between each first pre-order image frame and the last image frame, and determine the first target image frame corresponding to the last image frame among the first preset number of first pre-order image frames; the first target image frame corresponding to the last image frame is the image frame with the highest feature matching degree with the last image frame among the first preset number of first pre-order image frames.

[0137] (3) Determination process of the second optimized pose of each image frame

[0138] When the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame meets the first matching degree condition, determine the three-dimensional coordinates of the first target image frame corresponding to the last image frame based on the feature matching relationship between the first target image frame corresponding to the last image frame and each image frame and the initial pose of each image frame; determine the calculated pose of the last image frame based on the feature matching relationship between the first target image frame corresponding to the last image frame and the last image frame and the three-dimensional coordinates of the first target image frame corresponding to the last image frame.

[0139] When the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not meet the first matching degree condition, determine the initial pose of the first target image frame corresponding to the last image frame as the calculated pose of the last image frame.

[0140] Determine the second optimized pose of each image frame based on the initial pose of the last image frame, the calculated pose of the last image frame, and the second objective function.

[0141] (4) Determination process of the loop closure pose

[0142] Among multiple image frames, determine at least one peripheral image frame whose distance from the first target image frame corresponding to the last image frame satisfies a first distance condition; determine a second target image frame corresponding to the last image frame that has the highest feature matching degree with the first target image frame corresponding to the last image frame among the at least one peripheral image frame; when the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies a second matching degree condition, determine the first feature matching degree between the previous image frame of the first target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame, determine the first relative position difference between the first target image frame corresponding to the last image frame and the previous image frame of the first target image frame corresponding to the last image frame, determine the first pose difference between the second target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame, and determine the second pose difference between the previous image frame of the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame; when at least one of the following conditions is met: the first feature matching degree satisfies a third matching degree condition, the distance difference between the first relative position difference and the first pose difference satisfies a second distance condition, and the distance difference between the first relative position difference and the second pose difference satisfies a third distance condition, then determine that the loopback frame of the first target image frame corresponding to the last image frame is the second target image frame corresponding to the last image frame;

[0143] When the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies a fourth matching degree condition, determine a second preset number of consecutive candidate image frames and a third preset number of second previous image frames for each candidate image frame among the multiple image frames, respectively determine the feature matching degree between each second previous image frame and each candidate image frame, and determine the first target image frame corresponding to each candidate image frame among the third preset number of second previous image frames for each candidate image frame, and, respectively determine the second target image frame corresponding to the first target image frame corresponding to each candidate image frame among the multiple image frames, thereby determine the second feature matching degree between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame, determine the second relative position difference between the first target image frame corresponding to each candidate image frame and the previous image frame of the first target image frame corresponding to each candidate image frame, and determine the second pose difference between the second target image frame corresponding to each candidate image frame and the previous image frame of the second target image frame corresponding to each candidate image frame. When at least one of the following conditions is met: the second feature matching degree satisfies a fifth matching degree condition, and the distance difference between the second relative position difference and the second pose difference satisfies a fourth distance condition, then determine that the loopback frame of the first target image frame corresponding to the last image frame is the second preset number of candidate image frames.

[0144] Among them, the acquisition time nodes of each peripheral image frame are earlier than the acquisition time node of the first target image frame corresponding to the last image frame, and when the direction of the peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame, the time difference between the acquisition time node of the peripheral image frame and the acquisition time node of the first target image frame corresponding to the last image frame satisfies a preset time condition.

[0145] (5)Determination process of the calculated pose of each image frame

[0146] When at least one loop closure frame of the first target image frame corresponding to the last image frame is determined, the calculated pose of each image frame is determined based on the feature matching relationship between the image frames and the second optimized pose of each image frame.

[0147] (6)Determination process of the first optimized pose of each image frame

[0148] Based on the second optimized pose of each image frame, the calculated pose of each image frame, and the first objective function, the first optimized pose of each image frame is determined.

[0149] In this embodiment, by determining the first target image frame corresponding to the last image frame with the highest feature matching degree with respect to the last image frame, the calculated pose of the last image frame can be determined through the first target image frame corresponding to the last image frame, so as to determine the second optimized pose of each image frame based on the initial pose of the last image frame, the calculated pose of the last image frame, and the second objective function, and the trajectory drift of each image frame is preliminarily corrected through the last image frame. Further, the second target image frame corresponding to the last image frame that is close in distance and has a high similarity to the last image frame is determined through the first target image frame corresponding to the last image frame. Furthermore, based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop closure detection is performed among multiple image frames to obtain at least one loop closure frame of the first target image frame corresponding to the last image frame, and the accuracy of loop closure detection can be improved by ensuring the accuracy of the loop closure frame. It can be seen that the loop closure detection method provided in this embodiment enables the robot to perform accurate loop closure detection in a complex environment. Thus, the robot can distinguish correct loop closures from false loop closures in a complex environment. Furthermore, in a complex environment with similar ceiling features, it can avoid the adverse situation of large map construction errors caused by the robot detecting false loop closures. At the same time, in an environment with dissimilar features, the accuracy of loop closure detection can also be improved to improve the accuracy of map construction.

[0150] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless specifically stated herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least some of the steps or stages in other steps.

[0151] Based on the same inventive concept, an embodiment of the present application further provides a loop detection device for implementing the loop detection method described above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the loop detection device provided below can refer to the limitations on the loop detection method in the foregoing text, and will not be repeated here.

[0152] In an exemplary embodiment, as Figure 2 shown, a loop detection device is provided, including: an acquisition module 202, a first determination module 204, a second determination module 206, and a detection module 208, where:

[0153] The acquisition module 202 is configured to acquire a plurality of image frames collected by the robot, and determine the last image frame and the first preset number of first pre-order image frames of the last image frame among the plurality of image frames.

[0154] The first determination module 204 is configured to determine the feature matching degree between each first pre-order image frame and the last image frame, and determine the first target image frame corresponding to the last image frame among the first preset number of first pre-order image frames; the first target image frame corresponding to the last image frame is the image frame with the highest feature matching degree with the last image frame among the first preset number of first pre-order image frames.

[0155] The second determination module 206 is configured to determine the second target image frame corresponding to the first target image frame corresponding to the last image frame among the plurality of image frames; the second target image frame corresponding to the last image frame is the image frame whose distance from the first target image frame corresponding to the last image frame satisfies the first distance condition and has the highest feature matching degree with the first target image frame corresponding to the last image frame;

[0156] The detection module 208 is configured to perform loop detection on the second target image frame among multiple image frames based on the feature matching degree between the first target image frame and the second target image frame, and obtain the loop pose. Based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed among multiple image frames to determine at least one loop frame corresponding to the first target image frame corresponding to the last image frame.

[0157] In an exemplary embodiment, the above detection module 208 is further configured to optimize the initial pose of each image frame based on the initial pose of each loop frame, the calculated pose of each loop frame, and the first objective function, and determine the first optimized pose of each image frame.

[0158] In an exemplary embodiment, the above first determination module 204 is further configured to optimize the initial pose of each image frame based on the initial pose of the last image frame, the calculated pose of the last image frame, and the second objective function, and determine the second optimized pose of each image frame.

[0159] In an exemplary embodiment, the above first determination module 204 is further configured to, when the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame satisfies the first matching degree condition, determine the three-dimensional coordinates of the first target image frame corresponding to the last image frame based on the feature matching relationship between the first target image frame corresponding to the last image frame and each image frame and the initial pose of each image frame; determine the calculated pose of the last image frame based on the feature matching relationship between the first target image frame corresponding to the last image frame and the last image frame and the three-dimensional coordinates of the first target image frame corresponding to the last image frame.

[0160] In an exemplary embodiment, the above first determination module 204 is further configured to, when the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not satisfy the first matching degree condition, determine the initial pose of the first target image frame corresponding to the last image frame as the calculated pose of the last image frame.

[0161] In an exemplary embodiment, the above-mentioned second determination module 206 is further configured to determine, among multiple image frames, at least one peripheral image frame whose distance from the first target image frame corresponding to the last image frame satisfies a first distance condition; determine, among the at least one peripheral image frame, a second target image frame corresponding to the last image frame that has the highest feature matching degree with the first target image frame corresponding to the last image frame; wherein, the acquisition time node of each peripheral image frame is earlier than the acquisition time node of the first target image frame corresponding to the last image frame, and, when the direction of the peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame, the time difference between the acquisition time node of the peripheral image frame and the acquisition time node of the first target image frame corresponding to the last image frame satisfies a preset time condition.

[0162] In an exemplary embodiment, the above-mentioned detection module 208 is further configured to, when the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies a second matching degree condition, determine the first feature matching degree between the previous image frame of the first target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame, determine the first relative position difference between the first target image frame corresponding to the last image frame and the previous image frame of the first target image frame corresponding to the last image frame, determine the first pose difference between the second target image frame corresponding to the last image frame and the previous image frame of the second target image frame corresponding to the last image frame, and determine the second pose difference between the previous image frame of the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame; when at least one of the following conditions is met: the first feature matching degree satisfies a third matching degree condition, the distance difference between the first relative position difference and the first pose difference satisfies a second distance condition, and the distance difference between the first relative position difference and the second pose difference satisfies a third distance condition, then determine that the loopback frame of the first target image frame corresponding to the last image frame is the second target image frame corresponding to the last image frame.

[0163] In an exemplary embodiment, the above detection module 208 is further configured to, when the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame meets the fourth matching degree condition, determine a continuous second preset number of candidate image frames and a third preset number of second preceding image frames of each candidate image frame among multiple image frames; respectively determine the feature matching degree between each second preceding image frame and each candidate image frame, and determine the first target image frame corresponding to each candidate image frame among the third preset number of second preceding image frames of each candidate image frame, and, respectively determine the second target image frame corresponding to the first target image frame corresponding to each candidate image frame among multiple image frames; perform loop detection through the relationship between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame, so as to determine at least one loop frame of the first target image frame corresponding to the last image frame; wherein, the first target image frame corresponding to the candidate image frame is the image frame with the highest feature matching degree with the candidate image frame among the second preset number of second preceding image frames; the second target image frame corresponding to the candidate image frame is the image frame whose distance from the first target image frame corresponding to the candidate image frame meets the first distance condition and has the highest feature matching degree with the first target image frame corresponding to the candidate image frame.

[0164] In an exemplary embodiment, the above detection module 208 is further configured to determine the second feature matching degree between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame, determine the second relative position difference between the first target image frame corresponding to each candidate image frame and the previous image frame of the first target image frame corresponding to each candidate image frame, and, determine the second pose difference between the second target image frame corresponding to each candidate image frame and the previous image frame of the second target image frame corresponding to each candidate image frame; when at least one of the conditions that the second feature matching degree meets the fifth matching degree condition and the distance difference between the second relative position difference and the second pose difference meets the fourth distance condition is satisfied, then determine that the loop frame of the first target image frame corresponding to the last image frame is the second preset number of candidate image frames.

[0165] Each module in the above loop detection device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above respective modules.

[0166] In an exemplary embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 3As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image frame data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a loop detection method.

[0167] In an exemplary embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 4 shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC), or other technologies. When the computer program is executed by the processor, it implements a loop detection method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0168] Those skilled in the art can understand that Figure 4The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0169] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0170] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0171] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0172] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0173] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0174] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0175] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A loop detection method, characterized in that: The method comprises: Acquire multiple image frames captured by the robot, and determine a last image frame and a first preset number of first preceding image frames of the last image frame from the multiple image frames; Determine a feature matching degree between each of the first preceding image frames and the last image frame, and determine a first target image frame corresponding to the last image frame among a first preset number of the first preceding image frames; the first target image frame corresponding to the last image frame is an image frame having the highest feature matching degree with the last image frame among the first preset number of the first preceding image frames; Determine, among multiple image frames, a second target image frame corresponding to the first target image frame corresponding to the last image frame; the second target image frame corresponding to the last image frame is an image frame whose distance with the first target image frame corresponding to the last image frame among the multiple image frames satisfies a first distance condition and has the highest feature matching degree with the first target image frame corresponding to the last image frame; Based on the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame, loop detection is performed in multiple image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame.

2. The method according to claim 1, characterized in that In the case where at least one loop frame of the first target image frame corresponding to the last image frame is determined, the method further includes: Based on the initial pose of each of the loop frames, the calculated pose of each of the loop frames and the first objective function, the initial pose of each of the image frames is optimized to determine a first optimized pose of each of the image frames.

3. The method according to claim 1, characterized in that: After determining the first target image frame corresponding to the last image frame among the first preset number of the first preceding image frames, the method further includes: Based on the initial pose of the final image frame, the calculated pose of the final image frame and the second objective function, the initial pose of each image frame is optimized to determine a second optimized pose of each image frame.

4. The method according to claim 3, characterized in that The method further comprises: In a case where a feature matching degree between the first target image frame corresponding to the last image frame and the last image frame satisfies a first matching degree condition, determining the three-dimensional coordinates of the first target image frame corresponding to the last image frame based on a feature matching relationship between the first target image frame corresponding to the last image frame and each of the image frames and an initial position and posture of each of the image frames; The calculated position and posture of the last image frame is determined based on a feature matching relationship between a first target image frame corresponding to the last image frame and the last image frame and a three-dimensional coordinate of the first target image frame corresponding to the last image frame.

5. The method according to claim 4, characterized in that When the feature matching degree between the first target image frame corresponding to the last image frame and the last image frame does not satisfy the first matching degree condition, the initial pose of the first target image frame corresponding to the last image frame is determined as the calculated pose of the last image frame.

6. The method according to claim 1, characterized in that The step of determining, among the plurality of image frames, a second target image frame corresponding to the first target image frame corresponding to the last image frame comprises: Determine, among the plurality of image frames, at least one peripheral image frame whose distance to the first target image frame corresponding to the last image frame satisfies a first distance condition; Determine, from at least one peripheral image frame, a second target image frame corresponding to the last image frame having the highest feature matching degree with the first target image frame corresponding to the last image frame; Among them, the acquisition time node of each of the peripheral image frames is earlier than the acquisition time node of the first target image frame corresponding to the last image frame, and when the direction of the peripheral image frame is the same as or opposite to the direction of the first target image frame corresponding to the last image frame, the time difference between the acquisition time node of the peripheral image frame and the acquisition time node of the first target image frame corresponding to the last image frame satisfies the preset time condition.

7. The method according to claim 1, characterized in that The method of performing loop detection in a plurality of image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame based on a feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame comprises: In the case where the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the second matching degree condition, Determining a first feature matching degree between an image frame preceding an image frame of a first target image frame corresponding to the last image frame and an image frame preceding an image frame of a second target image frame corresponding to the last image frame, determining a first relative position difference between a first target image frame corresponding to the last image frame and an image frame preceding the first target image frame corresponding to the last image frame, Determine a first position difference between a second target image frame corresponding to the last image frame and an image frame preceding the second target image frame corresponding to the last image frame, And, determining a second posture difference between an image frame preceding an image frame of a first target image frame corresponding to the last image frame and an image frame preceding an image frame of a second target image frame corresponding to the last image frame; When the first feature matching degree satisfies the third matching degree condition, the distance difference between the first relative position difference and the first posture difference satisfies the second distance condition, and the distance difference between the first relative position difference and the second posture difference satisfies at least one of the third distance conditions, it is determined that the loop frame of the first target image frame corresponding to the last image frame is the second target image frame corresponding to the last image frame.

8. The method according to claim 1, characterized in that The method of performing loop detection in a plurality of image frames to determine at least one loop frame of the first target image frame corresponding to the last image frame based on a feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame comprises: In the case where the feature matching degree between the first target image frame corresponding to the last image frame and the second target image frame corresponding to the last image frame satisfies the fourth matching degree condition, Determining a second preset number of consecutive candidate image frames and a third preset number of second preceding image frames of each of the candidate image frames from the plurality of image frames; Determine the feature matching degree between each second preceding image frame and each candidate image frame respectively, and determine the first target image frame corresponding to each candidate image frame among a third preset number of second preceding image frames of each candidate image frame, and determine the second target image frame corresponding to the first target image frame corresponding to each candidate image frame among the plurality of image frames respectively; Perform loop detection based on the relationship between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame, so as to determine at least one loop frame of the first target image frame corresponding to the last image frame; Among them, the first target image frame corresponding to the candidate image frame is an image frame that has the highest feature matching degree with the candidate image frame among the third preset number of the second preceding image frames; the second target image frame corresponding to the candidate image frame is an image frame whose distance with the first target image frame corresponding to the candidate image frame among multiple image frames satisfies the first distance condition and has the highest feature matching degree with the first target image frame corresponding to the candidate image frame.

9. The method according to claim 8, characterized in that The loop detection is performed based on the relationship between the first target image frame corresponding to each candidate image frame and the second target image frame corresponding to each candidate image frame to determine at least one loop frame of the first target image frame corresponding to the last image frame, including: Determine a second feature matching degree between a first target image frame corresponding to each candidate image frame and a second target image frame corresponding to each candidate image frame, Determine a second relative position difference between a first target image frame corresponding to each candidate image frame and an image frame preceding the first target image frame corresponding to each candidate image frame, and determining a second position difference between a second target image frame corresponding to each candidate image frame and an image frame previous to the second target image frame corresponding to each candidate image frame; When the second feature matching degree satisfies the fifth matching degree condition and the distance difference between the second relative position difference and the second posture difference satisfies at least one of the fourth distance conditions, the loop frame of the first target image frame corresponding to the last image frame is determined to be a second preset number of candidate image frames.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

Images