Closed-loop detection method and device, storage medium and electronic device

Abstract

The present invention discloses a closed-loop detection method and device, a storage medium and an electronic device, and relates to the field of image processing technology. The closed-loop detection method comprises: obtaining each video frame image taken by a camera during motion, extracting the global features and local invariant features of each video frame image; constructing a feature index map according to the global features of each video frame image, and using the feature index map to determine the candidate video frame image to be closed-loop compared with the current video frame image; converting the local invariant features of the current video frame image into a first hash feature, converting the local invariant features of the candidate video frame image into a second hash feature, and using the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image; and determining the closed-loop detection result in combination with the result of whether the current video frame image matches the candidate video frame image. The present disclosure can improve the effect of closed-loop detection.

Description

Background Art

[0002] SLAM (Simultaneous Localization and Mapping) aims to solve the problem of how to perceive the surrounding environment, build an incremental map, and simultaneously localize the vehicle after entering an unknown environment. As one of the key technologies for autonomous navigation, SLAM has applications in a wide range of fields, including unmanned driving and robotic control.

[0003] In SLAM, an important issue is loop closure detection (LCD). Loop closure detection, also known as loop closure detection, refers to the technology that allows the robot to recognize that it has reached a certain scene before, making the constructed map closed.

[0004] However, in some processes of implementing closed-loop detection, large memory usage occurs and it is difficult to ensure a fast detection speed, resulting in unsatisfactory closed-loop detection results.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The purpose of the present disclosure is to provide a closed-loop detection method, a closed-loop detection device, a storage medium, and an electronic device, thereby overcoming, at least to a certain extent, the problem of poor closed-loop detection effect caused by the limitations and defects of related technologies.

[0007] According to one aspect of the present disclosure, a closed-loop detection method is provided, including: obtaining each video frame image taken by a camera during movement, and extracting global features and local invariant features of each video frame image; constructing a feature index map based on the global features of each video frame image, and using the feature index map to determine a candidate video frame image to be closed-loop compared with the current video frame image; converting the local invariant features of the current video frame image into a first hash feature, and converting the local invariant features of the candidate video frame image into a second hash feature, and using the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image; and determining a closed-loop detection result based on the result of whether the current video frame image matches the candidate video frame image.

[0008] Optionally, extracting the global features of each of the video frame images includes: inputting each of the video frame images into a trained convolutional neural network to determine the global features corresponding to each of the video frame images.

[0009] Optionally, extracting the local invariant features of each of the video frame images includes: determining a SURF feature of each of the video frame images as the local invariant feature corresponding to each of the video frame images.

[0010] Optionally, the feature index map is an HNSW map; wherein, using the feature index map to determine the candidate video frame image to be closed-loop compared with the current video frame image includes: configuring a first-in-first-out queue; writing the global features of the first predetermined number of video frame images before the current video frame image in the case of continuous shooting into the first-in-first-out queue, so that the determined candidate video frame image is different from any one of the first predetermined number of video frame images.

[0011] Optionally, the closed-loop detection method further includes: determining a camera frame rate and a time constant; and determining the first predetermined number according to the camera frame rate and the time constant.

[0012] Optionally, using the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image includes: using the first hash feature and the second hash feature to perform a binary ratio test to determine the feature matching points of the current video frame image and the candidate video frame image; using a random sampling consistency algorithm to eliminate erroneous matching points from the feature matching points; and judging whether the current video frame image matches the candidate video frame image based on the feature matching points after eliminating the erroneous matching points.

[0013] Optionally, judging whether the current video frame image matches the candidate video frame image based on the feature matching points after removing the erroneous matching points includes: determining the number of feature matching points after removing the erroneous matching points; if the number of feature matching points after removing the erroneous matching points is not less than a predetermined matching parameter, determining that the current video frame image matches the candidate video frame image.

[0014] Optionally, judging whether the current video frame image matches the candidate video frame image based on the feature matching points after removing the erroneous matching points includes: estimating a basic matrix based on the feature matching points after removing the erroneous matching points; and judging whether the current video frame image matches the candidate video frame image based on the basic matrix.

[0015] Optionally, determining the closed-loop detection result in combination with the result of whether the current video frame image matches the candidate video frame image includes: when the current video frame image matches the candidate video frame image, if a second predetermined number of video frame images before the current video frame image all match the candidate video frame image, determining that the closed-loop detection is completed.

[0016] According to one aspect of the present disclosure, a closed-loop detection apparatus is provided, comprising a feature extraction module, a candidate frame determination module, an image matching module, and a closed-loop detection module.

[0017] Specifically, the feature extraction module is used to obtain each video frame image taken by the camera during movement, and extract the global features and local invariant features of each video frame image; the candidate frame determination module is used to construct a feature index map based on the global features of each video frame image, and use the feature index map to determine the candidate video frame image to be compared with the current video frame image; the image matching module is used to convert the local invariant features of the current video frame image into a first hash feature, and convert the local invariant features of the candidate video frame image into a second hash feature, and use the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image; the closed-loop detection module is used to determine the closed-loop detection result based on the result of whether the current video frame image matches the candidate video frame image.

[0018] Optionally, the feature extraction module includes a first feature extraction unit.

[0019] Specifically, the first feature extraction unit is used to input each of the video frame images into a trained convolutional neural network to determine the global features corresponding to each of the video frame images.

[0020] Optionally, the feature extraction module includes a second feature extraction unit.

[0021] Specifically, the second feature extraction unit is used to determine the SURF features of each of the video frame images as the local invariant features corresponding to each of the video frame images.

[0022] Optionally, the feature index map is an HNSW map; wherein the candidate frame determination module is configured to execute: configuring a first-in-first-out queue; writing the global features of the first predetermined number of video frame images before the current video frame image in the case of continuous shooting into the first-in-first-out queue, so that the determined candidate video frame image is different from any one of the first predetermined number of video frame images.

[0023] Optionally, the candidate frame determination module is further configured to execute: determining a camera frame rate and a time constant; and determining the first predetermined number according to the camera frame rate and the time constant.

[0024] Optionally, the image matching module includes a matching point determination unit, an erroneous point elimination unit and an image matching unit.

[0025] Specifically, the matching point determination unit is used to perform a binary ratio test using the first hash feature and the second hash feature to determine the feature matching points between the current video frame image and the candidate video frame image; the error point elimination unit is used to eliminate error matching points from the feature matching points using a random sampling consistency algorithm; and the image matching unit is used to determine whether the current video frame image matches the candidate video frame image based on the feature matching points after eliminating the error matching points.

[0026] Optionally, the image matching unit is configured to perform: determining the number of feature matching points after removing the erroneous matching points; if the number of feature matching points after removing the erroneous matching points is not less than a predetermined matching parameter, determining that the current video frame image matches the candidate video frame image.

[0027] Optionally, the image matching unit is configured to perform: estimating a basic matrix based on the feature matching points after eliminating the erroneous matching points; and judging whether the current video frame image matches the candidate video frame image according to the basic matrix.

[0028] Optionally, the closed-loop detection module is configured to execute: when the current video frame image matches the candidate video frame image, if a second predetermined number of video frame images before the current video frame image all match the candidate video frame image, determining that the closed-loop detection is completed.

[0029] According to one aspect of the present disclosure, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above closed-loop detection methods is implemented.

[0030] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned closed-loop detection methods by executing the executable instructions.

[0031] In the technical solutions provided by some embodiments of the present disclosure, a feature index map is constructed by utilizing the global features of each video frame image to determine the candidate video frame image, and the local invariant features of each video frame image are hashed. Based on the hash features, it is determined whether the current video frame image matches the candidate video frame image, and the closed-loop detection result is determined in combination with the matching result. On the one hand, converting the local invariant features of the image into hash features can greatly reduce the memory usage. Therefore, the solution described in the present disclosure can be applied to robot systems with smaller memory, expanding the application scope of closed-loop detection and even SLAM technology; on the other hand, by constructing a feature index map and using the feature index map to determine the candidate video frame image, image matching can be quickly achieved, thereby improving the detection speed of closed-loop detection; on the other hand, the closed-loop detection solution disclosed in the present disclosure is a real-time and online detection solution that can be applied to various fields using autonomous navigation technology.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0034] Figure 1 Schematically shows a flow chart of a closed-loop detection method according to an exemplary embodiment of the present disclosure;

[0035] Figure 2 A schematic diagram illustrating matching of a current video frame image with a candidate video frame image according to an exemplary embodiment of the present disclosure is shown;

[0036] Figure 3 Schematically shows a block diagram of a closed-loop detection device according to an exemplary embodiment of the present disclosure;

[0037] Figure 4 Schematically shows a block diagram of a feature extraction module according to an exemplary embodiment of the present disclosure;

[0038] Figure 5 Schematically shows a block diagram of a feature extraction module according to another exemplary embodiment of the present disclosure;

[0039] Figure 6 Schematically shows a block diagram of an image matching module according to an exemplary embodiment of the present disclosure;

[0040] Figure 7 A block diagram schematically illustrates an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0042] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0043] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all steps. For example, some steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation. In addition, the terms "first," "second," etc., used below are for distinction purposes only and should not be construed as limitations of this disclosure.

[0044] It should be understood that the closed-loop detection method described below can generally be implemented by a terminal device. That is, the terminal device can perform each step of the closed-loop detection method. The terminal device may include, but is not limited to, a mobile phone, a tablet, a wearable device, various robotic devices, etc. In this case, the closed-loop detection device may be included in the terminal device.

[0045] Although the following description uses a terminal device executing the closed-loop detection method disclosed herein as an example, the various steps of the closed-loop detection method described below can also be executed by a server. Specifically, the server can obtain each video frame image of the scene from the camera and analyze and process each video frame image to determine the result of the closed-loop detection.

[0046] Figure 1 The flowchart of the closed-loop detection method according to an exemplary embodiment of the present disclosure is schematically shown. Figure 1 , the closed-loop detection method may include the following steps:

[0047] S12. Obtain each video frame image captured by the camera during the movement, and extract the global features and local invariant features of each video frame image.

[0048] In an exemplary embodiment of the present disclosure, the camera can be a camera integrated with the terminal device, for example, the camera can be the rear camera of a mobile phone. Alternatively, the camera can be a camera configured separately from the terminal device and capable of sending image data to the terminal device. In this case, the camera can capture video frames of a real environment and send the image data to the terminal device via a wired or wireless method.

[0049] The embodiment of the present disclosure can use a monocular camera to obtain video frame images captured during the motion process. In this case, the closed-loop detection method described in the present disclosure can be applied to the monocular SLAM technology. Among them, the monocular camera is a camera constructed based on the pinhole imaging principle. It is a single color (or grayscale) camera, which has the advantages of being cheap, easy to use, highly accurate, and convenient to configure. However, in some embodiments, binocular or multi-cameras can also be used to obtain images to perform subsequent analysis and processing. The present disclosure does not impose any special restrictions on the type, number, resolution, etc. of the camera.

[0050] In real-world scenarios, as the camera continuously moves, it can capture multiple video frames. After acquiring these video frames, the terminal device can extract corresponding global features and local invariant features from each video frame.

[0051] On the one hand, the global features of a video frame image can represent the overall information of the video frame image, and are pixel-level underlying features, which may involve the overall color features of the video frame image, the overall texture features of the image, the overall spatial layout features of the image, etc. On the other hand, the local invariant features of a video frame image refer to features that maintain stability of the image under geometric changes, illumination changes, and / or noise interference. The local invariant features described in the present disclosure will be described below using SURF (Speeded Up Robust Features) as an example. However, it should be understood that the local invariant features described in the present disclosure may also include Sift (Scale-invariant feature transform) features, Harris corners, ORB features, etc.

[0052] The process of determining the global features of a video frame image:

[0053] According to some embodiments of the present disclosure, a machine learning model can be used to extract global features from video frame images. Specifically, the present disclosure can use a lightweight deep convolutional neural network such as MobileNetV2 to extract global features through the last average pooling layer of the network.

[0054] The MobileNetV2 model simplifies the network structure by incorporating a batch normalization layer (BN layer), reducing the forward propagation time. The specific calculation process of incorporating the BN layer can be shown as follows:

[0055]

[0056] Among them, W BN ∈R C×C , b BN ∈R C , respectively represent the normalized versions of the weight matrix and feature map F Bias. The parameters before the BN layer can be expressed as where b conv ∈R C , and C prev Input convolutional layer feature map F prev The number of channels, k×k is the size of the convolution kernel. prev The k×k neighborhood of 2 ·C prev Vector i，j The BN layer and the previous convolution layer can be replaced by a single convolution layer, and the parameters of the single convolution layer can be expressed as:

[0057] W=W BN W conv

[0058] b=W BN b conv +b BN

[0059] It should be understood that the process of extracting each video frame image using MobileNetV2 is only an exemplary solution of the present disclosure. In other examples, the present disclosure may also use the output results of the fully connected layer of the convolutional neural network as global features. The present disclosure does not place any special restrictions on the specific structure of the convolutional neural network.

[0060] In addition, methods such as Histogram of Oriented Gradient (HOG) can be used to extract global features of each video frame image.

[0061] The process of determining the local invariant features of video frame images:

[0062] In the example of using SURF features as local invariant features, the Hessian matrix can be used to detect feature points. Specifically, the determinant value corresponding to the Hessian matrix can be determined, which represents the amount of change around the pixel point. Next, the local extreme value is determined among all the determinant values ​​as the local invariant feature.

[0063] In addition, in order to achieve high accuracy in loop closure detection, 128-dimensional SURF features can be used.

[0064] S14. Construct a feature index map based on the global features of each of the video frame images, and use the feature index map to determine a candidate video frame image to be closed-loop compared with the current video frame image.

[0065] In an exemplary embodiment of the present disclosure, the feature index graph may be a HNSW graph (Hierarchical Navigable Small World graphs). The HNSW graph is a graph-based incremental k-nearest neighbor search structure that has logarithmic or multi-logarithmic scalability in graph routing.

[0066] An HNSW graph can be constructed based on the global features of each video frame image. Each node in the constructed HNSW graph corresponds to the global feature vector of a video frame image. Specifically, for the global feature vector q of a newly acquired image, after inserting it into the HNSW graph, a KNN search algorithm can be used to query the global feature vectors closest to the global feature vector q. The video frame images corresponding to the determined global feature vectors are then used as candidate video frame images as described in this disclosure. These candidate video frame images are then compared with the current video frame image to determine the closed-loop detection result.

[0067] In the process of searching the HNSW graph, the scheme involving determining the similarity between global features can use the cosine distance to determine the similarity between features. For details, refer to the following formula:

[0068]

[0069] Among them, s pq represents the similarity between the global feature vector p and the global feature vector q, X p and X q denote the global eigenvector p and the global eigenvector q respectively, is the L2 norm of vector X.

[0070] According to some embodiments of the present disclosure, considering that video frame images are acquired sequentially and adjacent images have a high degree of similarity, which may affect the results of loop closure detection, a first-in-first-out (FIFO) queue can be configured to write the global features of the first predetermined number of video frame images before the current video frame image into the FIFO queue, thereby determining that the candidate video frame image is different from any of the first predetermined number of video frame images.

[0071] To determine the first predetermined number, first, the camera frame rate φ and a time constant ψ can be determined, where the time constant ψ can be manually set. Next, the first predetermined number can be determined based on the camera frame rate φ and the time constant ψ. Specifically, the first predetermined number can be ψ×φ. That is, the HNSW graph insertion and search process will only be performed if the number of image frames entering the first-in-first-out queue exceeds ψ×φ; otherwise, the images will be entered into the first-in-first-out queue first. In this case, when the global features of the current video frame image are used as the search features, only the database content of N-ψ×φ is searched, where N represents the total number of all images.

[0072] S16. Convert the local invariant features of the current video frame image into a first hash feature, convert the local invariant features of the candidate video frame image into a second hash feature, and use the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image.

[0073] It should be understood that the local invariant features are floating-point descriptors. For some terminal devices (e.g., mobile robots), the memory resources they are equipped with are limited, making it difficult to process these floating-point numbers. In this case, the present disclosure converts these features into hash features for ease of subsequent processing.

[0074] Specifically, the CasHash algorithm can be used to convert the local invariant features of the video frame image, converting the local invariant features of the current video frame image into a first hash feature and converting the local invariant features of the candidate video frame image into a second hash feature. Taking the 128-dimensional SURF feature as an example, 128 floating-point numbers can be converted to 128 bits, reducing memory consumption to only 1 / 32 of the original. This makes the disclosed closed-loop detection solution applicable to mobile robots with relatively small memory requirements.

[0075] After determining the first hash feature and the second hash feature, first, a binary ratio test can be performed using the first hash feature and the second hash feature to determine feature matching points between the current video frame image and the candidate video frame image. Specifically, for the binary ratio test process, the binary ratio threshold ε can be expressed as follows:

[0076]

[0077] Among them, d h (·) indicates Hamming distance calculation, C a It is picture I a The descriptor feature f in a The binary hash feature of and It is in the picture I b The two closest descriptor features in and Binary hash features of . Below the threshold ε 2 The feature points are regarded as matching feature points.

[0078] After determining the feature matching points between the current video frame image and the candidate video frame image, the Random Sample Consensus (RANSAC) algorithm can be used to eliminate incorrect matching points from these feature matching points. In addition, the RANSAC algorithm can record the correct matching points as inliers and the incorrect matching points as outliers.

[0079] Next, whether the current video frame image matches the candidate video frame image can be determined based on the feature matching points after removing the false matching points, that is, whether the current video frame image matches the candidate video frame image can be determined based on the inliers.

[0080] According to one embodiment of the present disclosure, the number of inliers can be determined. If the number of inliers is not less than a predetermined matching parameter τ, the current video frame image is determined to match the candidate video frame image. If the number of inliers is less than the predetermined matching parameter τ, the candidate video frame image for loop closure detection is ignored. The predetermined matching parameter τ can be set manually, and the present disclosure does not impose any specific restrictions on its value.

[0081] According to another embodiment of the present disclosure, during the process of eliminating outliers, the fundamental matrix of the current video frame image and the candidate frame image can be estimated based on the RANSAC algorithm. This fundamental matrix can be used to eliminate false matches between the images, thereby significantly improving the robustness and accuracy of the matching. In this case, whether the current video frame image matches the candidate video frame image can be determined based on the fundamental matrix. Specifically, if the fundamental matrix calculation fails, the candidate video frame image for loop closure detection can be ignored.

[0082] Figure 2 Schematic diagram showing the matching of the current video frame image and the candidate video frame image. Figure 2 It can be seen that there are many feature matching points as inliers. For example, the pixels corresponding to spires, trees, houses, etc. are all feature matching points.

[0083] S18. Determine a closed-loop detection result based on the result of whether the current video frame image matches the candidate video frame image.

[0084] According to some embodiments of the present disclosure, if the current video frame image matches the candidate video frame image, the closed-loop detection process ends. For example, in the application scenario of a mobile robot, at this time, the mobile robot confirms that it has arrived at the current scene.

[0085] According to other embodiments of the present disclosure, the closed-loop detection result can be determined in combination with the principle of temporal consistency. Specifically, when it is determined that the current video frame image matches the candidate video frame image, if the second predetermined number of video frame images before the current video frame image all match the candidate video frame image, the closed-loop detection process ends. Among them, the second predetermined number can be set to 1, that is, if two consecutive images match the candidate video frame image, it is determined that the closed-loop detection process ends. However, it should be understood that, considering the specific application scenarios and accuracy of closed-loop detection, the second predetermined number can also be set to other values, and this exemplary embodiment does not specifically limit this.

[0086] It should be noted that although the steps of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0087] Furthermore, this exemplary embodiment also provides a closed-loop detection device.

[0088] Figure 3Schematically shows a block diagram of a closed-loop detection device according to an exemplary embodiment of the present disclosure. Figure 3 The closed-loop detection apparatus 3 according to an exemplary embodiment of the present disclosure may include a feature extraction module 31 , a candidate frame determination module 33 , an image matching module 35 and a closed-loop detection module 37 .

[0089] Specifically, the feature extraction module 31 can be used to obtain each video frame image taken by the camera during movement, and extract the global features and local invariant features of each video frame image; the candidate frame determination module 33 can be used to construct a feature index map based on the global features of each video frame image, and use the feature index map to determine the candidate video frame image to be compared with the current video frame image; the image matching module 35 can be used to convert the local invariant features of the current video frame image into a first hash feature, and convert the local invariant features of the candidate video frame image into a second hash feature, and use the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image; the closed-loop detection module 37 can be used to determine the closed-loop detection result in combination with the result of whether the current video frame image matches the candidate video frame image.

[0090] According to the closed-loop detection device of the exemplary embodiment of the present disclosure, on the one hand, converting the local invariant features of the image into hash features can greatly reduce the memory usage. Therefore, the solution described in the present disclosure can be applied to robot systems with smaller memory, expanding the application scope of closed-loop detection and even SLAM technology; on the other hand, by constructing a feature index map and using the feature index map to determine the candidate video frame image, image matching can be quickly achieved, thereby improving the detection speed of closed-loop detection; on the other hand, the closed-loop detection solution of the present disclosure is a real-time and online detection solution, which can be applied to various fields using autonomous navigation technology.

[0091] According to an exemplary embodiment of the present disclosure, referring to Figure 4 , the feature extraction module 31 may include a first feature extraction unit 401.

[0092] Specifically, the first feature extraction unit 401 can be used to input each of the video frame images into a trained convolutional neural network to determine the global features corresponding to each of the video frame images.

[0093] According to an exemplary embodiment of the present disclosure, referring to Figure 5 , the feature extraction module 31 may include a second feature extraction unit 501.

[0094] Specifically, the second feature extraction unit 501 may be configured to determine the SURF features of each of the video frame images as the local invariant features corresponding to each of the video frame images.

[0095] According to an exemplary embodiment of the present disclosure, the feature index map is an HNSW map; wherein the candidate frame determination module 33 can be configured to execute: configuring a first-in-first-out queue; writing the global features of the first predetermined number of video frame images before the current video frame image in a continuous shooting situation into the first-in-first-out queue, so that the determined candidate video frame image is different from any one of the first predetermined number of video frame images.

[0096] According to an exemplary embodiment of the present disclosure, the candidate frame determination module 33 may be further configured to execute: determining a camera frame rate and a time constant; and determining the first predetermined number according to the camera frame rate and the time constant.

[0097] According to an exemplary embodiment of the present disclosure, referring to Figure 6 The image matching module 35 may include a matching point determination unit 601 , an error point elimination unit 603 and an image matching unit 605 .

[0098] Specifically, the matching point determination unit 601 can be used to perform a binary ratio test using the first hash feature and the second hash feature to determine the feature matching points between the current video frame image and the candidate video frame image; the error point elimination unit 603 can be used to eliminate error matching points from the feature matching points using a random sampling consistency algorithm; the image matching unit 605 can be used to determine whether the current video frame image matches the candidate video frame image based on the feature matching points after eliminating the error matching points.

[0099] According to an exemplary embodiment of the present disclosure, the image matching unit 605 can be configured to perform: determining the number of feature matching points after eliminating the erroneous matching points; if the number of feature matching points after eliminating the erroneous matching points is not less than a predetermined matching parameter, determining that the current video frame image matches the candidate video frame image.

[0100] According to an exemplary embodiment of the present disclosure, the image matching unit 605 may be configured to perform: estimating a basic matrix based on feature matching points after removing the erroneous matching points; and determining whether the current video frame image matches the candidate video frame image based on the basic matrix.

[0101] According to an exemplary embodiment of the present disclosure, the closed-loop detection module 37 can be configured to perform: when the current video frame image matches the candidate video frame image, if a second predetermined number of video frame images before the current video frame image all match the candidate video frame image, it is determined that the closed-loop detection is ended.

[0102] Since the functional modules of the program execution performance analysis device according to the embodiment of the present invention are the same as those in the above-mentioned method invention embodiment, they will not be described in detail here.

[0103] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0104] The program product for implementing the above-described method according to an embodiment of the present invention may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0105] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical disk, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0106] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0107] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0108] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0109] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0110] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0111] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present invention will be described. Figure 7 The electronic device 700 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0112] like Figure 7 As shown, electronic device 700 is implemented as a general-purpose computing device. Components of electronic device 700 may include, but are not limited to, the aforementioned at least one processing unit 710, the aforementioned at least one storage unit 720, a bus 730 connecting various system components (including storage unit 720 and processing unit 710), and a display unit 740.

[0113] The storage unit stores program codes that can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 may perform the following steps: Figure 1Step S12 shown in: obtaining each video frame image taken by the camera during movement, and extracting the global features and local invariant features of each video frame image; step S14: constructing a feature index map based on the global features of each video frame image, and using the feature index map to determine the candidate video frame image to be closed-loop compared with the current video frame image; step S16: converting the local invariant features of the current video frame image into a first hash feature, and converting the local invariant features of the candidate video frame image into a second hash feature, and using the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image; step S18: determining the closed-loop detection result based on the result of whether the current video frame image matches the candidate video frame image.

[0114] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache memory unit 7202 , and may further include a read-only memory unit (ROM) 7203 .

[0115] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0116] Bus 730 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0117] The electronic device 700 can also communicate with one or more external devices 800 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 700, and / or any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 750. Furthermore, the electronic device 700 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 700, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0118] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0119] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0120] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0121] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0122] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A closed-loop detection method, characterized in that: include: Acquire each video frame image captured by the camera during the motion process, and extract the global features and local invariant features of each video frame image; Constructing a feature index map according to the global features of each of the video frame images, and using the feature index map to determine a candidate video frame image to be closed-loop compared with the current video frame image; the feature index map is an HNSW map; Converting the local invariant features of the current video frame image into a first hash feature, converting the local invariant features of the candidate video frame image into a second hash feature, and using the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image; Determine a closed-loop detection result based on a result of whether the current video frame image matches the candidate video frame image; Wherein, using the feature index map to determine a candidate video frame image to be closed-loop compared with the current video frame image includes: Configure a first-in, first-out queue; The global features of the first predetermined number of video frame images before the current video frame image in the case of continuous shooting are written into the first-in-first-out queue, so that the determined candidate video frame image is different from any one of the first predetermined number of video frame images; the first predetermined number is determined according to the camera frame rate and a time constant.

2. The closed-loop detection method according to claim 1, characterized in that: Extracting the global features of each of the video frame images includes: Each of the video frame images is input into a trained convolutional neural network to determine the global features corresponding to each of the video frame images.

3. The closed-loop detection method according to claim 1, characterized in that: Extracting the local invariant features of each of the video frame images includes: The SURF features of each of the video frame images are determined as local invariant features corresponding to each of the video frame images.

4. The closed-loop detection method according to claim 1, characterized in that: Using the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image includes: Performing a binary ratio test using the first hash feature and the second hash feature to determine feature matching points between the current video frame image and the candidate video frame image; Using a random sampling consistency algorithm to remove mismatched points from the feature matching points; Based on the feature matching points after eliminating the erroneous matching points, it is determined whether the current video frame image matches the candidate video frame image.

5. The closed-loop detection method according to claim 4, characterized in that: Judging whether the current video frame image matches the candidate video frame image based on the feature matching points after eliminating the erroneous matching points includes: Determine the number of feature matching points after removing the mismatched points; If the number of feature matching points after removing the erroneous matching points is not less than a predetermined matching parameter, it is determined that the current video frame image matches the candidate video frame image.

6. The closed-loop detection method according to claim 4, characterized in that: Judging whether the current video frame image matches the candidate video frame image based on the feature matching points after eliminating the erroneous matching points includes: Estimate a basic matrix based on the feature matching points after removing the mismatched points; It is determined whether the current video frame image matches the candidate video frame image according to the basic matrix.

7. The closed-loop detection method according to claim 1, characterized in that: Determining the closed-loop detection result by combining the result of whether the current video frame image matches the candidate video frame image includes: When the current video frame image matches the candidate video frame image, if a second predetermined number of video frame images before the current video frame image all match the candidate video frame image, it is determined that the closed-loop detection is ended.

8. A closed-loop detection device, characterized in that: include: A feature extraction module is used to obtain each video frame image captured by the camera during movement, and extract the global features and local invariant features of each video frame image; A candidate frame determination module, used to construct a feature index map according to the global features of each of the video frame images, and use the feature index map to determine the candidate video frame image to be compared with the current video frame image; the feature index map is an HNSW map; An image matching module, configured to convert the local invariant features of the current video frame image into a first hash feature, convert the local invariant features of the candidate video frame image into a second hash feature, and use the first hash feature and the second hash feature to determine whether the current video frame image matches the candidate video frame image; A closed-loop detection module, configured to determine a closed-loop detection result based on a result of whether the current video frame image matches the candidate video frame image; The candidate frame determination module is also used to configure a first-in-first-out queue; write the global features of the first predetermined number of video frame images before the current video frame image in the case of continuous shooting into the first-in-first-out queue, so that the determined candidate video frame image is different from any one of the first predetermined number of video frame images; the first predetermined number is determined based on the camera frame rate and a time constant.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the closed-loop detection method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the closed-loop detection method according to any one of claims 1 to 7 by executing the executable instructions.

Images