A visual loop detector based on depth image fusion technology

Abstract

The present disclosure provides a visual loop detection method based on a depth image fusion technology, comprising: initializing loop detection data; extracting current picture features, comprising: using a pre-trained depth network model to extract features of a current picture with a frame number f; calculating the similarity between the current picture and a feature set, and determining a candidate frame based on the similarity; calculating the similarity between the current picture and the feature set through a similarity calculation method, and determining the candidate frame based on the similarity; grouping the candidate frame to form several candidate sets, and determining a final candidate set; and determining a loop frame through loop detection. The present disclosure also provides a visual loop detection device based on a depth image fusion technology, an electronic device, and a readable storage medium.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of self-localization and mapping (SLAM) of computer vision, and particularly relates to a visual loop closure detection method and device based on a depth image fusion technique, an electronic device, and a readable storage medium. BACKGROUND

[0002] With the development of artificial intelligence, robots are widely used in various fields such as transportation, home, industry, and production, and SLAM systems have become the focus of research by many scholars. The loop closure detection algorithm is a key component element in the visual SLAM architecture, which can effectively constrain the pose of the backend, thereby reducing the cumulative error brought in the process of estimating the robot motion, and plays a decisive role in realizing the simultaneous localization and mapping of mobile robots, and is an indispensable part of the SLAM system.

[0003] Loop closure detection is to identify the scenes that the camera has passed through, and when a loop is detected, a constraint can be added to the global trajectory to reduce the cumulative drift. The loop closure detection problem is essentially a scene recognition problem, and traditional methods match sparse features or pixel-level dense features designed by humans, while deep learning methods can learn deep features of images through neural networks, which can reduce the influence of subjectivity of human methods on accuracy.

[0004] (1) Traditional loop closure detection technology

[0005] Traditional loop closure detection module implementation methods can be divided into two types, a method dependent on a traditional odometer and a method based on visual appearance. The method dependent on the odometer needs to estimate whether the camera has returned to the previously passed position through the data of the odometer, and then make a loop closure judgment, but loop closure detection itself is to correct the error caused by the visual odometer, so the accuracy of this method is not high. The method based on visual appearance has become the mainstream method of loop closure detection because of its stability. Here, the traditional loop closure detection method based on visual appearance will be mainly introduced.

[0006] The earliest detection method is to perform feature matching on any two images in the picture sequence, and determine whether the two images are associated according to the number of correct matches. Although this method is effective, the number of images to be detected is too large, and the time complexity is very high. Randomly checking historical data and performing loop closure detection is another simple and practical method. However, when the number of frames increases, the probability of drawing a loop decreases significantly, which makes the operation efficiency not high. Therefore, these two methods have been gradually eliminated because they cannot meet the actual requirements.

[0007] In the early 21st century, a new appearance-based loop closure detection method was proposed, which can determine the loop closure detection relationship only according to the similarity of two images, and can be effectively used in many scenes. Many advanced appearance-based loop closure detection methods are based on the bag-of-words model. The bag-of-words model comes from the field of information retrieval and natural language processing. In 2005, Li et al. proposed applying the bag-of-words model to the field of image classification and retrieval. In loop closure detection, the bag-of-words model is to collect a large number of pictures, extract feature points in the pictures, and then cluster them by K-means method to divide the feature points into K categories, and establish a dictionary of feature point sets. When using, input a picture, extract the feature points of the picture, then consult the dictionary, and use a vector to represent the category to which the picture belongs, and then the representation vector can be compared to judge the similarity of the two pictures. The bag-of-words model has become the mainstream and has been applied to practical systems.

[0008] The effect of the bag-of-words model depends largely on the generation of feature descriptions. Considering that using image point features for dictionary training will lose a lot of useful image information, Lee et al. used line features for loop closure detection, using MSLD (mean standard-deviation line descriptor) as a line feature descriptor. Dong Ruifang et al. used LBD (line band descriptor) as a line feature descriptor, which has better description performance and higher computational efficiency. These methods have improved the bag-of-words model method to some extent, making it achieve better results.

[0009] However, the bag-of-words model also has some drawbacks. Visual features, which are actually the content of the "word", are often designed manually, and like traditional visual odometry, there are many types of features, and it is still unknown which method is the most suitable. On the other hand, visual features designed according to human perception are more dependent on human subjective perception. If we think that two pictures are similar, then their feature points in the feature space should be close, but in fact, manually designed feature points cannot guarantee this. Using deep learning methods to handle loop closure detection tasks is trying to solve these problems. The following describes the method of using deep learning to solve the loop closure detection task.

[0010] (2) Deep learning method

[0011] Traditional loop closure detection methods match sparse features or pixel-level dense features designed by hand, while deep learning can learn complex information in images through neural networks, and the recognition rate can reach a higher level.

[0012] Early deep learning based loop closure detection techniques mainly use pre-trained convolutional neural network architecture for image feature extraction, and then compare the image features to obtain a similarity score. Chen et al. first proposed a location recognition technology based on a convolutional neural network model. This method uses a convolutional neural network to extract image features and combines spatial continuity and sequential continuity filters to judge the results. The method has a very high average accuracy, which exceeds all loop closure detection technologies at that time. Hou et al. use the AlexNet model for feature extraction. Through experiments, the method performs similarly to the most advanced method in environments without significant light changes. In environments with significant light changes, the accuracy and speed of the method are superior to other methods.

[0013] However, due to the limitations of neural network structures pre-trained on a certain type of data in application, researchers try to find ways to improve or fine-tune existing neural networks. Sünderhauf et al. compared the performance of different methods in location recognition tasks and found that neural networks trained for semantic location classification are superior to networks trained for object recognition in cases of severe appearance changes. They also found that features from intermediate layers of convolutional neural networks have higher robustness to scene appearance changes caused by seasons and weather changes, features from deeper networks can better represent semantic information of locations, and features from shallower networks have higher robustness to changes in viewing angle. Bai et al. believe that in a sequence of images, adjacent images may be more similar than images with loops. To solve this problem, they provide a method to limit the range of image matching, and also propose a scheme that can use local sensitive hashing algorithm for image compression to improve real-time performance. Gao et al. use stacked autoencoders to extract image features. When training the network, they consider the influence of noise and pose continuity. This method treats images as a collection of image blocks when performing loop closure detection, compares the similarity between each image block in two frames to obtain a similarity score between the two frames, and then constructs a similarity matrix to determine whether a loop has occurred. Randjelovic et al. designed a new convolutional neural network framework that can be directly trained for location recognition tasks, and developed a training program based on weakly supervised ranking loss. The results of training using this framework not only have robustness to light and viewing angle changes, but also pay attention to information such as buildings that have distinctive features in images.

[0014] Compared with traditional loop closure detection algorithms, deep learning based methods use deep neural networks to extract image features, reducing the influence of human subjective judgment. The features can more significantly express image information and have stronger robustness to light and viewing angle changes. SUMMARY

[0015] To solve at least one of the above technical problems, the present disclosure provides a visual loop detection method and device based on depth image fusion technology, an electronic device, and a readable storage medium.

[0016] According to an aspect of the present disclosure, a visual loop detection method based on depth image fusion technology is provided, comprising:

[0017] Initializing loop detection data, comprising: setting a feature set, the feature set comprising a feature and a picture number, the feature set feature being initialized as empty, and the picture number N pic of the feature set being initialized as 0, setting a neighboring frame range threshold value as K, setting an initial loop appearing frame sequence number f1, the f1 being a last loop frame sequence number in a loop detection process, initializing a current picture frame sequence number as f, setting a similarity threshold value as M, and setting a grouping threshold value as m;

[0018] Extracting a current picture feature, comprising: using a pre-trained depth network model to extract a feature of a current picture with a frame sequence number f;

[0019] If a frame number between the current picture and a last loop frame is less than a threshold value K, or a picture number N pic of the feature set is less than a threshold value K, adding the feature of the current picture into the feature set, increasing the picture number N pic of the feature set by 1, increasing the frame sequence number f of the current picture by 1, and continuing the above-mentioned extraction of the current picture feature, otherwise, calculating a similarity between the current picture and the feature set by a similarity calculation method, determining a candidate frame based on the similarity;

[0020] Grouping the candidate frame to form a plurality of candidate sets, and determining a final candidate set; and,

[0021] Determining a loop frame by loop detection, comprising: selecting a picture corresponding to a feature with the highest similarity to the current picture in the final candidate set, and recording a frame sequence number of the picture as h; and inputting the current picture with the frame sequence number f and the picture with the frame sequence number h into a visual odometer to determine whether there is a loop.

[0022] According to the visual loop detection method based on depth image fusion technology of at least one embodiment of the present disclosure, calculating a similarity between a current picture and a feature set, and determining a candidate frame based on the similarity, comprises:

[0023] Removing part of features in the feature of the current picture to obtain remaining features as the current picture feature, the part of features referring to respective picture features in K frames adjacent to the current picture; and,

[0024] calculating a similarity of the current picture feature and the features in the feature set, and determining a relationship between the similarity S and a threshold M, if the similarity S is greater than the threshold M, the current picture is taken as a candidate frame, otherwise, the extracting of the current picture feature is continued.

[0025] According to the visual loop detection method based on the depth image fusion technology, the candidate frames are grouped to form several candidate sets, and a final candidate set is determined, including:

[0026] candidate frames with a frame number difference less than a grouping threshold m are taken as a candidate set L k ;

[0027] a similarity score sum of each candidate set L k is calculated; and,

[0028] the candidate set with the highest similarity score sum is taken as the final candidate set.

[0029] According to the visual loop detection method based on the depth image fusion technology, a current picture with a frame number f and a picture with a frame number h are input into a visual odometer to determine whether there is a loop, including:

[0030] a relative pose of the current picture with the frame number f and the picture with the frame number h is calculated by the visual odometer, and the calculated relative pose is compared with an existing result, if a difference is lower than a pose difference threshold, it is determined that there is a loop between the current picture with the frame number f and the picture with the frame number h, and the existing result is a result calculated in an existing pose graph.

[0031] According to the visual loop detection method based on the depth image fusion technology, further including:

[0032] The picture feature extraction is realized by a depth separable convolution of a MobileNet network model.

[0033] According to the visual loop detection method based on the depth image fusion technology, the picture feature is processed by de-redundancy and noise reduction, the de-redundancy is realized based on a principal component analysis method, and the noise reduction is realized by a whitening method.

[0034] According to the visual loop detection method based on the depth image fusion technology, the similarity calculation method includes an Euclidean distance similarity or a cosine similarity.

[0035] According to still another aspect of the present disclosure, a visual loop detection device based on a depth image fusion technology is provided, including:

[0036] The loop closure detection data initialization module is used to initialize loop closure detection data, including: setting a feature set, wherein the feature set includes features and the number of images, the features in the feature set are initialized to empty, and the number of images in the feature set is N. pic Initialize to 0, set the threshold to K, set the initial frame number of the loop closure to f1, where f1 is the frame number of the previous loop closure in the loop closure detection process, initialize the frame number of the current image to f, set the similarity threshold to M, and the grouping threshold to m;

[0037] The current image feature extraction module is used to extract the features of the current image, including: using a pre-trained deep network model to extract the features of the current image with frame number f;

[0038] The candidate frame determination module determines the frame if the number of frames between the current image and the last loopback frame is less than a threshold K, or if the number of images in the feature set is N. pic If the value is less than the threshold K, then the features of the current image are added to the feature set, and the number of images in the feature set is N. pic If the frame number f of the current image is incremented by 1, the current image feature extraction process continues. Otherwise, the similarity between the current image and the feature set is calculated, and candidate frames are determined based on the similarity.

[0039] The final candidate set determination module is used to group the candidate frames to form several candidate sets and determine the final candidate set; and,

[0040] The loop closure frame detection module is used to determine loop closure frames through loop closure detection, including: selecting the image corresponding to the feature with the highest similarity to the current image in the final candidate set, and recording its frame number as h; and inputting the current image with frame number f and the image with frame number h into the visual odometry to determine whether there is a loop closure.

[0041] According to at least one embodiment of the visual loop closure detection apparatus based on deep image fusion technology of this disclosure, the similarity between the current image and a feature set is calculated, and candidate frames are determined based on the similarity, including:

[0042] The remaining features after removing a subset of features from the current image's features are taken as the current image's features. These subset features refer to the image features corresponding to each image within the K adjacent frames of the current image; and...

[0043] Calculate the similarity between the features of the current image and the features in the feature set, and determine the relationship between the similarity S and the threshold M. If the similarity S is greater than the threshold M, then the current image is used as a candidate frame; otherwise, continue to extract the features of the current image.

[0044] According to at least one embodiment of the visual loop closure detection apparatus based on deep image fusion technology of this disclosure, the candidate frames are grouped to form several candidate sets, and a final candidate set is determined, including:

[0045] Candidate frames whose frame sequence number difference is less than the grouping threshold m are grouped into a candidate set L. k ;

[0046] Calculate each candidate set L k The sum of similarity scores; and,

[0047] The candidate set with the highest sum of similarity scores is selected as the final candidate set.

[0048] According to at least one embodiment of the visual loop closure detection device based on deep image fusion technology of this disclosure, the current image with frame number f and the image with frame number h are input into a visual odometer to determine whether there is a loop closure, including:

[0049] The relative pose of the current image with frame number f and the image with frame number h is calculated using visual odometry. The calculated relative pose is then compared with existing results. If the difference is lower than the pose difference threshold, it is determined that there is a loop closure between the current image with frame number f and the image with frame number h. The existing results are the results calculated from the existing pose graph.

[0050] A visual loop closure detection apparatus based on deep image fusion technology according to at least one embodiment of the present disclosure further includes:

[0051] The image feature extraction is achieved through depthwise separable convolutions in the MobileNet network model.

[0052] According to at least one embodiment of the visual loop closure detection device based on deep image fusion technology disclosed herein, the image features are subjected to redundancy removal and noise reduction processing, wherein the redundancy removal is achieved based on principal component analysis and the noise reduction is achieved through whitening.

[0053] According to at least one embodiment of the visual loop closure detection method based on deep image fusion technology of this disclosure, the similarity calculation method includes Euclidean distance similarity or cosine similarity.

[0054] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0055] The memory stores execution instructions; and,

[0056] A processor that executes execution instructions stored in the memory, causing the processor to perform the method described in any of the preceding statements.

[0057] According to still another aspect of the present disclosure, there is provided a readable storage medium having stored therein execution instructions for implementing any of the above-mentioned methods when executed by a processor. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings, which are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure.

[0059] Figure 1 is a flowchart of a visual loop detection method based on depth image fusion technology according to an embodiment of the present disclosure.

[0060] Figure 2 is a schematic diagram of a MobileNet depth separable convolution method.

[0061] Figure 3 is a flowchart of a visual loop detection method based on depth image fusion technology according to still another embodiment of the present disclosure.

[0062] Figure 4 is a structural schematic diagram of a visual loop detection device based on depth image fusion technology according to an embodiment of the present disclosure.

[0063] Reference numerals

[0064] 1000 visual loop detection device based on depth image fusion technology

[0065] 1002 loop detection data initialization module

[0066] 1004 current picture feature extraction module

[0067] 1006 candidate frame determination module

[0068] 1008 final candidate frame determination module

[0069] 1010 loop frame detection module

[0070] 1100 bus

[0071] 1200 processor

[0072] 1300 memory

[0073] 1400 other circuits. DETAILED DESCRIPTION

[0074] The present disclosure will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are merely exemplary and are not limiting to the present disclosure. In addition, it should also be understood that for the convenience and clarity of the description, only parts of the relevant are shown in the drawings.

[0075] It should be noted that the embodiments and features of the embodiments in the present disclosure can be combined with each other without conflict. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0076] Unless otherwise specified, the exemplary embodiments / examples shown will be understood as providing exemplary features of various details that can implement the technical concepts of the present disclosure in practice. Therefore, unless otherwise specified, the features of various embodiments / examples can be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of the present disclosure.

[0077] In the drawings, cross-hatching and / or shading are generally used to indicate that a portion of one component is positioned above another component. As such, unless otherwise specified, the presence of cross-hatching or shading is not a requirement for any particular material, material property, size, proportion, commonality of the illustrated components, or any other characteristic, attribute, property, or the like of the components. In addition, for clarity and / or descriptive purposes, the sizes of the components shown in the drawings can be exaggerated relative to other components. When the exemplary embodiments can be carried out in different ways, a specific process sequence can be performed in a different order from the described order. For example, two consecutively described processes can be performed substantially simultaneously or in the reverse order from the described order. In addition, the same reference numerals represent the same components.

[0078] When a component is referred to as being "on" or "above" another component, "connected to" or "coupled to" another component, it can be directly on, directly connected to, or directly coupled to the other component, or there can be intervening components. However, when a component is referred to as being "directly on", "directly connected to", or "directly coupled to" another component, there are no intervening components. For this reason, the term "connected" can refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0079] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, to the extent that the terms "comprise," "have," "contain," and / or "include" and variations thereof are used herein, such terms are intended to be inclusive, in an aspect, it is also noted that the terms "substantially," "approximately," and other similar terms are used as terms of approximation and not as terms of degree, as such, they are utilized to account for inherent variations in measurement, calculation, and / or provision of a value that would be appreciated by one of ordinary skill in the art.

[0080] Figure 1 is a flowchart of a visual loop detection method based on depth image fusion technology according to an embodiment of the present disclosure.

[0081] As shown in Figure 1 , the visual loop detection method based on depth image fusion technology S100 comprises:

[0082] S102: initializing loop detection data, comprising: setting a feature set, the feature set comprising a feature and a picture number, the feature set feature being initialized as empty, and the picture number N pic of the feature set being 0, setting a neighboring frame range threshold value as K, setting an initial appearing loop frame sequence number f1, the f1 being a last loop frame sequence number in a loop detection process, initializing a current picture frame sequence number as f, setting a similarity threshold value as M, and setting a grouping threshold value as m;

[0083] S104: extracting a current picture feature, comprising: using a pre-trained depth network model to extract a feature of a current picture with a frame sequence number f;

[0084] S106: if a frame number between the current picture and a last loop frame is less than a threshold value K, or a picture number N pic of the feature set is less than a threshold value K, adding the feature of the current picture into the feature set, increasing the picture number N pic of the feature set by 1, increasing the frame sequence number f of the current picture by 1, and continuing the above-mentioned extracting the current picture feature, otherwise, calculating a similarity between the current picture and the feature set through a similarity calculation method, determining a candidate frame based on the similarity;

[0085] S108: grouping the candidate frame to form a plurality of candidate sets, and determining a final candidate set; and,

[0086] S110: determining a loop frame through loop detection, comprising: selecting a picture corresponding to a feature pair with the highest similarity to the current picture in the final candidate set, and recording the frame number of the picture as h; and inputting the current picture with the frame number f and the picture with the frame number h into a visual odometer to determine whether there is a loop.

[0087] wherein the similarity between the current picture and the feature set is calculated, and the candidate frame is determined based on the similarity, comprising:

[0088] the remaining features after removing part of the features of the current picture are taken as the current picture features, and the part of the features refers to the respective picture features corresponding to the K frames adjacent to the current picture; and

[0089] the similarity between the current picture features and the features in the feature set is calculated, and the relationship between the similarity S and the threshold M is determined, if the similarity S is greater than the threshold M, the current picture is taken as the candidate frame, otherwise, the current picture features continue to be extracted.

[0090] wherein the candidate frames are grouped to form a plurality of candidate sets, and the final candidate set is determined, comprising:

[0091] the candidate frames with a frame number difference less than the grouping threshold m are taken as a candidate set L k ;

[0092] the similarity score sum of each candidate set L k is calculated; and

[0093] the candidate set with the highest similarity score sum is taken as the final candidate set.

[0094] wherein the current picture with the frame number f and the picture with the frame number h are input into the visual odometer to determine whether there is a loop, comprising:

[0095] the relative pose of the current picture with the frame number f and the picture with the frame number h is calculated through the visual odometer, and the calculated relative pose is compared with the existing result, if the difference is lower than the pose difference threshold, it is determined that there is a loop between the current picture with the frame number f and the picture with the frame number h, and the existing result is the result calculated in the existing pose graph.

[0096] wherein the visual loop detection method based on the depth image fusion technology can further comprise:

[0097] The picture feature extraction is realized by a deep separable convolution of a MobileNet network model. Optionally, the picture feature extraction can also select other networks, such as ResNet and Overfeat, and the feature extraction methods of the three neural networks are compared through experiments. Compared with the ResNet network, the Overfeat and MobileNet networks are more suitable for the loop detection task. Although the loop detection effect of the MobileNet network is slightly lower than that of the Overfeat, the applicability of the MobileNet network to mobile devices makes it more suitable for being carried on a UAV or a robot. The MobileNet model is selected as the feature extraction method of the loop detection task in the application, and the number of parameters is reduced and the operation speed is improved by using a new convolution method. Compared with other large networks, it is more suitable for visual applications of mobile and embedded devices and can be widely carried on a UAV or a robot.

[0098] In the application, the MobileNet model is used as the feature extraction part in the loop detection by means of a commonly used model in the image classification task. Since the neural network directly outputs high-dimensional features, the principal component analysis and whitening method are used to remove redundancy and denoise the features. Finally, the features of the pictures are compared, and the similarity between the pictures is calculated to determine whether there is a loop.

[0099] The loop detection algorithm of the application uses a deep learning method for feature extraction. The loop detection problem is essentially a scene recognition problem, similar to image classification and image retrieval. The key step of these problems is to obtain a suitable image representation. The MobileNet network model is used for feature extraction in the application. The MobileNet model is a lightweight network model, which realizes the reduction of the number of parameters and the improvement of the operation speed by using a new convolution method. Compared with other large networks, it is more suitable for visual applications of mobile and embedded devices. The MobileNet is characterized in that it uses a deep separable convolution to extract features. Figure 2 As shown in the figure, the deep separable convolution decomposes the convolution into a deep convolution and a pointwise convolution, completely separating the channel correlation and the spatial correlation. The channel number of the convolution kernel of the deep convolution is 1, and a convolution kernel of the deep convolution only calculates the convolution of one channel of the feature map. The number of convolution kernels is the same as the number of input channels, and the output and input channel numbers are the same. The deep convolution does not use the feature information of different channels at the same spatial position, so the pointwise convolution is used to combine the feature maps. The pointwise convolution is a convolution kernel with a size of 1x1xc, and c is the number of input channels. The convolution kernel weights and combines the features in the channel direction to generate a new feature map.

[0100] In the application, the picture features are subjected to redundancy removal and denoising. The redundancy removal is realized based on the principal component analysis method, and the denoising is realized by the whitening method.

[0101] Among the data dimension reduction algorithms, the principal component analysis method is most widely used. The main idea is to map n-dimensional features to k-dimensional, to realize the dimension reduction of data, and the reduced data can represent the original data set as much as possible. The work of principal component analysis is to select k orthogonal bases from the original space, which can make the covariance between each field be 0 after transforming the original data to the orthogonal bases.

[0102] Suppose that the neural network directly outputs n d-dimensional feature vectors, which are denoted as X = [V (1) , V (2) ...V (n) ] T The matrix is called a sample matrix, and each row is a sample and each column is a feature vector. Noise reduction and redundancy removal are specific to the sample matrix, which means that the correlation between the remaining sample matrix dimensions should be as small as possible, and the variance of the remaining dimensions should be as large as possible. The following is the processing process of the sample matrix:

[0103] First, the centering process is needed for all samples, that is, the samples are translated so that the center is at the origin:

[0104]

[0105]

[0106] The sample mean is 0 by the above formula;

[0107] The key to the principal component analysis method is the covariance matrix, and the covariance matrix of the sample matrix is calculated:

[0108]

[0109] The covariance matrix measures the relationship between dimensions, not between samples. The elements on the main diagonal of the covariance matrix are the variances of each dimension, and the other elements are the covariances between two dimensions, that is, the correlation. The noise reduction process aims to make the off-diagonal elements of the covariance matrix close to 0, that is, the correlation between the different dimensions of the remaining sample matrix is as small as possible. That is, the covariance matrix of the noise-reduced sample matrix is a diagonal matrix as much as possible, and this goal can be achieved through matrix basis transformation;

[0110] Let P be the matrix composed of the basis vectors of the matrix transformation, and Y be the transformed matrix:

[0111] Y = XP (4)

[0112] Let the covariance matrix of Y be Σ1, that is:

[0113]

[0114] Substitute formula (3) and formula (4) into formula (5):

[0115]

[0116] Formula (6) shows that the covariance matrix of the expected sample matrix is a diagonal matrix, and the matrix P of the base transformation of the sample matrix is P that diagonalizes its covariance matrix;

[0117] Then, the covariance matrix of the original sample matrix is decomposed, that is, it is diagonalized:

[0118] P T ΣP=Λ (7)

[0119] Wherein, P, Λ ∈ R d×d The eigenvalues on the diagonal of Σ are arranged in descending order, and P is an orthogonal matrix composed of the corresponding eigenvectors;

[0120] The diagonalized covariance matrix, the smaller value on the diagonal, that is, the minimum variance in its corresponding dimension, can be discarded, and the k dimensions with larger variance are retained to form a new diagonal matrix Λ1, and the first k eigenvectors of P are selected to form a new eigenvector matrix P1:

[0121] P1=P[:,k] (8)

[0122] The sample matrix is multiplied by P1, and the corresponding sample is:

[0123]

[0124] P1 is multiplied by the right with respect to each sample to perform linear transformation with P1 as the base, and the eigenvectors in P1 are the new coordinate system of the low-dimensional space. It is the eigenvector after dimensionality reduction;

[0125] The data processed by the principal component analysis method has a covariance matrix that is approximately diagonal, indicating that the different dimensions are basically independent and have eliminated the factors affecting the mutual influence between the original data components.

[0126] Wherein, the purpose of the whitening method is to further reduce the redundancy of the input, and in fact, the whitening is a standard deviation normalization processing on each dimension of the reduced vector:

[0127]

[0128] λ jis the eigenvalue corresponding to the j-th eigenvector in the principal component analysis, and ε is a small constant to avoid division by zero error. After the whitening process, the correlation between features is low and all features have the same variance.

[0129] The similarity calculation method includes Euclidean distance similarity or cosine similarity.

[0130] Euclidean distance is the distance between two points in Euclidean space. Given two feature vectors A and B of two images, we have:

[0131]

[0132] D(A,B) is the Euclidean distance between A and B.

[0133] The similarity score is defined as:

[0134]

[0135] Cosine distance similarity, also known as cosine similarity, is a measure of the similarity between two vectors by calculating the cosine of the angle between them. Given two feature vectors A and B, their cosine similarity is given by the dot product and vector length:

[0136]

[0137] where A i and B i represent the components of vectors A and B, respectively.

[0138] In geometric space, the definitions of Euclidean distance and cosine distance are visually represented. It can be seen that Euclidean distance calculates the absolute distance between two points corresponding to the space vector, while cosine distance is the cosine value of the angle between the space vectors. Compared to the absolute difference between individual numerical values measured by Euclidean distance, cosine distance measures the difference between the directions of two vectors. In this paper, we will choose to use the cosine distance between features to measure the difference between two images.

[0139] The visual loop detection method based on the depth image fusion technology provided by the application is tested and evaluated by using public data sets NewCollege and City Centre data sets, mainly including similarity matrix and accuracy-recall rate curve evaluation indexes. In the non-real-time offline loop detection task, the application may not be the best, but if it is mounted on a robot or a drone for real-time loop detection, the MobileNet method is more advantageous, and the accuracy thereof can be improved through fine-tuning, loop verification and the like, so that a high recognition accuracy can be realized, the scene passed by the camera can be well recognized, the existing closed loop in the track can be searched, the cumulative drift error caused by the visual odometer can be significantly reduced, and the absolute pose of the camera is optimized. Especially when the robot moves in a scene with rich vision, better results can be achieved.

[0140] Figure 2 Figure is a schematic diagram of the MobileNet depth separable convolution method.

[0141] As shown in Figure 2 , it is assumed that the input feature map size is D F ×D F ×M, the output feature map size is D F ×D F ×N, and the convolution kernel size is D K ×D K In the traditional convolution, the calculation amount is D K ·D K ·M·N·D F ·D F , and in the depth separable convolution, the calculation amount is D K ·D K ·M·D F ·D F +M·N·D F ·D F When N is relatively large and D K = 3, the depth separable convolution can reduce the calculation amount by 9 times. At the same time, since the depth separable convolution uses a large amount of 1x1 convolution, matrix multiplication can be directly used to complete, and the underlying preprocessing is no longer needed, so that the operation efficiency is greatly improved.

[0142] MobileNet has three versions, MobileNetV1, MobileNetV2 and MobileNetV3. Their structures and principles are roughly the same. This paper will use the first version of MobileNetV1 for feature extraction, Table 1 is the structure details of the prior art MobileNetV1, and the meaning of dw in the table is depth convolution, s1 represents step 1, and s2 represents step 2. The output of the last average pooling layer is selected as the feature required for loop detection in this paper, and the input picture size of the network is required to be 224x224x3, and the size of the one-dimensional feature output is 1024.

[0143] Table 1 Structure details of Mobile-Net V1

[0144]

[0145] Figure 3 is a flowchart of a visual loop detection method based on depth image fusion technology according to still another embodiment of the present disclosure.

[0146] As shown in Figure 3 , the visual loop detection method based on depth image fusion technology S200 includes:

[0147] S202: Before loop detection, the relevant parameters of input and output need to be set, wherein the output parameters include: the feature set M corresponding to the picture set that has been passed through before, the picture frame currently joined: f, the adjacent frame range: k, the pose difference threshold T1, T2, the grouping threshold m; wherein the output parameters include: the subscript l of the frame that has a loop relationship with the current frame;

[0148] After the output parameter setting is output, loop detection can be performed, including the following steps:

[0149] S204: using a pre-trained deep network model to extract the feature v f of the current picture;

[0150] If the picture is less than k frames away from the last loop, or the number of pictures in the current feature set is less than k frames, the picture feature is added to the feature set and returned to S204, otherwise S206 is continued;

[0151] S206: After removing the features corresponding to the local key frames (adjacent k frames) connected with the current frame from the feature set, the feature vector of the current frame is extracted and compared with the features in the feature set to calculate the similarity score:

[0152] S if =S(M i ,M f ),i∈(0,i-k)

[0153] S208: Select frames with similarity score greater than θ as candidate frames. If there is no candidate frame, return to S204;

[0154] S210: Group candidate frames, group pictures with picture index difference less than grouping threshold m as a group, denoted as L k , called candidate set, calculate the sum of similarity scores of each candidate set, select the one with the highest total score as the final candidate set C:

[0155]

[0156] S212: Select the picture index corresponding to the feature pair with the highest similarity score between the final candidate set and the current frame as h:

[0157]

[0158] S214: Input picture h and f as a picture pair into the visual odometry, the visual odometry calculates the relative pose (ΔP hf ,ΔΦ hf ) between them, and compares it with the existing result (the result calculated in the existing pose graph), if the difference is less than the specified threshold, it is considered that there is a loop between the two picture pairs, and the frame l is output, that is:

[0159] Output: h as l if:

[0160]

[0161] According to the algorithm in the above embodiment, for each frame in the picture sequence, the features of the frame will be extracted using the deep learning network model, and compared with the existing features in the feature set to calculate the similarity score and select the candidate frame. Then, the candidate frames are grouped and the sum of the similarity scores is calculated to select the candidate set with the highest total score as the final candidate set, and the frame with the highest similarity score in the candidate set is selected. The relative pose between it and the current frame is estimated using the visual odometry, and the estimated result is compared with the previous result. If it is less than the set threshold, then the frame is the frame that forms a loop with the current picture. According to the algorithm, each picture in the picture sequence either does not exist a frame that forms a loop with it, or outputs a unique frame that is most likely to form a loop with it.

[0162] Among them, the similarity score calculation in S206 can be obtained by Euclidean distance or cosine similarity calculation method.

[0163] Figure 4 is a structural schematic diagram of a visual loop detection device based on deep image fusion technology according to an embodiment of the present disclosure.

[0164] As Figure 4 shown, the visual loop detection device based on the depth image fusion technology comprises:

[0165] The loop detection data initialization module is configured to initialize loop detection data, including: setting a feature set, the feature set including a feature and a picture number, the feature set feature being initialized as empty, the picture number N pic of the feature set being initialized as 0, setting a neighboring frame range threshold value as K, setting an initial loop-occurring frame sequence number f1, the f1 being a last loop-occurring frame sequence number in a loop detection process, initializing a current picture frame sequence number as f, setting a similarity threshold value as M, and setting a grouping threshold value as m.

[0166] The current picture feature extraction module is configured to extract a current picture feature, including: using a pre-trained depth network model to extract a feature of a current picture with a frame sequence number f.

[0167] The candidate frame determination module is configured to: if a frame number between the current picture and a last loop-occurring frame is less than the threshold value K, or a picture number N pic of the feature set is less than the threshold value K, then adding the feature of the current picture to the feature set, increasing the picture number N pic of the feature set by 1, increasing the frame sequence number f of the current picture by 1, and continuing the above-mentioned extraction of the current picture feature, otherwise, calculating a similarity between the current picture and the feature set, and determining a candidate frame based on the similarity.

[0168] The final candidate set determination module is configured to group the candidate frames to form a plurality of candidate sets, and determine a final candidate set.

[0169] The loop frame detection module is configured to determine a loop frame through loop detection, including: selecting a picture corresponding to a feature with the highest similarity to the current picture in the final candidate set, and recording a frame sequence number of the picture as h; and inputting the current picture with the frame sequence number f and the picture with the frame sequence number h into a visual odometer to determine whether there is a loop.

[0170] According to still another aspect of the present disclosure, an electronic device is provided, comprising:

[0171] a memory storing an execution instruction; and

[0172] a processor executing the execution instruction stored in the memory, so that the processor executes the method of any one of the above.

[0173] According to still another aspect of the present disclosure, a readable storage medium is provided, the readable storage medium storing an execution instruction, the execution instruction being executed by a processor to implement the method of any one of the above.

[0174] Figure 4An example apparatus diagram showing a hardware implementation for a processing system is shown, which can include respective modules that perform one or more of the steps in the above-described flowcharts. Thus, each or several of the steps in the above-described flowcharts can be performed by respective modules, and the apparatus can include one or more of these modules. The modules can be one or more hardware modules specifically configured to perform the respective steps, or implemented by a processor configured to perform the respective steps, or stored within a computer-readable medium for implementation by a processor, or implemented through some combination.

[0175] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0176] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one connection line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0177] Any processes or methods described in the flowcharts or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) of the process, and / or that the various embodiments of the present disclosure can represent alternative process or methods including the steps thereof which can be implemented by other than the recited ordering and / or flow, such will be obvious to those having skill in the art of the disclosed embodiments' technical fields. The processors execute the various ones of the methods and processes described above. For example, the method embodiments of the present disclosure can be implemented as a software program which is tangibly embodied within a machine-readable medium, such as a memory. In some embodiments, portions of the software program can be loaded and / or installed via the memory and / or communication interface. When the software program is loaded into the memory and executed by the processor, one or more of the steps of the methods described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the methods described above by other means, such as by way of firmware.

[0178] Logic and / or steps represented in the flowcharts and / or otherwise described herein can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.

[0179] For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can specifically include the following, which are non-exhaustive examples: electrical connection (electrical device), portable computer diskette (magnetic device), random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber device, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable medium upon which the program can be printed, as the program can be electronically captured, for example via the optical scanner of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in the memory.

[0180] It should be understood that various aspects of the disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, various steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0181] A person of ordinary skill in the art of the present field can understand that all or part of the steps of the above-mentioned embodiment method can be completed by a program instructing the relevant hardware, and the program can be stored in a readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0182] In addition, each functional unit in each embodiment of the present disclosure can be integrated into one processing module, or each unit can be physically present separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0183] In the description of the present specification, the description of the terms "one embodiment / way", "some embodiments / ways", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily the same embodiment / way or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments / ways or examples. In addition, a person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples, without contradiction.

[0184] In addition, the terms "first", "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0185] Those skilled in the art will understand that the above-described embodiments are merely intended to clarify the present disclosure, and are not intended to limit the scope of the present disclosure. Other changes or modifications can be made by those skilled in the art based on the above disclosure, and the changes or modifications are still within the scope of the present disclosure.

Claims

1. A visual loop detection method based on depth image fusion technology, characterized in that, The method comprises: initializing loop detection data, including: setting a feature set, the feature set comprising features and picture quantity, the feature set features being initialized as empty, the picture quantity Npic of the feature set being initialized as 0, a threshold value of a range of adjacent frames being K, setting an initial frame sequence number f1 of a loop occurrence, the f1 being a last loop frame sequence number in a loop detection process, initializing a frame sequence number of a current picture as f, setting a similarity threshold value as M, a grouping threshold value as m, and a pose difference threshold value; extracting features of the current picture, including: using a pre-trained deep network model to extract features of the current picture with the frame sequence number f; if the frame quantity between the current picture and the last loop frame is less than the threshold value K or the picture quantity Npic in the feature set is less than the threshold value K, adding the features of the current picture to the feature set, increasing the picture quantity Npic of the feature set by 1, increasing the frame sequence number f of the current picture by 1, and continuing the above-mentioned extraction of the features of the current picture, otherwise, calculating the similarity between the current picture and the feature set by a similarity calculation method, and determining a candidate frame based on the similarity; grouping the candidate frame to form a plurality of candidate sets and determining a final candidate set; and determining a loop frame by loop detection, including: selecting a picture corresponding to the feature with the highest similarity to the current picture in the final candidate set, and recording the frame sequence number of the picture as h; and inputting the current picture with the frame sequence number f and the picture with the frame sequence number h into a visual odometer to determine whether there is a loop; calculating the similarity between the current picture and the feature set by a similarity calculation method, and determining a candidate frame based on the similarity, including: taking the remaining features after removing part of the features of the current picture as the features of the current picture, the part of the features referring to the features of each picture in the K adjacent frames of the current picture; and calculating the similarity between the features of the current picture and the features in the feature set, and determining the relationship between the similarity S and the threshold value M, if the similarity S is greater than the threshold value M, taking the current picture as the candidate frame, otherwise, continuing the extraction of the features of the current picture; grouping the candidate frame to form a plurality of candidate sets and determining a final candidate set, including: taking the candidate frames with a frame sequence number difference less than the grouping threshold value m as a candidate set Lk; calculating the total sum of the similarity scores of each candidate set Lk; and taking the candidate set with the highest total sum of the similarity scores as the final candidate set; inputting the current picture with the frame sequence number f and the picture with the frame sequence number h into a visual odometer to determine whether there is a loop, including: calculating the relative pose of the current picture with the frame sequence number f and the picture with the frame sequence number h by the visual odometer, and comparing the calculated relative pose with an existing result, if the difference is lower than the pose difference threshold value, determining that there is a loop between the current picture with the frame sequence number f and the picture with the frame sequence number h, the existing result being a result calculated in an existing pose graph.

2. The visual loop detection method based on depth image fusion technology according to claim 1, characterized in that, Further comprising: the picture feature extraction is implemented through a deep separable convolution of a MobileNet network model. 3.The visual loop detection method based on depth image fusion technology according to claim 1, characterized in that, the picture features are subjected to de-redundancy and noise reduction processing, the de-redundancy is implemented based on a principal component analysis method, and the noise reduction is implemented through a whitening method. 4.The visual loop detection method based on depth image fusion technology according to claim 1, characterized in that, The similarity calculation method includes Euclidean distance similarity or cosine similarity.

5. A visual loop detection device based on depth image fusion technology, characterized in that, The method comprises the steps of: The loop detection data initialization module is configured to initialize loop detection data, including: setting a feature set, the feature set including features and the number of pictures, initializing the feature set features as empty, initializing the number of pictures Npic of the feature set as 0, setting a threshold value as K, setting an initial frame sequence number f1 of loop occurrence, the f1 being the last frame sequence number of loop in the loop detection process, initializing the frame sequence number of the current picture as f, setting a similarity threshold value as M, and setting a grouping threshold value as m; The current picture feature extraction module is configured to extract the features of the current picture with the frame sequence number f, including: using a pre-trained deep network model to extract the features of the current picture with the frame sequence number f; The candidate frame determination module is configured to: if the number of frames between the current picture and the last frame of loop is less than the threshold value K, or the number of pictures Npic in the feature set is less than the threshold value K, add the features of the current picture to the feature set, increase the number of pictures Npic of the feature set by 1, increase the frame sequence number f of the current picture by 1, and continue to extract the features of the current picture; otherwise, calculate the similarity between the current picture and the feature set by using the similarity calculation method, and determine the candidate frame based on the similarity. The final candidate set determination module is configured to group the candidate frames to form a plurality of candidate sets, and determine a final candidate set. The loop frame detection module is configured to determine the loop frame by loop detection, including: selecting the picture corresponding to the features with the highest similarity to the current picture in the final candidate set, and recording the frame sequence number of the picture as h; and inputting the current picture with the frame sequence number f and the picture with the frame sequence number h into the visual odometry to determine whether there is a loop. The method for calculating the similarity between the current picture and the feature set by using the similarity calculation method includes: taking the remaining features after removing part of the features of the current picture as the features of the current picture, the part of the features referring to the features of each picture within the K frames adjacent to the current picture; calculating the similarity between the features of the current picture and the features in the feature set; and determining the relationship between the similarity S and the threshold value M, if the similarity S is greater than the threshold value M, taking the current picture as the candidate frame, otherwise, continuing to extract the features of the current picture. The method for grouping the candidate frames to form a plurality of candidate sets and determining a final candidate set includes: taking the candidate frames with a frame sequence number difference less than the grouping threshold value m as a candidate set Lk; calculating the total sum of the similarity scores of each candidate set Lk; and taking the candidate set with the highest total sum of the similarity scores as the final candidate set. The method for inputting the current picture with the frame sequence number f and the picture with the frame sequence number h into the visual odometry to determine whether there is a loop includes: calculating the relative pose of the current picture with the frame sequence number f and the picture with the frame sequence number h by using the visual odometry, and comparing the calculated relative pose with an existing result, if the difference is lower than a pose difference threshold value, determining that there is a loop between the current picture with the frame sequence number f and the picture with the frame sequence number h, the existing result being the result calculated in the existing pose graph.

6. An electronic device, comprising: The method comprises the steps of: a memory storing execution instructions; and a processor executing the execution instructions stored by the memory, so that the processor executes the method as claimed in any one of claims 1 to 4.

7. A readable storage medium, characterized by, The readable storage medium stores execution instructions, which are executed by a processor to implement the method as claimed in any one of claims 1 to 4.

Images