A visual low-light closed-loop detection algorithm, storage medium and device based on deep learning

Abstract

The present invention belongs to the technical field of synchronous positioning and map creation, and discloses a visual low-light closed-loop detection algorithm, storage medium and device based on deep learning. The algorithm includes: step S1, collecting environmental information based on an image acquisition device, and dividing semantic areas from the image of the environmental information; step S2, sampling the center line of the divided semantic area to generate a local superpixel semantic descriptor; step S3, enhancing the low-light data through a Fourier feature enhancement network, and performing feature matching and posture calculation on the enhanced data to estimate the motion state; step S4, associating the semantic descriptor with the closed-loop thread, and completing the closed-loop verification with the auxiliary word bag in the closed-loop detection to achieve global posture optimization. The present invention can determine whether the final loop fusion and global optimization can be performed through more stringent screening conditions, and is more suitable for dealing with the problem of low accuracy and easy occurrence of erroneous loops in the closed-loop detection thread under low-light conditions.

Description

Technical Field

[0001] The present invention belongs to the field of simultaneous location and mapping (SLAM) technology, and specifically relates to a visual low-light closed-loop detection algorithm, storage medium and device based on deep learning. Background Art

[0002] Simultaneous Location And Mapping (SLAM) refers to the process of using the sensors carried by the robot to simultaneously build a map of the current environment and locate its own position and posture in an unknown environment. However, in actual scenarios, due to factors such as uneven lighting or insufficient lighting conditions (such as underground mines, deep space, and outdoor at night), short exposure time, and backlit positions, the quality of images captured by the camera is likely to be poor, with problems such as low brightness, low contrast, object color distortion, and narrow grayscale in the image, which makes it difficult for the SLAM system to extract features, track, and match images, resulting in a series of problems that seriously affect the closed-loop and positioning accuracy of the system in low-light environments. Therefore, a visual SLAM algorithm suitable for low-light conditions is crucial to improving the robustness of the system. Summary of the invention

[0003] The purpose of the present invention is to provide a visual low-light closed-loop detection algorithm based on deep learning, which is used to solve the technical problems in the prior art that the SLAM system is difficult to extract and match features under low-light conditions, resulting in low closed-loop and positioning accuracy.

[0004] The deep learning-based visual low-light closed-loop detection algorithm comprises the following steps.

[0005] Step S1, collecting environmental information based on an image collection device, and dividing semantic areas from the image of the environmental information.

[0006] Step S2, perform centerline sampling on the divided semantic areas to generate local superpixel semantic descriptors; local superpixel semantic descriptors are referred to as descriptors for short, centerline sampling extracts pixel points located on the centerline of each semantic area on the image, and then calculates the three parameters of spectral transformation rate, contrast eigenvalue and color saturation rate of each semantic area based on the pixel points of the centerline of the semantic area, and then calculates and generates corresponding descriptors based on the above three parameters, and the descriptors of all semantic areas form the descriptor vectors of the corresponding images, and the descriptor vectors are stored in the descriptor library.

[0007] Step S3, feature enhancement is performed on the low-light data through a Fourier feature enhancement network, and feature matching and posture calculation are performed on the enhanced data to estimate the motion state.

[0008] Step S4, data association is performed between the semantic descriptor and the closed-loop thread, and the auxiliary word bag is used to jointly complete the closed-loop verification in the closed-loop detection to achieve global pose optimization; this step is used to assist in the screening of closed-loop candidate key frames by comparing the similarity of the mean descriptor vector.

[0009] Preferably, in step 1, each frame image is divided into a·b semantic regions, where a and b are the number of rows and columns of the semantic region respectively; in step 2, the descriptor T of each semantic region is calculated. ij A descriptor matrix of a·b is formed, and the rows of the descriptor matrix are expanded into a vector to obtain the descriptor vector T. For any key frame P that may be involved in the detection similarity, a common frame before and after the key frame image P is located respectively, and the descriptor vectors of the selected common frame and the corresponding key frame are averaged together as the basis for closed-loop detection. The descriptor of each semantic area is described by the following formula:

[0010]

[0011] In the above formula, T ij It represents the descriptor of the semantic region in the i-th row and j-th column of the image. Lt, Ce and Sr respectively represent the spectral transformation rate, contrast eigenvalue and color saturation rate of the corresponding semantic region. ω1, ω2 and ω3 represent the parameter weights corresponding to Lt, Ce and Sr respectively.

[0012] Preferably, the spectral transformation rate is used to describe the texture transformation trend of the central information of the semantic region, and is obtained by calculating the weighted sum of the first-order differences of the grayscale values ​​of all pixels on the center line and the grayscale values ​​of their neighborhood pixels; the formula for calculating the spectral transformation rate is as follows:

[0013]

[0014] In this semantic area, Lt is the spectral transformation rate of the semantic area, m d Represents the ordinate of the horizontal center line of the semantic region, n d represents the horizontal coordinate of the vertical center line of the semantic region, f(x,m d +1) indicates the xth row, m d +1 column pixel gray value, f(n d +1,y) indicates the nth d +1 row, y column pixel grayscale value, m, n are the number of rows and columns of pixels, s is the number of pixels collected.

[0015] Preferably, the contrast feature value is used to characterize the light and dark changes of each semantic area, so as to show the brightness difference of each part, and the center line sampling processing is performed on the three channels of R, G, and B so that they contain more stable feature contrast information; the formula for calculating the contrast feature value is as follows:

[0016]

[0017] In this semantic region, Ce represents the contrast feature value of the semantic region, y c represents the yth column in channel c, x c represents the xth row in channel c, where channel c is any of the three channels R, G, and B. f(n d ,y c ) represents the nth d The grayscale value of the pixel in row y and column y, f(x c ,m d ) represents the xth row, mth row in channel c d The grayscale value of the pixel in the column, m and n are the number of rows and columns of pixels respectively, and s is the number of pixels collected.

[0018] Preferably, the color saturation is more stable information obtained from the HSV color space. In the HSV color space, the algorithm separates the S channel with strong light resistance in HSV to obtain the color saturation information; the formula for calculating the color saturation is as follows:

[0019]

[0020] In this semantic area, Sr represents the color saturation rate of the semantic area, R, G, and B represent the three channel components of the RGB image in the semantic area, y represents the column number of the pixel, x represents the row number of the pixel, m and n are the number of rows and columns of the pixel points, and s is the number of collected pixels.

[0021] Preferably, in step S3, the Fourier feature enhancement network converts the input image into Fourier space, obtains the amplitude component and the phase component, and performs convolution and activation on the two components respectively to extract features; the amplitude component is divided by the amplitude transformation map estimated by the FP block to achieve amplitude amplification, and the amplified amplitude component and phase component are combined to perform inverse Fourier transformation to obtain output;

[0022] The low-light enhancement network also introduces a feature detail recovery module including a differential weight network DWNet. The input image is processed to separate high and low frequency components, and then feature extraction is performed to obtain an intermediate feature map. The intermediate feature map is input into the differential weight network DWNet for weight distribution to obtain pixel weights and channel weights, which are then combined with the input of the differential weight network DWNet to obtain the output result after feature detail enhancement; the output result after feature detail enhancement and the output of the inverse Fourier transform are fused to obtain the output image after low-light enhancement. Preferably, the channel attention module adaptively resizes each channel feature, focuses on different channel features, and gives priority to the dark channel, generating different weighted information for each channel feature; and the pixel attention module enables the network to filter irrelevant pixel information, pay more attention to important features such as low-light pixels, and provide differentiated weight information for them.

[0023] Preferably, in the feature detail recovery module, the input image obtains an intermediate feature map through convolution-activation-convolution operations, and then the global spatial information is mapped into channel descriptors and pixel descriptors by performing a global average pooling operation; thereafter, the pixel descriptor and the channel descriptor enter the pixel attention module and the channel attention module respectively to obtain differentiated pixel weights and channel weights; the channel attention module adaptively readjusts the size of each channel feature, focuses on different channel features, and gives priority to the dark channel, generating different weighted information for each channel feature; and the pixel attention module enables the network to filter irrelevant pixel information, pay more attention to important features such as low-light pixels, and provide differentiated weight information for them; finally, the pixel weights and channel weights are multiplied one by one with the input of the difference weight network DWNet to obtain the output result after feature detail enhancement.

[0024] Preferably, in step S4, for the current key frame input, it is first determined whether there is a candidate key frame. When there is a candidate key frame, the similarity is detected through a bag-of-words query; after the bag-of-words query detection passes, the similarity is measured based on the mean descriptor vector, and this step uses cosine similarity to measure the similarity.

[0025] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of a deep learning-based visual low-light closed-loop detection algorithm as described above are implemented.

[0026] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the steps of a deep learning-based visual low-light closed-loop detection algorithm as described above are implemented.

[0027] The present invention has the following advantages:

[0028] 1. The algorithm provided by the present invention introduces a new local superpixel semantic descriptor, which divides the semantic area, samples the center line of each semantic area, calculates the spectral conversion rate, contrast eigenvalue and color saturation rate according to the sampled pixel points, and fuses these three parameters to obtain a high-quality semantic descriptor. This descriptor can more accurately capture the characteristic texture and detail information of the image, while improving the data deviation and angular robustness caused by different camera data acquisition angles in the same scene, making it more suitable for SLAM system applications.

[0029] 2. In order to solve the problem of tracking loss and feature matching difficulty in low-light conditions, this algorithm proposes a new algorithm that integrates the FFENet network. The network also introduces the difference weight network DWNet to restore feature details. By performing cross-domain processing in the Fourier frequency domain and spatial domain, low-light image enhancement is achieved through joint optimization, which improves the number of feature matches and the overall positioning accuracy of the system under low-light conditions.

[0030] 3. In the closed-loop detection thread, the present invention first determines whether there are candidate key frames for the input key frames. When there are candidate key frames, the similarity is detected through bag-of-words query, and the semantic descriptor of the current key frame is queried at the same time. For the key frames determined as loops by the bag-of-words detection, the present invention calculates the descriptor similarity based on the mean descriptor vector of the key frame image, and the mean descriptor vector is obtained by averaging the descriptor vectors of the key frame and its two adjacent ordinary frames before and after, further improving the robustness of the algorithm. Therefore, the present invention can determine whether the final loop fusion and global optimization can be performed through more stringent screening conditions, making it more suitable for dealing with the problem of low accuracy and easy occurrence of erroneous loops in the closed-loop detection thread under low light conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a basic flow chart of a deep learning-based visual low-light closed-loop detection algorithm of the present invention.

[0032] Figure 2 The flowchart of the SLAM system of the present invention is applied.

[0033] Figure 3 It is a flow chart of the low-light enhancement network in the present invention.

[0034] Figure 4 Flow chart of the closed-loop detection process in the present invention.

[0035] Figure 5 It is a schematic diagram of the principle of calculating the mean descriptor vector in the present invention.

[0036] Figure 6 It is a comparison diagram of the enhancement results of the present invention and the prior art, wherein GT represents the ground truth value, Input represents the input image, IAT represents the image generated by the existing Illumination-Adaptive-Transformer network, EMNet represents the image generated by the existing Enhancement Module Net low-light enhancement network, FourLLIE represents the image generated by the existing Fourier low-light image enhancement network, and Our represents the image generated by the algorithm provided by the present invention.

[0037] Figure 7 The present invention obtains a description effect diagram from a descriptor vector in a similar scenario.

[0038] Figure 8 This is a comparison diagram of feature matching in low-light images before and after the improvement using the present invention.

[0039] Fig. 9 It is a comparison chart of the accuracy-recall rate curves of the present invention and other prior arts under different data sets, wherein the prior arts include SLAM3, OV2_SLAM, VINS_FUSION and Maplab, and OURS represents the SLAM system to which the present invention is applied.

[0040] Fig.10 This is the trajectory diagram obtained by the present invention under the room1 sequence of the TUM-VI dataset.

[0041] Fig.11 This is the trajectory diagram obtained by the present invention under the room2 sequence of the TUM-VI dataset.

[0042] Fig.12 This is the trajectory diagram obtained by the present invention under the room3 sequence of the TUM-VI dataset.

[0043] Fig.13 This is the trajectory diagram obtained by the present invention under the room4 sequence of the TUM-VI dataset.

[0044] Fig.14 It is an error line graph comparing the present invention with other prior arts under the KITTI data set, wherein the prior arts include SLAM3, OV2_SLAM and VINS_FUSION, and OURS represents the SLAM system to which the present invention is applied.

[0045] Fig.15This is a comparison diagram of the trajectory generated by the present invention in another set of experiments, the real scene layout, and the trajectories of other existing algorithms, where the left image is the real scene layout, and the right image is a comparison diagram of the trajectories, where groundtruth is the ground truth value, OURS is the trajectory generated by the algorithm provided by the present invention, and the existing technologies used for comparison include ORB_SLAM3 and GCNV2_SLAM. DETAILED DESCRIPTION

[0046] The specific implementation modes of the present invention will be further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0047] Embodiment 1.

[0048] like Figure 1-Figure 5 As shown, the present invention provides a visual low-light closed-loop detection algorithm based on deep learning, comprising the following steps.

[0049] Step S1, collecting environmental information based on an image collection device, and dividing semantic areas from the image of the environmental information.

[0050] In this embodiment, the image acquisition device is a camera that comes with the robot. Since the SLAM system will generate cumulative errors during operation, resulting in the same scene generating error loops due to different camera acquisition angles, it is necessary to improve the robustness to angles.

[0051] In this algorithm, each frame image is divided into a·b semantic regions, where a and b are the number of rows and columns of the semantic region, respectively. In this embodiment, a·b is 5×16, i.e., a total of 80 semantic regions. The above semantic regions will be used to generate descriptors later.

[0052] Step S2, performing centerline sampling on the divided semantic areas to generate local superpixel semantic descriptors.

[0053] In this step, the divided semantic regions are sampled by the centerline, and the pixels located on the centerline of each semantic region on the image, that is, the pixels located on the horizontal centerline and the vertical centerline, are extracted. Then, the three parameters of the spectral transformation rate, contrast eigenvalue and color saturation rate of each semantic region are calculated based on the pixels of the centerline of the semantic region, and then the corresponding descriptors are calculated based on the above three parameters. The specific calculation method of the three parameters is as follows.

[0054] The spectral transformation rate is used to describe the texture transformation trend of the central information of the semantic region. It is obtained by calculating the weighted sum of the first-order differences of the grayscale values ​​of all pixels on the center line and the grayscale values ​​of its neighbors. This parameter can well describe the grayscale changes near the two sides of the center line and the contrast changes of the tiny details of the image. The formula for calculating the spectral transformation rate is as follows:

[0055]

[0056] In this semantic area, Lt is the spectral transformation rate of the semantic area, m d Represents the ordinate of the horizontal center line of the semantic region, n d represents the horizontal coordinate of the vertical center line of the semantic region, f(x,m d +1) indicates the xth row, m d +1 column pixel gray value, f(n d +1,y) indicates the nth d +1 row, y column pixel grayscale value, m, n are the number of rows and columns of pixels, s is the number of pixels collected.

[0057] The contrast eigenvalue is used to characterize the light and dark changes of each semantic area, so as to show the brightness difference of each part and comprehensively consider the overall brightness distribution of the image. Since the image is composed of human visually perceptible color information by superimposing and mixing the changes and brightness of the three channels R (red), G (green), and B (blue), this means that the three channel components of RGB are closely related to the brightness of the image. Therefore, when the brightness changes, the components of the three channels will also change accordingly, thereby affecting the overall color information. Therefore, the center line sampling processing is performed on the three channels of R, G, and B at the same time to make them contain more stable feature contrast information. The formula for calculating the contrast eigenvalue is as follows:

[0058]

[0059] In this semantic region, Ce represents the contrast feature value of the semantic region, y c represents the yth column in channel c, x c represents the xth row in channel c, where channel c is any of the three channels R (red), G (green), and B (blue), f(n d ,y c ) represents the nth d The grayscale value of the pixel in row y and column y, f(x c ,m d ) represents the xth row, mth row in channel c d The grayscale value of the pixel in the column, m and n are the number of rows and columns of pixels respectively, and s is the number of pixels collected.

[0060] Color saturation is more stable information obtained from the HSV color space. Since RGB images are very sensitive to light, this algorithm also considers converting RGB images to HSV color space to obtain more stable information, where H represents hue, S represents saturation, and V represents brightness. Since the V channel selects the maximum component in the RGB image, the effect of light on the V channel of the HSV image is most significant. In contrast, the H channel and the S channel are obtained by taking the minimum component in the RGB image and performing difference and operation. Therefore, the effect of light on the H channel and S channel of the HSV image is relatively small. To this end, the algorithm separates the S channel in HSV, which has strong light resistance, to obtain color saturation information. The formula for calculating color saturation is as follows:

[0061]

[0062] In this semantic area, Sr represents the color saturation rate of the semantic area, R, G, and B represent the three channel components of the RGB image in the semantic area, y represents the column number of the pixel, x represents the row number of the pixel, m and n are the number of rows and columns of the pixel points, and s is the number of collected pixels.

[0063] Based on the above three parameters, the local superpixel semantic descriptor of each semantic region (hereinafter referred to as descriptor) is described by the following formula:

[0064]

[0065] In the above formula, T ij Descriptor of the semantic region in the i-th row and j-th column of the image. Lt, Ce and Sr correspond to the spectral transformation rate, contrast eigenvalue and color saturation rate of the corresponding semantic region, respectively. ω1, ω2 and ω3 represent the parameter weights corresponding to Lt, Ce and Sr, respectively. Calculate the descriptor T of each semantic region ij A descriptor matrix of a·b is formed, and the rows of the descriptor matrix are expanded into a vector, so that this embodiment obtains a 5×16=80-dimensional descriptor vector T for the frame image.

[0066] For any key frame P that may be involved in the detection of similarity, a normal frame is located before and after the key frame image P. In this embodiment, the two normal frames are the normal frame with sequence number P+3 and the normal frame with sequence number P-3. The descriptor vectors of the selected normal frames and the corresponding key frames are averaged together to obtain the basis for closed-loop detection. In this embodiment, the calculation formula is as follows:

[0067]

[0068] Where T0 represents the mean descriptor vector of key frame P, T Prepresents the descriptor vector of key frame P, T P-3 and T P+3 It represents the descriptor vectors of two ordinary frames selected from before and after the key frame P. The obtained mean descriptor vector T0 is stored in the descriptor library as the basis for subsequent closed-loop detection.

[0069] Step S3, feature enhancement is performed on the low-light data through a Fourier feature enhancement network, and feature matching and posture calculation are performed on the enhanced data to estimate the motion state.

[0070] This step uses an algorithm that integrates the FFENet network (Fourier Feature Enhanced Network). For a given input image X, the image parameters are H×W. In order to achieve brightness improvement, the image is converted to Fourier space, as shown in the following formula:

[0071]

[0072] Among them, h and w are spatial coordinates, u and v are Fourier space coordinates, and j is an imaginary unit.

[0073] For the input image X(u,v) in Fourier space, the corresponding amplitude component A(X(u,v)) and phase component P(X(u,v)) are expressed by the following formulas:

[0074]

[0075]

[0076] Among them, R(X(u,v)) and I(X(u,v)) represent the real part and imaginary part of the input image X(u,v), respectively. In this way, the amplitude component and phase component are separated from the converted image.

[0077] In this step, the FFENet network first converts the input image to Fourier space, obtains the amplitude component and phase component, and performs convolution and activation on these two components to extract features. Finally, these two components are converted back to the spatial domain and passed through a 3×3 convolution layer for input residual processing. The amplitude component of the input image is divided by the amplitude transform map estimated by the FP block (Fourier Processing Block) to achieve amplitude amplification. The amplified amplitude component and phase component are combined for inverse Fourier transform to obtain the output Img out . This process can be expressed as follows:

[0078] A in =A(F(input)),P in =P(F(input)),

[0079] A out =A in / (M+ε),

[0080] R out =A out ×cos(P in ), I out =A out ×sin(P in ),

[0081] Img out =F -1 (R out +j I out ),

[0082] Where F(·) represents Fourier transform, F -1 (·) represents inverse Fourier transform, A(·) represents the extracted amplitude, P(·) represents the extracted phase, A in and P in are the amplitude component and phase component of the input image, respectively. M is the amplitude transformation map estimated by the FP block. ε represents a very small constant used to avoid zero division. In this embodiment, it is set to 1.0×10 -8 The estimated amplitude transform map is generated by the FP block. The input image is sent to the FP block for processing, and the amplitude component before the inverse Fourier transform is obtained during the processing as the estimated amplitude transform map M. out is the output amplitude component obtained by amplification, R out and I out Represents the output Img out The real and imaginary parts in Fourier space.

[0083] There is a common problem in the current low-light enhancement network: channels and pixels are treated equally when processing low-light images, and the uneven distribution of illumination in the channel direction is not effectively handled. This processing method causes the network to be unable to fully distinguish the importance of channel features and pixel features in the image, thus affecting the effect of image enhancement. While the above processing achieves brightness improvement, it also causes serious loss in detail areas. To solve this problem, this algorithm introduces a feature detail recovery module, which includes a difference weight network DWNet. This network can assign different weights to different channel features and pixel area distributions on the feature map, thereby more flexibly processing different types of information and improving network performance.

[0084] The input image is processed to separate high- and low-frequency components, and the features of the separated high- and low-frequency components are extracted, that is, the intermediate feature map is obtained through convolution-activation-convolution operations. The shape of the feature map changes from H×W×C to 1×1×C after processing, so that the underlying network can fully use the global information. The intermediate feature map is input into the difference weight network DWNet. The pixel information and channel information of the intermediate feature map are respectively subjected to global average pooling operations, and the global spatial information is mapped to the channel descriptor I c and pixel descriptor I p .

[0085] Among them, channel descriptor I c The calculation formula is as follows:

[0086]

[0087] Among them, V c (m,n) represents the cth channel V of the image c The value at position (m,n), H p (·) represents the global pooling function, F inc Represents the corresponding image channel input of the difference weight network DWNet.

[0088] Similarly, the pixel descriptor I can be obtained p , the calculation formula is as follows:

[0089]

[0090] Among them, V p (m,n) represents the pixel V of the image p The value at position (m,n), H p (·) represents the global pooling function, F inp represents the image pixel value input of the difference weight network DWNet, F inp and F inc Both can be obtained from the input F of the difference weight network DWNet in Obtained from.

[0091] Thereafter, the pixel descriptor I p and channel descriptor I c They enter the pixel attention module and channel attention module respectively. The channel attention module adaptively resizes each channel feature, focuses on different channel features, gives priority to the dark channel, and generates different weighted information for each channel feature; while the pixel attention module enables the network to filter irrelevant pixel information, pay more attention to important features such as low-light pixels, and provide differentiated weight information for them. In order to obtain differentiated pixel weights and channel weights, the pixel descriptor I p and channel descriptor I cBoth of them will go through two convolutional layers and Sigmoid activation function and ELU activation function. The ELU activation function is an improved version of the ReLU function, and the function formula is as follows:

[0092]

[0093] Among them, x is the function variable, α is the hyperparameter, and its value range is usually [0,1]. Compared with ReLU, ELU also has a small slope in the negative area, avoiding the problem that its output is zero in the negative area, and the average value is close to zero, so it learns faster.

[0094] The pixel weights and channel weights are obtained by weight distribution through the difference weight network DWNet. c Indicates the weights of different channels obtained, CA p Represents the weights of different pixels obtained. The corresponding expression is as follows:

[0095]

[0096]

[0097]

[0098] Among them, τ is the ELU activation function, is the Sigmoid activation function, Conv is the convolution, and finally the pixel weights and channel weights are combined with the input F of the difference weight network DWNet. in Multiply them one by one to get the output result F after feature detail enhancement out .

[0099] The output result F of the feature detail recovery module out and the inverse Fourier transform output Img out The fusion results in a low-light enhanced output image, which is used for tracking and local mapping.

[0100] In step S4, the semantic descriptor is associated with the closed-loop thread, and the closed-loop verification is completed together with the auxiliary word bag in the closed-loop detection to achieve global pose optimization.

[0101] The closed-loop detection is based on the keyframes in the enhanced output image in step S3, and is mainly divided into two stages: finding the initial closed-loop candidate keyframes (hereinafter referred to as candidate keyframes) for the current keyframes and verifying the closed-loop candidate keyframes. The improvement of this algorithm is mainly in the verification of closed-loop candidate keyframes. The normal closed-loop detection algorithm only searches for common words based on a single bag-of-words query to obtain closed-loop candidate keyframes and verify them. On this basis, the present invention compares the similarity of the mean descriptor vector to assist in the screening of closed-loop candidate keyframes, improves the closed-loop conditions, and strictly screens out higher-quality closed-loop candidate keyframes to prevent them from erroneous loops under low-light conditions.

[0102] This algorithm performs similarity measurement based on the mean descriptor vector after the bag-of-words query detection is passed. The image number corresponding to the mean descriptor vector remains unchanged before and after low-light enhancement. This step uses cosine similarity to measure the similarity of the mean descriptor vector, and normalizes the measurement result to [0,1]. The closer the result is to 1, the higher the similarity. The calculation formula of cosine similarity is as follows:

[0103] In the above formula, T1 and T2 represent the mean descriptor vectors of the current keyframe and the candidate keyframe, respectively. After the cosine similarity calculation, strict screening is performed to determine whether to perform the final loop fusion and global optimization to achieve the final closed loop detection.

[0104] The following is an explanation of the process of the above-mentioned deep learning-based visual low-light closed-loop detection algorithm in combination with specific experiments.

[0105] like Figure 6-Figure 14As shown in the figure, the description experiment of the semantic descriptor in similar scenes is verified. Three popular low-light enhancement algorithms based on deep learning are selected to test and compare with the algorithm proposed in the present invention in 8 low-light data sets. For similar scenes, the semantic descriptor proposed by the present algorithm has a good description effect; the comparison results show that compared with other algorithms, the present algorithm has achieved better recovery effects in brightness and details. In order to verify the effectiveness of the present invention in the application of SLAM system, some low-light images in the public outdoor KITTI and indoor TUM_VI and Euroc data sets are selected for feature matching comparison experiments. It can be seen that the number of enhanced matches has increased significantly and the distribution is more balanced, which improves the robustness of feature matching. The accuracy and recall rate curves of the present algorithm and other algorithms in the KITTI data set and TUM-VI data set respectively show that the present algorithm still maintains a high accuracy rate at a high recall rate, which is better than other algorithms. The running estimation and error line graphs of the present algorithm in the indoor TUM-VI data set and the outdoor KITTI data set show that the trajectory formed by the present method is similar to the true trajectory. In the line graph, we can see that after combining this algorithm, the root mean square error (RMSE) and mean error (MEAN) are reduced compared with other algorithms.

[0106] like Fig.15 As shown in the figure, in another set of experiments, an indoor obstacle field was selected as the experimental environment, and a low-light environment was set for experimental verification. From the experimental results, it can be seen that due to the influence of the low-light environment, the original ORB_SLAM3 algorithm produced a large trajectory drift in the low-light area, and GCNV2_SLAM had a tracking loss phenomenon resulting in inaccurate positioning accuracy, while this algorithm showed a better effect in generating trajectories. The actual scene experiment proved that this algorithm is more superior when facing a low-light environment.

[0107] Embodiment 2.

[0108] Corresponding to the first embodiment of the present invention, the second embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following steps are implemented according to the method of the first embodiment.

[0109] Step S1, collecting environmental information based on an image collection device, and dividing semantic areas from the image of the environmental information.

[0110] Step S2, performing centerline sampling on the divided semantic areas to generate local superpixel semantic descriptors.

[0111] Step S3, feature enhancement is performed on the low-light data through a Fourier feature enhancement network, and feature matching and posture calculation are performed on the enhanced data to estimate the motion state.

[0112] Step S4, data association is performed between the semantic descriptor and the closed-loop thread, and the auxiliary word bag is used to jointly complete the closed-loop verification in the closed-loop detection to achieve global pose optimization.

[0113] The above storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), optical disk and other media that can store program codes.

[0114] The specific limitations on the steps implemented after the program in the computer-readable storage medium is executed can be found in Example 1, and will not be described in detail here.

[0115] Embodiment three.

[0116] Corresponding to the first embodiment of the present invention, the third embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, the following steps are implemented according to the method of the first embodiment.

[0117] Step S1, collecting environmental information based on an image collection device, and dividing semantic areas from the image of the environmental information.

[0118] Step S2, performing centerline sampling on the divided semantic areas to generate local superpixel semantic descriptors.

[0119] Step S3, feature enhancement is performed on the low-light data through a Fourier feature enhancement network, and feature matching and posture calculation are performed on the enhanced data to estimate the motion state.

[0120] Step S4, data association is performed between the semantic descriptor and the closed-loop thread, and the auxiliary word bag is used to jointly complete the closed-loop verification in the closed-loop detection to achieve global pose optimization.

[0121] The specific limitations on the above steps of implementing the computer device can be found in Example 1, and will not be described in detail here.

[0122] It should be noted that each box in the block diagram and / or flow chart in the accompanying drawings of the specification of the present invention, as well as the combination of boxes in the block diagram and / or flow chart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and machine instructions.

[0123] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the inventive concept and technical solution of the present invention, or the inventive concept and technical solution are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A visual low-light closed-loop detection algorithm based on deep learning, characterized by: The following steps are involved: Step S1, collecting environmental information based on an image acquisition device, and dividing semantic areas from an image of the environmental information; Step S2, performing centerline sampling on the divided semantic regions to generate local superpixel semantic descriptors; the local superpixel semantic descriptors are referred to as descriptors for short, centerline sampling extracts the pixels located on the centerline of each semantic region on the image, and then calculates the three parameters of the spectral transformation rate, contrast eigenvalue and color saturation rate of each semantic region based on the pixels of the centerline of the semantic region, and then calculates and generates corresponding descriptors based on the above three parameters, and the descriptors of all semantic regions form the descriptor vectors of the corresponding images, and the descriptor vectors are stored in the descriptor library; Step S3, feature enhancement is performed on the low-light data through a Fourier feature enhancement network, and feature matching and pose calculation are performed on the enhanced data to estimate the motion state; Step S4, data association between the semantic descriptor and the closed-loop thread, and the auxiliary word bag in the closed-loop detection jointly completes the closed-loop verification to achieve global pose optimization; Step S4 is divided into two stages: finding an initial closed-loop candidate key frame for the current key frame and verifying the closed-loop candidate key frame; in verifying the closed-loop candidate key frame, this step is used to assist in screening the closed-loop candidate key frame by comparing the similarity of the mean descriptor vector; The spectral transformation rate is used to describe the texture transformation trend of the central information of the semantic region. It is obtained by calculating the weighted sum of the first-order differences of the grayscale values ​​of all pixels on the center line and the grayscale values ​​of their neighborhood pixels. The formula for calculating the spectral transformation rate is as follows: In this semantic area, Lt is the spectral transformation rate of the semantic area, m d Represents the ordinate of the horizontal center line of the semantic region, n d represents the horizontal coordinate of the vertical center line of the semantic region, f(x,m d +1) indicates the xth row, m d +1 column pixel gray value, f(n d +1,y) indicates the nth d +1 row, y column pixel grayscale value, m, n are the number of rows and columns of pixels, s is the number of pixels collected; The contrast feature value is used to characterize the light and dark changes of each semantic area, so as to show the brightness difference of each part. At the same time, the R, G, and B channels are processed by centerline sampling to make them contain more stable feature contrast information. The formula for calculating the contrast feature value is as follows: In this semantic region, Ce represents the contrast feature value of the semantic region, y c represents the yth column in channel c, x c represents the xth row in channel c, where channel c is any of the three channels R, G, and B. f(n d ,y c ) represents the nth d The grayscale value of the pixel in row y and column y, f(x c ,m d ) represents the xth row, mth row in channel c d The pixel grayscale value of the column, m and n are the number of rows and columns of pixels respectively, and s is the number of pixels collected; In step S3, the Fourier feature enhancement network converts the input image into Fourier space, obtains the amplitude component and the phase component, and performs convolution and activation on the two components respectively to extract features; The amplitude component is divided by the amplitude transformation map estimated by the FP block to realize amplitude amplification, and the amplified amplitude component and phase component are combined to perform inverse Fourier transformation to obtain output; The Fourier feature enhancement network also introduces a feature detail recovery module including a difference weight network DWNet. The input image is processed to separate high and low frequency components, and then feature extraction is performed to obtain an intermediate feature map. The intermediate feature map is input into the difference weight network DWNet for weight distribution to obtain pixel weights and channel weights, which are then combined with the input of the difference weight network DWNet to obtain the output result after feature detail enhancement. The output result after feature detail enhancement and the output of inverse Fourier transform are fused to obtain the output image after low-light enhancement.

2. The deep learning-based visual low-light closed-loop detection algorithm according to claim 1, characterized in that: In step S1, each frame image is divided into a·b semantic regions, where a and b are the number of rows and columns of the semantic region respectively; in step S2, the descriptor T of each semantic region is calculated. ij A descriptor matrix of a·b is formed, and the rows of the descriptor matrix are expanded into a vector to obtain the descriptor vector T. For any key frame P that may be involved in the detection similarity, a common frame before and after the key frame image P is located respectively, and the descriptor vectors of the selected common frame and the corresponding key frame are averaged together as the basis for closed-loop detection. The descriptor of each semantic area is described by the following formula: In the above formula, T ij It represents the descriptor of the semantic region in the i-th row and j-th column of the image. Lt, Ce and Sr respectively represent the spectral transformation rate, contrast eigenvalue and color saturation rate of the corresponding semantic region. ω1, ω2 and ω3 represent the parameter weights corresponding to Lt, Ce and Sr respectively.

3. The deep learning-based visual low-light closed-loop detection algorithm according to claim 2, characterized in that: Color saturation is a more stable information obtained from the HSV color space. In the HSV color space, the algorithm separates the S channel with strong light resistance in HSV to obtain color saturation information; the formula for calculating color saturation is as follows: In this semantic area, Sr represents the color saturation rate of the semantic area, R, G, and B represent the three channel components of the RGB image in the semantic area, y represents the column number of the pixel, x represents the row number of the pixel, m and n are the number of rows and columns of the pixel points, and s is the number of collected pixels.

4. The deep learning-based visual low-light closed-loop detection algorithm according to claim 3, characterized in that: In the feature detail recovery module, the input image is converted into an intermediate feature map through convolution-activation-convolution operations, and then the global spatial information is mapped into channel descriptors and pixel descriptors by performing a global average pooling operation. After that, the pixel descriptors and channel descriptors enter the pixel attention module and the channel attention module respectively to obtain differentiated pixel weights and channel weights. The channel attention module adaptively resizes each channel feature, focuses on different channel features, and gives priority to the dark channel, generating different weighted information for each channel feature. The pixel attention module enables the network to filter irrelevant pixel information, pay more attention to important features such as low-light pixels, and provide differentiated weight information for them. Finally, the pixel weights and channel weights are multiplied one by one with the input of the difference weight network DWNet to obtain the output result after feature detail enhancement.

5. The deep learning-based visual low-light closed-loop detection algorithm according to claim 1, characterized in that: In step S4, for the current key frame input, it is first determined whether there is a candidate key frame. When there is a candidate key frame, the similarity is detected through a bag-of-words query. After the bag-of-words query is detected, the similarity is measured based on the mean descriptor vector. This step uses cosine similarity to measure the similarity.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a deep learning-based visual low-light closed-loop detection algorithm as described in any one of claims 1 to 5 are implemented.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of a deep learning-based visual low-light closed-loop detection algorithm as described in any one of claims 1 to 5 are implemented.

Images