An image location recognition method and device fusing monocular depth estimation

Abstract

The application discloses a kind of image position recognition method and device fusing monocular depth estimation, it is related to the cross field of computer vision and robot, wherein, recognition method includes: based on monocular depth estimation network from single visible light image obtains depth feature map;Depth feature map is connected with visible light image, and constructs lightweight feature extraction module as backbone network, extracts feature descriptor, obtains global feature descriptor by normalization operation;Based on anchor point image in dataset, according to anchor point image, positive and negative sample in dataset is selected to image and constructs loss function, extracts the global feature descriptor of each visible light image, and constructs feature database, calculates the feature similarity of image to be indexed image and feature database in image, according to feature similarity calculation result, from big to small order selects the first preset number of image as recognition result;In the scene of darker illumination, larger angle of view change, good recognition effect can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the intersection of computer vision and robotics, in particular to an image location recognition method and device fusing monocular depth estimation. BACKGROUND

[0002] In the past few decades, visual location recognition has been rapidly developed, which is a challenging task in the field of computer vision and robotics. Visual location recognition refers to the process of recognizing whether a place has been visited using visual information. Most existing methods are based on appearance, and the appearance of the same place may change greatly under different lighting conditions, viewpoints, seasons, distances, occlusions, etc. This ability to correctly and effectively recall previously seen locations using only visual input has many important applications, such as loop closure detection technology for correcting drift in the simultaneous localization and mapping process, image retrieval based on visual content, car navigation, and 3D model construction, etc.

[0003] The core of visual location recognition is image retrieval, and there are some specific problems. First, two different places may present similar or identical elements due to architectural structures, making them difficult to distinguish. Second, the same scene may present significantly different visual effects if viewed from different angles, and there may be little overlap between the query image and the database image. These problems make visual location recognition more challenging. In addition, the potential data characteristics have a significant impact on the difficulty of the resulting visual location recognition problem and the applicability of specific algorithms.

[0004] Three-dimensional geometric information helps to understand and perceive the scene, thereby alleviating the perceptual aliasing problem in visual location recognition and improving the robustness of visual location recognition. There are many ways to obtain three-dimensional geometric information, including but not limited to laser scanning sensors that can directly obtain three-dimensional data. In mobile scenarios, such as wearable devices, due to hardware limitations of mobile platforms, only one type of sensor can be mounted, so directly estimating depth from a single image becomes a very promising method in the field of visual location recognition. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide an image location recognition method fusing monocular depth estimation, which has strong robustness in scenes with large image illumination changes and large image viewpoint differences, and can achieve good recognition results in scenes with dark illumination and large viewpoint changes.

[0006] To achieve the above purpose, the technical solution adopted by the present application is:

[0007] Based on the monocular depth estimation network, depth information is obtained from a single visible light image to obtain a depth feature map.

[0008] concatenate the depth feature map and the visible light image at a channel level to obtain a fusion image, and construct a lightweight feature extraction module as a backbone network to extract a feature descriptor of the fusion image, and obtain a global feature descriptor through a normalization operation;

[0009] select a data set, obtain an anchor point image based on the data set, select positive and negative sample pair images in the data set according to the anchor point image, a preset distance threshold and a preset angle threshold, construct a loss function based on the positive and negative sample pair images, and train the monocular depth estimation network in a supervised manner combined with a difficult sample mining strategy;

[0010] extract a global feature descriptor of each visible light image based on the trained monocular depth estimation network, construct a feature database, calculate the feature similarity between the image to be indexed and the images in the feature database, and select the first preset number of images as the recognition result in descending order according to the feature similarity calculation result.

[0011] On the basis of the above technical solutions, the depth information is obtained from a single visible light image based on the monocular depth estimation network, and a depth feature map is obtained, and the specific steps include:

[0012] a standard Transformer module is used as a feature encoder of the depth prediction network to convert the input image into a vector representation, and the vector representation is passed through a second preset number of Transformer stages to recombine the vector representations of different levels into a pseudo image representation, and the pseudo image representation is obtained through step-by-step upsampling to obtain the depth feature map.

[0013] On the basis of the above technical solutions, the depth feature map is calculated by the following formula:

[0014]

[0015] wherein, is the depth feature map, Up is an upsampling operation, is a deeper pseudo image representation, c is a proportionality coefficient of the resolution of the input original image, represents a fusion process, T is a visual Transformer encoding, and token is a feature encoding of the visible light image after linear mapping.

[0016] On the basis of the above technical solutions, the lightweight feature extraction module is constructed as a backbone network to extract the feature descriptor of the fusion image, and the global feature descriptor is obtained through the normalization operation, and the specific steps include:

[0017] The deep neural network ResNet is used as a backbone network to extract fusion deep features of the fusion image, and the fusion image is subjected to image cropping, color transformation, tensor representation and image pixel value normalization operations to obtain a global description feature sub.

[0018] On the basis of the above technical solution, the loss function is a triplet loss function, and the triplet loss function is implemented by:

[0019]

[0020] Wherein, a i is the i th anchor image, is the j th positive sample image corresponding to the i th anchor image, is the k th negative sample image corresponding to the i th anchor image, f is the global feature extracted from the image, max represents the maximum value, m is the distance between the anchor image feature and the positive sample image feature, and the difference between the distance between the anchor image feature and the negative sample image feature.

[0021] On the basis of the above technical solution, the process of making the positive and negative sample pairs of images comprises:

[0022] The coordinate conversion relationship is determined, the corresponding relationship between the inertial measurement unit coordinate system and the camera coordinate system in the sensor is used to convert the pose, and the pose true value corresponding to each image is obtained;

[0023] The overall data set is divided into a training set, a validation set and a test set according to a ratio of 4:1:1, and the pose difference between any two images in the three data sets obtained by division is calculated, and the calculation formula is:

[0024]

[0025]

[0026] Wherein, Dis is the distance between images, D is the position difference between images, T is the position corresponding to the image, I is a data set containing N images, is the image currently selected as an anchor, I y is any image in the data set except the image selected as an anchor, a is the threshold of translation error, Ang is the angle between images, A is the pose difference between images, Q is the attitude corresponding to the image, and β is the threshold of rotation error;

[0027] It is judged whether Dis and Ang are less than 0 at the same time, if yes, it is a positive sample, if not, it is a negative sample.

[0028] On the basis of the above technical solution, the implementation steps of the difficult sample mining strategy are:

[0029] In the process of training the network model, a first operation is performed, the first operation is to select the samples identified as errors as difficult samples, use the difficult samples to train the network model, and recalculate the triplet loss, and evaluate the recognition effect of the network;

[0030] The first operation is repeatedly performed until the recognition effect on the verification set no longer improves.

[0031] On the basis of the above technical solution, the first preset number of images are selected as the recognition result according to the feature similarity calculation result from large to small, and the specific steps include:

[0032] Based on the to-be-indexed image, the feature similarity Sim(query, DB m ) between the to-be-indexed image and the images in the feature database is calculated, and the calculation method is:

[0033] Sim(query, DB m )=cos(f(I query ),f(I m ))

[0034] Wherein, cos is the cosine distance between two features, DB m is the number of the mth image in the feature database, I query is the to-be-indexed image, f is the image global feature of the image, and I m is the mth image in the feature database.

[0035] The first preset number of images with the largest similarity are selected as the recall and the result is returned.

[0036] On the basis of the above technical solution, the calculation method of the returned result is:

[0037]

[0038] Wherein, top_k is the returned result, Sort[] k is the top k results according to the quantitative value, is the anchor image in the vth triplet, I n is the image other than the to-be-indexed image in the feature database, and f is the image global feature of the image.

[0039] The application also provides an image position recognition device fusing monocular depth estimation, comprising:

[0040] An acquisition module is configured to acquire depth information from a single visible light image based on a monocular depth estimation network to obtain a depth feature map;

[0041] a fusion module, which is used to concatenate the depth feature map and the visible light image at a channel level to obtain a fusion image, and construct a lightweight feature extraction module as a backbone network to extract a feature descriptor of the fusion image and obtain a global feature descriptor through a normalization operation;

[0042] a training module, which is used to collect images and constitute a data set, obtain anchor point images based on the data set, select positive and negative sample pairs of images in the data set according to the anchor point images, a preset distance threshold and a preset angle threshold, construct a loss function based on the positive and negative sample pairs of images, and train the monocular depth estimation network in a supervised manner combined with a difficult sample mining strategy;

[0043] an execution module, which is used to extract a global feature descriptor of each visible light image based on the monocular depth estimation network trained by the training module, construct a feature database, calculate the feature similarity between a to-be-indexed image and images in the feature database, and select images of a first preset number as recognition results in descending order according to the feature similarity calculation results.

[0044] Compared with the prior art, the present application has the advantages that the present application can fuse monocular image depth estimation results to realize image position recognition, and the results obtained by the present application can be displayed from a global and local perspective. The present application has strong robustness in scenes with large image illumination changes and large image perspective differences, and can achieve good recognition results in scenes with dark illumination and large perspective changes. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 a flowchart of a kind of image position recognition method fusing monocular depth estimation in the embodiment of the present application;

[0047] Figure 2 a principle diagram of a kind of image position recognition method fusing monocular depth estimation in the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments.

[0049] The present application aims at the defects of the prior art in the single image location recognition task, and proposes an image location recognition method fusing monocular depth estimation, which estimates the depth from a single visible light image and obtains the depth information, and further obtains the depth feature map corresponding to each picture; the depth feature map and the visible light image are concatenated at the channel level to obtain the image feature representation fused with the depth information, then a lightweight feature extraction module is used as the backbone network to extract the features of the fused image, and a global feature description vector is obtained through batch normalization operation; an anchor-positive-negative triple loss function is constructed to realize network training, and a scene database is constructed based on the extracted features, so as to realize robust location recognition of a single visible light image.

[0050] Referring to Figure 1 The embodiment of the present application provides an image location recognition method fusing monocular depth estimation, which comprises the following steps:

[0051] S1: obtaining the depth information from a single visible light image based on a monocular depth estimation network, and obtaining a depth feature map;

[0052] S2: concatenating the depth feature map and the visible light image at the channel level to obtain a fused image, constructing a lightweight feature extraction module as a backbone network, extracting the feature descriptor of the fused image, and obtaining a global feature descriptor through normalization operation;

[0053] S3: selecting a data set, obtaining an anchor image according to the data set, selecting positive and negative sample images according to the anchor image, a preset distance threshold and a preset angle threshold, constructing a loss function based on the positive and negative sample images, and training the monocular depth estimation network in a supervised manner combined with a difficult sample mining strategy;

[0054] S4: extracting the global feature descriptor of each visible light image based on the trained monocular depth estimation network, constructing a feature database, calculating the feature similarity between the image to be indexed and the images in the feature database, and selecting the first preset number of images as the recognition result in descending order according to the feature similarity calculation result.

[0055] The embodiment of the present application provides a specific step of an image position recognition method fused with monocular depth estimation, which is first based on a monocular depth estimation network to obtain depth information from a single visible light image, to obtain a depth feature map, then the depth feature map and the visible light image are concatenated at the channel level to obtain a fused image, the channel level concatenation is specifically that the color image, i.e. the RGB three channels (red, green and blue three channels), and the depth image, i.e. one channel, are superimposed in the channel dimension to generate a four-channel image, then a lightweight feature extraction module is constructed as a backbone network to extract a feature descriptor of the fused image, and a global feature descriptor is obtained through a normalization operation; a dataset is selected according to actual application, and then an anchor image is obtained based on the dataset, wherein the anchor image is obtained from the dataset (training set and validation set), because the network uses the accuracy of the validation to select the most suitable network parameters in the training process, therefore the dataset includes the training set and the validation set, then positive and negative sample pairs of images in the dataset are selected according to the anchor image, a preset distance threshold and a preset angle threshold, a loss function is constructed based on the positive and negative sample pairs of images, and the monocular depth estimation network is trained in a supervised manner combined with a difficult sample mining strategy; the global feature descriptor of each visible light image is extracted based on the trained monocular depth estimation network to construct a feature database; based on a given image to be indexed, the feature similarity between the image to be indexed and each image in the feature database is calculated, the calculated feature similarity is arranged from large to small, and the first preset number of images are selected from large to small as the recognition result.

[0056] Referring to Figure 2 Fig. 1 is a principle diagram of an image position recognition method fused with monocular depth estimation in the embodiment of the present application, in Figure 2 DPT is a dense prediction transformer, and RGBD is a red-green-blue-depth image, i.e. a four-channel image; first, the monocular depth estimation network is used to accurately estimate the depth feature map corresponding to a single image; then the depth feature map and the visible light image are concatenated at the channel level to fuse, and ResNet (Deep residual network) is used to extract the features of the fused image, i.e. anchor features, positive sample features and negative sample features, to obtain global feature descriptors through normalization; an anchor-positive-negative sample triple is constructed, and a triple loss function is designed to realize end-to-end training of the network in a supervised manner, based on the trained feature extractor, a scene feature database is constructed, and any given image to be searched is implemented to realize robust single image position recognition.

[0057] Further, the depth information is obtained from a single visible light image based on the monocular depth estimation network to obtain a depth feature map, and the specific steps include:

[0058] The standard Transformer module is used as a special encoder of the deep prediction network to convert the input image into a vector representation, and the vector representation is passed through a second preset number of Transformer stages to recombine the vector representations of different levels into a pseudo-image representation and obtain a depth feature map through stepwise upsampling.

[0059] The monocular depth estimation network obtains depth information from a single visible light image to obtain a depth feature map. First, the input visible light image is divided into non-overlapping image blocks, and high-dimensional linear mapping is realized through a multi-layer perceptron. The token of the linearly mapped feature is input into a standard visual Transformer encoder to realize information exchange between different feature tokens in a global receptive field. A pyramid architecture is used to extract four feature maps of different resolutions at four different Transformer stages. The recombination operation is used to recombine the feature tokens of different resolutions into feature maps of corresponding resolutions, and stepwise upsampling is performed to obtain the final depth prediction feature, i.e., the depth feature map.

[0060] Further, the depth feature map is calculated by the following formula:

[0061]

[0062] wherein, is the depth feature map, and Up is the upsampling operation, is a deeper pseudo-image representation, represents the fusion process, T is a visual Transformer encoding, token is the feature encoding of the visible light image after linear mapping, c is a proportionality coefficient of the resolution of the input original image, and in the algorithm of the present embodiment, 4, 8, 16 and 32 are taken, respectively, representing that the feature map of this layer is 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the feature resolution of the input image.

[0063] Further, the light-weight feature extraction module is constructed as a backbone network to extract the feature descriptor of the fused image, and the global feature descriptor is obtained through normalization operation. The specific steps include:

[0064] The deep neural network ResNet is used as a backbone network to extract the fused depth feature of the fused image, and the fused image is subjected to image cropping, color transformation, tensor representation and image pixel value normalization operation to obtain the global feature descriptor.

[0065] The deep neural network ResNet is used as the backbone network, that is, a lightweight feature extraction module is constructed as the backbone network, then the deep features of the fusion image are extracted based on the deep neural network ResNet, and the fusion image is preprocessed, the preprocessing includes image cropping, color transformation, tensor representation and image pixel value normalization, wherein the image pixel value normalization parameter is pre-trained using the data in the ImageNet dataset, the network is composed of 34 layers of convolutional layers, batch normalization layers and activation layers, the input channel number is 4, and the output feature is 512 dimensions, that is, the final feature descriptor. The ImageNet dataset is a commonly used image database in the field of image processing.

[0066] Further, for the anchor point image, positive and negative sample pairs of images are selected according to distance threshold and angle threshold, and a three-tuple loss function is constructed, and the network is trained in a supervised manner combined with a difficult sample mining strategy, and the three-tuple loss function is realized as follows:

[0067]

[0068] Wherein, a i is the i th anchor point image, is the j th positive sample image corresponding to the i th anchor point image, is the k th negative sample image corresponding to the i th anchor point image, f is the global feature extracted from the image, max represents the maximum value, m is the distance between the anchor point image feature and the positive sample image feature, and the difference between the distance between the anchor point image feature and the negative sample image feature.

[0069] Further, the process of making the positive and negative sample pairs of images includes:

[0070] Determine the coordinate conversion relationship, use the corresponding relationship between the inertial measurement unit coordinate system and the camera coordinate system in the sensor to convert the pose, and obtain the pose true value corresponding to each image;

[0071] Divide the whole dataset into a training set, a validation set and a test set according to the ratio of 4:1:1, and calculate the pose difference between any two images in the three data sets obtained by division, the calculation formula is:

[0072]

[0073]

[0074] Wherein, Dis is the distance between the images, D is the position difference between the images, T is the position corresponding to the image, I is a data set containing N images, is the image currently selected as the anchor point, I yfor any image in the data set except the anchor image, a is the threshold of translation error, Ang is the angle between images, A is the pose difference between images, Q is the pose corresponding to the image, and β is the threshold of rotation error;

[0075] If Dis and Ang are both less than 0, it is a positive sample, otherwise, it is a negative sample. The positive sample indicates that the angle difference between the two images is within β degrees and the distance difference between the two images is within a meters, and the negative sample does not meet the condition.

[0076] The specific method for making the data triplets in the embodiment of the application is to determine the coordinate conversion relationship, use the corresponding relationship between the sensor inertial measurement unit coordinate system and the camera coordinate system in the image acquisition device to perform corresponding conversion on the pose, calculate the pose true value corresponding to each image, and then, for the whole image data set, divide it into three parts of a training set, a validation set and a test set according to a ratio of 4:1:1, wherein the training set is used for network model training, the validation set is used for network performance evaluation and optimal model parameter selection, and the test set is used for final result verification. In the three data subsets, the pose difference between any two images, including translation error and rotation error, is calculated to determine the positive sample image and the negative sample image corresponding to each image.

[0077] Further, the implementation steps of the difficult sample mining strategy are that in the process of training the network model, a first operation is performed, the first operation is to select the samples that are recognized incorrectly as difficult samples, use the difficult samples to train the network model, and recalculate the triplet loss to evaluate the recognition effect of the network; the first operation is repeatedly performed until the recognition effect on the validation set no longer improves, and when the recognition effect no longer improves, it means that the network model training is completed.

[0078] Further, the first preset number of images are selected as the recognition result according to the feature similarity calculation results from large to small, and the specific steps include:

[0079] Based on the to-be-indexed image, the feature similarity Sim(query, DB m ) between the to-be-indexed image and the images in the feature database is calculated, and the calculation method is:

[0080] Sim(query, DB m )=cos(f(I query ),f(I m ))

[0081] Wherein, cos is the cosine distance between two features, DB m is the number of the mth image in the feature database, and I queryis the image to be indexed, f is the image global feature of the extracted image, I m is the mth image in the feature database.

[0082] According to the trained network, a feature descriptor is extracted for each image in the database, a feature database is constructed, a to-be-indexed image is given, similarity between features is calculated, cos is the cosine distance between two features, the closer the two images are, the smaller the cosine distance is, indicating that the similarity between the two images is greater, then the first preset number of images are selected from large to small as the recognition result, and the first preset number can be set according to actual conditions.

[0083] The first preset number of images with the greatest similarity are selected as the recall and the result is returned.

[0084] Further, the calculation method of the returned result is:

[0085]

[0086] Wherein, top_k is the returned result, Sort[] k is the top k results according to the quantitative value, is the anchor image in the vth triple, I n is the image other than the to-be-indexed image in the feature database, f is the image global feature of the extracted image.

[0087] The present application firstly accurately estimates the depth feature map corresponding to a single image according to the depth estimation network, then fuses the depth feature map and the visible light image at the channel level, extracts the features of the fused image by using a lightweight feature extraction network, and obtains the global feature descriptor through batch normalization; a triple of anchor-positive-negative samples is constructed, and a triple loss function is designed to realize supervised end-to-end training of the network, a scene feature database is constructed according to the trained feature extractor, any to-be-searched image is given, and the final image position recognition is realized; through the above steps, under the condition that a to-be-identified visible light image is given, several images closest to the to-be-identified image can be returned from the database, in the urban scene automatic driving and robot positioning, the single-mode and consumer-level camera sensor can realize robust position recognition under the condition of image overexposure and large difference.

[0088] The embodiment of the present application also provides an image position recognition device fusing monocular depth estimation, comprising an acquisition module, a fusion module, a training module and an execution module.

[0089] The acquisition module is configured to acquire depth information from a single visible light image based on a monocular depth estimation network to obtain a depth feature map; the fusion module is configured to concatenate the depth feature map and the visible light image at a channel level to obtain a fusion image, and construct a lightweight feature extraction module as a backbone network to extract a feature descriptor of the fusion image, and obtain a global feature descriptor through a normalization operation; the training module is configured to collect images and form a data set, obtain an anchor image based on the data set, and select positive and negative samples in the data set according to the anchor image, a preset distance threshold and a preset angle threshold; based on the positive and negative sample images, a loss function is constructed, and the monocular depth estimation network is trained in a supervised manner combined with a difficult sample mining strategy; the execution module is configured to extract a global feature descriptor of each visible light image based on the monocular depth estimation network trained by the training module, and construct a feature database; the feature similarity between the image to be indexed and the images in the feature database is calculated, and the images of a first preset number are selected as the recognition result in descending order according to the feature similarity calculation result.

[0090] The acquisition module converts the input image into a vector representation, which can be generated by flattening and linearly projecting non-overlapping image blocks. The vector representation is recombined into pseudo-image representations with different resolutions through multiple Transformer stages, and then the fusion module gradually fuses the up-sampling of the next layer and the features of the current layer until a depth map prediction result with a specified dimension is generated. During the fusion process, a more detailed depth map prediction result is gradually generated; the fusion module is configured to concatenate the depth feature map and the visible light image at a channel level to obtain a fusion image, and construct a lightweight feature extraction module as a backbone network to extract a feature descriptor of the fusion depth feature image. The network consists of 34 convolutional layers, batch normalization layers and activation layers, with an input channel number of 4 and an output feature of 512 dimensions, which is a global feature descriptor; during the network model training process, the training module first selects samples incorrectly recognized by the model as difficult samples, then uses these samples to train the model again, and recalculates the triplet loss to evaluate the network effect. The process is repeated until the performance on the validation set no longer improves; the execution module uses the cosine distance between binary feature vectors as the similarity measure between two images, and selects a preset number of images in descending order of the calculated similarity as the recall, thereby realizing image position recognition.

[0091] The application can realize image position recognition by fusing monocular image depth estimation results, and can display the recognition results from global and local perspectives, and has strong robustness in scenes with large image light changes and large image perspective differences, and can achieve good results in scenes with dark light and large perspective changes, and in urban scene automatic driving and robot positioning, single sensor can realize robust position recognition in the case of image overexposure and large differences.

[0092] The above description is merely a specific implementation of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0093] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

Claims

1. An image location recognition method fusing monocular depth estimation, characterized in that, The method comprises the following steps: obtaining depth information from a single visible light image based on a monocular depth estimation network to obtain a depth feature map; concatenating the depth feature map and the visible light image at a channel level to obtain a fused image, constructing a lightweight feature extraction module as a backbone network, extracting a feature descriptor of the fused image, and obtaining a global feature descriptor through normalization operation; selecting a data set, obtaining an anchor image based on the data set, selecting positive and negative sample pairs of images in the data set according to the anchor image, a preset distance threshold and a preset angle threshold, constructing a loss function based on the positive and negative sample pairs of images, and training the monocular depth estimation network in a supervised manner combined with a difficult sample mining strategy; extracting a global feature descriptor of each visible light image based on the trained monocular depth estimation network, constructing a feature database, calculating the feature similarity between the image to be indexed and the images in the feature database, and selecting the first preset number of images in descending order of the feature similarity calculation results as the recognition results; the specific steps of constructing a lightweight feature extraction module as a backbone network, extracting a feature descriptor of the fused image, and obtaining a global feature descriptor through normalization operation include: using a deep neural network ResNet as a backbone network to extract a fused depth feature of the fused image, and performing image cropping, color transformation, tensor representation and image pixel value normalization on the fused image to obtain a global feature descriptor; the loss function is a triplet loss function, and the implementation method of the triplet loss function is: in, For the first Anchor point image, For the first The first anchor point image corresponding to the first One positive sample image, For the first The first anchor point image corresponding to the first One negative sample image, To extract global features from an image, This indicates taking the maximum value. This represents the distance between the anchor point image features and the positive sample image features, and the difference between the distance between the anchor point image features and the negative sample image features.

2. The image location recognition method of fusing monocular depth estimation as claimed in claim 1, wherein, the specific steps of obtaining a depth feature map from a single visible light image based on a monocular depth estimation network include: using a standard Transformer module as a feature encoder of the depth prediction network to convert the input image into a vector representation, and recombining the vector representations of different levels into a pseudo image representation through a second preset number of Transformer stages, and obtaining a depth feature map through step-by-step upsampling.

3. The image location recognition method of fusing monocular depth estimation as claimed in claim 2, wherein, the depth feature map is calculated by the following formula: wherein, is a deep feature map, is an up-sampling operation, is a deeper pseudo image representation, is a scale factor to the input original image resolution, denotes a fusion process, is a visual transformer encoding, is a feature encoding of the visible light image after linear mapping.

4. The image location recognition method of fusing monocular depth estimation as claimed in claim 1, wherein, the production process of the positive and negative sample pairs of images includes: determining the coordinate conversion relationship, using the corresponding relationship between the inertial measurement unit coordinate system and the camera coordinate system in the sensor to convert the pose to obtain the pose ground truth corresponding to each image; dividing the overall data set into a training set, a validation set and a test set according to a ratio of 4:1:1, and calculating the pose difference between any two images in the three data sets obtained by division, and the calculation formula is: wherein, is a distance between images, is a positional difference between images, is a position corresponding to an image, is a data set containing images, is an image currently selected as an anchor point, is any image in the data set other than the image selected as an anchor point, is a threshold value for a translation error, is an angle between images, is a pose difference between images, is a pose corresponding to an image, is a threshold value for a rotation error; determining and whether simultaneously less than 0, if yes, then positive sample, if no, then negative sample.

5. The image location recognition method of fusing monocular depth estimation as claimed in claim 4, wherein, the implementation steps of the difficult sample mining strategy are: in the process of training the network model, a first operation is performed, the first operation is to select the samples identified incorrectly as difficult samples, use the difficult samples to train the network model, and recompute the triplet loss to evaluate the recognition effect of the network; the first operation is repeatedly performed until the recognition effect on the validation set no longer improves.

6. The image location recognition method of fusing monocular depth estimation as claimed in claim 5, wherein, the specific steps of selecting the first preset number of images with the largest similarity as the recall and returning the results include: Based on the image to be indexed, a feature similarity between the image to be indexed and features of images in a feature database is calculated The calculation method is: wherein, is a cosine distance between two features, is a feature in a feature database, is a number of the i-th image, is an image to be indexed, is an image global feature of the extracted image, is a feature in a feature database, is the i-th image. selecting the first preset number of images with the largest similarity as the recall and returning the results.

7. The image location recognition method of fusing monocular depth estimation as claimed in claim 6, wherein, The calculation method of the return result is: wherein, is the returned result, is the first results, is the anchor image in the th triple, is the image other than the image to be indexed in the feature database, is the image global feature of the extracted image.

8. An image position recognition device that fuses monocular depth estimation, characterized by, Include: The acquisition module is used for acquiring depth information from a single visible light image based on a monocular depth estimation network to obtain a depth feature map; The fusion module is used for concatenating the depth feature map and the visible light image at a channel level to obtain a fusion image, constructing a lightweight feature extraction module as a backbone network, extracting a feature descriptor of the fusion image, and obtaining a global feature descriptor through a normalization operation; The training module is used for collecting images and constructing a data set, obtaining anchor point images based on the data set, selecting positive and negative sample pairs of images in the data set according to the anchor point images, a preset distance threshold and a preset angle threshold, constructing a loss function based on the positive and negative sample pairs of images, and training the monocular depth estimation network in a supervised manner combined with a difficult sample mining strategy; The execution module is used for extracting a global feature descriptor of each visible light image based on the monocular depth estimation network trained by the training module, constructing a feature database, calculating the feature similarity between the image to be indexed and the images in the feature database, and selecting the first preset number of images as the recognition result in descending order according to the feature similarity calculation result; The specific steps of constructing the lightweight feature extraction module as the backbone network, extracting the feature descriptor of the fusion image, and obtaining the global feature descriptor through the normalization operation include: A deep neural network ResNet is used as the backbone network to extract the fusion depth feature of the fusion image, and the fusion image is subjected to image cropping, color transformation, tensor representation and image pixel value normalization operation to obtain the global feature descriptor; The loss function is a triplet loss function, and the implementation method of the triplet loss function is: wherein, is the i-th anchor image, is the i-th anchor image, is the i-th anchor image pair, is the i-th positive sample image corresponding to the i-th anchor image pair, is the i-th negative sample image corresponding to the i-th anchor image pair, is the i-th anchor image, is the i-th anchor image, is the i-th anchor image, is the global feature extracted from the image, denotes taking the maximum value, is the distance between the anchor image feature and the positive sample image feature, and the difference between the distance between the anchor image feature and the negative sample image feature.

Images