A cross-modal place recognition method based on joint constraints of global and local features

Abstract

The present invention relates to the field of computer vision technology, and in particular to a cross-modal place recognition method based on joint constraints of global and local features. The present invention comprises the following steps: step 1, respectively inputting a color image and an infrared image into the GLFC‑Net network model; step 2, using the ResNet50 network and the local branch module, generating shared features rich in globality and locality from the images of the two modalities input in step 1; step 3, extracting corresponding global features from the shared features rich in globality; step 4, extracting corresponding local features from the shared features rich in locality; step 5, performing joint constraints at the global level and the local level on the global features obtained in step 3 and the local features obtained in step 4; step 6, if the specified number of training rounds is reached, then end; otherwise, continue to complete the training and return to step 1. The present invention increases the robustness to the problems of insufficient illumination and change of viewing angle, thereby improving the accuracy of place recognition.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-modal place recognition method based on joint constraints of global and local features. Background Art

[0002] Visual place recognition aims to help a robot or a vision-based navigation system determine whether it is located in a place it has visited before. It takes an image from a camera and identifies where the image is located. This can be seen as an image retrieval problem. The process is as follows: Figure 1 As shown. In SLAM, it can be used for re-positioning, map reuse or loop correction, and has broad application prospects in mobile robots and unmanned vehicle driving. For example, a mobile robot is placed in an unknown location in an unknown environment. While walking, it continuously acquires new feature landmarks and corrects the map model in the robot system. When the robot accidentally walks a long way and wants to know whether the current section has been there before, it will input the current location image and then search the data set of the location recognition system to find the corresponding location image.

[0003] In early studies, visual place recognition was applied to single-mode place recognition, and the place recognition task was color images throughout the entire recognition process. However, in real environments, color cameras cannot capture clear place images in low-light environments. For example, in the mobile robot mentioned in the previous paragraph, if only a color camera is used for shooting, the quality of the place images taken during the day will be better, and the quality of the photos taken at night will be poor. In this way, it is impossible to obtain clear place images in insufficient lighting environments.

[0004] With the development of technology, existing camera devices can be equipped with both color cameras and infrared cameras. The color camera captures location images in a well-lit environment, while the infrared camera captures location images in a low-light environment. For example, in the mobile robot mentioned in the previous paragraph, at night, when the lighting environment on the road is poor when the robot is walking, the infrared camera can be used to capture location images. This creates favorable conditions for the location recognition task to get rid of the lighting limitations, and the cross-modal location recognition task based on color modality and infrared modality came into being, namely the "color-infrared" visual place recognition (Visible Infrared Visual Place Recognition, VI-VPR) task. For example, in the robot mentioned in this paragraph, the camera equipped on the robot captured the image of a place that may have passed by during the day at night. In order to know whether the current road section has been visited before, it is necessary to match the infrared image with the set of color images captured during the day. An example is shown below. Figure 2 However, “color-infrared” visual place recognition is a multimodal recognition problem, which is more challenging than the unimodal recognition problem.

[0005] Figure 1 The figure shows the location recognition of the color query image in the Pittsburgh 30k dataset in a single mode. It is an image retrieval process that selects and sorts several images similar to the query image in the gallery. The color image on the left is the query set that is taken to identify the current location, and the color image on the right is the color gallery set recorded at the location that was passed before. The query image and the data in the gallery are searched and the images similar to the query image are selected.

[0006] Figure 2 The figure shows the cross-modal image matching of the Pittsburgh 30k infrared query set and the Pittsburgh 30k color gallery set obtained through simulation. The query set and the gallery set were taken at the same location, but at different angles. The color modality location image has 3 channels, while the infrared modality location image only contains 1 channel, which makes the difference between the color image and the infrared image larger, making the "color-infrared" visual location recognition task challenging. The infrared image on the left is a query set taken in an environment with insufficient lighting. The query set and the color gallery set recorded at the previously passed locations are used for image retrieval, and the images similar to the query image are screened out to obtain the color image on the right, where the infrared image on the left corresponds to the location of the color image on the right.

[0007] In recent years, many place recognition methods have been proposed, which have achieved good results in visible light image place recognition. However, due to the large difference between infrared images and visible light images, the existing place recognition methods are not very effective in solving multimodal recognition problems. Therefore, Huang Ze et al. designed an event-RGB cross-modal fusion network for visual place recognition, which has a cross-modal attention module, a self-attention module, and a pooling layer module. The specific network framework is as follows: Figure 3 As shown. The network is divided into three parts, namely the cross-modal attention module, the residual self-attention module and the pooling layer. First, the ResNet50 network is used for global feature extraction. Then, the features of the two modalities are fused through the cross-modal attention module. Then, the self-attention module is used to focus on important information. Finally, after the pooling layer, the number of parameters is reduced to speed up the calculation and prevent overfitting. Through the cross-modal attention module, the correspondence between the two modalities is established, and the features of the two modalities are fused with each other. Although the features of the two modalities are obtained, the ability to distinguish the features of the original image is reduced. The training flow chart is as follows: Figure 4 As shown in Figure 2, the shared features extracted by this model are based on global features. The samples contain a lot of noise with interfering properties such as illumination and occlusion. The model that only uses global features has poor robustness to noise.

[0008] Existing visual place recognition methods mostly use a single-modal approach to perform place recognition to solve problems such as GPS signal failure in mountainous areas and military areas and SLAM, but ignore the problem of incomplete feature extraction due to insufficient light at night or in severe weather conditions. Summary of the invention

[0009] In order to improve the accuracy of the prior art under insufficient lighting and changing viewing angles, the present invention proposes a cross-modal place recognition method based on joint constraints of global and local features.

[0010] The technical solution adopted by the present invention is: a cross-modal place recognition method based on joint constraints of global and local features, characterized in that it includes the following steps:

[0011] Step 1: Input a color image and an infrared image into the GLFC-Net network model respectively;

[0012] Step 2: Using the ResNet50 network and the local branch module, the two modal images input in step 1 are used to generate shared features rich in global and local features;

[0013] Step 3: Extract corresponding global features from the shared features rich in globality;

[0014] Step 4: extract corresponding local features from the shared features rich in locality;

[0015] Step 5: Perform global and local joint constraints on the global features obtained in step 3 and the local features obtained in step 4;

[0016] Step 6: If the specified number of training rounds is reached, the process ends; otherwise, the training continues and the process returns to step 1.

[0017] As a preferred technical solution of the present invention: Step 2 comprises the following steps:

[0018] Step 2-1: Color image and infrared images The input network is passed through a layer of convolution operation to obtain the shallow features of the color mode and infrared mode. and Step 2-2: and At the same time, the network consisting of the convolutional layers and local branch modules of the last four layers of ResNet50 is input to generate global shared features of the two modalities respectively. and local shared features

[0019] Step 2-3: Output global shared features and local shared features

[0020] As a preferred technical solution of the present invention: in the step 2, a local branch module is embedded after the third convolution layer in the four convolution layers of ResNet50, and the extraction of global shared features is performed through the four convolution layers of ResNet50, combined into a network for extracting global shared features and local shared features in step 2-2; the local branch module is composed of multiple void convolution modules and residual self-attention modules.

[0021] As a preferred technical solution of the present invention: Step 3 comprises the following steps:

[0022] Step 3-1: Global shared features and Perform global average pooling to obtain quasi-global features rich in globality and Step 3-2: and Input BN layer to generate the final global features and Step 3-3: Output global features and

[0023] As a preferred technical solution of the present invention: the multiple hole convolution modules perform the following steps:

[0024] Step S1: Global average pooling is performed on the global shared features obtained by convolution operation on the first three layers of ResNet50;

[0025] Step S2: Perform convolution operation on the features obtained in step S1;

[0026] Step S3: activating the features obtained in step S2;

[0027] Step S4: perform three dilated convolution operations on the global shared features obtained by performing convolution operations on the first three layers of ResNet50 respectively;

[0028] Step S5: stitching together the features obtained in step S3 and step S4;

[0029] Step S6: performing a convolution operation on the features obtained in step S5;

[0030] Step S7: activating the features obtained in step S6;

[0031] The residual self-attention module performs the following steps:

[0032] Step a: The features f obtained by multiple dilated convolution modulesMulti-Atrous Perform three convolution operations respectively to obtain f Query , f Key , f Value ;

[0033] Step b: Add f Query and f Key After multiplication, Softmax operation is performed to obtain the features

[0034] Step c: and f Value Multiply to get feature f;

[0035] Step d: Perform convolution operation on f to obtain feature f';

[0036] Step e: Input f' into the BN layer and f Multi-Atrous Add up to get the final local shared features and

[0037] As a preferred technical solution of the present invention: the step 4 comprises the following steps:

[0038] Step 4-1: Local shared features and Perform average pooling to obtain quasi-local features rich in locality and

[0039] Step 4-2: and Input BN layer to generate the final local features and

[0040] Step 4-3: Output local features and

[0041] As a preferred technical solution of the present invention: the global-level and local-level joint constraints in step 5 are composed of global-level constraint loss and local-level constraint loss; the global-level constraint loss and the local-level constraint loss are based on the difficult triplet loss and cross-entropy loss of the VPR task and the heterogeneous center loss inspired by cross-modal pedestrian re-identification.

[0042] As the preferred technical solution of the present invention: the difficult triplet loss and cross entropy loss based on the VPR task, the difficult triplet loss considers two different modes, and adds a difficult sampling process, and performs difficult sampling on the difficult samples of the two modes, and selects the most difficult to match positive sample pair and the easiest to match negative sample pair, and then calculates the loss; while the heterogeneous center loss is to shorten the distance between the centers of the two modes within the class, reduce the change of the cross-modality within the class, thereby improving the feature similarity across the modalities within the class, and then calculate the loss; the Euclidean distance is used to measure the distance between two feature vectors, as shown in formula (1):

[0043] D(f1,f2)=||f1-f2||2 (1)

[0044] Among them, f1 and f2 represent the feature vectors of the location image;

[0045] Suppose P different locations are selected in the training set, and K color location images and K infrared location images are randomly selected from each location. Each batch of training has 2PK location images. The difficult triplet loss for the VI-VPR task is shown in formula (2):

[0046]

[0047] Among them, f is the set of feature vectors, is the feature vector of the anchor point location image and is the set of two modal location images, is the feature vector of the positive sample location image in color or infrared modality, and Have the same location landmarks, is the feature vector of the negative sample location image in the color or infrared modality, and do not have the same place landmarks; when the sum of the Euclidean distance between the difficult positive sample pairs and α is less than the Euclidean distance between the difficult negative sample pairs, the anchor point place image can be correctly matched with all the place images in the batch; where α represents a manually set threshold parameter.

[0048] As a preferred technical solution of the present invention: when the cross entropy loss is adopted, P different locations are selected in the training set, and K color location images and K infrared location images are randomly selected from each location. Each batch of training has 2PK location images. The cross entropy loss for the VI-VPR task is shown in formula (3):

[0049]

[0050] Among them, f represents the set of feature vectors, p(y i |f i ) represents the eigenvector fi The model predicts that i The probability of , encoded by the softmax function;

[0051] When the heterogeneous center loss is used, P different locations are selected in the training set, and K color location images and K infrared location images are randomly selected from each location. The heterogeneous center loss for the VI-VPR task is shown in formula (4):

[0052]

[0053] in,

[0054]

[0055] Among them, c i,RGB and c i,IR represents the center of the two modal feature distributions at the ith location, f represents the feature vector set, and f RGB and f IR Represent the color image feature vector set and the infrared image feature vector set respectively, f i,RGB,j and f i,IR,j Two modal feature vectors representing the jth location image of the ith location.

[0056] As a preferred technical solution of the present invention: the global feature constraint loss is based on the difficult triplet loss and cross entropy loss of the VI-VPR task to constrain the global features; the global feature constraint loss utilizes the quasi-global features of step 3-1 and And the global features of step 3-3 and Its expression is shown in formula (5):

[0057]

[0058] The local feature constraint loss is based on the heterogeneous center loss and cross entropy loss of the VI-VPR task to constrain local features; the local feature constraint loss utilizes the quasi-local features of step 4-1 and And the local features of step 4-3 and Its expression is shown in formula (6);

[0059]

[0060] The global features and local features are jointly constrained, and finally the global level loss and the local level loss are combined, as shown in formula (7);

[0061] LGL =L Global +L Local (7)

[0062] Among them, L Global is the global feature constraint loss, L Local is the local feature constraint loss.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention proposes a global feature and local feature joint constraint network, learns and distinguishes the representation of two modal features, increases the robustness to insufficient lighting and perspective changes, and thus improves the accuracy of place recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of single-modal image location recognition of the present invention;

[0065] Figure 2 is a schematic diagram of dual-modal image location recognition of the present invention;

[0066] Figure 3 It is a schematic diagram of a cross-modal feature fusion network in the prior art;

[0067] Figure 4 It is a flow chart of the cross-modal fusion network training phase of the prior art;

[0068] Figure 5 It is a schematic diagram of the GLFC-Net network framework of the present invention;

[0069] Figure 6 It is a schematic diagram of the process of the present invention;

[0070] Figure 7 It is a schematic diagram of a local branch module of the present invention. DETAILED DESCRIPTION

[0071] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0072] This paper proposes a framework GLFC-Net (Global And Local Feature Constraints Network, GLFC-Net) based on a dual-stream network structure for the "color-infrared" visual place recognition task, which combines global features and local features to reduce cross-modal differences and intra-modal differences in cross-modal visual place recognition tasks, thereby improving the accuracy of place recognition. The framework diagram of the network is shown in the figure below. Figure 5As shown in the figure, the network architecture is divided into three parts, namely the global feature extraction module, the local branch module and the pooling layer. First, the ResNet50 network is used to extract global features. After the Res3 stage of the ResNet50 network structure, the local branch module is used to extract local features. Then, in order to prevent overfitting, the extracted global features and local features are pooled, and finally the loss value is calculated for the obtained features.

[0073] The training flow chart of GLFC-Net proposed in the present invention is as follows: Figure 6 As shown. The training process uses small batch training. Each time, P location images are randomly selected, and K color images and K infrared images are randomly selected for these location images. Next, we will take a color image and an infrared image as an example to introduce the training process. At this time, P is 1 and K is 1, as follows:

[0074] Step 1: Input a color image and an infrared image into the GLFC-Net network model respectively, and then proceed to step 2;

[0075] Step 2: Using the ResNet50 network and local branch module, the two modal images input in step 1 are used to generate shared features rich in global and local features and enter step 3;

[0076] Step 3: Extract the corresponding global features from the shared features rich in globality and proceed to step 4;

[0077] Step 4: Extract the corresponding local features from the shared features rich in locality, and proceed to step 5;

[0078] Step 5: Perform global and local joint constraints on the global features obtained in step 3 and the local features obtained in step 4, and proceed to step 6;

[0079] Step 6: If the specified number of training rounds is reached, proceed to step 7, otherwise continue training and return to step 1;

[0080] Step 7: End.

[0081] The color images in step 1 are all from standard data sets for visual place recognition, such as Pittsburgh 30k, Tokyo 24 / 7, etc., while the infrared images are simulated from data sets such as Pittsburgh 30k, Tokyo 24 / 7, etc. The color image is composed of three primary colors: red, green, and blue, and contains three channels, each channel corresponding to a primary color. In the present invention, the infrared image exists in the form of a single channel. The present invention assumes that there are n color place images and m infrared place images, and the samples of the color modality can be expressed as The infrared modal sample can be expressed as in, represents the i-th color location image, represents the jth infrared location image, and Respectively and Each color image and infrared image is randomly flipped horizontally with zero padding, and finally a location image with a height of 480 and a width of 640 is formed. Consistent with the above, the present invention takes a color modality sample as input and an infrared modality sample Take the example to introduce the working principle of the present invention in the training process.

[0082] The present invention adopts a dual-stream network structure and uses a residual network and a local branch module to extract global shared features and local shared features. The present invention adopts a ResNet50 network, which can increase the network depth and improve the feature expression capability. At the same time, the present invention embeds the local branch module in the ResNet50 network, which can better extract local shared features, so that the feature information will not affect the detection and matching of other features due to the disappearance of some features. The structure of the local branch module is as follows Figure 7 As shown in the figure, the main function of this module is to extract local features. It consists of a multi-hole convolution module and a self-attention module. First, after the feature extraction of the first three stages of ResNet50, the multi-hole convolution module is used to expand the receptive field, capture multi-scale contextual information, and obtain more local features. Then, the residual self-attention module is used to gather important information, and finally the extracted local features are directly output.

[0083] The steps of extracting global and local shared features in step 2 of the training process of the present invention are as follows:

[0084] Step 2-1: Color image and infrared images The input network is passed through a layer of convolution operation to obtain the shallow features of the color mode and infrared mode. and

[0085] Step 2-2: and At the same time, the network consisting of the convolutional layers and local branch modules of the last four layers of ResNet50 is input to generate global shared features of the two modalities respectively. and local shared features

[0086] Step 2-3: Output global shared features and local shared features

[0087] In step 2-1, the convolutional layers passed by the two modal images have the same structure but different parameters.

[0088] In step 2-2, the four convolutional layers that the images of the two modalities pass through have the same structure and parameters.

[0089] The present invention embeds a local branch module after the third convolutional layer in the four convolutional layers of ResNet50, and the extraction of global shared features is performed through the four convolutional layers of ResNet50, which are combined into a network for extracting global shared features and local shared features in step 2-2.

[0090] The local branch module of the present invention is composed of multiple hole convolution modules and residual self-attention modules.

[0091] The steps of extracting global features in step 3 of the training process of the present invention are as follows:

[0092] Step 3-1: Global shared features and Perform global average pooling to obtain quasi-global features rich in globality and

[0093] Step 3-2: and Input BN (Batch Normalization) layer to generate the final global features and

[0094] Step 3-3: Output global features and

[0095] In step 3-2, the BN layer used can make the data distribution approximate to a normal distribution and avoid the gradient vanishing problem.

[0096] The steps of multiple dilated convolution modules in the present invention are as follows:

[0097] Step S1: Perform global average pooling on the global shared features obtained by convolution operations on the first three layers of ResNet50.

[0098] Step S2: Perform a convolution operation on the features obtained in step S1.

[0099] Step S3: Activate the features obtained in step S2.

[0100] Step S4: Perform three dilated convolution operations on the global shared features obtained by convolution operations on the first three layers of ResNet50.

[0101] Step S5: Connect the features obtained in step S3 and step S4.

[0102] Step S6: Perform a convolution operation on the features obtained in step S5.

[0103] Step S7: Activate the features obtained in step S6.

[0104] In the present invention, the parameters of the three dilated convolution operations performed in step 4 of the multiple dilated convolution modules are different, in order to obtain the features of three different receptive fields.

[0105] The steps of the residual self-attention module in the present invention are as follows:

[0106] Step a: The features f obtained by multiple dilated convolution modules Multi-Atrous Perform three convolution operations respectively to obtain f Query , f Key , f Value .

[0107] Step b: Add f Query and f Key After multiplication, Softmax operation is performed to obtain the features

[0108] Step c: and f Value Multiply them together to get feature f.

[0109] Step d: Perform a convolution operation on f to obtain feature f'.

[0110] Step e: Input f' into the BN layer and f Multi-Atrous Add up to get the final local shared features and

[0111] In the present invention, the parameters of the three convolution operations performed in step 1 of the residual self-attention module are different.

[0112] The steps of extracting local features in step 4 of the training process of the present invention are as follows:

[0113] Step 4-1: Local shared features and Perform average pooling to obtain quasi-local features rich in locality and

[0114] Step 4-2: and Input BN layer to generate the final local features and

[0115] Step 4-3: Output local features and

[0116] The global level and local level joint constraints in step 5 of the training process of the present invention are composed of global level constraint loss and local level constraint loss.

[0117] The global level constraint loss and the local level constraint loss in step 5 of the training process of the present invention are based on the difficult triplet loss and cross entropy loss of the VPR task and the heterogeneous center loss inspired by cross-modal pedestrian re-identification.

[0118] The present invention proposes a difficult triplet loss and a cross entropy loss based on the VPR task. The difficult triplet loss takes into account two different modalities and adds a difficult sampling process. It also performs difficult sampling on the difficult samples of the two modalities, and selects the most difficult to match positive sample pair and the easiest to match negative sample pair, and then calculates the loss. The heterogeneous center loss is to shorten the distance between the centers of the two modalities within the class, reduce the changes in the cross-modalities within the class, thereby improving the feature similarity across modalities within the class, and then calculate the loss. The Euclidean distance is used to measure the distance between two feature vectors, as shown in Formula 1. Wherein, f1 and f2 represent the feature vectors of the location image.

[0119] D(f1,f2)=||f1-f2||2 (1)

[0120] Assume that P different locations are selected in the training set, and K color location images and K infrared location images are randomly selected from each location, then each batch of training has 2PK location images. The difficult triplet loss for the VI-VPR task is shown in Formula 2. Where f is the feature vector set, is the feature vector of the anchor point location image and is the set of two modal location images, is the feature vector of the positive sample location image in color or infrared modality, and Have the same location landmarks, is the feature vector of the negative sample location image in the color or infrared modality, and Do not have the same place landmarks. When the sum of the Euclidean distance between the difficult positive sample pairs and α is less than the Euclidean distance between the difficult negative sample pairs, the anchor point location image can be correctly matched with all the location images in the batch. Among them, α represents a manually set threshold parameter.

[0121]

[0122] The present invention adopts cross entropy loss. Similar to the difficult triplet loss for cross-modal place recognition tasks, assuming that P different places are selected in the training set, and K color place images and K infrared place images are randomly selected from each place, then each batch of training has 2PK place images. The cross entropy loss for the VI-VPR task is shown in Formula 3. Where f represents the feature vector set, p(y i |f i ) represents the eigenvector f i The model predicts that i The probability of , encoded by the softmax function.

[0123]

[0124] The present invention adopts heterogeneous center loss to improve the feature similarity of cross-modal within a class by shortening the distance between the centers of two modalities within a class and reducing the variation of cross-modal within a class. Similar to the difficult triplet loss for cross-modal location recognition tasks, it is assumed that P different locations are selected in the training set, and K color location images and K infrared location images are randomly selected from each location. The heterogeneous center loss for the VI-VPR task is shown in Formula 4. Where c i,RGB and c i,IR represents the center of the two modal feature distributions at the ith location, f represents the feature vector set, and f RGB and f IR Represent the color image feature vector set and the infrared image feature vector set respectively, f i,RGB,j and f i,IR,j Two modal feature vectors representing the jth location image of the ith location.

[0125]

[0126] in,

[0127]

[0128] The global feature constraint loss proposed in the present invention is based on the difficult triplet loss and cross entropy loss of the VI-VPR task, which constrains the global features and ensures the effectiveness of the global features. The global feature constraint loss utilizes the quasi-global feature in step 3-1. and And the global features of step 3-3 and Its expression is shown in Formula 5.

[0129]

[0130] The local feature constraint global feature proposed in this invention is based on the heterogeneous center loss and cross entropy loss of the VI-VPR task, which constrains local features and ensures the effectiveness of local features. The local feature constraint loss utilizes the quasi-local features in step 4-1. and And the local features of step 4-3 and Its expression is shown in Formula 6.

[0131]

[0132] The global features and local features of the present invention are jointly constrained, and ultimately the global level loss and the local level loss are combined, as shown in Formula 7.

[0133] L GL =L Global +L Local (7)

[0134] The test process of the present invention is:

[0135] Step A: Input the query dataset and gallery dataset, and proceed to step B;

[0136] Step B: Using the trained model, extract features from all the location images in the query dataset and gallery dataset input in step A, and proceed to step C;

[0137] Step C: perform similarity matching between the query dataset features and the gallery dataset features, and proceed to step D;

[0138] Step D: According to the similarity, the matching result of each location image in the query data set and the library data set is obtained, and then the process goes to step E;

[0139] Step E: End.

[0140] The query data set in step A of the test process represents a set of location images to be queried, and the gallery data set represents a set of location images to be matched by the query set.

[0141] In step B of the test process, the GLFC-Net model performs global shared feature extraction and local shared feature extraction, and uses global features and local features as the final feature representation.

[0142] The similarity calculation method in step C of the test process is to first screen out some images with close similarity between the query data set and the library data set through the nearest neighbor search method, and then perform similarity matching through the optimal matching algorithm.

[0143] In step D of the test process, each image in the query dataset has several matching images from the gallery set, and is evaluated using Rank-1, Rank-5, and Rank-10 indicators. The Rank-k accuracy measures the probability of the correct cross-modal place image appearing in the top K retrieval results.

[0144] In this embodiment, the model parameters of the model of the present invention are updated by mini-batch gradient descent. The batch size in the experiment is set to 4, that is, 4 different locations are randomly selected from the training data set. Each batch randomly selects 4 color images and 4 infrared images for each location, which contains a total of 16 color images and 16 infrared images, which facilitates the constraints of triple loss for difficult sample mining. Since infrared images have only a single channel, this experiment expands it into the same three channels as color images. All images are randomly horizontally flipped with zero padding to further enhance the generalization ability of the experiment. The images of the two modes are finally cropped to a size of 480×640.

[0145] The loss function of the present invention is set to the global feature and local feature joint constraint L described in the invention content. GL . The global feature loss and local feature loss parameters ρ are set to 0.3, P is set to 4, and K is set to 4. On each data set, the initial learning rate of this experiment is set to 0.01, the stochastic gradient descent optimization algorithm is used, the momentum parameter is set to 0.9, and the training is repeated for 60 rounds. The change of learning rate Learning_rate(epoch) with the training round epoch is shown in Formula 8.

[0146]

[0147] Embodiment 1:

[0148] This embodiment will use the 10,000-image library color dataset in Pittsburgh 30k and the query infrared dataset obtained by simulating the 6,816 query color datasets in Pittsburgh30k to complete the cross-modal place recognition task and test the performance of the model.

[0149] The Pittsburgh 30k dataset is a subset of the Pittsburgh 250k dataset and is a standard dataset in the field of visual place recognition. 200 different locations were manually selected in the Pittsburgh 250k dataset. Each location contains 10 color images and 10 infrared images, and the infrared images corresponding to the color images are taken at different angles. This experiment uses 4,000 location images from the 200 manually selected locations for training, and the processed Pittsburgh 30k dataset is used for testing. This experiment evaluates two retrieval modes: color image retrieval infrared image, and infrared image retrieval color image.

[0150] In this experiment, the two datasets are randomly split and the training set and test set are selected. The experimental results on the Pittsburgh30k dataset are shown in Table 1.

[0151] Table 1 shows the performance of the present invention on the Pittsburgh 30k dataset. Rank-1, Rank-5, and Rank-10 are evaluation criteria for accuracy.

[0152]

[0153] From the experimental data in Table 1, it can be seen that the performance of using infrared images to identify color images is better than that of using color images to identify infrared images. The possible reason is that the number of color images is greater than the number of infrared images, and more errors are generated in the color image retrieval infrared image mode.

[0154] Embodiment 2:

[0155] This embodiment will use the Tokyo 24 / 7 color dataset and the Tokyo 24 / 7 infrared dataset obtained through simulation to complete the cross-modal place recognition task and test the performance of the model.

[0156] Tokyo 24 / 7 dataset is a standard dataset in the field of visual place recognition. It contains a 76k database, and the query dataset is taken during the day, sunset and night. This experiment uses the model trained with Pittsburgh 250k dataset to test the Tokyo24 / 7 dataset. Since the database set has 76k images, it is too large. This experiment will randomly select 10k images for the database set. This experiment evaluates the two retrieval modes of color image retrieval infrared image and infrared image retrieval color image.

[0157] This experiment tests the model on the above test set. The experimental results on the Tokyo 24 / 7 dataset are shown in Table 2.

[0158] Table 2 shows the performance of the present invention on the Tokyo 24 / 7 dataset. Rank-1, Rank-5, and Rank-10 are the evaluation criteria for accuracy.

[0159]

[0160] The present invention conducts simulation experiments in the Pittsburgh 30k simulation data set and the Tokyo 24 / 7 simulation data set. In the Pittsburgh 30k simulation data set, in the mode of using color images to identify infrared images, the Rank-1 of the method of the present invention is 33.5%, the Rank-5 is 48.5%, and the Rank-10 is 54.6%. In the mode of using infrared images to identify color images, the Rank-1 of the method of the present invention is 39.4%, the Rank-5 is 56.7%, and the Rank-10 is 65.1%. In the Tokyo 24 / 7 simulation data set, in the mode of using color images to identify infrared images, the Rank-1 of the method of the present invention is 27.6%, the Rank-5 is 38.2%, and the Rank-10 is 49.6%. In the mode of using infrared images to identify color images, the Rank-1 of the method of the present invention is 32.2%, the Rank-5 is 45.1%, and the Rank-10 is 61.3%.

[0161] Embodiment 3:

[0162] This embodiment will introduce an applicable scenario of the present invention.

[0163] Recently, a school put unmanned express delivery into the campus. During the day, the test successfully delivered the express from the starting point to the destination. At night, the starting point and destination of the express delivery were the same as during the day. The pictures taken were infrared images. Because they were taken at night, the clarity was not high and the color information was not rich, which increased the difficulty of location recognition. The method proposed in the present invention can solve this problem.

[0164] First, the video of the route taken during the day is captured at a standard of 60 frames, and it is stipulated that the area within 5 meters is considered to be the same location. Then, the images are given corresponding names and used as a gallery set.

[0165] Secondly, the same standard as capturing images during the day is used to capture images at night as a query set, and both are input into the model proposed by the present invention.

[0166] Next, the model proposed in the present invention obtains an output of a set of image sequences, and determines whether the locations are consistent based on the names of the images.

[0167] Finally, by matching the given location, we can learn whether the unmanned delivery vehicle has taken the wrong road at night.

[0168] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A cross-modal place recognition method based on joint constraints of global and local features, characterized in that: The following steps are involved: Step 1: Input a color image and an infrared image into the GLFC-Net network model respectively; Step 2: Using the ResNet50 network and the local branch module, the two modal images input in step 1 are used to generate shared features rich in global and local features; Step 3: Extract corresponding global features from the shared features rich in globality; Step 4: extract corresponding local features from the shared features rich in locality; Step 5: Perform global and local joint constraints on the global features obtained in step 3 and the local features obtained in step 4; Step 6: If the specified number of training rounds is reached, the training ends; otherwise, the training continues and returns to step 1; The step 2 comprises the following steps: Step 2-1: Color image and infrared images The input network is passed through a layer of convolution operation to obtain the shallow features of the color mode and infrared mode. and Step 2-2: and At the same time, the network consisting of the convolutional layers and local branch modules of the last four layers of ResNet50 is input to generate global shared features of the two modalities respectively. and local shared features Step 2-3: Output global shared features and local shared features In the step 2, a local branch module is embedded after the third convolution layer in the four convolution layers of ResNet50, and the extraction of global shared features is performed through the four convolution layers of ResNet50, and is combined into a network for extracting global shared features and local shared features in step 2-2; the local branch module is composed of multiple void convolution modules and residual self-attention modules; The step 3 comprises the following steps: Step 3-1: Global shared features and Perform global average pooling to obtain quasi-global features rich in globality and Step 3-2: and Input BN layer to generate the final global features and Step 3-3: Output global features and The multiple hole convolution modules execute the following steps: Step S1: Global average pooling is performed on the global shared features obtained by convolution operation on the first three layers of ResNet50; Step S2: Perform convolution operation on the features obtained in step S1; Step S3: activating the features obtained in step S2; Step S4: perform three dilated convolution operations on the global shared features obtained by performing convolution operations on the first three layers of ResNet50 respectively; Step S5: stitching together the features obtained in step S3 and step S4; Step S6: performing a convolution operation on the features obtained in step S5; Step S7: activating the features obtained in step S6; The residual self-attention module performs the following steps: Step a: The features f obtained by multiple dilated convolution modules Multi-Atrous Perform three convolution operations respectively to obtain f Query , f Key , f Value ; Step b: Add f Query and f Key After multiplication, Softmax operation is performed to obtain features Step c: and f Value Multiply to get feature f; Step d: Perform convolution operation on f to obtain feature f'; Step e: Input f' into the BN layer and f Multi-Atrous Add up to get the final local shared features and The step 4 comprises the following steps: Step 4-1: Local shared features and Perform average pooling to obtain quasi-local features rich in locality and Step 4-2: and Input BN layer to generate the final local features and Step 4-3: Output local features and The global-level and local-level joint constraints in step 5 are composed of a global-level constraint loss and a local-level constraint loss; the global-level constraint loss and the local-level constraint loss are based on the difficult triplet loss and cross entropy loss of the VPR task and the heterogeneous center loss inspired by cross-modal person re-identification; The difficult triplet loss and cross entropy loss based on the VPR task, the difficult triplet loss considers two different modes, and adds a difficult sampling process, and performs difficult sampling on the difficult samples of the two modes, from which the most difficult to match positive sample pair and the easiest to match negative sample pair are selected, and then the loss is calculated; while the heterogeneous center loss is to shorten the distance between the centers of the two modes in the class, reduce the change of the cross-modality in the class, thereby improving the feature similarity across the modalities in the class, and then calculate the loss; the Euclidean distance is used to measure the distance between two feature vectors, as shown in formula (1): D(f1,f2)=||f1-f2||2 (1) Among them, f1 and f2 represent the feature vectors of the location image; Suppose P different locations are selected in the training set, and K color location images and K infrared location images are randomly selected from each location. Each batch of training has 2PK location images. The difficult triplet loss for the VI-VPR task is shown in formula (2): Among them, f is the set of feature vectors, is the feature vector of the anchor point location image and is the set of two modal location images, is the feature vector of the positive sample location image in color or infrared modality, and Have the same location landmarks, is the feature vector of the negative sample location image in the color or infrared modality, and do not have the same place landmarks; when the sum of the Euclidean distance between the difficult positive sample pairs and α is less than the Euclidean distance between the difficult negative sample pairs, the anchor point location image can be correctly matched with all the location images in the batch; where α represents a manually set threshold parameter; When cross entropy loss is used, P different locations are selected in the training set, and K color location images and K infrared location images are randomly selected from each location. Each batch of training has 2PK location images. The cross entropy loss for the VI-VPR task is shown in formula (3): Among them, f represents the set of feature vectors, p(y i |f i ) represents the eigenvector f i The model predicts that i The probability of , encoded by the softmax function; When the heterogeneous center loss is used, P different locations are selected in the training set, and K color location images and K infrared location images are randomly selected from each location. The heterogeneous center loss for the VI-VPR task is shown in formula (4): Among them, c i,RGB and c i,IR represents the center of the two modal feature distributions at the ith location, f represents the feature vector set, and f RGB and f IR Represent the color image feature vector set and the infrared image feature vector set respectively, f i,RGB,j and f i,IR,j The two modal feature vectors representing the jth location image of the ith location; The global level constraint loss is based on the difficult triplet loss and cross entropy loss of the VI-VPR task to constrain the global features; the global level constraint loss utilizes the quasi-global features of step 3-1 and And the global features of step 3-3 and Its expression is shown in formula (5): The local level constraint loss is based on the heterogeneous center loss and cross entropy loss of the VI-VPR task to constrain local features; the local level constraint loss utilizes the quasi-local features of step 4-1 and And the local features of step 4-3 and Its expression is shown in formula (6); The global features and local features are jointly constrained, and finally the global level loss and the local level loss are combined, as shown in formula (7); L GL =L Global +L Local (7) Among them, L Global is the global level constraint loss, L Local is the local level constraint loss.

Images