A lightweight open-set landmark recognition method for mobile terminals

Abstract

A lightweight open-set landmark recognition method for mobile terminals, belonging to the field of computer vision. The present invention first improves the MobileNet‑V2 lightweight network to make it suitable for landmark recognition tasks, then uses an auxiliary training set and constructs a new loss function to improve the network's external distribution anomaly detection capability, and finally uses multiple indicators to evaluate network performance. The present invention is based on a lightweight neural network model and combines it with an external distribution detection method, so that the model deployed on the mobile terminal can not only eliminate abnormal image interference, but also efficiently identify landmark buildings within the task, while having the advantages of low latency and light weight.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically involves knowledge such as neural networks, landmark recognition, and external distribution detection. Background Art

[0002] Landmark buildings are important features in human daily activity areas. If they can be accurately detected and identified in real-world environments for specific building locations, they can provide important support for third-party industries such as autonomous navigation and tourism services. Currently, with the rapid development of artificial intelligence and Internet of Things technologies characterized by deep learning, an important foundation has been laid for the recognition of high-resolution natural scene images on mobile terminals.

[0003] Traditional recognition methods include support vector machines, K nearest neighbors, maximum correlation coefficient, adaptive Gaussian classifier, convolutional neural networks, etc. Taking convolutional neural networks as an example, it is assumed that the input sample belongs to a certain category of known distribution, and the result is predicted by outputting probability through the Softmax layer after feature extraction. Due to the open set environment in actual applications, the network often encounters a large number of unknown images. If the traditional closed set recognition method continues to be used, all inputs will be mistakenly judged as known categories. The network model using the open set recognition method can not only reject out-of-distribution abnormal images in the real environment, but also correctly identify landmark buildings within the task. From the perspective of anomaly detection, traditional learning methods usually assume that the data for model training and testing are independent and identically distributed, but in fact, after the network is deployed, the test images are often uncontrolled. At present, the landmark recognition method based on deep learning can easily identify any image as a category within the task and give a high confidence, which is obviously unreasonable. Therefore, enabling deep neural networks to exclude out-of-distribution anomalies and achieve accurate recognition will give the artificial intelligence industry higher application value.

[0004] On the other hand, with the development of mobile Internet of Things technology, smart mobile devices such as mobile phones have become an important part of people's daily life. To complete landmark recognition based on mobile terminals, the neural network model needs to meet strict requirements such as size, latency and accuracy. Although deep convolutional neural networks are currently widely used in the field of computer vision and have achieved excellent performance, due to excessive pursuit of recognition accuracy, the models are getting larger and more complex. For example, the number of layers of ResNet can reach up to 152, and although VGG-16 has 16 layers, its parameter volume is as high as 138M, which is difficult to apply in mobile terminals. First, the model is too large and faces the problem of insufficient memory. Secondly, it does not meet the requirements of fast response. For example, in an autonomous driving system, too slow recognition will significantly increase the possibility of danger. Therefore, it is crucial to reasonably design a lightweight and efficient neural network for mobile terminals.

[0005] In summary, for the open set landmark recognition problem of mobile terminals, the present invention is based on a lightweight neural network and combined with an external distribution detection method, so that the mobile terminal can complete the landmark recognition task in an open set environment in a lightweight and efficient manner. Summary of the invention

[0006] In order to solve the problems of limited mobile terminal resources and abnormal image interference during landmark recognition, the present invention is based on a lightweight neural network model and combines it with an external distributed detection method. The model deployed on the mobile terminal can not only eliminate abnormal image interference, but also efficiently recognize landmark buildings, while having the advantages of low latency and light weight.

[0007] First, we improve the MobileNet-V2 lightweight network to make it suitable for landmark recognition tasks; then we use auxiliary training sets and construct new loss functions to improve the out-of-distribution anomaly detection capabilities, and finally use multiple indicators to evaluate network performance. In addition, we also include details such as data set processing, model training and testing. The specific steps are as follows.

[0008] 1 Lightweight network based on MobileNet-V2

[0009] MobileNet-V2 is a neural network structure specially designed for mobile phones and devices with limited resources. It ensures accuracy while reducing the amount of parameter calculations and required memory to make the network run efficiently. The improved network structure based on MobileNet-V2 is as follows Figure 1 shown.

[0010] The first layer of the network is a two-dimensional convolution layer (Conv2D) containing 32 standard convolution kernels with a stride of 2, and then connected to 17 inverse residual linear bottleneck blocks (Bottleneck). When the number of input and output channels of the inverse residual linear bottleneck block is the same and the stride is 1, a shortcut connection is used; then a 1x1 two-dimensional convolution kernel with a dimension of 1280 and a 7x7 average pooling layer are connected, and two fully connected layers are added at the end. Finally, the orientations of K landmark buildings are predicted. K can be set for different landmark recognition data sets. For example, if a tourist attraction has 50 landmarks that need to be recognized, K is set to 50; if all landmark buildings have a total of 118 different orientations that need to be recognized (the east, west, south and north orientations of each building), K is set to 118. The changes in the network input and output structure are shown in Table 1.

[0011] Table 1 Network structure table

[0012] enter Convolutional structure t c n s 224x224x3 Conv2d - 32 1 2 112x112x32 Bottleneck 1 16 1 1 112x112x16 Bottleneck 6 24 2 2 56x56x24 Bottleneck 6 32 3 2 28x28x32 Bottleneck 6 64 4 2 14x14x64 Bottleneck 6 96 3 1 14x14x96 Bottleneck 6 160 3 2 7x7x160 Bottleneck 6 320 1 1 7x7x320 Conv2d 1x1 - 1280 1 1 7x7x1280 Avgpool 7x7 - 1280 1 - 1x1x1280 FC-512 - 512 1 - 1x1x512 FC-K - K 1 -

[0013] Among them, c represents the number of output channels of the network; n represents the number of repetitions of the linear inverse residual bottleneck block; t represents the multiple of the expansion of the number of input channels in the linear inverse residual bottleneck block, so other layers except the linear inverse residual bottleneck block do not need this parameter; s represents the stride of the first convolution kernel in each sequence. Since the pooling layer and the fully connected layer are non-convolution operations, this parameter is not required. Each row of the table represents a sequence. When n is 1, the sequence contains 1 layer, corresponding to Figure 1 There is a linear inverted residual bottleneck block (Bottleneck) in the middle; when n=4, the sequence contains 4 layers, corresponding to Figure 1 There are 4 linear inverted residual bottleneck blocks in the same sequence. The number of output channels c of each linear inverted residual bottleneck block in the same sequence is the same. In the sequence where n≠1 and s≠1, s≠1 is only valid for the first linear inverted residual bottleneck block, and the stride s of other linear inverted residual bottleneck blocks is still 1. At this time, the number of input and output channels is the same, so shortcut connections are used. t means that the number of input channels of the first linear inverted residual bottleneck block in each sequence is t times the dimension increase. For example, when the number of input channels of the linear inverted residual bottleneck block is 16 and t=6, the number of intermediate channels after 1x1 point-by-point convolution expansion is 96, and the number of output channels c is 24 after 3x3 depth-wise convolution and 1x1 point-by-point convolution. The output feature size calculated by the 7x7 average pooling layer is 1x1x1280. Finally, two fully connected layers are connected to obtain the final landmark category.

[0014] MobileNet-V2 uses depth-wise separable convolution instead of standard convolution, which can be decomposed into depth-wise convolution and point-wise convolution. The structures of the two are as follows: Figure 2 As shown in the figure, depth-wise convolution uses one convolution kernel in each input channel, so the number of input channels does not change; point-wise convolution uses a 1x1 standard convolution kernel to convolve on all channels to fuse the output of the depth-wise convolution kernel.

[0015] Depthwise separable convolution can significantly reduce the model size and computational complexity while approaching the performance of standard convolution. When the convolution kernel size is 3x3, its computational complexity is approximately 1 / 2 of that of standard convolution. times, calculated as shown in formula 1, where D F is the height and width of the input feature map, and M is the number of channels of the input feature map. K is the convolution kernel size, and N is the number of output feature maps.

[0016]

[0017] The linear inverted residual bottleneck block (Bottleneck) adopts a structure of first dimension increase, then depth-wise convolution and finally compression. The dimension increase operation uses 1x1 point-by-point convolution to increase the number of input channels with a dilation factor of t=6, so its output is 6 times the number of input channels, and then connects batch normalization and ReLU6 nonlinear activation function; depth-wise convolution is as follows Figure 2 As shown, a 3x3 convolution kernel is used to perform a convolution operation on each channel of the feature map, followed by batch normalization and ReLU6 nonlinear activation function, with the same number of input and output channels; compression is performed using Figure 2 The 1x1 point-by-point convolution kernel shown in the figure is followed by batch normalization and a linear activation function. In this process, the purpose of increasing the dimension is to enrich the number of features and improve accuracy; after the depth convolution, the number of channels is compressed, so that useful features are automatically selected to reduce the number of parameters. Since ReLU6 will destroy features after compression, the 1x1 point-by-point convolution kernel used for compression is followed by a linear activation function. The structure is as follows Figure 3 shown.

[0018] 2 Out-of-distribution detection loss function

[0019] For the landmark recognition problem, it is necessary to meet the external distribution detection requirements while achieving accurate recognition, that is, first determine whether the input image is a landmark building within the task. If so, predict the corresponding category, otherwise refuse to further recognize it.

[0020] By constructing a new loss function, the model can be trained on the abnormal auxiliary training set and landmark recognition training set Given a network model f and an anomaly auxiliary training set and landmark recognition training set Using the maximum Softmax probability (MSP), we learn through training whether the sample to be tested is in-distributed or out-distributed. The loss function constructed is as shown in Formula 2:

[0021]

[0022] Where x∈X is the landmark input image, y∈Y={1,2,3,...,k} is the landmark corresponding category, and the function f(x) represents the network model. For any x, 1 T f(x)=1 and f(x)≥0. It means to find the mathematical expectation of the previous loss when the landmark image follows the distribution within the task; Indicates the mathematical expectation of the latter loss when the input is an abnormal image; L represents the original classification loss function, U represents the uniform distribution on K categories, H represents the cross entropy loss function from the output of f(x) to the uniform distribution, and λ is a hyperparameter set to 0.5. In the training stage, since the landmark images in the task contain the category information y, and the images in the abnormal auxiliary training set do not need to provide this information, when the input image belongs to the distribution within the task, the network model is recorded as f y(x). Through multiple rounds of training and fine-tuning of the network model, the MSP corresponding to the correctly predicted landmark building category is close to 1, while the MSP corresponding to the abnormal image is close to uniform distribution. The calculation of MSP is as shown in Formula 3:

[0023]

[0024] Among them, z i represents the output of the i-th neuron in the fully connected layer FC-K, and K represents the landmark building category. Formula 3 normalizes the vector z containing K real numbers to a probability distribution containing K numbers proportional to the input exponent, and outputs the maximum Softmax probability MSP, that is, P msp .

[0025] 3 Data preprocessing

[0026] In order to enable the model to have the ability to detect out-of-distribution, an anomaly auxiliary training set is introduced The training is performed, so the dataset contains three parts that are unrelated to each other. The first part is the landmark recognition distribution training set D in , the second part is the external distribution auxiliary training set The third part is the test set for model evaluation, including internal distribution and external distribution test sets. and Data preprocessing includes the following steps:

[0027] 1) For landmark recognition tasks in different places, we collect data sets based on the landmark buildings or the detailed locations of each building, and then split them into training sets according to the ratio of 8:1:1 for each category. Validation set and test set Taking the self-built landmark building dataset BJUT-118 as an example, more than 30,000 landmark building images were collected, including all buildings in the school in 118 directions, so there are 118 categories in total, which are then divided into training sets at a ratio of 8:1:1. Validation set and test set

[0028] 2) Abnormal auxiliary training set Use the public dataset Places365;

[0029] 3) The test set includes the internal distribution test set and out-of-distribution test set in Using public datasets

[0030] ImageNet's landmark class, which is There is no intersection;

[0031] 4) During the network training phase, data enhancement is performed on the landmark images, including random cropping and horizontal flipping of the original image size and normalization. The normalization uses a mean u of (0.485, 0.456, 0.406) and a standard deviation σ of

[0032] (0.229, 0.224, 0.225) normalizes the RGB channels of the input image. The calculation is shown in Formula 4:

[0033]

[0034] Among them, x is the input tensor data of the landmark image, and x′ is the normalized data.

[0035] 4 Training and Testing

[0036] In order to evaluate the open set landmark recognition performance of the network, the present invention first uses the improved network as the base network, then modifies the loss function and fine-tunes the network. The training and testing steps are as follows Figure 4 .

[0037] 1) Train the landmark recognition base network

[0038] Based on the improved base network, 200 rounds of training were performed on the landmark recognition training and validation sets. The cross entropy loss function was used. The SGD optimizer with Nesterov momentum was used, with the initial learning rate of 0.1, the momentum of 0.9, the weight decay of 0.0005, and the LambdaLR learning rate adjustment strategy. The model parameters were trained and saved.

[0039] 2) Modify the out-of-distribution detection loss function

[0040] Based on the landmark recognition base network trained in step 1, the loss function of formula 2 is used to introduce the auxiliary training set Fine-tune the network so that it can learn to distinguish whether an image is a landmark or an abnormal image within the task. The network is trained for 20 rounds. The optimizer uses the SGD optimizer with Nesterov momentum, with an initial learning rate of 0.001, a momentum of 0.9, and a weight decay of 0.0005. In addition, the LambdaLR learning rate adjustment strategy is used to train and save the model parameters.

[0041] 3) Model testing

[0042] Based on the network model trained in step 1 and step 2, test the network's open set landmark recognition performance. This includes the following steps:

[0043] 1. Load the network model and read the landmark test set within the task and out-of-distribution test set

[0044] 2. When the landmark image to be tested obeys the distribution within the task, its landmark information y can be obtained. At this time, the opposite of MSP (maximum Softmax probability) is used as the OOD score S in , so the OOD score is less than 0. The correctly predicted OOD score is recorded as The OOD score of the wrong prediction is recorded as The recognition accuracy can be obtained by dividing the number of correctly classified samples by the total number of test samples.

[0045] 3. When the image to be tested is not a landmark category within the task, the network will make its corresponding MSP close to uniform distribution. The out-of-distribution image is regarded as the positive class, the landmark sample within the task is regarded as the negative class, and the opposite number of MSP is used as the OOD score S out . The AUROC, AUPR, and FPR95 indicators are used to evaluate the out-of-distribution detection capability of the network. AUROC and AUPR can reflect the overall performance of the network across different thresholds. AUROC can be regarded as the probability that the out-of-distribution sample obtains a larger OOD score than the in-distribution sample. The closer the value is to 1, the better the model is. AUPR considers the frequency of abnormal samples, that is, the ratio of out-of-distribution samples to normal samples. The ratio is 1:5. The closer the value is to 1, the better the performance. FPR95 represents the false positive rate when the true positive rate reaches 95%. The closer the value is to 0, the better.

[0046] 4. Based on the OOD scores counted in steps 2 and 3, the OOD score corresponding to FPR95 is set as the threshold δ. Based on δ, it is determined whether the image to be tested obeys the distribution within the task. If so, its landmark category is predicted, otherwise the prediction is rejected. For the landmark recognition task implemented in the BJUT-118 dataset, the threshold δ is set to -0.98, which can be set for various landmark recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is an improved network structure based on MobileNet-V2.

[0048] Figure 2 It is the structure of MobileNet-V2 decomposed into depth-wise convolution and point-wise convolution.

[0049] Figure 3 It is a linear inverse residual bottleneck block structure.

[0050] Figure 4 are the training and testing steps. DETAILED DESCRIPTION

[0051] In order to solve the problems of limited mobile terminal resources and abnormal interference from external distribution during the identification process, the present invention designs a lightweight network and combines it with an external distribution detection method, so that the mobile terminal can correctly identify landmark buildings in an open environment, with the advantages of low latency, small computational complexity and light weight.

[0052] Taking the self-built campus landmark dataset BJUT-118 as an example, the landmark recognition accuracy of the base network is 91.39%. After introducing the outer distribution loss function and fine-tuning, the landmark recognition accuracy is 91.52%. The present invention enables the network to have open-set landmark recognition capabilities while ensuring high accuracy. The model size of the present invention is 9.3M, which has the advantages of light weight and small computational complexity compared to ResNet-18 with a network model of 43M. In order to verify the open-set landmark recognition capability of the network, by adding synthetic anomalies of Gaussian noise, Radmacher noise, and Blob noise and real anomalies of ImageNet, the present invention can generalize the open-set landmark recognition capability of the network to a larger unknown range, and has good open-set landmark recognition capability. The performance comparison is shown in Table 2.

[0053] Table 2 Network performance comparison

[0054]

Claims

1. A lightweight open-set landmark recognition method for mobile terminals, characterized by: 1) Improve the lightweight network based on MobileNet-V2 The first layer of the network is a two-dimensional convolution layer containing 32 standard convolution kernels with a step size of 2, and then connected to 17 inverted residual linear bottleneck blocks Bottleneck, followed by a 1x1 two-dimensional convolution kernel with a dimension of 1280 and a 7x7 average pooling layer. At the same time, two fully connected layers are added at the end. Finally, the orientation of K landmark buildings is predicted, where K is the number of landmarks to be identified. The network structure is as follows: Where c represents the number of output channels of the network; n represents the number of repetitions of the linear inverse residual bottleneck block; t represents the multiple of the expansion of the number of input channels in the linear inverse residual bottleneck block, and s represents the stride of the first convolution kernel in each sequence; Each row of the table represents a sequence. The number of output channels c of each linear inverse residual bottleneck block in the same sequence is the same. In the sequence where n≠1 and s≠1, s≠1 is only valid for the first linear inverse residual bottleneck block, and the stride s of other linear inverse residual bottleneck blocks is still 1. At this time, the number of input and output channels is the same. t means that the number of input channels of the first linear inverse residual bottleneck block in each sequence is multiplied by t. The output feature size after calculation by the 7x7 average pooling layer is 1x1x1280. Finally, two fully connected layers are connected to obtain the final landmark category. The improved network was trained for 200 rounds on the landmark recognition training set and validation set. The loss function used cross entropy and the SGD optimizer with Nesterov momentum was used. The initial learning rate was 0.1, the momentum was set to 0.9, and the weight decay was 0.0005. 2) Constructing out-of-distribution detection loss function Construct a new loss function to allow the model to be trained on the abnormal auxiliary training set at the same time and landmark recognition training set Learning on the network model to generalize the model's out-of-distribution detection ability; given the network model f, the anomaly auxiliary training set and landmark recognition training set Using MSP (maximum Softmax probability), we learn through training whether the input image is an in-distribution landmark image or an out-distribution anomaly; the loss function constructed is as shown in formula (2): Where x∈X is the landmark input image, y∈Y={1,2,3,...,k} is the category corresponding to the landmark, and the function f(x) represents the network model. For any x, 1 T f(x) = 1 and f(x) ≥ 0; It means to find the mathematical expectation of the previous loss when the landmark image follows the distribution within the task; Indicates the mathematical expectation of the posterior loss when the input is an abnormal image; L represents the original classification loss function, U represents the uniform distribution on K categories, H represents the cross entropy loss function from the output of f(x) to the uniform distribution, and λ is a hyperparameter set to 0.5; in the training stage, since the landmark images in the task contain the category information y, and the images in the abnormal auxiliary training set do not need to provide this information, when the input image belongs to the distribution within the task, the network model is recorded as f y (x); After modifying the loss function, the network was fine-tuned for 20 rounds. The optimizer used the SGD optimizer with Nesterov momentum, with an initial learning rate of 0.001, a momentum of 0.9, and a weight decay of 0.0005. Finally, the MSP corresponding to the correctly predicted landmark building category of the network tended to 1, while the MSP corresponding to the abnormal image was close to uniform distribution. The MSP calculation is as shown in formula (3): Among them, z i represents the output of the i-th neuron of the fully connected layer FC-K, and K represents the total number of landmark categories to be identified; Formula (3) normalizes the vector z containing K real numbers to a probability distribution containing K numbers proportional to the input exponent, and outputs the maximum Softmax probability MSP, that is, P msp ; 3) Data preprocessing By introducing an abnormal auxiliary training set The training is performed, so the dataset contains three parts that are unrelated to each other; the first part is the landmark recognition distribution training set D in , the second part is the external distribution auxiliary training set The third part is the test set for model evaluation, including internal distribution and external distribution test sets. and Data preprocessing includes the following steps: (1) For landmark recognition tasks in different scenarios, we collect datasets based on landmark buildings or the detailed locations of each building as categories, and then split them into training sets according to the ratio of 8:1:1 for each category. Validation set and test set Abnormal auxiliary training set Use the public dataset Places365; (2) The test set includes the internal distribution test set and out-of-distribution test set in Using public datasets ImageNet's landmark class, which is There is no intersection; (3) During the model training phase, data enhancement is performed on the landmark images, including random cropping and scaling of the original images. Flip and normalize. The RGB channels of the landmark image are normalized using the mean u of (0.485, 0.456, 0.406) and the standard deviation σ of (0.229, 0.224, 0.225). The calculation is shown in formula (4): Among them, x is the input tensor data of the landmark image, and x′ is the normalized data; 4) Training and testing 4.1) Training the landmark recognition base network Based on the improved base network, 200 rounds of training were performed on the landmark recognition training and validation sets. The loss function used cross entropy and the SGD optimizer with Nesterov momentum was used. The initial learning rate was 0.1, the momentum was set to 0.9, the weight decay was 0.0005, and the LambdaLR learning rate adjustment strategy was combined to train and save the model parameters. 4.2) Modify the out-of-distribution detection loss function Based on the landmark recognition base network trained in step 4.1), the loss function of formula (2) is used to introduce the auxiliary training set Fine-tune the network so that it can learn to distinguish whether the input is a landmark building or an abnormal image within the task; the network is trained for 20 rounds in total; the optimizer uses the SGD optimizer with Nesterov momentum, with an initial learning rate of 0.001, a momentum of 0.9, and a weight decay of 0.0005, combined with the LambdaLR learning rate adjustment strategy, to train and save the model parameters; 4.3) Model testing Based on the network model trained in step 4.1) and step 4.2), the open set landmark recognition performance of the network is tested, including the following steps: (I) Load the network model and read the landmark test set within the task and out-of-distribution test set (ii) When the landmark image to be tested obeys the distribution within the task, its landmark information y can be obtained. In this case, the opposite of MSP is used as the OOD score S in , so the OOD score is less than 0; the correctly predicted OOD score is recorded as The OOD score of the wrong prediction is recorded as The recognition accuracy is obtained by dividing the number of correctly classified samples by the total number of test samples; (III) When the image to be tested is not a landmark category within the task, the network will make its corresponding MSP close to uniform distribution; the out-of-distribution image is taken as the positive class, the landmark sample within the task is taken as the negative class, and the opposite number of MSP is used as the OOD score S out ; Use AUROC, AUPR, and FPR95 indicators to evaluate the network's out-of-distribution detection capabilities; (iv) Based on the OOD scores counted in steps (ii) and (iii), the OOD score corresponding to FPR95 is set as the threshold δ. Based on δ, it is judged whether the image to be tested obeys the task internal distribution. If so, its landmark category is predicted, otherwise the prediction is rejected. In the landmark recognition task whose internal distribution training dataset is BJUT-118, the threshold δ is set to -0.98.

Images