Visual address identification method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a visual address identification method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the steps of inputting a query image into a pre-trained feature extraction model to perform local feature extraction to obtain at least one query local feature, obtaining a category weight of the query local feature under each clustering category, and obtaining a representation matrix based on the category weight and the query local feature, and obtaining a query global descriptor according to the at least one representation matrix, calculating the similarity between the query global descriptor and the address global descriptor of each address image, selecting a target recognition image based on the similarity, and taking an address identifier of the target recognition image as a target address of the query image. When the query local features are clustered, only category weights corresponding to different clustering categories are used, and a residual error does not need to be calculated with a specific clustering center, so that the calculation efficiency of an address global descriptor is improved, and the recognition efficiency of visual address recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a visual address recognition method, apparatus, device and storage medium. Background Art

[0002] Visual address recognition is to compare the query image taken at the current address with the images in the historical database to determine whether the address exists in the database. If it does, the matching database image is given. Since the images in the database are usually labeled with geographic location, the rough positioning of the current address can be completed in a purely visual way.

[0003] In the related art, visual address recognition is usually achieved through image retrieval methods. First, the global features of all images are obtained, and then the nearest neighbor search is used based on these global features to obtain the database image that best matches the query address image. In the related art, the global features are usually obtained by calculating the first-order statistical information of the residuals of local features and cluster centers. The global features obtained in this way have a higher dimension, so an additional dimensionality reduction process is required, which will result in additional running time and memory consumption. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a visual address recognition method, device, equipment and storage medium to improve the recognition efficiency of visual address recognition.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a visual address recognition method, comprising:

[0006] Acquire a query image and a plurality of address images, wherein the address images include corresponding address identifiers;

[0007] Inputting the query image into a pre-trained feature extraction model to extract local features, and obtaining at least one query local feature;

[0008] Obtaining a category weight of the query local feature in each cluster category, and calculating a representation matrix corresponding to the query local feature based on the category weight and the query local feature;

[0009] Obtaining a query global descriptor corresponding to the query image according to at least one of the representation matrices;

[0010] The similarity between the query global descriptor and the address global descriptor of each address image is calculated, and based on the similarity, one of the address images is selected as the target recognition image of the query image, and the address identifier of the target recognition image is used as the target address of the query image.

[0011] In some embodiments, obtaining the category weight of the query local feature in each cluster category includes:

[0012] Inputting the query local feature into a linear layer for feature extraction to obtain a linear local feature;

[0013] The linear local features are input into the softmax layer for feature extraction to obtain the category weight corresponding to each cluster category.

[0014] In some embodiments, the clustering category includes a first preset category and a second preset category, and the calculating based on the category weight and the query local feature to obtain the representation matrix corresponding to the query local feature includes:

[0015] Obtaining the category weight corresponding to the second preset category as the target category weight;

[0016] The representation matrix is ​​calculated based on the target category weight and the query local feature.

[0017] In some embodiments, the calculating the representation matrix according to the target category weight and the query local feature includes:

[0018] Multiplying the target category weight by each of the query local features respectively to obtain a classification feature;

[0019] All the classification features are accumulated to obtain the representation matrix corresponding to the second preset category.

[0020] In some embodiments, obtaining a query global descriptor corresponding to the query image according to at least one of the representation matrices includes:

[0021] Obtaining a total representation matrix corresponding to the query local feature according to the representation matrix;

[0022] Normalizing at least one of the total representation matrices to obtain a normalized representation matrix;

[0023] The normalized representation matrices are concatenated to obtain a global representation matrix, and the global representation matrix is ​​globally normalized to obtain the query global descriptor.

[0024] In some embodiments, the linear local features are input into a softmax layer for feature extraction to obtain the category weight corresponding to each cluster category, including:

[0025] Inputting the linear local features into a softmax layer for feature extraction to obtain an intermediate weight corresponding to each cluster category;

[0026] The intermediate weights are accumulated to obtain the total weight, and the ratio of the intermediate weight to the total weight is calculated to obtain the category weight.

[0027] In some embodiments, the number of the first preset categories is greater than one, and the number of the second preset categories is one.

[0028] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a visual address recognition device, comprising:

[0029] Image acquisition module: used to acquire a query image and a plurality of address images, wherein the address images include corresponding address identifiers;

[0030] Local feature extraction module: used for inputting the query image into a pre-trained feature extraction model to extract local features, and obtaining at least one query local feature;

[0031] A representation matrix calculation module is used to obtain the category weight of the query local feature in each cluster category, and calculate the representation matrix corresponding to the query local feature based on the category weight and the query local feature;

[0032] A global feature extraction module: used for obtaining a query global descriptor corresponding to the query image according to at least one of the representation matrices;

[0033] Query module: used to calculate the similarity between the query global descriptor and the address global descriptor of each address image, select one of the address images as the target recognition image of the query image based on the similarity, and use the address identifier of the target recognition image as the target address of the query image.

[0034] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0035] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0036] The visual address recognition method, apparatus, device and storage medium proposed in the embodiments of the present application obtain a query image and multiple address images, input the query image into a pre-trained feature extraction model to extract local features to obtain at least one query local feature, obtain the category weight of the query local feature under each clustering category, and calculate the representation matrix corresponding to the query local feature based on the category weight and the query local feature, obtain the query global descriptor corresponding to the query image according to at least one representation matrix, calculate the similarity between the query global descriptor and the address global descriptor of each address image, select an address image as the target recognition image of the query image based on the similarity, and use the address identifier of the target recognition image as the target address of the query image. When clustering the query local features, the embodiments of the present application only use the category weights corresponding to different clustering categories, and do not need to calculate the residual with the specific cluster center, thereby improving the calculation efficiency of the address global descriptor, and then improving the recognition efficiency of visual address recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the visual address recognition method provided in an embodiment of the present application.

[0038] Figure 2 This is a flowchart of obtaining the category weight of the query local feature in each cluster category provided by an embodiment of the present application.

[0039] Figure 3 It is the process of using aggregation units to obtain the query global descriptor based on the query local features.

[0040] Figure 4 It is a flowchart of calculating a representation matrix corresponding to a query local feature based on category weights and query local features provided in an embodiment of the present application.

[0041] Figure 5 It is a flowchart of obtaining a query global descriptor corresponding to a query image according to at least one representation matrix provided by an embodiment of the present application.

[0042] Figure 6 It is an overall flow chart of the visual address recognition method provided in the embodiment of the present application.

[0043] Figure 7 It is a schematic diagram of the recognition results of the visual address recognition method provided in an embodiment of the present application.

[0044] Figure 8 This is a structural block diagram of a visual address recognition device provided in yet another embodiment of the present application.

[0045] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0049] First, some nouns involved in this application are analyzed:

[0050] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0051] Visual address recognition refers to comparing the query image taken at the current address with the images in the historical database to determine whether the address exists in the database. If it does, a matching database image is given. Since the images in the database usually have geographic location tags, the rough positioning of the current address can be completed in a purely visual way. Simply put, visual address recognition can be understood as a content-based image retrieval task, which is widely used in autonomous driving, augmented reality and smart transportation.

[0052] In the related art, visual address recognition is usually achieved through image retrieval methods. First, the global features of all images (including the query image and all images in the database) are obtained, and then the nearest neighbor search is used based on these global features to obtain the database image that best matches the query address image. In the related art, the global features are obtained by calculating the first-order statistical information of the residuals of the local features and the cluster centers. On the one hand, the data dimension of the global features obtained in this way is relatively high, and an additional dimensionality reduction process is usually required, resulting in additional running time and memory consumption. On the other hand, due to changes in weather, seasons, lighting and other conditions, as well as changes in shooting perspectives, images from the same address may have great visual differences, or images from different addresses may be visually similar. The recognition accuracy of the global features obtained in this way is low.

[0053] Based on this, the embodiments of the present application provide a visual address recognition method, apparatus, device and storage medium. When clustering the local features of the query, only the category weights corresponding to different clustering categories are used, and there is no need to calculate the residual with the specific cluster center, thereby improving the calculation efficiency of the global address descriptor and further improving the recognition efficiency of visual address recognition.

[0054] The embodiments of the present application provide a visual address recognition method, apparatus, device and storage medium, which are specifically illustrated by the following embodiments. First, the visual address recognition method in the embodiments of the present application is described.

[0055] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0056] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0057] The visual address recognition method provided in the embodiment of the present application relates to the field of image processing technology. The visual address recognition method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be a computer program running in a terminal or a server side. For example, a computer program can be a native program or a software module in an operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports visual address recognition, or it can be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server via a network. The visual address recognition method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0058] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the visual address recognition method of this embodiment to provide related services and enhance the driving experience. The server may be an independent server, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, and each service node in the blockchain system forms a point-to-point (P2P, Peer To Peer, P2P) network, and the P2P protocol is an application layer protocol running on the Transmission Control Protocol (Transmission Control Protocol, TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (Universal Serial Bus, USB) or network communication connection methods, which are not limited in this embodiment.

[0059] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0060] The following describes the visual address recognition method in an embodiment of the present application.

[0061] Figure 1 is an optional flow chart of the visual address recognition method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps 110 to 150. It can also be understood that this embodiment Figure 1 The order of step 110 to step 150 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0062] Step 110: Acquire a query image and a plurality of address images.

[0063] In one embodiment, the query image is an image that needs to be visually recognized, and the address image may be an image with an address identifier stored in a database. For example, in an application scenario, after a query image is captured, a visual address recognition operation is performed to match it with the address images in the database, determine which address image is closest to the query image, and use the address identifier of the address image as the recognized address corresponding to the query image.

[0064] Step 120: Input the query image into a pre-trained feature extraction model to extract local features, and obtain at least one query local feature.

[0065] In one embodiment, a Transformer model is pre-trained as a feature extraction model, such as a visual basic model DINOv2. Then, the query image is input into the pre-trained feature extraction model to extract local features, and at least one query local feature is obtained. The number of query local features here is the dimension, which is denoted as D, and the dimension can be set according to actual needs.

[0066] The feature extraction model includes multiple cascaded encoder modules, each of which includes a multi-head self-attention layer and a feedforward neural network layer. Inside the encoder module, the input data is first processed by the multi-head self-attention layer so that the encoder module can understand the input data from a global perspective. Next, the data output by the multi-head self-attention layer is passed to the feedforward neural network layer for nonlinear transformation through a residual connection to enhance the expressive power of the encoder module. After the output data of the last encoder module is segmented, it is used as the output of the feature extraction model to obtain at least one query local feature.

[0067] Step 130: Obtain the category weight of the query local feature in each cluster category, and calculate the representation matrix corresponding to the query local feature based on the category weight and the query local feature.

[0068] In one embodiment, a specific clustering category can be set according to the actual visual recognition task. In the embodiment of the present application, the clustering category includes a first preset category and a second preset category, wherein the first preset category can be regarded as a "useless" category, which is used to correspond to "useless" objects in the address recognition process, such as the sky, the ground, dynamic objects, etc. The second preset category is a "useful" category, which belongs to the category that can clearly indicate the address information in the visual recognition process, such as traffic signs, residential buildings, etc.

[0069] In one embodiment, referring to Figure 2 , Figure 2 This is a flowchart of obtaining the category weight of the query local feature in each cluster category provided by an embodiment of the present application, which specifically includes the following steps:

[0070] Step 210: Input the query local feature into the linear layer for feature extraction to obtain a linear local feature.

[0071] In one embodiment, referring to Figure 3 , Figure 3 It is the process of using the aggregation unit to obtain the query global descriptor based on the query local features. The aggregation unit includes a linear layer, which is a 1*1*D*K convolution layer with a convolution kernel of 1*1, an input dimension of D, and an output dimension of K, where K is the number of clustering categories. That is, the linear layer includes K convolution kernels, each of which is a D-dimensional vector, and the kth convolution kernel can be expressed as w k In this embodiment, the linear local feature is extracted by the linear layer to obtain the linear local feature. The dimension of the linear local feature is K, and the i-th query local feature x i The dimension is D.

[0072] At this time, for each convolution kernel wk , that is, for the kth cluster category, the convolution operation is calculated as is a dot product operation that produces a scalar result. In addition, each convolution kernel has a corresponding bias b k Therefore, for the kth convolution kernel, the scalar output plus the corresponding bias becomes At this time, for all K convolution kernels, a K-dimensional linear local feature can be obtained, in which each element is the dot product of the corresponding convolution kernel and the query local feature plus the bias.

[0073] Among them, the kth element in the linear local feature is expressed as:

[0074]

[0075] Step 220: Input the linear local features into the softmax layer for feature extraction to obtain the category weight corresponding to each cluster category.

[0076] In one embodiment, the K-dimensional linear local features are input into a softmax layer for feature extraction, and the linear local features are normalized using the softmax layer so that the sum of the elements is 1, thereby obtaining the category weight of each cluster category.

[0077] The specific process is described as follows: input the linear local features into the softmax layer for feature extraction, obtain the intermediate weight corresponding to each clustering category, accumulate the intermediate weights to obtain the total weight, calculate the ratio of the intermediate weight to the total weight, and obtain the category weight.

[0078] Among them, the category weight a of the kth cluster category k (x i ) is expressed as:

[0079]

[0080] in, represents the intermediate weight of the k-th cluster category, represents the total weight, k' represents any integer value between 1 and K, and the category weight of the kth cluster category represents the category weight of the kth cluster category given the input x i Next, predict the probability that the input belongs to cluster category k.

[0081] Through the above steps, the linear layer combined with the softmax layer can convert the D-dimensional query local features into category weights corresponding to K cluster categories. These category weights represent the probability that the query local features belong to each cluster category.

[0082] Next, refer to Figure 4 , Figure 4The flowchart of the embodiment of the present application is to calculate the representation matrix corresponding to the query local feature based on the category weight and the query local feature, which specifically includes the following steps:

[0083] Step 410: Obtain the category weight corresponding to the second preset category as the target category weight.

[0084] In one embodiment, each clustering category has been calculated to the corresponding category weight. At this time, the category weight corresponding to the second preset category representing the "useful" category is selected as the target category weight to participate in the subsequent calculation process. That is to say, the "useless" category participates in the weight allocation process, but is removed in the subsequent aggregation process and will not be directly used in the final output. The advantage of this is that it improves the robustness of the processing process, reduces the interference of useless objects on the final recognition results, and improves the accuracy of recognition.

[0085] Step 420: Calculate a representation matrix based on the target category weight and the query local features.

[0086] In one embodiment, after selecting the target category weight, the embodiment of the present application does not calculate the residual by subtracting the corresponding cluster center from the query local feature. This is because when the training data distribution is different, the learned cluster center is also different, and the difference in cluster centers can cause the similarity measurement results of the global descriptors calculated accordingly to be completely different. In other words, when there is a distribution inconsistency between the training data and the reasoning data, the cluster center learned on the training set is usually not suitable for reasoning, which will lead to a significant performance degradation. Therefore, the embodiment of the present application improves the cross-data domain robustness of the aggregation method by eliminating the cluster center, thereby improving the accuracy of the recognition result.

[0087] In one embodiment, without using dimensionality reduction technology, the representation matrix is ​​calculated by directly multiplying the target category weight and the query local feature. The specific process is: multiplying the target category weight with each query local feature to obtain the classification feature, and then accumulating all the classification features to obtain the representation matrix corresponding to the second preset category. This process is expressed as:

[0088]

[0089] Among them, x i,j Represents the i-th query local feature x i The jth component of , N represents the total number of local features of the query, V k,j The jth component of the representation matrix representing the kth cluster category (the cluster category at this time is one of the second preset categories), a k (x i ) represents the category weight of the k-th cluster category.

[0090] In one embodiment, a small number of second preset categories are set to reduce the data dimension, so that no additional dimensionality reduction operation is required. For example, the number of second preset categories is one, and the number of first preset categories is more than one, for example, two. At this time, since the number of second preset categories is 1, the aggregation process is also called single cluster aggregation, which means that all local features of objects related to address recognition are assigned to the same cluster category, and the recognition accuracy becomes higher. At the same time, compared with the global pooling method, the dimension of the query global descriptor generated by the embodiment of the present application is very low, and the recognition performance is improved.

[0091] Step 140: Obtain a query global descriptor corresponding to the query image according to at least one representation matrix.

[0092] In one embodiment, referring to Figure 5 , Figure 5 The flowchart of obtaining a query global descriptor corresponding to a query image according to at least one representation matrix provided by an embodiment of the present application specifically includes the following steps:

[0093] Step 510: Obtain a total representation matrix corresponding to the query local feature according to the representation matrix.

[0094] Step 520: Normalize at least one total representation matrix to obtain a normalized representation matrix.

[0095] Step 530: concatenate the normalized representation matrices to obtain a global representation matrix, and globally normalize the global representation matrix to obtain a query global descriptor.

[0096] In one embodiment, if the number of the second preset categories is 1, k in the above characterization matrix is ​​1. At this time, the j components are concatenated to obtain the total characterization matrix V corresponding to the i-th query local feature, and then the total characterization matrices corresponding to all query local features are normalized within the class to obtain the normalized characterization matrix corresponding to the total characterization matrix of each query local feature. The normalization process here is to divide each total characterization matrix by the sum of the total characterization matrices.

[0097] Next, all normalized representation matrices are concatenated to obtain the corresponding global representation matrix. At this time, the global representation matrix is ​​globally normalized with the L2 norm to obtain the query global descriptor, which is expressed as:

[0098]

[0099] Among them, H′ represents the query global descriptor, H represents the global representation matrix, ||H‖2 represents a diagonal matrix, and the elements on the diagonal of the diagonal matrix are the L2 norm of each row of the global representation matrix H.

[0100] In the above process, removing the cluster center makes the cross-domain robustness of the calculated query global descriptor better, and the number of parameters is greatly reduced.

[0101] Step 150: Calculate the similarity between the query global descriptor and the address global descriptor of each address image, select an address image as the target recognition image of the query image based on the similarity, and use the address identifier of the target recognition image as the target address of the query image.

[0102] In one embodiment, the address global descriptor of each address image is pre-calculated in accordance with the above-mentioned manner of querying the global descriptor of the query image, wherein the address global descriptor may be stored together with the address image in a database.

[0103] Next, based on the method of calculating the Euclidean distance between vectors, the similarity between the query global descriptor and each address global descriptor is calculated, and the address image corresponding to the largest similarity is selected as the target recognition image of the query image. Since each address image contains a corresponding address identifier, the address identifier of the target recognition image is used as the target address of the query image, and the target address is the recognition result of visual address recognition of the query image.

[0104] In one embodiment, referring to Figure 6 , Figure 6 This is an overall flow chart of the visual address recognition method provided by an embodiment of the present application. First, a query image and multiple address images are obtained, and the following global descriptor calculation process is performed for each image obtained. The query image is used as an example for explanation below. The query image is input into a pre-trained feature extraction model for local feature extraction to obtain at least one query local feature, and then the category weight of the query local feature under each clustering category is obtained. Then the category weight corresponding to the second preset category is obtained as the target category weight, and a characterization matrix is ​​obtained by aggregation based on the target category weight and the query local feature. Then, the query global descriptor corresponding to the query image is obtained based on at least one characterization matrix. In this way, the address global descriptor of the address image obtained in advance is stored in the database. Finally, a search is performed in the database to calculate the similarity between the query global descriptor and the address global descriptor of each address image, and an address image is selected as the target recognition image of the query image based on the similarity, and the address identifier of the target recognition image is used as the target address of the query image.

[0105] Reference Figure 7 , Figure 7 4 different query images in the figure can retrieve relevant address images of different viewing angles, different weather or different seasons in the database according to the visual address recognition method of the present application, thereby obtaining the correct target address.

[0106] Next, the experimental verification process of the visual address recognition method of the embodiment of the present application is described. In the verification process, some models in the relevant technology are selected as references, such as the NetVLAD model, the SFRS (Spatial Feature Relational Networks) model, the TransVPR (Transformer Visual Place Recognition) model, the CosPlace model, the MixVPR (Mixture of Visual Place Recognition) model, the EigenPlaces model, the SelaVPR model, the CricaVPR model, the SALAD (Self-Aware Learning for Autonomous Driving) model, etc. And M1 represents the NetVLAD model, M2 represents the SFRS model, M3 represents the TransVPR model, M4 represents the CosPlace model, M5 represents the MixVPR model, M6 represents the EigenPlaces model, M7 represents the SelaVPR model, M8 represents the CricaVPR model, M9 represents the SALAD model, and M10 represents the visual address recognition method provided in the embodiment of the present application.

[0107] In one embodiment, the NetVLAD model is first used as an example to verify the recognition performance of different data set distributions. Referring to Table 1 below, the data sets are the Pitts30k data set and the MSLS (Multi-column Convolutional Neural Networks for Semantic Segmentation on CityScapes) data set, wherein the Pitts30k data set is a large-scale urban street scene understanding data set, containing high-quality stereo image pairs and corresponding dense pixel-level annotations, and the MSLS data set is a semantic segmentation data set for multi-column convolutional neural networks. Compared with the NetVLAD model, when the distribution of the training set and the test set is consistent, the performance gap between the two is basically the same. When the distribution of the training set and the test set is inconsistent, the embodiment of the present application removes the cluster center during the aggregation process for this situation, and the performance advantage of the embodiment of the present application is more obvious. Among them, the performance indicators are: R@1 (recall@10), R@5, and R@10, where 1, 5, and 10 are used to indicate the probability of the recognition result being in the first place, the top five places, and the top ten places in the ranking.

[0108]

[0109]

[0110] Table 1 Recognition performance examples of the visual address recognition method of the present application embodiment

[0111] In one embodiment, referring to Table 2, the recognition performance of other different models is compared with the embodiment of the present application, and the Nordland dataset and SPED (Spatial Pyramid for Efficient Deep Learning) dataset are added, wherein the Nordland dataset is a large-scale aerial image dataset, which is mainly used for tasks such as remote sensing image analysis, object classification and target detection, and the SPED dataset is a dataset used to evaluate the performance of image recognition and retrieval algorithms. As can be seen from Table 2, the recognition performance of the visual address recognition method of the embodiment of the present application is better than other related models, especially in datasets with drastic condition changes such as the Nordland dataset and the SPED dataset.

[0112]

[0113] Table 2 Another example of recognition performance of the visual address recognition method of the embodiment of the present application

[0114] In one embodiment, referring to Table 3, the recognition performance of the single clustering aggregation in the embodiment of the present application is compared with the generalized mean pooling and class token in the related art. In Table 2, according to the same dimension of the global descriptor, the performance of the visual address recognition method in the embodiment of the present application has obvious advantages.

[0115]

[0116]

[0117] Table 3 Another example of recognition performance of the visual address recognition method of the embodiment of the present application

[0118] The technical solution provided by the embodiment of the present application obtains a query image and multiple address images, inputs the query image into a pre-trained feature extraction model to extract local features to obtain at least one query local feature, obtains the category weight of the query local feature under each clustering category, and calculates the representation matrix corresponding to the query local feature based on the category weight and the query local feature, obtains the query global descriptor corresponding to the query image according to at least one representation matrix, calculates the similarity between the query global descriptor and the address global descriptor of each address image, selects an address image as the target recognition image of the query image based on the similarity, and uses the address identifier of the target recognition image as the target address of the query image. When clustering the query local features, the embodiment of the present application only uses the category weights corresponding to different clustering categories, does not need to calculate the residual with the specific cluster center, and reduces the number of parameters in the aggregation process, improves the calculation efficiency of the address global descriptor, and thus improves the recognition efficiency of visual address recognition. And the cross-domain robustness is improved through single cluster aggregation, which can adapt to recognition scenarios under extreme conditions and perspective changes.

[0119] The present application also provides a visual address recognition device that can implement the above-mentioned visual address recognition method. Figure 8 , the device comprises:

[0120] Image acquisition module 810: used to acquire a query image and multiple address images, where the address images include corresponding address identifiers.

[0121] The local feature extraction module 820 is used to input the query image into a pre-trained feature extraction model to perform local feature extraction to obtain at least one query local feature.

[0122] The representation matrix calculation module 830 is used to obtain the category weight of the query local feature in each cluster category, and calculate the representation matrix corresponding to the query local feature based on the category weight and the query local feature.

[0123] The global feature extraction module 840 is configured to obtain a query global descriptor corresponding to the query image according to at least one representation matrix.

[0124] Query module 850: used to calculate the similarity between the query global descriptor and the address global descriptor of each address image, select an address image as the target recognition image of the query image based on the similarity, and use the address identifier of the target recognition image as the target address of the query image.

[0125] The specific implementation of the visual address recognition device of this embodiment is basically the same as the specific implementation of the above-mentioned visual address recognition method, and will not be repeated here.

[0126] The present application also provides an electronic device, including:

[0127] at least one memory;

[0128] at least one processor;

[0129] at least one program;

[0130] The program is stored in the memory, and the processor executes the at least one program to implement the visual address recognition method of the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0131] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0132] The processor 901 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0133] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the visual address recognition method of the embodiment of this application;

[0134] The input / output interface 903 is used to realize information input and output; the communication interface 904 is used to realize communication interaction between the present device and other devices, and communication can be realized through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 905 transmits information between the various components of the device (such as the processor 901, the memory 902, the input / output interface 903 and the communication interface 904); wherein the processor 901, the memory 902, the input / output interface 903 and the communication interface 904 realize communication connection with each other within the device through the bus 905.

[0135] An embodiment of the present application also provides a storage medium, which is a storage medium that stores a computer program. When the computer program is executed by a processor, the above-mentioned visual address recognition method is implemented.

[0136] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The visual address recognition method, device, equipment and storage medium proposed in the embodiment of the present application obtains a query image and multiple address images, inputs the query image into a pre-trained feature extraction model to extract local features to obtain at least one query local feature, obtains the category weight of the query local feature under each cluster category, and calculates the characterization matrix corresponding to the query local feature based on the category weight and the query local feature, obtains the query global descriptor corresponding to the query image according to at least one characterization matrix, calculates the similarity between the query global descriptor and the address global descriptor of each address image, selects an address image as the target recognition image of the query image based on the similarity, and uses the address identifier of the target recognition image as the target address of the query image. When clustering the query local features, the embodiment of the present application only uses the category weights corresponding to different cluster categories, does not need to calculate the residual with the specific cluster center, and reduces the number of parameters in the aggregation process, improves the calculation efficiency of the address global descriptor, and thus improves the recognition efficiency of visual address recognition. And the cross-domain robustness is improved through single cluster aggregation, which can adapt to recognition scenarios under extreme conditions and perspective changes.

[0138] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0139] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0140] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0142] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0143] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0144] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store programs.

[0148] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A visual address recognition method, characterized in that: include: Acquire a query image and a plurality of address images, wherein the address images include corresponding address identifiers; Inputting the query image into a pre-trained feature extraction model to extract local features, and obtaining at least one query local feature; Obtaining a category weight of the query local feature in each cluster category, and calculating a representation matrix corresponding to the query local feature based on the category weight and the query local feature; Obtaining a query global descriptor corresponding to the query image according to at least one of the representation matrices; The similarity between the query global descriptor and the address global descriptor of each address image is calculated, and based on the similarity, one of the address images is selected as the target recognition image of the query image, and the address identifier of the target recognition image is used as the target address of the query image.

2. The visual address recognition method according to claim 1, characterized in that: The obtaining of the category weight of the query local feature in each cluster category includes: Inputting the query local feature into a linear layer for feature extraction to obtain a linear local feature; The linear local features are input into the softmax layer for feature extraction to obtain the category weight corresponding to each cluster category.

3. The visual address recognition method according to claim 1, characterized in that: The clustering category includes a first preset category and a second preset category, and the characterization matrix corresponding to the query local feature is calculated based on the category weight and the query local feature, including: Obtaining the category weight corresponding to the second preset category as the target category weight; The representation matrix is ​​calculated based on the target category weight and the query local feature.

4. The visual address recognition method according to claim 3, characterized in that: The calculating and obtaining the representation matrix according to the target category weight and the query local feature includes: Multiplying the target category weight by each of the query local features respectively to obtain a classification feature; All the classification features are accumulated to obtain the representation matrix corresponding to the second preset category.

5. The visual address recognition method according to claim 1, characterized in that: The obtaining a query global descriptor corresponding to the query image according to at least one of the representation matrices comprises: Obtaining a total representation matrix corresponding to the query local feature according to the representation matrix; Normalizing at least one of the total representation matrices to obtain a normalized representation matrix; The normalized representation matrices are concatenated to obtain a global representation matrix, and the global representation matrix is ​​globally normalized to obtain the query global descriptor.

6. The visual address recognition method according to claim 2, characterized in that: Inputting the linear local features into the softmax layer for feature extraction to obtain the category weight corresponding to each cluster category includes: Inputting the linear local features into a softmax layer for feature extraction to obtain an intermediate weight corresponding to each cluster category; The intermediate weights are accumulated to obtain the total weight, and the ratio of the intermediate weight to the total weight is calculated to obtain the category weight.

7. The visual address recognition method according to claim 3, characterized in that: The number of the first preset categories is greater than one, and the number of the second preset categories is one.

8. A visual address recognition device, characterized in that: include: Image acquisition module: used to acquire a query image and a plurality of address images, wherein the address images include corresponding address identifiers; Local feature extraction module: used for inputting the query image into a pre-trained feature extraction model to extract local features, and obtaining at least one query local feature; A representation matrix calculation module is used to obtain the category weight of the query local feature in each cluster category, and calculate the representation matrix corresponding to the query local feature based on the category weight and the query local feature; A global feature extraction module: used for obtaining a query global descriptor corresponding to the query image according to at least one of the representation matrices; Query module: used to calculate the similarity between the query global descriptor and the address global descriptor of each address image, select one of the address images as the target recognition image of the query image based on the similarity, and use the address identifier of the target recognition image as the target address of the query image.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the visual address recognition method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the visual address recognition method according to any one of claims 1 to 7 is implemented.

Images