A retrieval method based on semantic concept extraction

Abstract

The application discloses a retrieval method based on semantic concept extraction, which comprises: obtaining query features; performing similarity calculation on the query features and candidate features in a candidate feature database to obtain a similarity ranking result; and returning the similarity ranking result as a retrieval result. The candidate features are obtained by performing feature extraction on images in an image database: extracting basic semantic elements from the images; performing semantic concept cutting to obtain semantic concept features; and performing L2 regularization, PCA whitening and a further round of L2 regularization on the semantic concept features to obtain the candidate features. The semantic concept features extracted by the application can cover instance levels and image levels, so that the extracted features can describe global and local image semantic information, thereby unifying image retrieval and instance retrieval in a set of frameworks, and thus the application can be used for both instance retrieval tasks and image retrieval tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to computer vision technology and information retrieval technology, in particular to a retrieval method based on semantic concept extraction, which can be applied to search engines in Internet companies, e-commerce and security monitoring scenes, etc. BACKGROUND

[0002] For a long time, many researchers have regarded image retrieval and instance retrieval as two different problems to solve. Looking at the main ideas of many current methods, image retrieval extracts image-level features for each image, while instance retrieval extracts instance-level features for each instance in the image. In image-level features, the features of local instances may be covered by background or main body instances; in instance-level features, the center of expression of the features is each visual instance in the image. An image usually contains multiple visual instances, and similar images mean that some or several visual instances between images have similarities. It can be seen that the two problems are related. However, due to the inconsistency of the target feature granularity of image retrieval and instance retrieval, the current solution is still to solve the two problems separately.

[0003] In the feature map generated by the deep convolutional neural network for an image, due to the translational invariance of convolution and pooling operations, each pixel on the feature map corresponds to a spatial region in the original image. Therefore, the feature vector of a certain pixel position on the feature map expresses the semantic information in the corresponding spatial range in the original image. Image retrieval methods usually use global pooling or feature encoding methods to aggregate all feature vectors in a feature map to obtain image-level features of the image; while instance retrieval methods respectively aggregate feature vectors in a local range in a feature map according to the instance positioning box to generate features for the instance appearing in the local position. However, the high-level semantic concepts of an image may not be continuous in space, and they may be expressed by the semantics of several scattered parts in the image. Therefore, the current image-level features and instance-level features cannot express the high-level semantic concepts in the image, and on the contrary, these semantic concept features can represent image-level and instance-level features.

[0004] In the feature map of the convolutional neural network, the features of different spatial positions contain specific semantic information, and the spatial features that collectively express a certain semantic concept should be associated with each other, and if these associated features can be aggregated, the semantic concept features in the image can be extracted. However, not all regions in the image contribute to the aggregation of semantic concept features, and the interference of non-main part information needs to be excluded in the process of aggregating semantic features. In addition, among all the semantic concepts extracted from an image, a part of the non-main semantic concepts may not be important in the data set, and these non-main semantic concept features stored in the database will occupy a large amount of space and consume additional comparison, thereby reducing the retrieval speed. SUMMARY

[0005] In view of the problems in the prior art, the purpose of the present application is to provide a retrieval method based on semantic concept extraction, which can realize image retrieval and instance retrieval in a unified framework.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is:

[0007] A retrieval method based on semantic concept extraction, the method comprising

[0008] Obtaining query features;

[0009] Calculating the similarity of the query features and the candidate features in the candidate feature database to obtain a similarity ranking result;

[0010] Returning the similarity ranking result as a retrieval result;

[0011] The candidate features in the candidate feature database are obtained by feature extraction from the images in the image database; the feature extraction method is as follows:

[0012] Extracting basic semantic elements from the image;

[0013] Performing semantic concept segmentation:

[0014] Constructing an undirected graph G for the basic semantic elements image The weight of the edge between two nodes in the undirected graph G image is defined as:

[0015]

[0016] Wherein, cos(v i ,v j ) is the cosine similarity between v i and v j ;

[0017] Cut out the connected components in the undirected graph, each connected component contains similar basic semantic elements; calculate the average features of the nodes in each connected component to aggregate the features of the basic semantic elements, and obtain the semantic concept features;

[0018] The semantic concept features are processed by L2 regularization, PCA whitening and another round of L2 regularization to obtain candidate features.

[0019] The basic semantic elements are extracted from the image as follows:

[0020] The image is input into the convolutional neural network to obtain an output feature map X of HxWxC dimensions;

[0021] The average activation map of HxW dimensions is obtained by averaging the feature map X in the channel dimension C

[0022] The peak point set is obtained by using an N*N window on the average activation map

[0023] For each peak point on the average activation map , the contribution probability formula is used to calculate the contribution probability of each peak point from back to front layer by layer until the input image, and the contribution probability map M of the same size as the original image is obtained.

[0024] The spatial position information of the basic semantic elements is estimated in the form of a rectangular frame on the contribution probability map M.

[0025] The contribution probability formula is as follows:

[0026]

[0027] Where P(I i,j ) is the contribution probability of a pixel position I i,j in the input feature map to a pixel position O p,q in the output feature map; the conditional probability P(I i,j |O p,q ) is defined as:

[0028]

[0029] Where, is the bottom-up activation value of I at spatial position (i,j) calculated by forward propagation, and Z p,q is a regularization term to ensure that p,q P(I i,j |O p,q ) = 1.

[0030] Before estimating the spatial position of the basic semantic elements, the contribution probability map M is processed as follows:​

[0031] The values on the contribution probability map M are normalized to have a range of [0, 1];

[0032] A threshold τ is set a Pixels that do not contribute to the peak point are filtered out;

[0033] The active region on the contribution probability map M is estimated to be an elliptical shape, and the parameters of the ellipse are modeled by the pixel positions in the contribution probability map M that are greater than τ a The second moment of the image is obtained as:

[0034]

[0035] The rectangular frame corresponding to the basic semantic element is obtained by the circumscribed rectangle of the ellipse, and the feature of the basic semantic element is obtained by average pooling of each basic semantic element corresponding rectangular frame on the feature map.

[0036] The undirected graph G image is represented as:

[0037] G image = <V image , E image >,

[0038] Where V image is the node of the undirected graph, which is composed of basic semantic element features

[0039] E image is a set of edges, represented as

[0040] After the semantic concept cutting to obtain the semantic concept feature, non-semantic concept removal is performed; specifically as follows:

[0041] First, the data set level undirected graph G dataset = <V dataset , E dataset > is established, where V dataset is the node in the data set level undirected graph, which is the semantic concept feature E dataset is a set of edges, represented as The edge weight in G dataset is defined as: if the similarity between two nodes is greater than a threshold τ c , the weight between the two nodes is 1, otherwise 0;

[0042] Then, the degree centrality of the nodes in the data set level undirected graph G dataset is used to measure the importance of the nodes; the definition of the degree centrality is the degree of the node, in the undirected graph Gdataset In G, the degree centrality is the number of edges for each node; the higher the degree centrality of a node, the more nodes it is connected to. Therefore, in an undirected graph G... dataset China and Vietnam are important; a node v i Degree D i The following formula is used to calculate:

[0043] Finally, exclude those with degree centrality scores below the threshold τ. d Non-primary semantic concepts retain meaningful semantic features.

[0044] The query features are either image query features or instance query features.

[0045] When the query feature is an image query feature, the query feature is obtained as follows:

[0046] Input a query image, extract a feature map through a convolutional neural network, and perform global pooling on the feature map to obtain the query image features; or, perform semantic concept segmentation on the feature map to obtain multiple semantic concept features, and select one of the semantic concept features as the image query features.

[0047] When the query feature is an instance query feature, the query feature is obtained as follows:

[0048] Input an instance image or a query image, crop the query image using a query instance bounding box to obtain the instance image; extract feature maps from the instance image using a convolutional neural network; and perform global pooling on the feature maps to obtain the instance query features.

[0049] After adopting the above scheme, the semantic concept feature granularity extracted by the present invention can cover both instance level and image level, so that the extracted features can describe global and local image semantic information, thereby unifying image retrieval and instance retrieval within a single framework. Therefore, it can be used for both instance retrieval tasks and image retrieval tasks.

[0050] Specifically, in the semantic concept segmentation of the proposed method, the construction of the undirected graph is based on basic semantic elements, and the similarity between the basic semantic elements is determined according to whether it exceeds a threshold τ. b To determine whether there is an edge connecting them, therefore the threshold τ bThe high and low of the degree of connectivity of the nodes in the undirected graph affects the degree of connectivity of the nodes in the undirected graph. Subsequently, the semantic concept feature is fused by the basic semantic element features in the connected component in the undirected graph. Therefore, if the degree of connectivity of the nodes in the undirected graph is higher, the more nodes are contained in the connected component in the graph, and the semantic concept feature generated by the connected component is closer to the image-level feature. Conversely, if the degree of connectivity of the nodes in the undirected graph is lower, the semantic concept feature generated by the connected component is closer to the instance-level feature. In summary, the proposed semantic concept feature can take into account both the instance retrieval and image retrieval tasks. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of the method of the present application;

[0052] Figure 2 A flowchart of the acquisition of the image query feature;

[0053] Figure 3 A flowchart of the acquisition of the instance query feature. DETAILED DESCRIPTION

[0054] As shown in Figure 1 , the present application discloses a retrieval method based on semantic concept extraction, which comprises the following steps:

[0055] Step 1, acquiring a query feature;

[0056] The query feature is an image query feature or an instance query feature.

[0057] As shown in Figure 2 , when the query feature is an image query feature, the acquisition of the query feature is as follows:

[0058] An input query image is extracted by a convolutional neural network to obtain a feature map, and global pooling is performed on the feature map to obtain an image query feature.

[0059] Alternatively, the feature map is subjected to semantic concept segmentation to obtain a plurality of semantic concept features, and one of the semantic concept features is selected as the image query feature. The semantic concept segmentation here is the same as the semantic concept segmentation described below, and specific reference can be made to the following.

[0060] As shown in Figure 3 , when the query feature is an instance query feature, the acquisition of the query feature is as follows:

[0061] An input instance image, or an input query image, is cropped by a query instance rectangular frame to obtain an instance image;

[0062] A feature map is extracted from the instance image by a convolutional neural network;

[0063] Global pooling is performed on the feature map to obtain an instance query feature.

[0064] Step 2, similarity calculation is performed between the query feature and the candidate features in the candidate feature database, and a similarity ranking result is obtained.

[0065] Step 3, the similarity ranking result is returned as a search result.

[0066] In the above step 2, the candidate features in the candidate feature database are obtained by feature extraction from the images in the image database. The feature extraction method is as follows:

[0067] S1, extract basic semantic elements

[0068] Specifically, for the HxWxC-dimensional feature map X output by the convolutional neural network, the average activation map HxW can be obtained by averaging X in the channel dimension C. Since different channels in the convolutional neural network usually represent different semantics, the value at each pixel position on the average activation map represents the sum of the high-level semantic responses in the spatial range of the original image. Since the main part of the image usually has more semantic information than the non-main part, it will show a higher response value in the average activation map. Therefore, the set of peak points found by using a 3x3 window on the average activation map summarizes the positions of the main parts of the image, i.e., the positions of the basic semantic elements.

[0069] Since the feature map is obtained by downsampling the original image through multiple pooling layers, the peak point on the feature map actually corresponds to a local region in the original image. Therefore, by finding the local region in the original image corresponding to the peak point and then performing feature pooling, the features of each basic semantic element can be obtained more accurately.

[0070] For a convolutional layer in the convolutional neural network, for the sake of simplicity, let the H f xW f dimensional convolution kernel be F, where H f and W f are the dimensions of the convolution kernel. Let the input feature map and the output feature map of the convolutional layer be I and O, respectively. In the process of forward propagation, the value I i,j at position (i,j) on the input feature map is related to the value O p,q at position (p,q) on the output feature map through the weight F i-p,j-q at position (i-p,j-q) on the convolution kernel. Therefore, the output feature map O is calculated by the following formula:

[0071]

[0072] where b represents the bias value in the convolutional layer, and σ represents the nonlinear activation function after the convolutional layer.

[0073] Thus, the contribution probability of a pixel position I i,j in the input feature map to a pixel position O p,q in the output feature map can be expressed as:

[0074]

[0075] where the conditional probability P(I ij |O pq ) is defined as:

[0076]

[0077] where, is the bottom-up activation value of I at spatial position (i, j) calculated by forward propagation, Z p,q is the regularization term to ensure that p,q P(I i,j |O p,q ) = 1.

[0078] Equation (2) gives the way to calculate the contribution probability of each position in the input feature map to a position in the output feature map. Thus, for each peak point on the average activation map , the contribution probability map M of the same size as the original image can be obtained by backward propagation from the last layer to the first layer.

[0079] Next, the spatial position of the basic semantic element needs to be estimated in the form of a rectangular box on the contribution probability map M. First, the values on M are normalized to the range [0, 1]. Since the starting point of the backward propagation is the feature map output by the last convolutional layer, which has the largest receptive field, when the backward propagation reaches the input image, a large range of pixel positions will be activated, even if the pixel has a very small contribution to the peak point. Therefore, by setting a threshold τ a to filter out pixels that have almost no contribution to the peak point, τ a is set to 0.1 in this method. Subsequently, the activation region of M is estimated as an elliptical shape, and the parameters of the ellipse are modeled by the second moment of all pixel positions in M that are greater than τ a :

[0080]

[0081] The rectangular box of each basic semantic element activation region is obtained from the circumscribed rectangle of the ellipse. Finally, the features of each basic semantic element are obtained by average pooling in the feature map through the rectangular box corresponding to each basic semantic element.

[0082] S2, semantic concept segmentation

[0083] After extracting the basic semantic elements from an image, an undirected graph G is constructed from these basic semantic elements. image = <V image E image >,

[0084] Among them, V image For nodes in the graph, the basic semantic element features are... Composition; E image Let be the set of edges, denoted as

[0085] G image The weight of an edge between two nodes is defined as follows:

[0086]

[0087] In equation (5), cos(v i ,v j ) for v i With v j The cosine similarity between them is used. Thus, similar basic semantic elements in the graph are connected by edges. Then, a breadth-first search algorithm is used to cut connected components in the graph. Specifically, initially, each node is marked as unvisited, and then from an unvisited node v... i First, a new connected component is created. Then, all its unvisited adjacent nodes are visited, and these adjacent nodes are added to the connected component and marked as visited. After all nodes have been visited, the resulting connected component is added to the connected component set S, and the above process is repeated starting from the next unvisited node until all nodes have been visited. Thus, each connected component eventually contains similar basic semantic elements, which collectively describe a semantic concept in the image. Next, the features of the basic semantic elements are aggregated by calculating the average features of the nodes within each connected component to obtain the features of the semantic concept:

[0088]

[0089] Among them, S i,j For connected component S i The features of the j-th basic semantic element in S, where N is the connected component S i The number of basic semantic elements within.

[0090] S3. Removal of non-essential semantic concepts

[0091] The semantic concept extraction process described above is performed on a single image. Besides the primary semantic concepts (such as the main subject instance), the extracted semantic concepts also include non-primary semantic concepts (such as cluttered backgrounds and irrelevant instances). The presence of these non-primary semantic concept features not only increases storage burden and slows down retrieval, but also affects the salience of the primary semantic features in the dataset, thus impacting retrieval performance. However, it is difficult to distinguish non-primary semantics from a single image because, in the above category-independent semantic concept extraction process, any semantic concept extracted from an image is not classified, making it impossible to determine which semantic concepts should be retained or removed. However, at the level of an image dataset, if a semantic concept is primary in the dataset, it will inevitably appear frequently; conversely, if a semantic concept appears very infrequently, it may be a non-primary semantic concept. The steps for removing non-primary semantics are as follows:

[0092] First, construct an undirected graph G at the dataset level. dataset = <V dataset E dataset >, where V dataset For nodes in the graph, the semantic concept features are... E dataset Let be the set of edges, denoted as G dataset The edge weights and G in image The definition of similarity is defined as follows: if the similarity between two nodes is greater than a threshold τ. c If the weight is 1, then the weight between the two nodes is 1; otherwise, it is 0.

[0093] Then, the degree centrality of nodes in the graph is used to measure the importance of nodes. Degree centrality is defined as the degree of a node, which is the number of edges for each node in an undirected graph. The higher the degree centrality of a node, the more nodes it is connected to, and therefore the more important it is in the graph. Thus, a node v i Degree D i The following formula is used to calculate:

[0094]

[0095] Finally, exclude those with degree centrality scores below the threshold τ. d The removal of non-primary semantic concepts is achieved by retaining meaningful semantic concept features.

[0096] S4. Semantic concept features are processed through L2 regularization, PCA whitening, and another round of L2 regularization to obtain candidate features.

[0097] The semantic concept feature extracted by the method can cover instance level and image level, so that the extracted feature can describe global and local image semantic information, thereby unifying image retrieval and instance retrieval in a set of frameworks, and thus can be used for both instance retrieval task and image retrieval task. Specifically, in the semantic concept segmentation of the method, the construction of the undirected graph is based on basic semantic elements, and whether there is an edge between the basic semantic elements is determined according to whether the similarity between the basic semantic elements is higher than a threshold τ b The threshold τ b affects the node connectivity in the undirected graph. Subsequently, the semantic concept feature is fused by the basic semantic element features in the connected component in the undirected graph. Therefore, the higher the node connectivity in the undirected graph is, the more nodes contained in the connected component in the graph are, and the closer the semantic concept feature generated by the connected component is to the image level feature. Conversely, the lower the node connectivity in the undirected graph is, the closer the semantic concept feature generated by the connected component is to the instance level feature. In summary, the proposed semantic concept feature can take into account both instance retrieval and image retrieval tasks.

[0098] In the instance retrieval task, since the retrieval target is the instance in the image, the target feature tends to be local spatial level. Therefore, when extracting the semantic concept feature on the instance retrieval dataset, the threshold τ b is set to a higher value, so that the generated semantic concept feature is more inclined to the instance level. When the query requirement of instance retrieval comes, the method uses the same feature extraction network to extract the feature map of the query image on the same output layer, and then uses the query instance frame to perform average region pooling on the feature map of the query image to obtain the query feature. Subsequently, the query feature obtains the similarity ranking results of all semantic concept features in the candidate database through the nearest neighbor search algorithm, and the ranking of each image is the highest ranking of the semantic concept feature in the image. Finally, the ranking results of the images are returned as the retrieval results.

[0099] In the image retrieval task, since the retrieval target is the whole image, the target feature tends to be global image level. Therefore, unlike in the instance retrieval task, when extracting the semantic concept feature on the image retrieval dataset, the threshold τ b is set to a lower value, so that the generated semantic concept feature is more inclined to the image level. When the query requirement of image retrieval comes, the feature map of the query image is extracted on the same output layer of the same feature extraction network, and then the feature map is averaged in the spatial dimension to obtain the query feature. The subsequent feature retrieval step is the same as in the instance retrieval task, and the retrieval ranking results of the images are obtained according to the images to which each semantic concept feature belongs.

[0100] In summary, the present application can describe global and local image semantic information by mining semantic concepts on the feature map, so that image retrieval and instance retrieval can be unified in a framework. That is, when the retrieval method based on the present application is used, the developer only needs to develop a system to apply it to image retrieval and instance retrieval, specifically, only needs to adjust the size of threshold τ b , so as to greatly reduce the development cost. Moreover, the present application can achieve excellent retrieval performance on multiple instance retrieval and image retrieval data sets.

[0101] To illustrate the effect of the present application, the retrieval method of the present application is compared with the existing retrieval methods. Specifically as follows:

[0102] Table 1 is the retrieval accuracy comparison of the method of the present application on the Instance-335 data set with R-MAC, CroW, CAM, BLCF, BLCF-SalGAN, Regional Attention, DeepVision, FCIS+XD, PCL*+SPN, and DASR.

[0103] Method Top 50 Top 100 All R-MAC 23.4 31.5 37.5 CroW 15.9 22.5 32.1 CAM 19.4 26.3 34.7 BLCF 24.6 35.8 48.3 BLCF-SalGAN 24.5 35.0 46.9 Regional Attention 24.2 35.1 48.8 DeepVision 40.2 52.1 62.0 FCIS+XD 40.3 50.0 59.3 PCL*+SPN 38.5 47.9 57.9 DASR 41.9 55.8 69.9 Method of the invention 43.6 57.4 72.1

[0104] Table 1

[0105] Table 2 is the retrieval accuracy comparison of the method of the present application on the INSTRE data set with R-MAC, CroW, CAM, BLCF, BLCF-SalGAN, Regional Attention, DeepVision, FCIS+XD, PCL*+SPN, and DASR.

[0106] Method Search precision R-MAC 52.3 CroW 41.6 CAM 32.0 BLCF 63.6 BLCF-SalGAN 69.8 Regional Attention 54.2 DeepVision 19.7 FCIS+XD 6.7 PCL*+SPN 56.9 DASR 62.9 Method of the invention 69.7

[0107] Table 2

[0108] Table 3 is the retrieval accuracy comparison of the method of the present application on the Holidays, Oxford5k and Paris6k data sets with SIFT+VLAD, BoVW+HE, Neural Codes, R-MAC, CroW, BLCF, BLCF-SalGAN, CAM, Regional Attention, DeepVision, and DASR+VLAD.

[0109] Method Holidays Oxford5k Paris6k SIFT+VLAD 66.4 35.9 39.1 BoVW+HE 74.2 50.3 50.1 Neural Codes 74.9 43.5 - R-MAC - 66.9 83.0 CroW 85.1 70.8 79.7 BLCF 85.4 72.2 79.8 BLCF-SalGAN 83.5 74.6 81.2 CAM 78.5 71.2 80.5 Regional Attention - 76.8 87.5 DeepVision - 71.0 79.8 DASR+VLAD 83.4 59.4 69.0 Algorithm of the invention 91.9 73.5 84.2

[0110] Table 3

[0111] In the above table 1-3, the method corresponding to R-MAC is the method proposed by Tolias G et al. (Tolias G, Sicre R, Jégou H. Particular object retrieval with integral max-pooling of CNN activations [J]. arXiv preprint arXiv:1511.05879, 2015.);

[0112] The method corresponding to CroW is the method proposed by Kalantidis Y et al. (Kalantidis Y, Mellina C, Osindero S. Cross-dimensional weighting for aggregated deep convolutional features [C] / / European conference on computer vision. Springer, 2016: 685-701);

[0113] The method corresponding to CAM is the method proposed by Jimenez A et al. (Jimenez A, Alvarez J M, Giró-i-Nieto X. Class weighted convolutional features for visual instance search [C] / / 2017);

[0114] The method corresponding to BLCF and BLCF-SalGAN is the method proposed by Mohedano E et al. (Mohedano E, McGuinness K, Giró-i Nieto X, et al. Saliency weighted convolutional features for instance search [C] / / 2018 international conference on content-based multimedia indexing (CBMI). IEEE, 2018: 1-6);

[0115] The method corresponding to Regional Attention is the method proposed by Kim J et al. (Kim J, Yoon SE. Regional attention based deep feature for image retrieval. [C] / / BMVC. 2018: 209);

[0116] The method corresponding to DeepVision is the method proposed by Salvador A et al.(Salvador A, Giró-i Nieto X, Marqués F, et al. Faster r-cnn features for instance search[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2016: 9-16);

[0117] The method corresponding to FCIS+XD is the method proposed by Zhan Y et al.(Zhan Y, Zhao W L. Instance search via instance level segmentation and feature representation[J]. Journal of Visual Communication and Image Representation, 2021, 79: 103253.);

[0118] The method corresponding to PCL*+SPN is the method proposed by Lin J et al.(Lin J, Zhan Y, Zhao W L. Instance search based on weakly supervised feature learning[J]. Neurocomputing, 2021, 424: 117-124.);

[0119] The method corresponding to DASR and DASR+VLAD is the method proposed by Xiao H C et al.(Xiao H C, Zhao W L, Lin J, et al. Deeply activated salient region for instance search[J]. ACM Trans. Multimedia Comput. Commun. Appl., 2021);

[0120] The method corresponding to SIFT+VLAD and BoVW+HE is the method proposed by Zhao W L et al.(Zhao W L, Ngo CW, Wang H. Fast covariant vlad for image search[J]. IEEE Transactions on Multimedia, 2016, 18(9): 1843-1854);

[0121] The method corresponding to the neural code is the method proposed by Babenko A et al. (Babenko A, Slesarev A, Chigorin A, et al. Neural codes for image retrieval [C] / / European conference on computer vision. Springer, 2014: 584-599.).

[0122] As can be seen from Tables 1, 2 and 3, the method of the present application achieves the best effect on two example retrieval data sets and three image retrieval data sets. In addition, the algorithm of the present application only depends on a pre-trained convolutional neural network, does not need additional human-labeled data for fine-tuning training and can extract semantic concept features of any category.

[0123] The above merely illustrates the embodiments of the present application, and does not limit the technical scope of the present application in any way. Any slight modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments still falls within the scope of the technical solutions of the present application.

Claims

1. A retrieval method based on semantic concept extraction, characterized in that: The method comprises: Obtaining query features; Calculating the similarity of the query features with candidate features in a candidate feature database to obtain a similarity ranking result; Returning the similarity ranking result as a search result; The candidate features in the candidate feature database are obtained by feature extraction from images in an image database; the feature extraction method is as follows: Extracting basic semantic elements from the images; Inputting the images into a convolutional neural network to obtain an output feature map X of HxWxC dimensions; average activation map of dimension H x W is obtained after averaging the feature map X over the channel dimension C ; Finding the set of peak points using a N x N window on the average activation map the set of peak points is found For each peak point on the average activation map , the contribution probability map M of the same size as the original image is calculated by back-propagation from the last layer to the first layer through the contribution probability formula until the input image. The contribution probability formula is as follows: wherein a convolution kernel of a convolution layer in the convolutional neural network is F, and a size of the convolution kernel F is a contribution probability of a pixel position in an input feature map to a pixel position in an output feature map conditional probability is defined as: wherein is In spatial locations The bottom-up activation values calculated by forward propagation, is a regularization term to ensure ; Estimating the spatial position information of the basic semantic elements in the form of a rectangular frame on the contribution probability map M; Performing semantic concept segmentation: Building an undirected graph of basic semantic elements undirected graph The weight of the edge between two nodes in the undirected graph is defined as: wherein, is and cosine similarity between Cutting out connected components in the undirected graph, each connected component containing similar basic semantic elements; calculating the average features of the nodes in each connected component to aggregate the features of the basic semantic elements to obtain semantic concept features; The semantic concept features are processed by L2 regularization, PCA whitening and another round of L2 regularization to obtain candidate features.

2. The retrieval method based on semantic concept extraction according to claim 1, characterized in that: Before estimating the spatial position of the basic semantic elements, the contribution probability map M is processed as follows: Normalizing the values on the contribution probability map M to a range of [0, 1]; Setting a threshold Pixels that do not contribute to the peak point are filtered out; The activation region on the contribution probability map M is estimated as an elliptical shape, the parameters of which are derived from the pixel positions in the contribution probability map M that are greater than The modeling of the image second moment gives:​ 。 3. The retrieval method based on semantic concept extraction according to claim 2, characterized in that: The rectangular frame corresponding to the basic semantic elements is obtained from the circumscribed rectangle of the ellipse, and the features of the basic semantic elements are obtained by average pooling of the rectangular frame corresponding to each basic semantic element on the feature map.

4. The retrieval method based on semantic concept extraction according to claim 1, characterized in that: The undirected graph is represented as: ， wherein nodes of an undirected graph, composed of basic semantic element features ; is a set of edges, denoted as .

5. The retrieval method based on semantic concept extraction according to claim 1, characterized in that: After performing semantic concept segmentation to obtain semantic concept features, non-semantic concept removal is performed; specifically as follows: First, a dataset-level undirected graph is established wherein is a node in the dataset-level undirected graph, which is a set of semantic concept features, denoted as , is a semantic concept feature: is a set of edges, denoted as ; The edge weight in is defined as: if the similarity between two nodes is greater than a threshold , the weight between the two nodes is 1, otherwise 0. Then, the importance of a node is measured by its degree centrality in the dataset-level undirected graph The definition of degree centrality is the degree of a node, which is the number of edges for each node in the undirected graph The higher the degree centrality of a node, the more nodes it is connected to, and thus the more important it is in the undirected graph The degree of a node is calculated by the following formula: ​​ Finally, exclude those with degree centrality scores below the threshold. Non-primary semantic concepts retain meaningful semantic features.

6. The retrieval method based on semantic concept extraction according to claim 1, characterized in that: The query features are image query features or instance query features.

7. The retrieval method based on semantic concept extraction according to claim 6, characterized in that: When the query features are image query features, the query features are obtained as follows: Input the query image, extract the feature map through the convolutional neural network, and perform global pooling on the feature map to obtain the query image features; or perform semantic concept segmentation on the feature map to obtain multiple semantic concept features, and select one of the semantic concept features as the image query feature.

8. The retrieval method based on semantic concept extraction according to claim 6, characterized in that: When the query features are instance query features, the query features are obtained as follows: Input the instance image, or input the query image, crop the query image using the query instance rectangular frame to obtain the instance image; extract the feature map from the instance image through the convolutional neural network; and perform global pooling on the feature map to obtain the instance query feature.

Images