A method for segment learning hash codes of different levels of information in large-scale fine-grained image retrieval

Abstract

The application discloses a method for segmental learning of hash coding of different levels of information in large-scale fine-grained image retrieval, comprising the following steps: segmentally extracting global feature information and component feature information of an image by a segmental convolutional neural network; mining potential component feature information and related information in different components of the image; proposing a channel self-changing mechanism, taking a channel slice as a basic unit of model input and mapping to a continuous vector space, and mining related information of each part of a fine-grained object; and segmentally mapping the global feature information and the component feature information of the image extracted to hash coding. The application separates features and coding, so that hash coding of different parts has independent actual meanings; through an "inhibition-enhancement" attention module and the channel self-changing mechanism, the application further grasps the relationship between the independent coding, so as to obtain higher image retrieval accuracy and faster image retrieval rate.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a method for learning hash coding of different levels of information in segment in large-scale fine-grained image retrieval. BACKGROUND

[0002] Fine-grained image retrieval, as an important part of fine-grained image analysis, has been paid more and more attention in recent years. Fine-grained image recognition is a basic research topic in the field of computer vision and pattern recognition, aiming to study the visual recognition task of different sub-class categories under a certain traditional semantic category, such as different sub-classes of dogs and different car models. Due to the subtle visual differences between object objects in fine-grained images in class differences, and the great changes in posture, size and other intra-class differences, there is great difficulty in retrieval.

[0003] Hash learning maps data into binary codes through machine learning method, which can significantly reduce the storage and communication overhead of data, thereby effectively improving the retrieval efficiency of the system. The purpose of hash learning is to learn the binary hash code representation of data, so that the hash code can preserve the neighbor relationship in the original space as much as possible, that is, the similarity. Hashing methods are divided into data-independent and data-dependent methods. In the data-independent hashing method, the hash function in the model is usually randomly generated and independent of any training data. In order to achieve a high retrieval accuracy, a very long code bit number is required for such a hash method. Data-dependent hash method learns hash function through training data and obtains hash code in turn. Compared with data-independent method, data-dependent hash algorithm can get higher retrieval accuracy with shorter hash code. With the rise of deep learning, some data-dependent hash methods integrate deep learning into the hash framework and obtain good performance. In the past work, many deep hash methods have been proposed for large-scale image retrieval.

[0004] Attention mechanism originates from the study of human vision. In cognitive science, due to the bottleneck of information processing, humans selectively focus on part of all information while ignoring other visible information, and these focused information is also considered to be the most important part of memory or distinguishing the whole information. The above mechanism is usually called attention mechanism. In order to reasonably use the limited visual information processing resources, humans need to select a specific part in the visual area and then focus on it. In the field of computer vision, attention mechanism is used to simulate the way humans focus on the whole information. However, attention mechanism also has some shortcomings. Due to the potential characteristics of neural networks, attention mechanism often only focuses on those features that are decisive for distinguishing objects, while ignoring other important features of objects, resulting in lower robustness of the trained neural network. Therefore, some improved attention mechanism methods begin to guide the model to focus on those ignored information.

[0005] Existing deep hashing methods are mostly limited to coarse-grained data retrieval. Coarse-grained data has large differences between classes and is easy to distinguish, such as humans can quickly distinguish cats from dogs after a short learning. However, fine-grained features have very small differences between classes, such as the varieties of persimmons, and untrained humans can hardly accurately distinguish them. At the same time, fine-grained images often have great intra-class differences, such as apple trees. Even if the images are taken at the same location, the differences between them are great in different seasons. Therefore, existing coarse-grained hashing retrieval methods cannot meet the current application requirements. Existing fine-grained hashing methods are limited to the existing paradigm of fine-grained image analysis, and different levels of features are mixed together and hashed by feature extraction, resulting in that different levels of feature information cannot be effectively utilized. Therefore, a new method of hashing coding that can segmentally learn different levels of information in fine-grained retrieval environment is needed. SUMMARY

[0006] The purpose of the present application is to provide a method of hashing coding that can segmentally learn different levels of information in large-scale fine-grained image retrieval.

[0007] The technical solution for achieving the purpose of the present application is as follows: In a first aspect, the present application provides a method of hashing coding that can segmentally learn different levels of information in large-scale fine-grained image retrieval, comprising the following steps:

[0008] Step 1: segmentally extracting global feature information and component feature information of the image by a segmented convolutional neural network;

[0009] Step 2: on the basis of step 1, constructing an "inhibition-enhancement" attention module to mine potential component feature information and related information in different components in the image;

[0010] Step 3: on the basis of step 1, proposing a channel self-changing mechanism, taking channel slices as the basic unit of model input and mapping them to a continuous vector space to mine the related information of each part of the fine-grained object;

[0011] Step 4: segmentally mapping the extracted global feature information and component feature information of the image into hash coding.

[0012] In a second aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the program.

[0013] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the steps of the method of the first aspect.

[0014] In a fourth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of the first aspect.

[0015] Compared with the prior art, the present application has the following advantages: (1) the traditional fine-grained image analysis method extracts the overall features and local features of the image and performs softening to obtain better image representation, and the extended hash release retains this feature softening manner, so that the obtained hash code is too mixed. The present application separates the image features and maps them one by one into relatively independent hash codes, so that each part of the hash code has an actual physical meaning; (2) by constructing the "inhibition-enhancement" attention module and proposing the channel self-changing mechanism, the internal correlation degree of the separated features is improved and the correlation information between the separated features is mined, so that the learned hash code has higher precision on the basis of having actual meaning; (3) the present application has a performance result comparable to traditional hash learning methods in coarse-grained image retrieval, and has a retrieval accuracy much higher than traditional hash learning methods in fine-grained image retrieval; compared with existing fine-grained hash retrieval methods, the present application also has higher retrieval accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A schematic diagram of a method for learning hash codes of different levels of information in large-scale fine-grained image retrieval according to the present application. DETAILED DESCRIPTION

[0017] In combination Figure 1 A method for learning hash codes of different levels of information in large-scale fine-grained image retrieval, specifically comprising the following steps:

[0018] Step 1: extracting global feature information and component feature information of the image by a segmented convolutional neural network;

[0019] A segmented convolutional neural network is constructed, which not only separates the global information and component information, but also separates the information of different components, the purpose of which is to extract relatively independent features, so that each part of the hash code corresponding to the image has a specific and different meaning. Specifically, first, the shallow activation features of the input image are extracted by a convolutional neural network:

[0020]

[0021] wherein C, H and W represent the number of channels, height and width of the shallow activation feature vector T, and Φ CNN represents a pre-trained convolutional neural network. On the basis of the activation feature vector T, the global transformation branch network φ(·) is used to extract the global feature vector of the image:

[0022]

[0023] where θ global represent the parameters of the global transformation branch network φ(·), C', H' and W' represent the channel number, height and width of the global feature vector The local feature extraction branch network φ'(·) contains m attention guiding vectors where c represents the channel number of these vectors, and their width H and height W have the same values as the shallow layer activation feature vector T. The calculation of these attention guiding vectors will be described in step 2. For each attention guiding vector P i , a network composed of a 1x1 convolutional layer and an activation layer will be used to obtain an activation map M i :

[0024]

[0025] where H and W represent the width and height of the activation map M i , which have the same values as the width and height of the shallow layer activation feature vector T. For each activation map M i , it is applied to the activation feature vector t to obtain the activation feature t' of different parts i :

[0026]

[0027] where ⊙ represents the Hadamard product of matrices. Then, the local feature extraction branch network φ'(·) generates feature vectors of different parts in the image through these activation features T' i

[0028]

[0029] where θ local represent the parameters of the local feature extraction branch network φ'(·), C', H' and W' represent the channel number, height and width of the global feature vector , which have the same values as the channel number, height and width of the global feature vector . The overall object feature x global and m part-level features are obtained by performing global average pooling on and m

[0030] ​Step 2, on the basis of step 1, further excavate the potential component feature information in the image and the correlation information in different components by constructing the "inhibition-enhancement" attention module;

[0031] Since different components in the image or object often have correlation information, by constructing the "inhibition-enhancement" attention module, the potential component feature information in the image and the correlation information in different components are excavated in the local feature extraction branch network φ'(·). Specifically, for the m attention guide vectors P i described in step 1, the first attention guide vector P1 will be obtained through an attention guide network φ att (·):

[0032] P1=φ att (T; θ att ) (6)

[0033] Where θ att represents the parameters of the attention guide network φ att (·). The remaining m-1 attention guide vectors P i ,i={2,3,…,m} will be calculated by the "inhibition-enhancement" attention module. Specifically, for the activation map M i-1 corresponding to the i-1 attention guide vector P i-1 (refer to formula 3), the following operation is performed:

[0034]

[0035] Where softmax(·) is a common activation function in deep learning, and H and W represent the height and width of M' i-1 , which are exactly the same in value as the height and width of M i-1 . Record and represent the standard deviation and mean of all elements in M' i-1 . For each element in M' i-1 , the following calculation is performed:

[0036]

[0037] In this formula, "=" represents assignment operation, that is, the after assignment will replace the original α represents a hyperparameter that adjusts the degree of inhibition and enhancement. We denote formula (6) as f SEM (·). Then, for the m attention guide vectors P i , the calculation method is:

[0038]

[0039] Step 3, based on step 1, a channel self-changing mechanism is proposed, taking channel slices as the basic unit of model input and mapping them to continuous vector space, to mine the association information of each part of the fine-grained object;

[0040] For the global feature vector of the image and the feature vectors of m different parts A two-stage method is proposed, taking channel slices as the basic unit of model input and mapping them to continuous vector space, to mine the association information of each part of the fine-grained object.

[0041] Specifically, according to the correlation calculation in step 1, it is known that and Since these feature vectors are of the same size, these m+1 feature vectors are taken as a whole. In the first stage, for any one feature vector in the whole It is equally divided into N feature blocks of the same size: G = [G1; G2; …; G N ]. Among them, each feature block d = C' / N. For each feature block G i , a 1x1 convolution layer is used to generate Q i ,K i and V i of the same size as it. Q i represents the query vector, K i and V i represent the key-value pair vector, and then the following operations are performed:

[0042]

[0043] where K i T represents the transpose of K i . For each the following operations are performed:

[0044]

[0045] where softmax(·) is a common activation function in deep learning, sign(·) represents a common sign function in deep learning, and δ is a small positive bias, which is a kind of hyperparameter. The size of G′ i is completely consistent with G i . That is, after the first stage, each feature block G i is transformed into G′ i . For each transformed feature block G′ iIt contains d channel slices, denoted as G′. i ={G i1 G i2 ;…;G id In the second stage, for N transformed feature blocks G′ of the same size... i The slices at the same position in each feature block are recombined to obtain a new feature block G″. i :

[0046]

[0047] That is, the i-th reconstructed feature block G″ i The i-th channel slice G of the original feature block G′1 1i The i-th channel slice G′2 2i ... and G′ N The i-th channel slice G Ni There are d such new feature blocks. For each new feature block G″ i Perform the same calculations as in formulas (10) and (11). Cover the calculated eigenvectors with... The corresponding eigenvectors in the equation.

[0048] Step 4: The extracted global feature information and component feature information of the image are segmented and mapped into hash codes;

[0049] Based on the optimization of steps 2 and 3, the overall object feature x global With m component-level features Segment mapping is hash encoding.

[0050] Specifically, the hash mapping layer first contains an encoder consisting of m+1 linear mapping layers. The encoder will x global and Mapped to floating-point vectors v respectively global and Finally, the hash coding layer will v global and Mapped to binary hash code u global and

[0051]

[0052] Where tanh(·) represents the hyperbolic tangent activation function commonly used in deep learning, and sign(·) represents the sign function commonly used in deep learning, for images... Its hash code is

[0053] In one embodiment, the method for learning hash codes of different levels of information in large-scale fine-grained image retrieval can be used for pistachio nut species identification.

[0054] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for segmental learning of hash codes of different levels of information in large-scale fine-grained image retrieval, characterized in that, The method comprises the following steps: Step 1, segmentally extracting global feature information and component feature information of an image through a segmented convolutional neural network, specifically: Extracting input images by convolutional neural networks Shallow activation features of wherein C, H and W represent the channel number, height and width of the shallow activation feature vector T, and Φ CNN represents a pre-trained convolutional neural network; on the basis of the activation feature vector T, the global transformation branch network φ(·) is used to extract the global feature vector of the image: Where, θ global The parameters representing the global transformation branch network φ(·), C ′ H ′ With W ′ Represents global feature vectors Channel number, height, and width; local feature extraction branch network φ ′ (·) contains m attention-guided vectors Where c represents the number of channels in these vectors, and their height H and width W are the same as the height and width of the shallow activation feature vector T; for each attention-guided vector P i It will be passed through a network consisting of 1×1 convolutional layers and activation layers. To obtain an activation graph M i : wherein, H and W represent the height and width of the activation map M i whose values are the same as the height and width of the shallow activation feature vector T; for each activation map M i , it is applied to the activation feature vector T to obtain the activation features T' of different parts i : where represents the Hadamard product of matrices; and then, the local feature extraction branch network φ'(·) extracts the feature vectors of different components in the generated image i generating feature vectors of different components in the image wherein θ local represent parameters of the local feature extraction branch network φ'(·), and C', H' and W' represent the number of channels, height and width of φ'(·), respectively, which are completely identical in value to C, H and W of φ(·); global ​​​​​ Step 2, on the basis of step 1, mining potential component feature information and related information in different components in the image through construction of a "suppression-enhancement" attention module; Step 3, on the basis of step 1, proposing a channel self-changing mechanism, taking a channel slice as a basic unit of model input and mapping to a continuous vector space, and mining related information of each part of a fine-grained object; Step 4, segmentally mapping the extracted global feature information and component feature information of the image into hash codes. 2.The method for hashing coding of segment learning different levels of information in large-scale fine-grained image retrieval according to claim 1, characterized in that, By constructing the "inhibition-enhancement" attention module, in the local feature extraction branch network φ ′ (·) The potential component feature information in the image and the related information in different components are mined. For m attention guiding vectors P i where the first attention guiding vector P1will be obtained by an attention guiding network φ att (·) gives: P1 = φ att (T; θ att ) (6) where θ att represent the parameters of the attention guiding network φ att ; the remaining m-1 attention guiding vectors P i , i = {2, 3,..., m} will be computed by the "inhibition- enhancement" attention module; for the i-1th attention guiding vector P i-1 corresponding to the activation map M i-1 , it is operated as follows: where softmax(·) is an activation function, H and W represent the height and width of M ′ i-1 respectively, and the values of H and W are exactly the same as those of M i-1 respectively; record and respectively, and the values of H and W are exactly the same as those of M i-1 respectively, and the values of H and W are exactly the same as those of M i-1 respectively, and the values of H and W are exactly the same as those of M respectively, and the values of H and W are exactly the same as those of M The "=" in the formula represents an assignment operation, i.e., after being assigned The original Alpha represents a hyperparameter for adjusting the degree of inhibition and enhancement; formula (6) is denoted as f SEM (·); then for m attention guide vectors P i The calculation method is: 3.The method of claim 2, wherein, Global feature vector for an image Feature vectors for m different components A channel self-changing mechanism is proposed, which takes channel slices as the basic unit of model input and maps them to a continuous vector space, mining the correlation information of each part of the fine-grained object. It is known And Since the size of these feature vectors is exactly the same, we regard the m+1 feature vectors as a whole; in the first stage, for any feature vector in this whole We divide it into N feature blocks with the same size: G = [G1; G2;...; G N ]; where each feature block d = C' / N; for each feature block G i , we generate Q i ,K i and V i with the same size as it through a 1x1 convolution layer; Q i represents the query vector, K i and V i represent the key-value pair vector, and then the following operations are performed: where K i T denotes the transpose of K i ; for each the following operations are performed: where sign(·) is the sign function; δ is a small positive bias, G′ i has the same size as G i ; that is, after the first stage, each feature block G i is transformed to G′ i ; for each transformed feature block G′ i , which contains d channel slices, denoted as G′ i = {G i1 ; G i2 ;...; G id}, in the second stage, for N transformed feature blocks G′ i of the same size, the slices at the same position of each feature block are recombined to obtain a new feature block G" i : That is, the i-th reconstructed feature block G” i The i-th channel slice G of the original feature block G′1 1i The i-th channel slice G′2 2i ... and G′ N The i-th channel slice G Ni Composition; There are d such new feature blocks; For each new feature block G” i Perform the same operations as in formulas (10) and (11); cover the calculated eigenvectors with... The corresponding eigenvectors in the equation.

4. The method of claim 3, wherein the different levels of information are learned by segmenting the hash codes. characterizing x global with m component-level features segmented mapping into hash codes; The hash mapping layer first contains an encoder consisting of m+1 linear mapping layers The encoder maps x global to a floating point vector v global Finally the hash encoding layer maps v global to a binary hash code u global ​​​​ wherein tanh(·) is the hyperbolic tangent activation function, for the image with hash code 5. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1-4 when executing the program.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method of any one of claims 1-4.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-4.

Images