Feature vector determination method for image global representation, image retrieval method and device

Abstract

The application discloses a feature vector determination method of image global representation, an image retrieval method and device. In the application, the gradient of the parameter of the calculation model is taken as the feature, a second tensor of the fourth order formed by each gradient is determined, the weight value of the corresponding position in the weight value matrix is determined based on each gradient in the second tensor of each position, the feature value matrix is aggregated as the first feature vector of the global representation of the image in the convolution layer, the first feature vector corresponding to each convolution layer is subjected to splicing processing, and the target feature vector of the global representation of the target image is obtained, so that the image retrieval can be realized based on the feature vector of the image global representation.

Description

Technical Field

[0001] This invention relates to the fields of image analysis and image retrieval technology, and in particular to a method for determining feature vectors of global image representation, an image retrieval method, and an apparatus. Background Technology

[0002] Fine-grained image retrieval has wide applications in both academia and industry. For example, it can be used in the protection of endangered wild animals and plants and in the product retrieval of e-commerce websites. Furthermore, applications such as face verification and pedestrian re-identification can be regarded as generalized fine-grained image retrieval problems.

[0003] However, the accuracy of image retrieval based on the feature vectors of the global image representation determined by existing technologies is low. Therefore, how to determine new feature vectors of the global image representation and improve the accuracy of image retrieval has become an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides a method for determining the feature vector of global image representation, an image retrieval method, apparatus, device, and medium to solve the problem of low accuracy in image retrieval using the feature vector of global image representation determined in the prior art.

[0005] This invention provides a method for determining the feature vector of a global image representation, the method comprising:

[0006] The target image is acquired and input into the feature extraction model to obtain the first feature map. Based on the pixel value of each position in the first feature map, a third-order first tensor is obtained. The first tensor is input into the pooling layer of the network model for pooling processing to obtain the target vector. The target loss value is determined based on the element value of each element in the target vector.

[0007] For each convolutional layer in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix of each channel of the convolutional layer, each gradient in the corresponding position of the second tensor of the fourth order is determined. Based on each gradient in the second tensor of each position in the second tensor, the weight value in the corresponding position of the weight value matrix is ​​determined. The mean of each column in the weight value matrix is ​​calculated to obtain the first feature vector of the global representation of the convolutional layer.

[0008] The first feature vector corresponding to each convolutional layer is concatenated to obtain the target feature vector of the global representation of the target image.

[0009] Furthermore, the weight value matrix is ​​a second-order matrix, and the first feature vector and the target feature vector are both row vectors.

[0010] Further, determining the weight value at the corresponding position in the weight value matrix based on each gradient of the second-order tensor at each position in the second tensor includes:

[0011] For each position of the second tensor in the second tensor, if the row and column of the second tensor are both preset values, then the absolute value of the gradient in the second tensor is determined as the weight value of the corresponding position. If the row or column of the second tensor is not a preset value, then the difference between the maximum gradient and the minimum gradient in the second tensor is determined, and the sum of the difference and the standard deviation of the gradient is determined as the weight value of the corresponding position. A weight value matrix is ​​formed according to the weight values ​​corresponding to the second tensor at each position.

[0012] Further, the step of converting the pixel value at each position in the first feature map into a third-order first tensor includes:

[0013] For each channel feature map in the first feature map, the pixel value of each pixel is used as the element value of the corresponding position in the matrix of the channel feature map according to the position of each pixel in the channel feature map;

[0014] Based on the arrangement order of the feature maps composed of each channel feature map, the matrices corresponding to each channel feature map are arranged in the order of arrangement in a direction perpendicular to the plane where the matrix corresponding to each channel feature map is located, thereby transforming the first feature map into a third-order first tensor.

[0015] Furthermore, after converting the pixel values ​​at each position in the first feature map into a third-order first tensor, and before inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector, the method further includes:

[0016] Based on each matrix in the first tensor, determine the sum matrix of each matrix, and perform binarization on the sum matrix to obtain a binarized Boolean matrix;

[0017] The Boolean matrix is ​​copied and arranged in a direction perpendicular to the plane containing the Boolean matrix to obtain a third tensor of the same order as the first tensor;

[0018] The updated first tensor is obtained by multiplying the first tensor and the third tensor point-to-point.

[0019] Further, the step of inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector includes:

[0020] The updated first tensor is input into the pooling layer of the network model. Global average pooling is performed on the updated first tensor to obtain the first vector after global average pooling. Global max pooling is then performed on the updated first tensor to obtain the second vector after global max pooling.

[0021] The first vector and the second vector are concatenated, and the concatenated vector is used as the target vector.

[0022] Further, after performing global average pooling on the updated first tensor to obtain a first vector after global average pooling, and before performing global max pooling on the updated first tensor to obtain a second vector after global max pooling, the method further includes:

[0023] Based on the first number of element points with set values ​​in the Boolean matrix and the second number of element points contained in each channel feature map, the ratio of the second number to the first number is determined, and the ratio is multiplied by the first vector to obtain the updated first vector.

[0024] Furthermore, before concatenating the first feature vectors corresponding to each convolutional layer, the method further includes:

[0025] The first feature vector corresponding to each convolutional layer is subjected to L2 normalization to obtain each normalized first feature vector, and the normalized first feature vectors are then concatenated.

[0026] Accordingly, the present invention provides an image retrieval method, the method comprising:

[0027] The steps of the above-described image global representation feature vector determination method are performed on the image to be retrieved and each image in the retrieval database to obtain the second feature vector of the image to be retrieved and each third feature vector of each image;

[0028] Based on the second feature vector and each of the third feature vectors, the similarity between the second feature vector and each of the third feature vectors is determined, and each image is sorted in descending order of similarity.

[0029] Accordingly, the present invention provides a feature vector determination device for global image representation, the device comprising:

[0030] The acquisition module is used to acquire the target image and input it into the feature extraction model to obtain the first feature map as output.

[0031] The determination module is used to convert the pixel value at each position in the first feature map into a third-order first tensor, input the first tensor into the pooling layer of the network model for pooling processing, and obtain the output target vector; determine the target loss value based on the element value of each element in the target vector; for each convolutional layer set in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix of each channel of the convolutional layer, determine each gradient at the corresponding position in the fourth-order second tensor; based on each gradient in the second-order tensor at each position in the second tensor, determine the weight value at the corresponding position in the weight value matrix; calculate the mean of each column in the weight value matrix to obtain the first feature vector of the global representation of the convolutional layer; and concatenate the first feature vectors corresponding to each convolutional layer to obtain the target feature vector of the global representation of the target image.

[0032] Further, the determining module is specifically used for each position of the second tensor in the second tensor. If the rows and columns of the second tensor are both preset values, then the absolute value of the gradient in the second tensor is determined as the weight value of the corresponding position. If the rows or columns of the second tensor are not preset values, then the difference between the maximum gradient and the minimum gradient in the second tensor is determined, and the sum of the difference and the standard deviation of the gradient is determined as the weight value of the corresponding position. A weight value matrix is ​​formed according to the weight values ​​corresponding to the second tensor at each position.

[0033] Further, the determining module is specifically used to, for each channel feature map in the first feature map, take the pixel value of each pixel as the element value of the corresponding position in the matrix of the channel feature map according to the position of each pixel in the channel feature map; and arrange the matrices corresponding to each channel feature map in the order in which the feature map is composed of the channel feature maps according to the order in which the matrix of each channel feature map is located in a direction perpendicular to the plane where the matrix of each channel feature map is located, so as to obtain the first feature map transformed into a third-order first tensor.

[0034] Furthermore, the determining module is also used to, after converting the pixel value at each position in the first feature map to obtain a third-order first tensor, and before inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector, determine the sum matrix of each matrix in the first tensor, perform binarization processing on the sum matrix to obtain a binarized Boolean matrix; copy the Boolean matrix and arrange it in a direction perpendicular to the plane where the Boolean matrix is ​​located to obtain a third-order tensor of the same size as the first tensor; and perform point-to-point multiplication of the first tensor and the third tensor to obtain the updated first tensor.

[0035] Further, the determining module is specifically used to input the updated first tensor into the pooling layer of the network model, perform global average pooling on the updated first tensor to obtain a first vector after global average pooling, perform global max pooling on the updated first tensor to obtain a second vector after global max pooling, and concatenate the first vector and the second vector to use the concatenated vector as the target vector.

[0036] Furthermore, the determining module is also used to determine the ratio of the second number to the first number based on the first number of elements with set values ​​in the Boolean matrix and the second number of elements contained in each channel feature map, and multiply the ratio by the first vector to obtain the updated first vector.

[0037] Furthermore, the determining module is also used to perform L2 normalization on the first feature vector corresponding to each convolutional layer before performing the concatenation process on the first feature vector corresponding to each convolutional layer, to obtain each normalized first feature vector, and to perform concatenation process on each normalized first feature vector.

[0038] Accordingly, the present invention provides an image retrieval device, the device comprising:

[0039] The determination module is used to perform the above-described image global representation feature vector determination method on the image to be retrieved and each image in the retrieval library, to obtain the second feature vector of the image to be retrieved and each third feature vector of each image;

[0040] The retrieval module is used to determine the similarity between the second feature vector and each third feature vector based on the second feature vector and each third feature vector, and to sort each image in descending order of similarity.

[0041] Accordingly, the present invention provides an electronic device, which includes a processor and a memory. The memory is used to store program instructions, and the processor is used to execute the computer program stored in the memory to implement the steps of any of the above-described methods for determining the feature vector of global image representation, or to implement the steps of any of the above-described methods for image retrieval.

[0042] Accordingly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for determining feature vectors of global image representation, or implements the steps of any of the above-described image retrieval methods.

[0043] This invention provides a method for determining feature vectors of global image representation, an image retrieval method, apparatus, device, and medium. In this invention, a target image is acquired and input into a feature extraction model to obtain a first output feature map. A third-order first tensor is obtained based on the pixel values ​​at each position in the first feature map. This first tensor is then input into the pooling layer of a network model for pooling processing to obtain the output target vector. A target loss value is determined based on the element value of each element in the target vector. For each convolutional layer in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix for each channel of that convolutional layer, each gradient at the corresponding position in a fourth-order second tensor is determined. Based on each gradient value in the second-order tensor at each position of the second tensor, the weight matrix is ​​determined. The weight values ​​at corresponding positions are calculated, and the mean value of each column in the weight value matrix is ​​used to obtain the first feature vector of the global representation of the convolutional layer. The first feature vectors corresponding to each convolutional layer are concatenated to obtain the target feature vector of the global representation of the target image. Since the gradient of the model parameters is used as a feature in this invention, and the second tensor of the fourth order composed of each gradient is determined, the weight value at the corresponding position in the weight value matrix is ​​determined based on each gradient in the second order tensor at each position in the second tensor. The feature value matrix is ​​then aggregated to form the first feature vector of the global representation of the image in the convolutional layer. The first feature vectors corresponding to each convolutional layer are concatenated to obtain the target feature vector of the global representation of the target image, thereby enabling image retrieval based on the feature vector of the global representation of the image. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A schematic diagram illustrating the process of a method for determining feature vectors for global image representation provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram illustrating the process of an image retrieval method provided in an embodiment of the present invention;

[0047] Figure 3A schematic diagram illustrating the process of a method for determining feature vectors for global image representation provided in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the structure of an image retrieval device provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0051] A fine-grained image retrieval method based on a contrastive similarity loss function in the prior art includes the following steps: Step 1: Sampling effective samples by utilizing inter-class similarity and asymmetric distance; Step 2: Analyzing the similarity of sample pairs in the training set and assigning different weights to different sample pairs according to the degree to which the similarity of sample pairs violates their corresponding boundary values; Step 3: Constructing a contrastive similarity loss function; Step 4: Using the Inception network with BatchNormalization as the baseline network, pre-training and initializing the network parameters in ILSVRC2012-CLS; Step 5: Training the network initialized in Step 4 using the contrastive similarity loss function, and adjusting the network parameters through Adam during the training process to obtain a network more suitable for fine-grained image retrieval tasks; Step 6: Inputting the test set images into the fine-tuned network after data augmentation by single-center cropping, and extracting discriminative embedding features through the trained network to output feature vectors; Step 7: Calculating the cosine similarity of the embedding features between the query image and the remaining images in the test set, using the calculated value to measure the similarity of the images, and then sorting the images according to their similarity to return the retrieval sequence of the query images.

[0052] In order to select more valuable sample pairs in the prior art, a new sample pair sampling method has been proposed. This sample pair sampling method utilizes inter-class similarity and selects more effective sample pairs for training through asymmetric distance. At the same time, a new weighting scheme is designed to assign different weights to the selected effective sample pairs to achieve full utilization of the sample pairs. The weights of different sample pairs are determined by the degree to which they violate the positive and negative boundaries.

[0053] Existing technologies employ typical end-to-end deep metric learning algorithms. While deep metric learning leads to better feature extraction, this end-to-end global representation does not fully reflect this improvement and cannot further approximate the upper bound of the neural network model used to extract feature vectors for global image representation.

[0054] To determine new feature vectors for global image representation and improve the accuracy of image retrieval, embodiments of the present invention provide a method for determining feature vectors for global image representation, an image retrieval method, an apparatus, a device, and a medium.

[0055] Example 1:

[0056] Figure 1 This invention provides a schematic diagram of a method for determining feature vectors for global image representation, which includes the following steps:

[0057] S101: Acquire the target image and input it into the feature extraction model to obtain the first feature map output; based on the pixel value of each position in the first feature map, convert it into a third-order first tensor, input the first tensor into the pooling layer of the network model for pooling processing, and obtain the target vector output; determine the target loss value based on the element value of each element in the target vector.

[0058] In order to determine the feature vector of the global representation of an image, the feature vector determination method of the global representation of an image provided in this embodiment of the invention is applied to an electronic device, wherein the electronic device may be a host, tablet computer, smartphone or other smart terminal device, or a server, wherein the server may be a local server or a cloud server, and the comparison of this embodiment of the invention is not limited.

[0059] The electronic device acquires a target image, wherein the target image is a fine-grained image. Specifically, the electronic device may acquire a target image sent by other devices connected to it, or it may receive a target image input by the user through an input device.

[0060] After acquiring the target image, the electronic device also pre-stores a feature extraction model in order to retrieve the target image. The feature extraction model can be a pre-trained model or a network model fine-tuned by a deep metric learning algorithm. For example, the feature extraction model is a residual neural network (ResNet-50). The feature extraction model is used to extract the features of the image. After the electronic device inputs the target image into the feature extraction model, it obtains the first feature map output by the feature extraction model.

[0061] Since the structure of the feature map can be represented by a third-order tensor T with M×y×z elements, where T can be described by a set S consisting of 2D channel feature maps, S={Sm}, m=(1,…,M)M represents the number of channels, and y and z represent the number of pixels in the channel feature map; therefore, the electronic device uses the pixel value at each position in the first feature map as the element value of each position in the first tensor obtained by transformation, thereby obtaining the transformed third-order first tensor; the ResNet-50 in this embodiment of the invention ends with the Rectified Linear Unit (ReLU) activation function, so all elements in the first tensor take non-negative values.

[0062] To determine the target loss value of the loss function in the feature extraction model, the electronic device also stores the pooling layer of the network model. Based on the pre-stored pooling layer of the network model, the electronic device performs pooling processing on the first tensor. After pooling processing, the target vector is obtained as the output. According to the element value of each element in the target vector, the sum of the element values ​​of each element is determined as the target loss value of the loss function in the feature extraction model.

[0063] S102: For each convolutional layer set in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix of each channel of the convolutional layer, determine each gradient at the corresponding position in the second tensor of the fourth order. Based on each gradient in the second tensor at each position in the second tensor, determine the weight value at the corresponding position in the weight value matrix. Calculate the mean of each column in the weight value matrix to obtain the first feature vector of the global representation of the convolutional layer.

[0064] To determine the feature vector of the global representation of the target image, the electronic device pre-stores the parameter matrix for each channel of each convolutional layer in the feature extraction model. The parameter matrix contains multi-row, multi-column parameters of the model, where each convolutional layer is any two convolutional layers in the feature extraction model. For example, when the feature extraction model is ResNet-50, the last six convolutional layers are designated as layers... 4-1-1 layer 4-1-2 layer 4-1-3 layer 4-2-1 layer 4-2-2 layer 4-2-3 Each convolutional layer can be a layer 4-1-1 and layer 4-1-2 .

[0065] The parameter matrix for each channel is a second-order tensor f containing H×W elements. n,c f n,c This represents the parameter matrix of the c-th channel of the n-th convolutional kernel in this convolutional layer; the parameter matrices of all channels of the n-th convolutional kernel constitute a third-order tensor F.n F n It contains C×H×W elements and determines the value of the nth channel in the output feature map of the convolutional layer, F. n It can be viewed as a set f consisting of 2D convolutional kernels. n f n ={f n,c}(c=1,…,C); the third-order tensor F of all convolutional kernels of this convolutional layer. n This forms a fourth-order tensor P, which is composed of the third-order tensors F of all convolution kernels. n The set F is composed of F = {F n}(n=1,…,N), P contains N×C×H×W elements.

[0066] For each pre-saved convolutional layer, the electronic device determines each gradient at the corresponding position in the second tensor (which is a fourth-order tensor with partial derivatives of the target loss value and each parameter) based on the target loss value and the position of the parameters in each row and column of the parameter matrix for each channel of the convolutional layer. Based on each gradient in the second tensor at each position in the second tensor, each gradient in the second tensor is input to a pre-saved weight determination function, yielding the output value of the weight determination function. This value is the weight value at the corresponding position in the second tensor of the weight matrix; where the weight matrix is ​​a second-order matrix, and the weight matrix M... p It is a matrix of size N×C, M p A set consisting of column vectors m, where m = {m c}(c=1,…,C).

[0067] Based on the weight matrix corresponding to the convolutional layer, the mean of the weight values ​​in each column is calculated. Using the mean of the weight values ​​in each column as the element value of the corresponding column in the row vector, a row vector is obtained. This row vector is determined as the first feature vector of the global representation of the convolutional layer, where the size of the row vector is 1×C, and the row vector V={mean(m c (c = 1, ..., C).

[0068] S103: The first feature vector corresponding to each convolutional layer is concatenated to obtain the target feature vector of the global representation of the target image.

[0069] Based on the first feature vector corresponding to each convolutional layer, the electronic device performs concatenation processing on each first feature vector. Since the first feature vector is a row vector, the last element of each row of the first feature vector is connected to an element of the next first feature vector to obtain the target feature vector of the global representation of the target image. Therefore, the target feature vector is also a row vector. For example, the two first feature vectors that were originally 1 row and n columns are concatenated into a feature vector of 1 row and 2n columns.

[0070] In this embodiment of the invention, the target image is acquired and input into the feature extraction model to obtain the output first feature map. A third-order first tensor is obtained based on the pixel values ​​at each position in the first feature map. The first tensor is then input into the pooling layer of the network model for pooling processing to obtain the output target vector. The target loss value is determined based on the element value of each element in the target vector. For each convolutional layer in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix for each channel of that convolutional layer, each gradient at the corresponding position in the fourth-order second tensor is determined. Based on each gradient value in the second-order tensor at each position of the second tensor, the weight value at the corresponding position in the weight matrix is ​​determined. The mean value of each column in the weight matrix is ​​calculated to obtain the first feature vector representing the global representation of that convolutional layer. The first feature vectors corresponding to each convolutional layer are then concatenated. The method obtains the target feature vector of the global representation of the target image. Since the gradient of the model parameters is used as the feature in this invention, and the second tensor of the fourth order composed of each gradient is determined, the weight value of the corresponding position in the weight value matrix is ​​determined based on each gradient in the second tensor of each position in the second tensor. The feature value matrix is ​​then aggregated into the first feature vector of the global representation of the image in the convolutional layer. The first feature vectors corresponding to each convolutional layer are concatenated to obtain the target feature vector of the global representation of the target image. Thus, image retrieval can be achieved based on the feature vector of the global representation of the image. Furthermore, this method does not require any additional modules or special training processes and can be directly applied to any fine-tuned or pre-trained neural network model used to obtain the feature vector of the global representation of the image, thereby exploring the potential of the neural network model and further approaching the upper bound of the neural network model.

[0071] Example 2:

[0072] To determine the weight matrix, based on the above embodiments, in this embodiment of the invention, determining the weight value at the corresponding position in the weight matrix based on each gradient of the second-order tensor at each position in the second tensor includes:

[0073] For each position of the second tensor in the second tensor, if the row and column of the second tensor are both preset values, then the absolute value of the gradient in the second tensor is determined as the weight value of the corresponding position. If the row or column of the second tensor is not a preset value, then the difference between the maximum gradient and the minimum gradient in the second tensor is determined, and the sum of the difference and the standard deviation of the gradient is determined as the weight value of the corresponding position. A weight value matrix is ​​formed according to the weight values ​​corresponding to the second tensor at each position.

[0074] For each second-order tensor in the second tensor of the convolutional layer, if the number of rows and columns in the second-order tensor is 1, the electronic device determines the absolute value of the element value as the weight value of the corresponding position of the second-order tensor based on the gradient in the second-order tensor; if the number of rows or columns in the gradient matrix is ​​not 1, the electronic device determines the maximum gradient and the minimum gradient in the second-order tensor, determines the difference between the maximum gradient and the minimum gradient based on the maximum gradient and the minimum gradient, and determines the standard deviation based on each gradient. Based on the difference between the maximum gradient and the minimum gradient and the standard deviation, the sum of the difference and the standard deviation is determined, and the sum is determined as the weight value of the corresponding position of the second-order tensor, where the weight value corresponding to the second-order tensor represents the response strength of the convolutional layer to the current input image.

[0075] Specifically, the second tensor of this convolutional layer is a fourth-order tensor G of the loss function with respect to the fourth-order tensor P, where The tensor G contains N×C×H×W elements, and the parameter matrix f for each channel of this convolutional layer... n,c It functions as a whole; therefore, each second-order tensor in the second tensor is the loss function with respect to f. n,c tensor G n,c Tensor G n,c It is a tensor containing H×W elements, where

[0076] Therefore, this electronic device will store each second-order tensor G n,c The input is given to the function that determines the weight values, where the function that determines the weight values ​​is... Get the output of each tensor G n,c The corresponding weight values, where H represents the second-order tensor G. n,c The row number in the text, W represents the second-order tensor G. n,c The number of columns, f(G) n,c ) represents the order tensor G n,c The corresponding weight value.

[0077] For each second-order tensor G in the second tensor G n,cPerforming the operation in the above formula, the second tensor G is transformed into a second-order tensor trans(G) containing N×C weight values, which is an N-row, C-column matrix of weight values; specifically, based on each second-order tensor G... n,c The position in the second tensor G corresponding to this convolutional layer, and each second-order tensor G n,c The corresponding weight values ​​are used to form a weight matrix for the convolutional layer according to their corresponding positions.

[0078] Example 3:

[0079] To convert the first feature map into a third-order first tensor, based on the above embodiments, in this embodiment of the invention, the conversion of the first feature map into a third-order first tensor according to the pixel value at each position includes:

[0080] For each channel feature map in the first feature map, the pixel value of each pixel is used as the element value of the corresponding position in the matrix of the channel feature map according to the position of each pixel in the channel feature map;

[0081] Based on the arrangement order of the feature maps composed of each channel feature map, the matrices corresponding to each channel feature map are arranged in the order of arrangement in a direction perpendicular to the plane where the matrix corresponding to each channel feature map is located, thereby transforming the first feature map into a third-order first tensor.

[0082] To improve the accuracy of determining the target loss value, the electronic device, for each channel feature map in the first feature map, determines the corresponding position of each pixel in the matrix of the channel feature map based on the position of each pixel in the channel feature map, that is, the row or column where each pixel is located in the channel feature map, and uses the pixel value of each pixel as the element value of the corresponding position in the matrix.

[0083] Based on the arrangement order of the feature maps composed of each channel feature map, the electronic device arranges the matrix corresponding to each channel feature map. Specifically, based on the plane where the matrix corresponding to the channel feature map is located, the matrix corresponding to each channel feature map is arranged in the direction perpendicular to the plane to obtain the first tensor of the third order corresponding to the first feature map as a whole.

[0084] Example 4:

[0085] To obtain a more accurate target loss value, based on the above embodiments, in this embodiment of the invention, after converting the pixel value at each position in the first feature map into a third-order first tensor, and before inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector, the method further includes:

[0086] Based on each matrix in the first tensor, determine the sum matrix of each matrix, and perform binarization on the sum matrix to obtain a binarized Boolean matrix;

[0087] The Boolean matrix is ​​copied and arranged in a direction perpendicular to the plane containing the Boolean matrix to obtain a third tensor of the same order as the first tensor;

[0088] The updated first tensor is obtained by multiplying the first tensor and the third tensor point-to-point.

[0089] To improve the accuracy of determining the target loss value, the electronic device performs matrix summation on each matrix in the first tensor to obtain a sum matrix for each matrix, and then binarizes the sum matrix to obtain a binarized Boolean matrix.

[0090] When performing binarization on the sum matrix, the electronic device first uses the mean function to take the mean of the sum matrix, mapping the sum matrix to a scalar, and then uses the broadcast function to map the scalar back to the first matrix.

[0091] If the feature extraction model is a fine-tuning model, based on the sum matrix and the element value of each element in the first matrix, if the element value of an element in the sum matrix is ​​greater than the element value of the corresponding element in the first matrix, then the element value of the corresponding element in the Boolean matrix is ​​set to 1; if the element value of an element in the sum matrix is ​​not greater than the element value of the corresponding element in the first matrix, then the element value of the corresponding element in the Boolean matrix is ​​set to 0. If the feature extraction model is a pre-trained model, after determining the Boolean matrix, an LCC operation is performed on the Boolean matrix to determine the region matrix of the largest connected region in the Boolean matrix, and the region matrix is ​​used as the updated Boolean matrix.

[0092] For example, the electronic device determines the Boolean matrix based on a pre-stored function. Where S m Let M represent each matrix in the first tensor, M represent the total number of two-dimensional matrices contained in the first tensor, and B2 represent the final Boolean matrix.

[0093] After determining the Boolean matrix, the electronic device copies the Boolean matrix, determines the direction perpendicular to the plane based on the plane where the Boolean matrix is ​​located, and arranges them in the direction perpendicular to the plane until a third tensor of the same order as the first tensor is obtained.

[0094] Based on the first and third tensors, the element values ​​of each corresponding element in the first and third tensors are multiplied point-to-point to obtain a fourth tensor of the same size as the first and third tensors. The fourth tensor is then used as the updated first tensor and input into the pooling layer of the network model for pooling processing.

[0095] Example 5:

[0096] To obtain a more accurate target loss value, based on the above embodiments, in this embodiment of the invention, the step of inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector includes:

[0097] The updated first tensor is input into the pooling layer of the network model. Global average pooling is performed on the updated first tensor to obtain the first vector after global average pooling. Global max pooling is then performed on the updated first tensor to obtain the second vector after global max pooling.

[0098] The first vector and the second vector are concatenated, and the concatenated vector is used as the target vector.

[0099] To improve the accuracy of determining the target loss value, the electronic device, based on the updated first tensor and the pre-saved network model pooling layer, inputs the updated first tensor into the average pooling layer of the network model pooling layer to perform global average pooling on the updated first tensor, obtaining a first vector after global average pooling; then inputs the updated first tensor into the max pooling layer of the network model pooling layer to perform global max pooling on the updated first tensor, obtaining a second vector after global max pooling.

[0100] Based on the first vector and the second vector, the electronic device performs a splicing process, that is, splicing the first element of the second vector after the last element of the first vector, thereby obtaining the spliced ​​vector.

[0101] To obtain an accurate target loss value, in this embodiment of the invention, after performing global average pooling on the updated first tensor to obtain a first vector after global average pooling, and before performing global max pooling on the updated first tensor to obtain a second vector after global max pooling, the method further includes:

[0102] Based on the first number of element points with set values ​​in the Boolean matrix and the second number of element points contained in each channel feature map, the ratio of the second number to the first number is determined, and the ratio is multiplied by the first vector to obtain the updated first vector.

[0103] After determining the first vector, the electronic device determines the ratio of the first number to the second number based on the first number of element points with a set value in the Boolean matrix and the second number of element points contained in each channel feature map, where the set number is 1.

[0104] The electronic device multiplies the ratio of the first quantity to the second quantity and the first vector, and determines the resulting vector as the updated first vector.

[0105] Specifically, the electronic device performs point-to-point multiplication of the first tensor T and the second tensor B3 to obtain a third tensor (B3·T). Based on the first number of elements with a value of 1 in the Boolean matrix, sum (B2==1), the second number of elements contained in each channel feature map, y×z, and the third tensor, it performs global average pooling to obtain a first vector v1. The third tensor is subjected to global max pooling to obtain the second vector v2, where v1 = GMP(B3·T); the first vector and the second vector are concatenated to obtain the concatenated vector [v1,v2], and the concatenated vector is summed to obtain the target loss value loss, where loss = sum([v1,v2]).

[0106] In this embodiment of the invention, the gradient tensor of the convolution kernel parameter tensor of the feature extraction model is used as image features and feature aggregation is performed. No additional modules or special training processes are required. It can be directly applied to any fine-tuned or pre-trained convolutional neural network model, thereby exploring the potential of the feature extraction model and further approximating the inherent upper bound of the feature extraction model.

[0107] Example 6:

[0108] To improve the accuracy of fine-grained image retrieval, based on the above embodiments, in this embodiment of the invention, before concatenating the first feature vectors corresponding to each convolutional layer, the method further includes:

[0109] The first feature vector corresponding to each convolutional layer is subjected to L2 normalization to obtain each normalized first feature vector, and the normalized first feature vectors are then concatenated.

[0110] The electronic device performs L2 normalization on the first feature vector corresponding to each convolutional layer to obtain each normalized first feature vector, and then concatenates each normalized first feature vector.

[0111] Specifically, in this embodiment of the invention, the convolutional layer...4-1-1 The corresponding first feature vector CFGA 4-1-1 and convolutional layer 4-1-2 The first feature vectors of the two layers are CFGA 4-1-2 After performing l2-normlization on each image and concatenating the results, the target feature vector CFGA+ = [l2-normlization(CFGA)](link to l2-normlization(CFGA)) is obtained as the global representation of the query image. 4-1-1 ), l2-normlization(CFGA) 4-1-2 )).

[0112] Example 7:

[0113] To achieve image retrieval of fine-grained images, embodiments of the present invention provide an image retrieval method based on the feature vector determination method of global image representation according to any of the above embodiments. Figure 2 This is a schematic diagram of an image retrieval method provided in an embodiment of the present invention. The process includes the following steps:

[0114] S201: The step of performing the above-described method for determining the feature vector of the global image representation on the image to be retrieved and each image in the retrieval database, to obtain the second feature vector of the image to be retrieved and each third feature vector of each image.

[0115] To achieve fine-grained image retrieval, the image retrieval method provided in this embodiment of the invention is applied to an electronic device, which may be a host, tablet computer, smartphone or other smart terminal device, or a server, which may be a local server or a cloud server. The electronic device and the electronic device executing the feature vector determination method may be the same device or different devices, and this embodiment of the invention does not impose any restrictions on this.

[0116] The electronic device acquires the image to be retrieved and each image in the retrieval library. The image to be retrieved may be pre-saved or sent by other electronic devices connected to the electronic device. Each image in the retrieval library is pre-saved by the electronic device. Both the image to be retrieved and each image in the retrieval library are fine-grained images. The steps of the image global representation feature vector determination method in the above embodiment are performed on the image to be retrieved and each image in the retrieval library to determine the second feature vector of the image to be retrieved and each third feature vector of each image.

[0117] S202: Based on the second feature vector and each of the third feature vectors, determine the similarity between the second feature vector and each of the third feature vectors, and sort each image in descending order of similarity.

[0118] Based on the second feature vector of the global representation of the image to be queried, and each third feature vector corresponding to each image in the pre-saved retrieval database, the electronic device determines the similarity between the second feature vector and each third feature vector. Specifically, it can calculate the cosine similarity between the second feature vector and each third feature vector, or it can calculate the Euclidean distance between the second feature vector and each third feature vector, and use the Euclidean distance as the similarity. Based on the similarity between the second feature vector and each third feature vector, each image in the retrieval database is sorted in descending order of similarity to obtain the sorted images.

[0119] Example 7:

[0120] Figure 3 This is a schematic diagram of the structure of a feature vector determination device for global image representation provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the device includes:

[0121] The acquisition module 301 is used to acquire the target image and input it into the feature extraction model to obtain the first feature map of the output;

[0122] The determining module 302 is used to convert the pixel value at each position in the first feature map into a third-order first tensor, input the first tensor into the pooling layer of the network model for pooling processing, and obtain the output target vector; determine the target loss value based on the element value of each element in the target vector; for each convolutional layer set in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix of each channel of the convolutional layer, determine each gradient at the corresponding position in the fourth-order second tensor; based on each gradient in the second-order tensor at each position in the second tensor, determine the weight value at the corresponding position in the weight value matrix; calculate the mean of each column in the weight value matrix to obtain the first feature vector of the global representation of the convolutional layer; and concatenate the first feature vectors corresponding to each convolutional layer to obtain the target feature vector of the global representation of the target image.

[0123] Further, the determining module 302 is specifically used for each position of the second tensor in the second tensor. If the rows and columns of the second tensor are both preset values, then the absolute value of the gradient in the second tensor is determined as the weight value of the corresponding position. If the rows or columns of the second tensor are not preset values, then the difference between the maximum gradient and the minimum gradient in the second tensor is determined, and the sum of the difference and the standard deviation of the gradient is determined as the weight value of the corresponding position. A weight value matrix is ​​formed according to the weight values ​​corresponding to the second tensor at each position.

[0124] Further, the determining module 302 is specifically used to, for each channel feature map in the first feature map, take the pixel value of each pixel as the element value of the corresponding position in the matrix of the channel feature map according to the position of each pixel in the channel feature map; and arrange the matrices corresponding to each channel feature map in the order in which the feature map is composed of the channel feature maps according to the order in which the matrix of each channel feature map is located in a direction perpendicular to the plane where the matrix of each channel feature map is located, so as to obtain the first feature map transformed into a third-order first tensor.

[0125] Furthermore, the determining module 302 is also used to, after converting the first tensor into a third-order first tensor based on the pixel values ​​at each position in the first feature map, and before inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector, determine the sum matrix of each matrix in the first tensor, perform binarization processing on the sum matrix to obtain a binarized Boolean matrix; copy the Boolean matrix and arrange it in a direction perpendicular to the plane where the Boolean matrix is ​​located to obtain a third tensor of the same size as the first tensor; and perform point-to-point multiplication of the first tensor and the third tensor to obtain the updated first tensor.

[0126] Further, the determining module 302 is specifically used to input the updated first tensor into the pooling layer of the network model, perform global average pooling on the updated first tensor to obtain a first vector after global average pooling, perform global max pooling on the updated first tensor to obtain a second vector after global max pooling, and concatenate the first vector and the second vector to use the concatenated vector as the target vector.

[0127] Furthermore, the determining module 302 is also used to determine the ratio of the second number to the first number based on the first number of element points with set values ​​in the Boolean matrix and the second number of element points contained in each channel feature map, and multiply the ratio by the first vector to obtain the updated first vector.

[0128] Furthermore, the determining module 302 is also used to perform L2 normalization on the first feature vector corresponding to each convolutional layer before performing the concatenation process on the first feature vector corresponding to each convolutional layer, to obtain each normalized first feature vector, and to perform concatenation process on each normalized first feature vector.

[0129] Example 8:

[0130] Figure 4 This is a schematic diagram of the structure of an image retrieval device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the device includes:

[0131] The determination module 401 is used to perform the above-described image global representation feature vector determination method on the image to be retrieved and each image in the retrieval library to obtain the second feature vector of the image to be retrieved and each third feature vector of each image;

[0132] The retrieval module 402 is used to determine the similarity between the second feature vector and each third feature vector based on the second feature vector and each third feature vector, and to sort each image in descending order of similarity.

[0133] Example 9:

[0134] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Based on the above embodiments, this application also provides an electronic device, such as... Figure 5 As shown, it includes: processor 501, communication interface 502, memory 503 and communication bus 504, wherein processor 501, communication interface 502 and memory 503 communicate with each other through communication bus 504.

[0135] The memory 503 stores a computer program, which, when executed by the processor 501, causes the processor 501 to perform the following steps:

[0136] The target image is acquired and input into the feature extraction model to obtain the first feature map. Based on the pixel value of each position in the first feature map, a third-order first tensor is obtained. The first tensor is input into the pooling layer of the network model for pooling processing to obtain the target vector. The target loss value is determined based on the element value of each element in the target vector.

[0137] For each convolutional layer in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix of each channel of the convolutional layer, each gradient in the corresponding position of the second tensor of the fourth order is determined. Based on each gradient in the second tensor of each position in the second tensor, the weight value in the corresponding position of the weight value matrix is ​​determined. The mean of each column in the weight value matrix is ​​calculated to obtain the first feature vector of the global representation of the convolutional layer.

[0138] The first feature vector corresponding to each convolutional layer is concatenated to obtain the target feature vector of the global representation of the target image.

[0139] Furthermore, the processor 501 is also configured such that the weight value matrix is ​​a second-order matrix, and the first feature vector and the target feature vector are both row vectors.

[0140] Further, the processor 501 is specifically used to determine the weight value at the corresponding position in the weight value matrix for each gradient of the second-order tensor based on each position in the second tensor, including:

[0141] For each position of the second tensor in the second tensor, if the row and column of the second tensor are both preset values, then the absolute value of the gradient in the second tensor is determined as the weight value of the corresponding position. If the row or column of the second tensor is not a preset value, then the difference between the maximum gradient and the minimum gradient in the second tensor is determined, and the sum of the difference and the standard deviation of the gradient is determined as the weight value of the corresponding position. A weight value matrix is ​​formed according to the weight values ​​corresponding to the second tensor at each position.

[0142] Further, the processor 501 is specifically used to convert the pixel value at each position in the first feature map into a third-order first tensor, including:

[0143] For each channel feature map in the first feature map, the pixel value of each pixel is used as the element value of the corresponding position in the matrix of the channel feature map according to the position of each pixel in the channel feature map;

[0144] Based on the arrangement order of the feature maps composed of each channel feature map, the matrices corresponding to each channel feature map are arranged in the order of arrangement in a direction perpendicular to the plane where the matrix corresponding to each channel feature map is located, thereby transforming the first feature map into a third-order first tensor.

[0145] Furthermore, the processor 501 is further configured to, after converting the pixel value at each position in the first feature map into a third-order first tensor, and before inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector, the method further includes:

[0146] Based on each matrix in the first tensor, determine the sum matrix of each matrix, and perform binarization on the sum matrix to obtain a binarized Boolean matrix;

[0147] The Boolean matrix is ​​copied and arranged in a direction perpendicular to the plane containing the Boolean matrix to obtain a third tensor of the same order as the first tensor;

[0148] The updated first tensor is obtained by multiplying the first tensor and the third tensor point-to-point.

[0149] Further, the processor 501 is specifically used to input the first tensor into the pooling layer of the network model for pooling processing, and obtain the output target vector, including:

[0150] The updated first tensor is input into the pooling layer of the network model. Global average pooling is performed on the updated first tensor to obtain the first vector after global average pooling. Global max pooling is then performed on the updated first tensor to obtain the second vector after global max pooling.

[0151] The first vector and the second vector are concatenated, and the concatenated vector is used as the target vector.

[0152] Furthermore, the processor 501 is further configured to perform global average pooling on the updated first tensor to obtain a first vector after global average pooling, and before performing global max pooling on the updated first tensor to obtain a second vector after global max pooling, the method further includes:

[0153] Based on the first number of element points with set values ​​in the Boolean matrix and the second number of element points contained in each channel feature map, the ratio of the second number to the first number is determined, and the ratio is multiplied by the first vector to obtain the updated first vector.

[0154] Furthermore, before the processor 501 performs the concatenation process on the first feature vector corresponding to each convolutional layer, the method further includes:

[0155] The first feature vector corresponding to each convolutional layer is subjected to L2 normalization to obtain each normalized first feature vector, and the normalized first feature vectors are then concatenated.

[0156] Alternatively, the processor 501 may also perform the following steps:

[0157] The steps of the above-described image global representation feature vector determination method are performed on the image to be retrieved and each image in the retrieval database to obtain the second feature vector of the image to be retrieved and each third feature vector of each image;

[0158] Based on the second feature vector and each of the third feature vectors, the similarity between the second feature vector and each of the third feature vectors is determined, and each image is sorted in descending order of similarity.

[0159] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0160] Communication interface 502 is used for communication between the above-mentioned electronic device and other devices.

[0161] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0162] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0163] Example 9:

[0164] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by a processor. When the program is run on the processor, the processor executes the following steps:

[0165] The target image is acquired and input into the feature extraction model to obtain the first feature map. Based on the pixel value of each position in the first feature map, a third-order first tensor is obtained. The first tensor is input into the pooling layer of the network model for pooling processing to obtain the target vector. The target loss value is determined based on the element value of each element in the target vector.

[0166] For each convolutional layer in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix of each channel of the convolutional layer, each gradient in the corresponding position of the second tensor of the fourth order is determined. Based on each gradient in the second tensor of each position in the second tensor, the weight value in the corresponding position of the weight value matrix is ​​determined. The mean of each column in the weight value matrix is ​​calculated to obtain the first feature vector of the global representation of the convolutional layer.

[0167] The first feature vector corresponding to each convolutional layer is concatenated to obtain the target feature vector of the global representation of the target image.

[0168] Furthermore, the weight value matrix is ​​a second-order matrix, and the first feature vector and the target feature vector are both row vectors.

[0169] Further, determining the weight value at the corresponding position in the weight value matrix based on each gradient of the second-order tensor at each position in the second tensor includes:

[0170] For each position of the second tensor in the second tensor, if the row and column of the second tensor are both preset values, then the absolute value of the gradient in the second tensor is determined as the weight value of the corresponding position. If the row or column of the second tensor is not a preset value, then the difference between the maximum gradient and the minimum gradient in the second tensor is determined, and the sum of the difference and the standard deviation of the gradient is determined as the weight value of the corresponding position. A weight value matrix is ​​formed according to the weight values ​​corresponding to the second tensor at each position.

[0171] Further, the step of converting the pixel value at each position in the first feature map into a third-order first tensor includes:

[0172] For each channel feature map in the first feature map, the pixel value of each pixel is used as the element value of the corresponding position in the matrix of the channel feature map according to the position of each pixel in the channel feature map;

[0173] Based on the arrangement order of the feature maps composed of each channel feature map, the matrices corresponding to each channel feature map are arranged in the order of arrangement in a direction perpendicular to the plane where the matrix corresponding to each channel feature map is located, thereby transforming the first feature map into a third-order first tensor.

[0174] Furthermore, after converting the pixel values ​​at each position in the first feature map into a third-order first tensor, and before inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector, the method further includes:

[0175] Based on each matrix in the first tensor, determine the sum matrix of each matrix, and perform binarization on the sum matrix to obtain a binarized Boolean matrix;

[0176] The Boolean matrix is ​​copied and arranged in a direction perpendicular to the plane containing the Boolean matrix to obtain a third tensor of the same order as the first tensor;

[0177] The updated first tensor is obtained by multiplying the first tensor and the third tensor point-to-point.

[0178] Further, the step of inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector includes:

[0179] The updated first tensor is input into the pooling layer of the network model. Global average pooling is performed on the updated first tensor to obtain the first vector after global average pooling. Global max pooling is then performed on the updated first tensor to obtain the second vector after global max pooling.

[0180] The first vector and the second vector are concatenated, and the concatenated vector is used as the target vector.

[0181] Further, after performing global average pooling on the updated first tensor to obtain a first vector after global average pooling, and before performing global max pooling on the updated first tensor to obtain a second vector after global max pooling, the method further includes:

[0182] Based on the first number of element points with set values ​​in the Boolean matrix and the second number of element points contained in each channel feature map, the ratio of the second number to the first number is determined, and the ratio is multiplied by the first vector to obtain the updated first vector.

[0183] Furthermore, before concatenating the first feature vectors corresponding to each convolutional layer, the method further includes:

[0184] The first feature vector corresponding to each convolutional layer is subjected to L2 normalization to obtain each normalized first feature vector, and the normalized first feature vectors are then concatenated.

[0185] Alternatively, the processor may perform the following steps during execution:

[0186] The steps of the above-described image global representation feature vector determination method are performed on the image to be retrieved and each image in the retrieval database to obtain the second feature vector of the image to be retrieved and each third feature vector of each image;

[0187] Based on the second feature vector and each of the third feature vectors, the similarity between the second feature vector and each of the third feature vectors is determined, and each image is sorted in descending order of similarity.

[0188] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0192] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for determining feature vectors for global image representation, characterized in that, The method includes: The target image is acquired and input into the feature extraction model to obtain the first feature map. Based on the pixel value of each position in the first feature map, a third-order first tensor is obtained. The first tensor is input into the pooling layer of the network model for pooling processing to obtain the target vector. The target loss value is determined based on the element value of each element in the target vector. For each convolutional layer in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix of each channel of the convolutional layer, each gradient in the corresponding position of the second tensor of the fourth order is determined. Based on each gradient in the second tensor of each position in the second tensor, the weight value in the corresponding position of the weight value matrix is ​​determined. The mean of each column in the weight value matrix is ​​calculated to obtain the first feature vector of the global representation of the convolutional layer. The first feature vector corresponding to each convolutional layer is concatenated to obtain the target feature vector of the global representation of the target image; Wherein, the weight value matrix is ​​a second-order matrix, and the first feature vector and the target feature vector are both row vectors; Determining the weight value at the corresponding position in the weight value matrix based on each gradient of the second-order tensor at each position in the second tensor includes: For each position of the second tensor in the second tensor, if the row and column of the second tensor are both preset values, then the absolute value of the gradient in the second tensor is determined as the weight value of the corresponding position. If the row or column of the second tensor is not a preset value, then the difference between the maximum gradient and the minimum gradient in the second tensor is determined, and the sum of the difference and the standard deviation of the gradient is determined as the weight value of the corresponding position. A weight value matrix is ​​formed according to the weight values ​​corresponding to the second tensor at each position.

2. The method according to claim 1, characterized in that, The step of converting the pixel value at each position in the first feature map to obtain the third-order first tensor includes: For each channel feature map in the first feature map, the pixel value of each pixel is used as the element value of the corresponding position in the matrix of the channel feature map according to the position of each pixel in the channel feature map; Based on the arrangement order of the feature maps composed of each channel feature map, the matrices corresponding to each channel feature map are arranged in the order of arrangement in a direction perpendicular to the plane where the matrix corresponding to each channel feature map is located, thereby transforming the first feature map into a third-order first tensor.

3. The method according to claim 1, characterized in that, After converting the pixel values ​​at each position in the first feature map into a third-order first tensor, and before inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector, the method further includes: Based on each matrix in the first tensor, determine the sum matrix of each matrix, and perform binarization on the sum matrix to obtain a binarized Boolean matrix; The Boolean matrix is ​​copied and arranged in a direction perpendicular to the plane containing the Boolean matrix to obtain a third tensor of the same order as the first tensor; The updated first tensor is obtained by multiplying the first tensor and the third tensor point-to-point.

4. The method according to claim 3, characterized in that, The step of inputting the first tensor into the pooling layer of the network model for pooling processing to obtain the output target vector includes: The updated first tensor is input into the pooling layer of the network model. Global average pooling is performed on the updated first tensor to obtain the first vector after global average pooling. Global max pooling is then performed on the updated first tensor to obtain the second vector after global max pooling. The first vector and the second vector are concatenated, and the concatenated vector is used as the target vector.

5. The method according to claim 4, characterized in that, After performing global average pooling on the updated first tensor to obtain a first vector after global average pooling, and before performing global max pooling on the updated first tensor to obtain a second vector after global max pooling, the method further includes: Based on the first number of element points with set values ​​in the Boolean matrix and the second number of element points contained in each channel feature map of the first feature map, the ratio of the second number to the first number is determined, and the ratio is multiplied by the first vector to obtain the updated first vector.

6. The method according to claim 1, characterized in that, Before concatenating the first feature vectors corresponding to each convolutional layer, the method further includes: The first feature vector corresponding to each convolutional layer is subjected to L2 normalization to obtain each normalized first feature vector, and the normalized first feature vectors are then concatenated.

7. An image retrieval method for determining the feature vector of a global image representation according to any one of claims 1-6, characterized in that, The method includes: The steps of the above-described image global representation feature vector determination method are performed on the image to be retrieved and each image in the retrieval database to obtain the second feature vector of the image to be retrieved and each third feature vector of each image; Based on the second feature vector and each of the third feature vectors, the similarity between the second feature vector and each of the third feature vectors is determined, and each image is sorted in descending order of similarity.

8. A feature vector determination device for global image representation, characterized in that, The device includes: The acquisition module is used to acquire the target image and input it into the feature extraction model to obtain the first feature map as output. The determination module is used to convert the pixel values ​​at each position in the first feature map into a third-order first tensor, input the first tensor into the pooling layer of the network model for pooling processing, and obtain the output target vector; determine the target loss value based on the element value of each element in the target vector; for each convolutional layer in the feature extraction model, based on the target loss value and the position of the parameters in each row and column of the parameter matrix of each channel of the convolutional layer, determine each gradient at the corresponding position in the fourth-order second tensor; based on each gradient in the second-order tensor at each position of the second tensor, determine the weight value at the corresponding position in the weight value matrix; calculate the mean of each column in the weight value matrix to obtain the first feature vector of the global representation of the convolutional layer; and process the first feature vector corresponding to each convolutional layer. Row concatenation is performed to obtain the target feature vector of the global representation of the target image; wherein, the weight value matrix is ​​a second-order matrix, and the first feature vector and the target feature vector are both row vectors; the step of determining the weight value of the corresponding position in the weight value matrix based on each gradient of the second-order tensor at each position in the second tensor includes: for each position of the second-order tensor in the second tensor, if the row and column of the second-order tensor are both preset values, then the absolute value of the gradient in the second-order tensor is determined as the weight value of the corresponding position; if the row or column of the second-order tensor is not a preset value, then the difference between the maximum gradient and the minimum gradient in the second-order tensor is determined, and the sum of the difference and the standard deviation of the gradient is determined as the weight value of the corresponding position; and a weight value matrix is ​​formed according to the weight values ​​corresponding to the second-order tensor at each position.

9. An image retrieval device, characterized in that, The device includes: The determination module is used to perform the step of determining the feature vector of the global representation of the image as described in any one of claims 1-6 for the image to be retrieved and each image in the retrieval library, to obtain the second feature vector of the image to be retrieved and each third feature vector of each image; The retrieval module is used to determine the similarity between the second feature vector and each third feature vector based on the second feature vector and each third feature vector, and to sort each image in descending order of similarity.

10. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the feature vector determination method for global image representation as described in any one of claims 1-6, or the steps of the image retrieval method as described in claim 7.

11. A computer-readable storage medium, characterized in that, It stores a computer program executable by a processor, which, when run on the processor, causes the processor to perform the steps of the feature vector determination method for global image representation as described in any one of claims 1-6, or the steps of the image retrieval method as described in claim 7.

Images