A method and apparatus for image re-identification feature enhancement

Abstract

This invention discloses a method and apparatus for image re-identification feature enhancement. The method includes step 1, model construction: (1) backbone network layer: the backbone network uses HRNet to extract features; (2) feature representation layer: after the image passes through the backbone network layer, preliminary features are obtained, followed by a flattening operation; (3) feature enhancement layer: the feature representation matrix M obtained through the backbone network layer and the feature representation layer is fed into two feature enhancement layers to enhance the features; step 2, model training; step 3, predicting whether the image is in the base database. By using the feature enhancement layer, the base database images and the image to be queried are combined into a batch input to achieve the purpose of enhancing the features, making the features more discriminative in the base database. The feature enhancement layer is used to enhance each feature according to the features of a batch to improve the discriminativeness between the features of that batch. The base database and the query image are enhanced together to improve the accuracy and recall of the recognition.

Description

Technical Field

[0001] This invention relates to the field of face recognition and re-recognition research, specifically to a method and apparatus for enhancing image re-recognition features. Background Technology

[0002] Currently, facial recognition technology has many limitations in general scenarios, especially when performing large-scale facial recognition, which can lead to many false alarms. This is because facial recognition systems encode each facial image into a fixed-length feature vector, so the similarity between different facial images is completely fixed. This results in multiple different people potentially being located in close proximity in the feature space, making it impossible for the model to distinguish them. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides a method and apparatus for image re-identification feature enhancement, which enhances both the base database and the query image to improve recognition accuracy and recall. The technical solution is as follows:

[0004] This invention provides a method for image re-identification feature enhancement, which mainly includes the following steps:

[0005] Step 1, build the model;

[0006] Backbone network layer: The backbone network uses HRNet to extract features;

[0007] Feature representation layer:

[0008] After the image passes through the backbone network layer, preliminary features are obtained. Then, a flattening operation is performed, followed by a fully connected layer. The output dimension K of the fully connected layer becomes the final feature dimension, resulting in the final image feature representation vector F: F∈R. K ;

[0009] Feature enhancement layer:

[0010] Images in a batch of size N are processed through a backbone network layer and a feature representation layer to obtain a feature representation matrix M∈R. N*K Each row of M is used as a feature representation vector. Then, M is transposed, and each column of M can be regarded as a component sampling vector of the feature representation vector. Then, M is fed into two feature enhancement layers to strengthen the features. The feature enhancement layer is calculated as follows:

[0011]

[0012] L = concat(H1, H2, ..., H...) i …, H m W (O) ;

[0013] Where m is the width of the reinforcing layer, W i (Q) W i (K) W i (V) W (O) The learnable weight matrix is ​​i = 1, 2, 3, ..., m, d k As a constant, after Q i K i T Next, the pairwise feature component sampling vectors in M ​​are multiplied by an inner product, meaning each component incorporates information from the other components. The resulting weights are then returned to V. i The corresponding components are enhanced to obtain new features, where each component of the feature is enhanced and information from very similar feature components is excluded in order to distinguish different targets to the greatest extent, and finally the feature matrix L is obtained.

[0014] Step 2, Model Training;

[0015] The loss function during training is calculated as follows:

[0016]

[0017]

[0018] Among them W (C) W is a learnable weight matrix. x (C) For matrix W (C) The xth line, L y T y is the y-th column of matrix L; i Let cos(θ) represent the true class of the i-th sample. x，y ) represents the cosine of the angle between category x and the y-th sample, a is the interval hyperparameter, and s is the radius of the feature hypersphere.

[0019] Step 3: Predict whether the image is in the base database;

[0020] After training the model using step 2, put all the base database images and query images together to form a batch of size B;

[0021] If B < N, then fill with a completely black image until B = N. After extracting features from the batch using steps 1-2, obtain the feature matrix L and calculate the similarity matrix S: S = LL'.

[0022] Find the highest similarity smax and its corresponding base library l:

[0023] l, smax=argmax(S[end, 1: end-1]);

[0024] Where S[end, 1: end-1] represents the vector consisting of the first end-1 elements of the last row of matrix S;

[0025] If smax is greater than the threshold, output base library 1; otherwise, output "match failed".

[0026] If N <= B < 2N, first take the first N-1 base images and form a batch with the query image, compare them, then replace the base images with the lowest similarity in the first BN-1 base images with base images N to B, and compare them again. If the highest similarity between the two comparisons is greater than the threshold, output the corresponding base image; otherwise, output "match failed".

[0027] If B >= 2N, divide the base database into 2B / (N-1) blocks. Each time, take two blocks and add the query image to form a batch. If the batch is insufficient, fill it with a completely black image. Continue until any two blocks have been matched together. If the highest similarity among these matches is greater than the threshold, return the corresponding base database; otherwise, return a matching failure.

[0028] Preferably, the backbone network described in step 1 uses HRNet to extract features. HRNet obtains a relatively complete information abstraction by fusing image spatial information and semantic information.

[0029] Preferably, the output dimension K in step 1 is 512.

[0030] Preferably, the radius s of the feature hypersphere in step 2 is 64.

[0031] Compared with existing technologies, one of the above technical solutions has the following beneficial effects: By employing a feature enhancement layer, the base database images and the query image are combined into a batch input to enhance the features, making the features more discriminative within the base database. The feature enhancement layer strengthens each feature based on the features of a batch, thereby improving the discriminative power among the features in that batch. Strengthening the base database and the query image together improves the accuracy and recall of the recognition. Attached Figure Description

[0032] Figure 1 This is a schematic diagram illustrating the extraction of basic features according to an embodiment of the present disclosure;

[0033] Figure 2 This is a schematic diagram illustrating a feature enhancement layer for strengthening features according to an embodiment of this disclosure. Detailed Implementation

[0034] To clarify the technical solutions and working principles of the present invention, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings. All the above-described optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be elaborated upon here. The terms "step 1," "step 2," "step 3," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those described herein.

[0035] In a first aspect, embodiments of this disclosure provide a method for image re-identification feature enhancement, with appended... Figure 1 This is a schematic diagram illustrating the extraction of basic features according to an embodiment of the present disclosure. Figure 2 This invention provides a schematic diagram of a feature enhancement layer for enhancing features according to an embodiment of the present disclosure. Combining these two diagrams, the main steps include:

[0036] This method mainly includes the following steps:

[0037] Step 1, build the model;

[0038] Backbone network layer:

[0039] The backbone network uses HRNet to extract features. HRNet effectively integrates spatial and semantic information of the image to obtain a relatively complete information abstraction.

[0040] Feature representation layer:

[0041] After the image passes through the backbone network layer, preliminary features are obtained. Then, a flattening operation is performed, followed by a fully connected layer. The output dimension K of the fully connected layer is the final feature dimension (preferably K = 512), resulting in the final image feature representation vector F: F ∈ R. K ;

[0042] Feature enhancement layer:

[0043] Images in a batch of size N are processed through a backbone network layer and a feature representation layer to obtain a feature representation matrix M∈R. N*K Each row of M serves as a feature representation vector, such as Figure 1 As shown, M is then transposed. Each column of M can be considered as a component sampling vector of the feature representation vector. M is then fed into two feature enhancement layers to strengthen the features. The feature enhancement layer calculation method is as follows:

[0044]

[0045] L = concat(H1, H2, ..., H...) i …, H m W (O) ;

[0046] Where m is the width of the reinforcing layer, preferably m = 3, W i (Q) W i (K) W i (V) W (O) Let be a learnable weight matrix, i = 1, 2, 3, ..., m, d k As a constant, the preferred d k =64, after Q i K i T Next, the pairwise feature component sampling vectors in M ​​are multiplied by an inner product, meaning each component incorporates information from the other components. The resulting weights are then returned to V. i The corresponding components are enhanced to obtain new features. Each component in the feature matrix is ​​enhanced, and information from very similar feature components is eliminated to maximize the differentiation of different targets. The final feature matrix L is obtained, such as... Figure 2 As shown.

[0047] Step 2, Model Training;

[0048] The loss function during training is calculated as follows:

[0049]

[0050]

[0051] Among them W (C) W is a learnable weight matrix. x (C) For matrix W (C) The xth line, L y T y is the y-th column of matrix L; i Let cos(θ) represent the true class of the i-th sample. x，y ) represents the cosine of the angle between category x and the y-th sample, a is the interval hyperparameter, preferably a = 0.4, and s is the radius of the feature hypersphere, preferably s = 64.

[0052] Step 3: Predict whether the image is in the base database;

[0053] After training the model using step 2, put all the base database images and query images together to form a batch of size B;

[0054] If B < N, then fill with a completely black image (RBG = #00000000) until B = N. After extracting features from the batch using steps 1-2, obtain the feature matrix L and calculate the similarity matrix S: S = LL'.

[0055] Find the highest similarity smax and its corresponding base library l:

[0056] l, smax=argmax(S[end, 1: end-1]);

[0057] Where S[end,1:end-1] represents the vector consisting of the first end-1 elements of the last row of matrix S;

[0058] If smax is greater than the threshold, output the base library l; otherwise, output "match failed".

[0059] If N <= B < 2N, first take the first N-1 base images and form a batch with the query image, compare them, then replace the base images with the lowest similarity in the first BN-1 comparisons with base images N to B, and compare them again. If the highest similarity between the two comparisons is greater than the threshold, output the corresponding base image; otherwise, output "match failed".

[0060] If B >= 2N, divide the base database into 2B / (N-1) blocks. Each time, take two blocks and add the query image to form a batch. If the batch is insufficient, fill it with a completely black image. Continue until any two blocks have been matched together. If the highest similarity among these matches is greater than the threshold, return the corresponding base database; otherwise, return a matching failure.

[0061] Secondly, embodiments of this disclosure provide an apparatus for image re-identification feature enhancement. Based on the same technical concept, this apparatus can implement or execute any one of the image re-identification feature enhancement methods in all possible implementations. The apparatus includes a construction unit, a training unit, and a prediction unit.

[0062] The building unit is used to perform step 1 of an image re-identification feature enhancement method according to any one of all possible implementations.

[0063] The training unit is used to perform step 2 of an image re-identification feature enhancement method according to any one of the possible implementations.

[0064] The prediction unit is used to perform step 3 of an image re-identification feature enhancement method according to any one of all possible implementations.

[0065] It should be noted that the image re-identification feature enhancement apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when executing an image re-identification feature enhancement method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image re-identification feature enhancement apparatus and the image re-identification feature enhancement method embodiment provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiment, which will not be repeated here.

[0066] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without improvement or equivalent substitution, are all within the protection scope of the present invention.

Claims

1. A method for image re-identification feature enhancement, characterized in that, The method mainly comprises the following steps: Step 1, model construction; The backbone network layer: the backbone network uses hrnet to extract features; The feature representation layer: After the image passes through the backbone network layer, the preliminary feature is obtained, then the flatten operation is performed, and then a fully connected layer is used. The output dimension K of the fully connected layer is the final feature dimension, and the final image feature representation vector F is obtained: F∈R K ; The feature enhancement layer: After the image in a batch of size N is processed by the backbone network layer and the feature representation layer, a feature representation matrix M ∈ R N*K where each row of M is a feature representation vector, then the transpose operation is performed on M, and each column of M can be regarded as a component sampling vector of the feature representation vector, then M is sent to two layers of feature enhancement layers to strengthen the features, and the feature enhancement layer is calculated as follows: L = concat(H1, H2,..., H i ..., H m )W (O) ; where m is the width of the reinforcement layer, W i (Q) ,W i (K) ,W i (V) ,W (O) is a learnable weight matrix, i = 1, 2, 3, …, m, d k is a constant, after going through Q i K i T After that, the inner product of the two feature component sampling vectors in M is taken, that is, each component fuses the information of other components, and then the fused weight is returned to V i The corresponding component is reinforced, and a new feature L is obtained, where each component in the feature is reinforced, the information of very similar feature components is excluded, and the different targets are distinguished to the greatest extent, and finally the feature matrix L is obtained. Step 2, model training; The loss function Loss during training is calculated as follows: where W (C) is a learnable weight matrix, W x (C) is the xth row of the matrix W (C) , L y T is the yth column of the matrix L; y i represents the true class of the ith sample, cos(θ x,y ) represents the cosine value of the angle between class x and the yth sample, a is a margin hyperparameter, and s is a feature hypersphere radius. Step 3, predicting whether the image is in the base library or not; After the model is trained in step 2, all base library images and query images are put together to form a batch with a size of B; If B < N, black images are used to fill until B = N, and after the features of the batch are extracted using steps 1-2, a feature matrix L is obtained, and a similarity matrix S is calculated: S = LL'; Find the highest similarity smax and the corresponding base library l: l, smax = argmax(S[end, 1:end-1]); Where S[end, 1:end-1] represents a vector composed of the last row of the matrix S and the first end-1 elements; If smax is greater than the threshold value, output the base library l, otherwise output a match failure; If N <= B < 2N, first take the first N-1 base libraries and the query image to form a batch, then replace the first B-N-1 base libraries with the Nth to B base libraries, and then compare again, if the highest similarity of the two comparisons is greater than the threshold value, output the corresponding base library, otherwise output a match failure; If B >= 2N, divide the base library into 2B / (N-1) blocks, take two blocks each time to form a batch with the query image, and fill the insufficient part with black images, until any two blocks are matched together, if the highest similarity in these matches is greater than the threshold value, return the corresponding base library, otherwise return a match failure. 2.The method of claim 1, wherein, The backbone network in step 1 uses hrnet to extract features, which obtains relatively complete information abstraction by fusing image spatial information and semantic information.

3. The method of claim 1, wherein, The output dimension K in step 1 is 512.

4. The method of claim 1-3, wherein, The feature hypersphere radius s in step 2 is 64.

5. An apparatus for image re-identification feature enhancement, the apparatus comprising: The device can implement the image re-identification feature enhancement method of any one of claims 1-4.

Images