Image feature extraction, retrieval method, device, equipment and storage medium

Abstract

The present invention belongs to the field of video processing, and specifically provides an image feature extraction, retrieval method, device, equipment and storage medium, aiming to solve the problem of how to effectively extract image features, and accurately and quickly find the target image corresponding to the retrieval image in the target base library according to the image features. To this end, the method of the present invention includes: using a re-recognition model with two-way outputs including a self-attention structure network to extract standard features and hash features of the image; using hash features for initial retrieval to reduce the retrieval range; and then using standard features for secondary retrieval in the reduced target base library to obtain target images within a more accurate range. Applying the method of the present invention, the feature effect can be further improved through the self-attention structure network in feature extraction; the method of joint retrieval using binary hash features and standard features has the advantages of accurate retrieval and fast calculation speed, and is very suitable for ultra-large-scale base library retrieval tasks.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and specifically provides an image feature extraction, retrieval method, device, equipment and storage medium. Background Art

[0002] With the advent of the big data era, it is becoming increasingly convenient to obtain information and data. A large amount of video and image data provides strong data support for intelligent information management systems such as security systems and intelligent search engines, enabling these systems to operate effectively. In systems such as intelligent information management, pedestrian features captured by the camera are usually extracted based on the re-identification model and used to build a big data pedestrian database. When a specific pedestrian needs to be retrieved and analyzed, the pedestrian features are usually compared one by one with all the pedestrian features in the database to obtain the complete movement trajectory of the pedestrian in a cross-camera situation.

[0003] With the development and progress of deep learning, the re-identification model method based on deep learning has gained more and more favor in the industry. In order to improve the accuracy of retrieval matching, the capacity of the re-identification model is usually increased. However, the increase in the model will inevitably lead to an increase in the related computing processing overhead. In practical applications, the speed of image retrieval is very important. For example, in the field of intelligent information management, in order to determine the identity, location and other information of a person as quickly as possible, the security system must accurately and quickly find the complete movement trajectory of the pedestrian. Therefore, how to effectively extract pedestrian image features and search in the underlying database based on pedestrian image features, while ensuring accuracy and reducing retrieval time has become an urgent problem to be solved.

[0004] Accordingly, a new solution is needed in the art to solve the above problems. Summary of the invention

[0005] The present invention aims to solve the above technical problem, namely, how to effectively extract image features and accurately and quickly find the target image corresponding to the image in the target base library based on the image features.

[0006] In a first aspect, the present invention provides an image feature extraction method, the method comprising:

[0007] Get the image to be processed;

[0008] Inputting the image to be processed into a trained target re-identification model to obtain standard features and hash features of the image to be processed, wherein the target re-identification model includes a re-identification model basic network, a self-attention structure network, a first output network and a second output network;

[0009] The method of “inputting the image to be processed into a trained target re-identification model to obtain the standard features and hash features of the image to be processed” includes:

[0010] Using the re-identification model basic network to extract features of the image to be processed, to obtain a first tensor of the image to be processed;

[0011] Processing the first tensor using the self-attention structure network to obtain a seventh tensor of the image to be processed;

[0012] Inputting the seventh tensor into the first output network for processing to obtain the standard features of the image to be processed;

[0013] The seventh tensor is input into the second output network for processing to obtain the hash feature of the image to be processed.

[0014] In one embodiment of the above-mentioned image feature extraction method, the self-attention structure network includes a first convolution kernel, a second convolution kernel, a third convolution kernel, a first operation unit, a second operation unit and a logistic regression unit.

[0015] The method of “processing the first tensor using the self-attention structure network to obtain the seventh tensor of the image to be processed” includes:

[0016] Using the first convolution kernel, the second convolution kernel, and the third convolution kernel to process the first tensor respectively, to obtain a second tensor, a third tensor vector, and a sixth tensor respectively;

[0017] Inputting the second tensor and the third tensor into the first operation unit for processing to obtain a fourth tensor;

[0018] Inputting the fourth tensor into the logistic regression unit for processing to obtain a fifth tensor;

[0019] The fifth tensor and the sixth tensor are input into the second operation unit for processing to obtain the seventh tensor, which is the output of the self-attention network structure.

[0020] In one embodiment of the above-mentioned image feature extraction method, the first output network includes a fourth convolution kernel, a first pooling layer, and a first batch normalization layer; the second output network includes a fifth convolution kernel, a second pooling layer, a second batch normalization layer, and a first hash unit; the method further includes:

[0021] The seventh tensor is input into the first output network, and is processed by the fourth convolution kernel, the first pooling layer, and the first batch normalization layer in sequence to obtain the standard features of the image to be processed.

[0022] The seventh tensor is input into the second output network, and is processed by the fifth convolution kernel, the second pooling layer, the second batch normalization layer and the first hash unit in sequence to obtain the hash feature of the image to be processed.

[0023] In one embodiment of the above-mentioned image feature extraction method, the re-identification model further includes a loss function network for model training, and the method further includes:

[0024] Obtaining the trained target re-identification model;

[0025] The method of “obtaining the trained target re-identification model” includes:

[0026] Get a sample image;

[0027] Inputting the sample image into the target re-identification model to be trained to perform feature extraction, and obtaining the standard feature and the hash feature of the sample image;

[0028] Inputting the standard feature and the hash feature of the sample image into the loss function network to obtain the standard feature loss and the hash feature loss of the sample image respectively;

[0029] Based on the standard feature loss and the hash feature loss, obtaining a loss calculation result of the sample image;

[0030] The target re-identification model to be trained is adjusted based on the loss calculation result to obtain the trained target re-identification model.

[0031] In one embodiment of the above-mentioned image feature extraction method, the loss function network includes a first loss function unit, the first loss function unit includes at least a first fully connected layer, and the method for obtaining the standard feature loss includes:

[0032] Inputting the standard feature of the sample image into the first fully connected layer of the first loss function unit to obtain a standard feature classification result of the sample image;

[0033] Based on the standard features of the sample image, the standard feature classification results and the inter-class distance penalty loss function F(u), a standard feature first loss is obtained;

[0034] Based on the standard features of the sample image, the standard feature classification result and the cross entropy classification loss function G(v), obtaining a standard feature second loss;

[0035] Based on the first standard feature loss and the second standard feature loss, obtaining the standard feature loss of the sample image;

[0036] Among them, the general expression of the inter-class distance penalty loss function F(u) is:

[0037]

[0038] Where u is the feature tensor of a single batch of samples in a training batch, N is the number of single batch samples in the training batch, and u i is the eigenvector corresponding to sample i, u n is the sum of u in the training batch i The classification label is different and is different from u i The eigenvector corresponding to the nearest sample n, dist(u i ,u n ) represents u i and u n A first distance between, wherein the first distance is a cosine distance or a Euclidean distance, and margin is a preset parameter;

[0039] The general expression of the cross entropy classification loss function G(v) is:

[0040]

[0041] Among them, C is the total number of categories of all training data, v ij For feature v i In the j dimension, k is the category label of sample i, v ik For feature v i The value of k dimension, v, is obtained through the fully connected layer, specifically:

[0042] v = f(u) = weight · u + bias

[0043] Among them, weight is the weight parameter of the fully connected layer, and bias is the bias parameter of the fully connected layer.

[0044] In one embodiment of the above-mentioned image feature extraction method, the loss function network further includes a second loss function unit, the second loss function unit at least includes the second hash unit and a second fully connected layer, and the method for obtaining the hash feature loss includes:

[0045] Inputting the hash feature of the sample image into the second hash unit of the second loss function unit for processing to obtain an adjusted hash feature;

[0046] According to the adjusted hash feature, obtaining a hash feature classification result of the sample image through the second fully connected layer;

[0047] Determine a first hash feature loss based on the adjusted hash feature, the hash feature classification result and the inter-class distance penalty loss function F(u);

[0048] Determine a hash feature second loss based on the adjusted hash feature, the hash feature classification result and the cross entropy classification loss function G(v);

[0049] Based on the first hash feature loss and the second hash feature loss, obtaining the hash feature loss of the sample image;

[0050] The method of “inputting the hash feature of the sample image into the second hash unit for processing to obtain an adjusted hash feature” includes:

[0051] Amplifying the hash feature of the sample image by a preset ratio to obtain the amplified hash feature g(y) of the sample image, wherein g(y)=scale*y;

[0052] The hash feature g(y) of the enlarged sample image is adjusted based on the following formula to obtain the adjusted hash feature h(g(y)):

[0053]

[0054] Among them, y is the value of the hash feature of the sample image, scale is a preset parameter, and boundry is a preset parameter.

[0055] In one embodiment of the above-mentioned image feature extraction method, the method further includes:

[0056] The output of the second batch normalization layer is binarized using the first hash unit to obtain the hash feature of the image to be processed, specifically:

[0057]

[0058] Wherein, x is the output of the second batch normalization layer, and hash1(x) is the hash feature of the image to be processed.

[0059] In a second aspect, the present invention provides an image feature extraction method, the method comprising:

[0060] Get the image to be retrieved;

[0061] Based on the image feature extraction method described in any of the above solutions, the hash features and standard features of the image to be retrieved are obtained;

[0062] Retrieving the target base library according to the hash feature to obtain a reduced base library;

[0063] Search the reduced base library according to the standard features to obtain search results;

[0064] The target database is a database constructed by obtaining the standard features and the hash features from the database images through a trained target re-identification model.

[0065] In one embodiment of the above-mentioned image retrieval method, the step of “retrieve the target base library according to the hash feature to obtain a reduced base library” specifically includes:

[0066] Obtaining a second distance between a first hash feature and a second hash feature, wherein the first hash feature is the hash feature of the image to be retrieved, the second hash feature is the hash feature of the base image in the target base, and the second distance is a Hamming distance;

[0067] The second distances are sorted from small to large, and the reduced base library is obtained from the target base library according to the second distances M before sorting, where M is a preset parameter.

[0068] In one embodiment of the above-mentioned image retrieval method, the step of "searching the reduced base library according to the standard features to obtain the retrieval results" specifically includes:

[0069] Acquire a third distance between a first standard feature and a second standard feature, wherein the first standard feature is the standard feature of the image to be retrieved, the second standard feature is the standard feature of the base image in the target base, and the third distance is a cosine distance or a Euclidean distance;

[0070] The third distances are sorted from small to large, and the search result is obtained from the reduced base library according to the third distances L before sorting, where L is a preset parameter.

[0071] In a third aspect, the present invention provides an image feature extraction device, the device comprising:

[0072] An image acquisition module, wherein the image acquisition module is configured to acquire an image to be processed;

[0073] A re-identification model module, wherein the re-identification model module is configured to input the image to be processed into a trained target re-identification model to obtain standard features and hash features of the image to be processed, wherein the target re-identification model includes a re-identification model basic network, a self-attention structure network, a first output network, and a second output network;

[0074] The trained target re-identification model specifically performs the following operations:

[0075] Using the re-identification model basic network to extract features of the image to be processed, to obtain a first tensor of the image to be processed;

[0076] Processing the first tensor using the self-attention structure network to obtain a seventh tensor of the image to be processed;

[0077] Inputting the seventh tensor into the first output network for processing to obtain the standard features of the image to be processed;

[0078] The seventh tensor is input into the second output network for processing to obtain the hash feature of the image to be processed.

[0079] In a fourth aspect, the present invention provides an image retrieval device, the device comprising:

[0080] An image acquisition module, wherein the image acquisition module is configured to acquire an image to be retrieved;

[0081] An image feature extraction module, wherein the image feature extraction module is configured to obtain hash features and standard features of the image to be retrieved based on the image feature extraction method described in any of the above solutions;

[0082] A hash feature retrieval module, wherein the hash feature retrieval module is configured to retrieve a target base library according to the hash feature to obtain a reduced base library;

[0083] A standard feature retrieval module, wherein the standard feature retrieval module is configured to search the reduced base library according to the standard feature to obtain a search result;

[0084] The target database is a database constructed by the standard features and the hash features obtained from the database images through the image feature extraction module.

[0085] In a fifth aspect, the present invention proposes a computer device, comprising a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the image feature extraction method or image retrieval method described in any of the above-mentioned schemes.

[0086] In a sixth aspect, the present invention proposes a storage medium, which is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by a processor to execute the image feature extraction method or image retrieval method described in any of the above-mentioned schemes.

[0087] In the case of adopting the above technical solution, the present invention extracts the features of the image through a re-recognition model with two-way output including a self-attention structure network, and obtains information-rich standard features and binary hash features at the same time, wherein the self-attention structure network can enhance the expressiveness of the features and effectively improve the effect of feature extraction. In the image retrieval application, the hash features are first used for the initial retrieval to quickly reduce the size of the base database; then the standard features are used to perform a secondary retrieval in the reduced base database to obtain the target base database image. The image feature extraction method and the joint retrieval method of the present invention can effectively extract image features, and perform fast and accurate base database retrieval based on the standard features and binary hash features of the image, which is very suitable for ultra-large-scale base database retrieval tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings, in which:

[0089] Figure 1 It is a flow chart of the main steps of the image feature extraction method according to an embodiment of the present invention.

[0090] Figure 2 It is a structural diagram of a re-identification model for image feature extraction according to an embodiment of the present invention.

[0091] Figure 3 Schematic diagram of the structure of a re-identification model used for model training according to an embodiment of the present invention.

[0092] Figure 4 It is a flow chart of the main steps of the image retrieval method according to an embodiment of the present invention.

[0093] Figure 5 It is a detailed step flow chart of the image retrieval method according to an embodiment of the present invention.

[0094] Figure 6 It is a schematic diagram of the composition structure of the image feature extraction device of the present invention.

[0095] Figure 7 It is a schematic diagram of the composition structure of the image retrieval device of the present invention. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0097] It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention. Those skilled in the art may make adjustments to them as needed to adapt to specific application scenarios.

[0098] Read First Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of the image feature extraction method according to an embodiment of the present invention. Figure 1 As shown, the image feature extraction method of the present invention includes:

[0099] Step S101: obtaining an image to be processed;

[0100] Step S102: input the image to be processed into the trained target re-identification model to obtain the standard features and hash features of the image to be processed.

[0101] In step S101, the image to be processed (hereinafter referred to as: image) usually comes from an image acquisition device such as a video camera, a digital camera, etc.

[0102] Continue reading Figure 2 , combined with Figure 2 The implementation process of step S102 is described below. Figure 2 It is a structural diagram of a re-identification model for image feature extraction according to an embodiment of the present invention.

[0103] like Figure 2 As shown, the re-identification model for image feature extraction in an embodiment of the present invention includes: a re-identification model basic network 1, a self-attention structure network 2, a first output network 3 and a second output network 4.

[0104] The basic network 1 of the re-identification model is used to extract the features of the image. The basic network of the re-identification model uses convolutional neural networks (CNN), which is a type of feedforward neural networks (FNN) with deep structure and convolution calculation. It is one of the representative algorithms of deep learning. Convolutional neural networks can learn features from data and transform the results into unknown data of the same type. It has been widely used in image recognition, image segmentation, natural language processing and other fields.

[0105] Preferably, the convolutional neural network of the embodiment of the present invention uses a ResNet network. The ResNet network is modified based on the VGG19 network with reference to the VGG19 network, and the residual unit is added through the short-circuit mechanism to solve the network degradation phenomenon well.

[0106] Before the image is sent to the re-identification model network, it is necessary to perform image preprocessing according to the input requirements of the re-identification model basic network 1 to obtain an image that meets the input dimension requirements. The image preprocessing method includes but is not limited to image scaling, image padding, image storage format conversion, normalization, etc. As an example, the image is converted into an RGB format image with a size of 384*192 or 256*128 pixels and normalized.

[0107] The processed image is passed through the re-identification model basic network 1 to extract the feature data of the image to obtain the first tensor data. The dimension of the first tensor is L*D, where L is the area of ​​the feature map, D is the number of channels of the feature map, and the first tensor is the input of the self-attention structure network 2.

[0108] The self-attention structure network 2 includes a first convolution kernel 21, a second convolution kernel 22, a third convolution kernel 25, a first operation unit 23, a second operation unit 26 and a logistic regression unit 24. The inputs of the first convolution kernel 21, the second convolution kernel 22 and the third convolution kernel 25 are all first tensors of dimension L*D.

[0109] The first convolution kernel 21 is a convolution kernel of size 1×1, and its output is a second tensor, and the dimension of the second tensor is L*D; the second convolution kernel 22 is a convolution kernel of size 1×1, and its output is a third tensor, and the dimension of the third tensor is L*D. The second tensor and the third tensor are simultaneously input into the first operation unit 23, and the third tensor is first transposed in the first operation unit 23, and then multiplied with the second tensor, that is, ((second tensor)·(third tensor) T ), a fourth vector is obtained, and the dimension of the fourth tensor is L*L. The fourth tensor is input into the logistic regression unit 24 to obtain a fifth tensor, and the dimension of the fifth tensor is L*L. Preferably, the logistic regression unit 24 is a Softmax function or a Sigmoid function.

[0110] The third convolution kernel 25 is a convolution kernel of size 1×1, and its output is the sixth tensor, and the dimension of the sixth tensor is L*D; the fifth tensor and the sixth tensor are simultaneously input into the second operation unit 26, and a multiplication operation is performed in the second operation unit 26, i.e. ((fifth tensor)·(sixth tensor)), to obtain the seventh tensor, and the dimension of the seventh tensor is L*D.

[0111] It should be noted that after the fourth tensor in the self-attention structure network passes through the logistic regression unit 24, the value of the fifth tensor is between 0 and 1, and each value in the fifth tensor represents the importance of each position to other positions. The higher the importance, the closer the value of the position is to 1, and vice versa, it is close to 0. After the fifth tensor is multiplied by the sixth tensor, a seventh tensor with a feature importance tendency is obtained, which focuses the features on more important positions, thereby effectively improving the effect of feature extraction.

[0112] The seventh tensor is input into the first output network 3 and the second output network 4 respectively to obtain the standard features and hash features of the image to be processed in the embodiment of the present invention.

[0113] The first output network 3 sequentially includes a fourth convolution kernel 31, a first pooling layer 32, and a first batch normalization layer 33. The seventh tensor is input into the fourth convolution kernel 31 with a convolution kernel size of 1×1 to transform the number of feature channels to obtain an eighth tensor, and the dimension of the eighth tensor is L*d1, where d1 is the number of channels of the standard feature, that is, the dimension of the standard feature.

[0114] The eighth tensor is input into the first pooling layer 32 for downsampling to reduce the dimension of the feature data and obtain a first output feature vector with a dimension of d1. The first output feature vector is then input into the first batch normalization layer 33 to obtain the standard features of the image of the embodiment of the present invention.

[0115] It should be noted that, in the embodiment of the present invention, after the image features are processed by the first pooling layer 32 to obtain the image features of the required dimensions, they are further processed by the first batch normalization layer 33. The addition of the first batch normalization layer 33 can well solve the gradient vanishing problem, improve the generalization ability of the network, and further enhance the effect of the features.

[0116] The second output network 4 sequentially includes a fifth convolution kernel 41, a second pooling layer 42, a second batch normalization layer 43 and a first hash unit 44. The seventh tensor is input into the fifth convolution kernel 41 with a convolution kernel size of 1×1 to transform the number of feature channels to obtain a ninth tensor with a dimension of L*d2, where d2 is the number of channels of the hash feature, that is, the dimension of the hash feature.

[0117] The ninth tensor is input into the second pooling layer 42 for downsampling to reduce the dimension of the feature data and obtain a second output feature vector with a dimension of d2. The second output feature vector is then input into the second batch normalization layer 43 to obtain a third output feature vector with a dimension of d2.

[0118] The third output feature is input into the first hash unit 44 to obtain a binary hash feature of the image of the embodiment of the present invention. The binarization function of the first hash unit 44 is:

[0119]

[0120] It should be noted that, in the second output network 4, the second batch normalization layer 43 has the same function as the aforementioned first batch normalization layer 33, which will not be described in detail here.

[0121] Before the re-identification model is used, the model needs to be trained. The structural diagram of the re-identification model for model training in an embodiment of the present invention is as follows: Figure 3 As shown in Figure 5, the re-identification model used for training adds a loss function network 5.

[0122] The loss function network 5 includes a first loss function unit 51, a second loss function unit 52 and a third operation unit.

[0123] The first loss function unit 51 also includes a first fully connected layer 511 and a first loss function acquisition module 512; the input of the first fully connected layer 511 is the standard features of the image output by the first output network 3, and the standard features of the image are classified through the first fully connected layer 511 to obtain the standard feature classification results of the image.

[0124] The standard feature classification result of the image output by the first fully connected layer 511 and the standard feature of the image output by the first output network 3 (the output of the first batch normalization layer 33) are sent to the first loss function acquisition module 512 to obtain the output standard feature loss Loss_n of the first loss function unit.

[0125] The second loss function unit 52 includes a second hash unit 521, a second fully connected layer 522 and a second loss function acquisition module 523. The output third output feature vector of the second batch normalization layer 43 of the second output network 4 is sent to the second hash unit 521 to obtain an adjusted hash feature; the adjusted hash feature is classified through the second fully connected layer 522 to obtain a hash feature classification result of the image.

[0126] The hash feature classification result of the image output by the second fully connected layer 522 and the adjusted hash feature output by the second hash unit 521 are sent to the second loss function acquisition module 523 to obtain the output hash feature loss Loss_h of the second loss function unit.

[0127] The standard feature loss and the hash feature loss are sent to the third operation unit 53 to obtain the loss calculation result of the embodiment of the present invention. The loss calculation result is obtained by:

[0128] Loss = Loss_n + Loss_h (Formula 2)

[0129] The first loss function acquisition module 512 and the second loss function acquisition module 523 have the same structure, both of which are composed of an inter-class distance penalty loss function F(u) and a cross entropy classification loss function G(v);

[0130] The general expression of the inter-class distance penalty loss function F(u) is:

[0131]

[0132] In an embodiment of the present invention, the re-identification model training model has only one loss calculation result output Loss, and the first output network 3 and the second output network 4 are trained synchronously, so the samples used to train Loss_n and Loss_h are the same, and the corresponding sample quantity, sample classification category and other data are the same.

[0133] In Formula 3, u is the feature tensor of a single batch of samples in a training batch, N is the number of single batch samples in a training batch, and u i is the eigenvector corresponding to sample i, u n For a training batch, u i The classification label is different and is different from u i The eigenvector corresponding to the nearest sample n, dist(u i ,u n ) represents u i and u n For example, the first distance may be a cosine distance or a Euclidean distance, and margin is a preset parameter, such as 0.3.

[0134] The general expression of the cross entropy classification loss function G(v) is:

[0135]

[0136] In Formula 4, C is the total number of categories of all training data, v ij For feature v i In the j dimension, k is the category label of sample i, v ik For feature v i The value of k dimension, v, is obtained through the fully connected layer, specifically:

[0137] v=f(u)=weight·u+bias (Formula 5)

[0138] In Formula 5, weight is the weight parameter of the fully connected layer, and bias is the bias parameter of the fully connected layer.

[0139] Combination Figure 3 It can be seen that for the first loss function unit, if the output of the first batch normalization layer 33 is x, then the output of the first fully connected layer 511 is:

[0140] f(x) = weight1 x + bias1 (Equation 6)

[0141] In Formula 6, weight1 is the weight parameter of the first fully connected layer 511, and bias1 is the bias parameter of the first fully connected layer 511.

[0142] It can be seen that u in Formula 3 and u in Formula 5 are the output x of the first batch normalization layer 33, and the standard feature loss Loss_n is:

[0143] Loss_n=F(x)+G(f(x)) (Equation 7)

[0144] Right now,

[0145]

[0146] In Formula 7, F(x) is the standard feature of the first loss, and G(f(x)) is the standard feature of the second loss.

[0147] For the second loss function unit, assuming that the output of the second batch normalization layer 43 is y, the output of the second hash unit 521 is hash2(y), and hash2(y) is specifically:

[0148] hash2(y)=h(g(y)) (Formula 8)

[0149] In formula 8,

[0150] g(y)=scale*y (Equation 9)

[0151] In Formula 9, scale is a preset parameter. As an example, scale can be set to 10.

[0152] The specific form of h(g(y)) is:

[0153]

[0154] Continue to combine Figure 3 It can be seen that for the second loss function unit, the output of the second fully connected layer 522 is:

[0155] f(hash2(y))=weight2·hash2(y)+bias2 (Equation 11)

[0156] In Formula 11, weight2 is the weight parameter of the second fully connected layer 522, and bias2 is the bias parameter of the second fully connected layer 522.

[0157] It can be seen that u in Formula 3 and u in Formula 5 are the output hash2(y) of the second hash function 523, and the hash feature loss Loss_h is:

[0158] Loss_h=F(hash2(y))+G(f(hash2(y))) (Formula 12)

[0159] Right now,

[0160]

[0161] In Formula 12, F(hash2(y)) is the first loss of hash feature, and G(f(hash2(y))) is the first loss of hash feature.

[0162] It should be noted that, in the embodiment of the present invention, the numerical value of the image feature is first amplified by formula 9, and then the numerical value is clipped to the range of [-boundry,boundry] by formula 10. The purpose of this processing is to make the image feature numerical value distributed on both sides of 0 as much as possible after amplification, and close to the boundary value boundry, while retaining a certain degree of discretization within the boundary range, thereby ensuring the training effect of the model of the embodiment of the present invention.

[0163] It should be noted that when the re-identification model of the present invention is trained, the first output network 3 and the second output network 4 are trained synchronously, and the loss calculation result Loss is calculated once for each batch, and the model network weight is updated once. The method of adjusting the model according to the loss calculation result Loss is a technology well known to those skilled in the art and will not be repeated here.

[0164] The present invention does not limit the dataset of the re-identification model training samples. As an example, the Market-1501 dataset, the MSMT17 dataset, etc. can be selected.

[0165] After the re-identification model is trained, you can use the trained re-identification model to extract image features and perform image retrieval in the target database. Figure 4 , Figure 4 It is a flow chart of the main steps of an embodiment of the present invention, such as Figure 4 As shown, the image retrieval method of the present invention comprises:

[0166] Step S401: obtaining an image to be retrieved;

[0167] Step S402: obtaining standard features and hash features of the image to be retrieved through the trained re-identification model;

[0168] Step S403: searching the target base database according to the hash feature to obtain a reduced base database;

[0169] Step S404: Retrieve and narrow the base database according to standard features to obtain search results.

[0170] Combination Figure 5, illustrating the application of the method of the present invention in pedestrian re-identification, the image to be retrieved in step S401 is a pedestrian image obtained by a camera. The present invention does not limit the method of detecting pedestrians in video images. As an example, a method based on Haar wavelet features, a method based on HOG features, etc. can be used to implement pedestrian detection. Those skilled in the art can use a suitable pedestrian detection method according to actual conditions.

[0171] In step S402, through the trained Figure 2 The re-identification model shown obtains the standard features and hash features of the pedestrian image to be retrieved, the standard features of the pedestrian image to be retrieved are called the first standard features, and the hash features of the pedestrian image to be retrieved are called the first hash features (step S4021).

[0172] In step S403, the base database image is usually a historical pedestrian image acquired by a camera, and the target base database is a database constructed by standard features and hash features acquired from the base database images through a trained re-identification model. The standard features of the base database images are called second standard features, and the hash features of the base database images are called second hash features.

[0173] Calculate the Hamming distance between the first hash feature and the second hash feature to obtain the second distance between the first hash feature of the image to be detected and the second hash features of all the base library images in the base library (step S4031); sort the second distances from small to large, the smaller the distance, the higher the matching degree of the two images, and the higher the ranking (step S4032); obtain the target base library data corresponding to the second distance ranked in the top M, and generate a reduced base library for secondary retrieval (step S4033), where M is a preset parameter. For example, M can be set to 10% of all images in the target base library, that is, the top 10% of the target base library data with the smallest second distance are selected as the reduced base library.

[0174] In step S404, the third distance between the first standard feature and the second standard feature of all the base images in the reduced base library is calculated (step S4041); the third distances are sorted from small to large, and the smaller the distance, the higher the matching degree of the two images, and the higher the sorting (step S4042); the images in the reduced base library corresponding to the third distances ranked in the top L are obtained to obtain the retrieval results (step S4043). For example, for an image to be retrieved, L can be set to 20, that is, the top 20 base images in the reduced base library with the smallest third distance are selected as the retrieval results. Among them, the third distance can be a cosine distance or a Euclidean distance.

[0175] It should be noted that in this embodiment, the dimension value of the standard feature is 256, and the data in the standard feature is floating-point data; the dimension value of the hash feature is 512, and the data in the hash feature is binary 0 or 1, which is unsigned integer data. Unsigned integer data occupies less storage space and has a faster calculation speed than floating-point data, and has more obvious speed and video memory advantages in the process of ultra-large matrix calculation.

[0176] According to the image retrieval method of the present invention, firstly, the hash features of unsigned integer data are used for initial retrieval to quickly narrow the scope of the target database, and then the standard features of floating-point data with richer feature information are used to perform precise retrieval in the narrowed target database, thereby realizing fast and accurate image retrieval, which is particularly suitable for ultra-large-scale target database retrieval tasks.

[0177] It should be noted that, in this embodiment, the dimension of the hash feature is usually larger than the dimension of the standard feature. This is because the hash feature is binary data and will lose some image feature information. Increasing the dimension of the hash feature can partially compensate for the image features lost due to binarization, which helps to improve the accuracy of retrieval.

[0178] It should be noted that, although the re-identification model used in the method of the present invention adds the output of hash features compared to the general re-identification model that only obtains standard features, the main data for image feature extraction has been completed in the basic model of the re-identification model, and the added hash function unit part will not affect the processing speed of the re-identification model.

[0179] In another embodiment, the method of the present invention is also applicable to applications of other types of targets, such as vehicles, animals, etc., or multiple mixed types of targets, such as mixed image retrieval of people and vehicles. In this case, when training the re-identification model, it is sufficient to use a data set containing the target type for training.

[0180] Furthermore, the present invention also provides an image feature extraction device, such as Figure 6 As shown, the image feature extraction device 6 of the present invention mainly includes an image acquisition module 61 and a re-identification model module 62, wherein the re-identification model module 62 further includes a re-identification model basic network 621, a self-attention structure network 622, a first output network 623 and a second output network 624.

[0181] In one embodiment of the present invention, the image acquisition module 61 is configured to acquire an image to be processed. After preprocessing, the image to be processed is input into the re-identification model module 62, and features are extracted through the re-identification model basic network 621, and then features are enhanced through the self-attention structure network 622, and then respectively input into the first output network 623 and the second output network 624, and the standard features of the image to be processed are obtained through the first output network 623, and the hash features of the image to be processed are obtained through the second output network 624.

[0182] Furthermore, the present invention also provides an image retrieval device, such as Figure 7 As shown, the image retrieval device 7 of the present invention mainly includes an image acquisition module 71 , an image feature extraction module 72 , a hash feature retrieval module 73 and a standard feature retrieval module 74 .

[0183] In one embodiment of the present invention, the image acquisition module 71 is configured to execute step 401 to acquire the image to be retrieved. The image feature extraction module 72 is configured to execute the operations in step S402 and step S4021. The hash feature retrieval module 73 is configured to execute the operations in step S403, step S4031, step S4032, and step S4033. The standard feature retrieval module 74 is configured to execute the operations in step S404, step S4041, step S4042, and step S4043. And the image acquisition module 71 is also configured to obtain the standard features and hash features of the base library image.

[0184] Furthermore, the present invention also provides a computer device, which includes a processor and a storage device, the storage device can be configured to store and execute the program of the image feature extraction method and the image retrieval method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program of executing the image feature extraction method and the image retrieval method of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The image detection device can be a control device device formed by various electronic devices.

[0185] Furthermore, the present invention also provides a storage medium, which can be configured to store a program for executing the image feature extraction method and image retrieval method of the above method embodiment, and the program can be loaded and run by a processor to implement the above image feature extraction method and image retrieval method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The storage medium can be a storage device formed by various electronic devices. Optionally, the storage medium in the embodiment of the present invention is a non-temporary readable and writable storage medium.

[0186] Those skilled in the art should be able to appreciate that the method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0187] It should be noted that the terms "first" to "ninth" and other ordinal numbers in the specification and claims of the present invention and the above-mentioned drawings are only used to distinguish similar objects, rather than to describe or indicate a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0188] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for extracting image features, characterized in that: The method comprises: Get the image to be processed; Inputting the image to be processed into a trained target re-identification model to obtain standard features and hash features of the image to be processed, wherein the target re-identification model includes a re-identification model basic network, a self-attention structure network, a first output network and a second output network; The method of “inputting the image to be processed into a trained target re-identification model to obtain standard features and hash features of the image to be processed” includes: Using the re-identification model basic network to extract features of the image to be processed, to obtain a first tensor of the image to be processed; Processing the first tensor using the self-attention structure network to obtain a seventh tensor of the image to be processed that characterizes the tendency of feature importance, wherein the self-attention structure network includes a first convolution kernel, a second convolution kernel, a third convolution kernel, a first operation unit, a second operation unit, and a logistic regression unit, and the seventh tensor is the output of the self-attention network structure; Inputting the seventh tensor into the first output network for processing to obtain the standard features of the image to be processed, wherein the first output network sequentially includes a fourth convolution kernel, a first pooling layer, and a first batch normalization layer; The seventh tensor is input into the second output network for processing to obtain the hash feature of the image to be processed, wherein the second output network sequentially includes a fifth convolution kernel, a second pooling layer, a second batch normalization layer, and a first hash unit.

2. The image feature extraction method according to claim 1, characterized in that: The method of “processing the first tensor using the self-attention structure network to obtain the seventh tensor of the image to be processed” includes: Using the first convolution kernel, the second convolution kernel, and the third convolution kernel to process the first tensor respectively, to obtain a second tensor, a third tensor vector, and a sixth tensor respectively; Inputting the second tensor and the third tensor into the first operation unit for processing to obtain a fourth tensor; Inputting the fourth tensor into the logistic regression unit for processing to obtain a fifth tensor; The fifth tensor and the sixth tensor are input into the second operation unit for processing to obtain the seventh tensor.

3. The image feature extraction method according to claim 1, characterized in that: The method further comprises: Inputting the seventh tensor into the first output network, and processing it sequentially through the fourth convolution kernel, the first pooling layer, and the first batch normalization layer to obtain the standard features of the image to be processed; The seventh tensor is input into the second output network, and is processed by the fifth convolution kernel, the second pooling layer, the second batch normalization layer and the first hash unit in sequence to obtain the hash feature of the image to be processed.

4. The image feature extraction method according to claim 1, characterized in that: The re-identification model also includes a loss function network for model training, and the method further includes: Obtaining the trained target re-identification model; The method of “obtaining the trained target re-identification model” includes: Get a sample image; Inputting the sample image into the target re-identification model to be trained to perform feature extraction, and obtaining the standard feature and the hash feature of the sample image; Inputting the standard feature and the hash feature of the sample image into the loss function network to obtain the standard feature loss and the hash feature loss of the sample image respectively; Based on the standard feature loss and the hash feature loss, obtaining a loss calculation result of the sample image; The target re-identification model to be trained is adjusted based on the loss calculation result to obtain the trained target re-identification model.

5. The image feature extraction method according to claim 4, characterized in that: The loss function network includes a first loss function unit, the first loss function unit includes at least a first fully connected layer, and the method for obtaining the standard feature loss includes: Inputting the standard feature of the sample image into the first fully connected layer of the first loss function unit to obtain a standard feature classification result of the sample image; Based on the standard features of the sample image, the standard feature classification results and the inter-class distance penalty loss function F(u), a standard feature first loss is obtained; Based on the standard features of the sample image, the standard feature classification result and the cross entropy classification loss function G(v), obtaining a standard feature second loss; Based on the first standard feature loss and the second standard feature loss, obtaining the standard feature loss of the sample image; Among them, the general expression of the inter-class distance penalty loss function F(u) is: Where u is the feature tensor of a single batch of samples in a training batch, N is the number of single batch samples in the training batch, and u i is the eigenvector corresponding to sample i, u n is the sum of u in the training batch i The classification label is different and is different from u i The eigenvector corresponding to the nearest sample n, dist(u i ,u n ) represents u i and u n A first distance between, wherein the first distance is a cosine distance or a Euclidean distance, and margin is a preset parameter; The general expression of the cross entropy classification loss function G(v) is: Among them, C is the total number of categories of all training data, v ij For feature v i In the j dimension, k is the category label of sample i, v ik For feature v i The value of k dimension, v, is obtained through the fully connected layer, specifically: v = f(u) = weight · u + bias Among them, weight is the weight parameter of the fully connected layer, and bias is the bias parameter of the fully connected layer.

6. The image feature extraction method according to claim 5, characterized in that: The loss function network further includes a second loss function unit, the second loss function unit at least includes a second hash unit and a second fully connected layer, and the method for obtaining the hash feature loss includes: Inputting the hash feature of the sample image into the second hash unit of the second loss function unit for processing to obtain an adjusted hash feature; According to the adjusted hash feature, obtaining a hash feature classification result of the sample image through the second fully connected layer; Determine a first hash feature loss based on the adjusted hash feature, the hash feature classification result and the inter-class distance penalty loss function F(u); Determine a hash feature second loss based on the adjusted hash feature, the hash feature classification result and the cross entropy classification loss function G(v); Based on the first hash feature loss and the second hash feature loss, obtaining the hash feature loss of the sample image; The method of "inputting the hash feature of the sample image into the second hash unit for processing to obtain an adjusted hash feature" includes: Amplifying the hash feature of the sample image by a preset ratio to obtain the amplified hash feature g(y) of the sample image, wherein g(y)=scale*y; The hash feature g(y) of the enlarged sample image is adjusted based on the following formula to obtain the adjusted hash feature h(g(y)): Among them, y is the value of the hash feature of the sample image, scale is a preset parameter, and boundry is a preset parameter.

7. The image feature extraction method according to claim 3, characterized in that: The method further comprises: The output of the second batch normalization layer is binarized using the first hash unit to obtain the hash feature of the image to be processed, specifically: Wherein, x is the output of the second batch normalization layer, and hash1(x) is the hash feature of the image to be processed.

8. An image retrieval method, characterized in that: The method comprises: Get the image to be retrieved; Based on the image feature extraction method according to any one of claims 1 to 7, obtaining hash features and standard features of the image to be retrieved; Retrieving the target base library according to the hash feature to obtain a reduced base library; Search the reduced base library according to the standard features to obtain search results; The target database is a database constructed by obtaining the standard features and the hash features from the database images through a trained target re-identification model.

9. The image retrieval method according to claim 8, characterized in that: The step of "retrieve the target base library according to the hash feature to obtain a reduced base library" specifically includes: Obtaining a second distance between a first hash feature and a second hash feature, wherein the first hash feature is the hash feature of the image to be retrieved, the second hash feature is the hash feature of the base image in the target base, and the second distance is a Hamming distance; The second distances are sorted from small to large, and the reduced base library is obtained from the target base library according to the second distances M before sorting, where M is a preset parameter.

10. The image retrieval method according to claim 9, characterized in that: The step of "searching the reduced base library according to the standard features to obtain search results" specifically includes: Acquire a third distance between a first standard feature and a second standard feature, wherein the first standard feature is the standard feature of the image to be retrieved, the second standard feature is the standard feature of the base image in the target base, and the third distance is a cosine distance or a Euclidean distance; The third distances are sorted from small to large, and the search result is obtained from the reduced base library according to the third distances L before sorting, where L is a preset parameter.

11. An image feature extraction device, characterized in that: The device comprises: An image acquisition module, wherein the image acquisition module is configured to acquire an image to be processed; A re-identification model module, wherein the re-identification model module is configured to input the image to be processed into a trained target re-identification model to obtain standard features and hash features of the image to be processed, wherein the target re-identification model includes a re-identification model basic network, a self-attention structure network, a first output network, and a second output network; The trained target re-identification model specifically performs the following operations: Using the re-identification model basic network to extract features of the image to be processed, to obtain a first tensor of the image to be processed; Processing the first tensor using the self-attention structure network to obtain a seventh tensor of the image to be processed; Inputting the seventh tensor into the first output network for processing to obtain the standard features of the image to be processed; The seventh tensor is input into the second output network for processing to obtain the hash feature of the image to be processed.

12. An image retrieval device, characterized in that: The device comprises: An image acquisition module, wherein the image acquisition module is configured to acquire an image to be retrieved; An image feature extraction module, wherein the image feature extraction module is configured to obtain hash features and standard features of the image to be retrieved based on the image feature extraction method according to any one of claims 1 to 7; A hash feature retrieval module, wherein the hash feature retrieval module is configured to retrieve a target base library according to the hash feature to obtain a reduced base library; A standard feature retrieval module, wherein the standard feature retrieval module is configured to search the reduced base library according to the standard feature to obtain a search result; The target database is a database constructed by the standard features and the hash features obtained from the database images through the image feature extraction module.

13. A computer device, comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the image feature extraction method according to any one of claims 1 to 7 or the image retrieval method according to any one of claims 8 to 10.

14. A storage medium, the storage medium being suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the image feature extraction method according to any one of claims 1 to 7 or the image retrieval method according to any one of claims 8 to 10.

Images