A multi-label image retrieval method and system based on object scale perception

Abstract

The present invention discloses a multi-label image retrieval method based on object scale perception, which utilizes the hash centroid of the perceived image object scale to construct an end-to-end multi-label image retrieval method with high retrieval performance. The method proposed in the present invention first utilizes the Swin Transformer backbone network as an image feature extractor, uses label semantic information to guide the generation of hash center points, and proposes a calculation method for the label affinity coefficient related to the object scale, and the two together generate a hash centroid that includes the object scale. These hash centroids serve as anchor points in the Hamming space, guiding the multi-label image hash code to continuously approach its corresponding hash centroid, and through continuous optimization based on the center similarity loss function, the model learns a good hash mapping relationship. The hash centroid used in the present invention contains the scale information of different objects in the image, and combined with the powerful feature extraction capability of the Swin Transformer backbone network, the retrieval performance of the image retrieval method is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition and image retrieval, and more specifically, relates to a multi-label image retrieval method and system based on object scale perception. Background Art

[0002] Multi-label image retrieval has been increasingly applied in computer vision, including multi-object recognition, databases, storage systems, and search engines. Because each image contains multiple objects, leveraging the positive effects of different object scales on models and efficiently integrating this scale information with retrieval algorithms remain challenges.

[0003] Generally speaking, existing multi-label image retrieval methods first use a feature extraction network to obtain the feature vectors of the entire image, then use a hash layer to map these feature vectors into hash codes of a preset bit length, and finally directly use a hash loss function based on pairwise loss to guide model training.

[0004] However, existing multi-label image retrieval methods have some non-negligible flaws: First, due to the huge differences in scale between different objects, this will have different impacts on the image's feature vectors, making it impossible for the retrieval model to obtain the optimal mean average precision (MAP), which in turn affects the final retrieval performance; second, because existing methods often use a hash loss function based on pairwise loss, it leads to huge computational overhead during the model training and update process. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a multi-label image retrieval method and system based on object scale perception, which aims to solve the technical problems in the existing multi-label image retrieval method, such as the huge differences in scale between different objects, which will have different impacts on the feature vectors of the image, making it impossible for the retrieval model to obtain the optimal MAP, thereby affecting the final retrieval performance; and the technical problem that the existing method often uses a hash loss function based on pairwise loss, resulting in huge computational overhead in the model training and updating process.

[0006] To achieve the above object, according to one aspect of the present invention, a multi-label image retrieval method based on object scale perception is provided, comprising the following steps:

[0007] (1) Obtain an image to be retrieved and multiple image samples in an image database, and use the PIL library to convert the image to be retrieved and each image sample into multidimensional tensors respectively, and all multidimensional tensors constitute a multidimensional tensor set;

[0008] (2) Inputting the multidimensional tensor set obtained in step (1) into a pre-trained retrieval model to obtain the hash code corresponding to the image to be retrieved and the hash code corresponding to each image sample;

[0009] (3) For each hash code corresponding to the image sample obtained in step (2), calculate the similarity between the hash code and the hash code corresponding to the image to be retrieved obtained in step (2), and select multiple image samples that match the image to be retrieved and output them based on the multiple similarity calculation results obtained.

[0010] Preferably, the retrieval model includes a Swin Transformer network, a hash map layer and a hash center generation network connected in sequence.

[0011] Preferably, the Swin Transformer network receives an input tensor of dimension bs×3×448×448 and outputs a matrix of dimension bs×768, where bs is the batch size set during training.

[0012] The hash map layer consists of three fully connected layers. The first layer's input is the bs×768 matrix output by the Swin Transformer network. This layer uses a 768×512-dimensional weight matrix, and its output is a bs×512-dimensional matrix. The second layer's input is the bs×512-dimensional matrix output by the first layer. This layer uses a 512×256-dimensional weight matrix, and its output is a bs×256-dimensional matrix. The third layer's input is the bs×256-dimensional matrix output by the second layer. This layer uses a 256×K-dimensional weight matrix, and its output is a bs×K-dimensional matrix. K represents the hash code length.

[0013] The hash center generation network consists of three stacked fully connected layers. The first layer inputs a C×D-dimensional label word vector matrix, uses a D×512-dimensional weight matrix, and outputs a C×512-dimensional matrix. The second layer inputs the C×512-dimensional matrix output from the first layer, uses a 512×256-dimensional weight matrix, and outputs a C×256-dimensional hash center matrix. The third layer inputs the C×256-dimensional matrix output from the second layer, uses a 256×K-dimensional weight matrix, and outputs a C×K-dimensional hash center matrix, where K represents the hash code bit length, with K∈{16,32,48,64}, C represents the number of all label categories in the image database, and D represents the dimension of each label word vector.

[0014] Preferably, the retrieval model is trained by the following steps:

[0015] (2-1) Get N train The training set consists of images, N test The test set consists of images, and Ndb The training set image samples are composed of a matrix Each image sample x in the training set i Input into the Swin Transformer network to extract the feature vector of each image sample in the training set, and then input the feature vector of each image sample into the hash map layer to obtain the image sample x at training round t. i The corresponding continuous value hash code with a bit length of K The continuous value hash code corresponding to all image samples at training round t constitutes a matrix Samples per image x i The corresponding continuous value hash code is:

[0016]

[0017] where N train , N test and N db are all positive integers, F Swin represents the Swin Transformer network, represents the learnable parameters of the SwinTransformer network at training round t, f hash represents the hash map layer, represents the learnable parameters in the hash map layer at training epoch t.

[0018] (2-2) Get each image sample x in the training set i The label word vector corresponding to each label is obtained using BERT word vector mapping technology. All label word vectors form a C×D dimensional label word vector matrix The label word vector matrix M label Input the hash center generation network to obtain a hash center point matrix with a dimension of C×K at training round t

[0019] (2-3) For each image sample x in the training set i For the image sample x i Establishing tag affinity coefficients Its initial value is equal to t represents the training round, and t∈[0,50]:

[0020] (2-4) For each image sample x in the training set i For example, the image sample x obtained according to step (2-3) i The label affinity coefficient and the hash center point matrix obtained in step (2-2) Get the image sample xi The corresponding hash centroid r i , the hash centroids corresponding to all image samples at training round t constitute the hash centroid set

[0021] (2-5) The continuous value hash code matrix at training round t obtained according to step (2-1) And the hash centroid set at training round t obtained in steps (2-4) Get the loss function.

[0022] (2-6) According to the loss functions L1 and L2 obtained in step (2-5), the learnable parameters of the Swin Transformer network at training round t-1 for L1 are and Find the gradients separately, that is and Regarding L2 Taking the derivative we get Then use the following formula to get the updated learnable parameters and

[0023] (2-7) The initialized image obtained in step (2-3) is compared with each image sample x i The corresponding tag affinity coefficient As the initial value, according to the label affinity coefficient at training round t-1 And obtain the label affinity coefficient at training round t through mapping operation and iterative optimization

[0024] (2-8) Using the result obtained in step (2-6) Update the hash centroid matrix in steps (2-4)

[0025] (2-9) Using the N obtained in step (2-1) test The test set consists of images and N db The database consists of images, the retrieval model at training round t is tested and its MAP value is obtained;

[0026] (2-10) Repeat the above steps (2-1) to (2-9) until the upper limit of the training rounds (i.e., 50 times), and select the retrieval model with the largest MAP value as the final trained retrieval model.

[0027] Preferably, in step (2-2), the calculation process of the hash center point matrix is:

[0028]

[0029] Among them, g represents the hash center generation network, represents the learnable parameters of the hash center generation network at training round t;

[0030] Steps (2-3) are to set The total number of elements in is C.

[0031] Steps (2-4) are calculated using the following formula:

[0032]

[0033] in, Indicates that at training round t, the hash center point matrix E t The jth element in , and j∈[1,C], Represents image sample x i Label affinity coefficient at training round t The jth element of .

[0034] Preferably, step (2-5) is specifically as follows: first, according to the image sample x i Corresponding continuous value hash code and its corresponding hash centroid r i t Get the similarity loss value;

[0035] This process uses the following formula:

[0036]

[0037]

[0038] Among them, the superscript t represents the training round, Represents a continuous value hash code Its corresponding hash centroid r i t The distance between them, γ represents a scaling factor based on the Cauchy distribution, preferably γ = 0.15.

[0039] Then, according to the image sample x i The consecutive value hash code and its binary hash code The difference between them is used to obtain the quantized loss value;

[0040] This process uses the following formula:

[0041]

[0042] Then, according to the label word vector obtained in step (2-2) And the hash centroid set obtained in steps (2-4) at training round t Get the loss value based on KL divergence (i.e. Kullback-Leibler divergence):

[0043] This process uses the following formula:

[0044]

[0045]

[0046]

[0047] Among them, the superscript t represents the training round, and Respectively represent the hash center point matrix E obtained in step (2-2) t The mth and nth elements in d m and d n Respectively represent the label word vector matrix M obtained in step (2-2) label The mth and nth elements in , and m,n∈[1,C].

[0048] Then, according to the hash center point matrix obtained in step (2-2) Get the inter-class loss value;

[0049] This process uses the following formula:

[0050]

[0051] Finally, the first loss value is obtained based on the similarity loss value and the quantization loss value obtained above, and the second loss value is obtained based on the similarity loss value, the loss value based on KL divergence, and the inter-class loss value:

[0052] This process uses the following formula:

[0053] L1=L center +L q

[0054] L2=L center +L kl +L class

[0055] Preferably, steps (2-6) are specifically performed using the following formula:

[0056]

[0057]

[0058]

[0059] Among them, α and β represent different learning rates, preferably α = 2×10 -6 , β=5×10 -4 .

[0060] Preferably, step (2-7) comprises the following sub-steps:

[0061] (2-7-1) According to the loss function L1 obtained in step (2-5), obtain the intermediate coefficient at training round t The jth element in Then we get the intermediate coefficient at training round t

[0062] This step specifically uses the following formula:

[0063]

[0064]

[0065] in, Indicates the label affinity coefficient at training round t-1 The jth element in , η=5×10 -4 .

[0066] (2-7-2) The label affinity coefficient of the tth training round All elements in are arranged in descending order to obtain the sorting result u=[u1,u2,...,u C ];

[0067] (2-7-3) Calculate the quantization factor based on the sorting result u obtained in step (2-7-2)

[0068] (2-7-4) Obtain the normalized bias based on the quantization factor p obtained in step (2-7-3) And obtain the label affinity coefficient at training round t based on the calculated normalized bias q Each element in Then obtain the label affinity coefficient of the tth training round

[0069]

[0070]

[0071] According to another aspect of the present invention, a multi-label image retrieval system based on object scale perception is provided, comprising:

[0072] The first module is used to obtain the image to be retrieved and multiple image samples in the image database, and use the PIL library to convert the image to be retrieved and each image sample into multidimensional tensors respectively, and all multidimensional tensors constitute a multidimensional tensor set;

[0073] The second module is used to input the multidimensional tensor set obtained by the first module into a pre-trained retrieval model to obtain the hash code corresponding to the image to be retrieved and the hash code corresponding to each image sample;

[0074] The third module is configured to calculate the similarity between the hash code corresponding to each image sample obtained by the second module and the hash code corresponding to the image to be retrieved obtained by the second module, and select and output multiple image samples that match the image to be retrieved based on the multiple similarity calculation results.

[0075] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0076] (1) Due to the use of steps (2-2), (2-3), and (2-4), the label of each image is converted into a label word vector during the model training phase, and these word vectors are used to generate hash center points containing semantic relationships. In addition, a label affinity coefficient is proposed to reflect the contribution of objects of different sizes to the final image feature vector. The hash center calculated by the two together allows the target scale to affect the feature extraction process and thus obtain a better image feature vector, thereby improving the retrieval performance of the model.

[0077] (2) Since the present invention adopts steps (2-5), a hash function based on center similarity loss is used in the model training stage, which can enable the model to have a global perspective on the distribution of the data set. Compared with the method based on pairwise hash loss, it can reduce the computational overhead in the model training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 Schematic diagram of the overall framework of the multi-label image retrieval method based on object scale perception of the present invention;

[0079] Figure 2 is a curve showing how the average Hamming distance between hash center points generated by the object scale-aware multi-label image retrieval method of the present invention changes with the number of training rounds;

[0080] Figure 3 This is a workflow diagram of the multi-label image retrieval method based on object scale perception of the present invention. DETAILED DESCRIPTION

[0081] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0082] The present invention is implemented on the basis of a good hash centroid calculation method and a center similarity loss function. By extracting the semantic relationship contained in the label word vector, a hash center point containing semantic information is generated, and then a well-designed center similarity hash loss function is used to guide the training of the model, thereby improving the image retrieval performance of the model.

[0083] In experiments conducted on multiple multi-label image datasets, the present invention further improved its retrieval performance on these datasets, based on traditional retrieval performance metrics. This is because the hash centroid generated based on the hash center point containing semantic information and the label affinity coefficient is more accurate, guiding the model to generate more reasonable hash codes, thereby further improving the model's retrieval performance.

[0084] The basic idea of ​​this paper is to propose a label affinity coefficient related to the scale of different objects in multi-label images. This is combined with the hash center points obtained from the label word vector mapping to calculate the hash centroid that contains object scale information. In addition, a hash loss function based on the center similarity hash loss is designed to continuously narrow the distance between the sample hash code and the corresponding hash centroid, thereby obtaining a retrieval model with good retrieval performance.

[0085] like Figure 1 and Figure 3 As shown, the present invention provides a multi-label image retrieval method based on object scale perception, comprising the following steps:

[0086] (1) Obtain an image to be retrieved and multiple image samples in an image database, and use the Python Image Library (PIL library) to convert the image to be retrieved and each image sample into multidimensional tensors, and all multidimensional tensors constitute a multidimensional tensor set;

[0087] Specifically, this step uses the Image.open().convert('RGB') function and the Image.resize() function in the PIL library to convert the image to be retrieved and the image sample into a 3×448×448-dimensional tensor.

[0088] (2) Inputting the multidimensional tensor set obtained in step (1) into a pre-trained retrieval model to obtain the hash code corresponding to the image to be retrieved and the hash code corresponding to each image sample;

[0089] (3) For each hash code corresponding to the image sample obtained in step (2), calculate the similarity between the hash code and the hash code corresponding to the image to be retrieved obtained in step (2), and select multiple image samples that match the image to be retrieved and output them based on the multiple similarity calculation results obtained.

[0090] The retrieval model of the present invention includes a Swin Transformer network, a hash mapping layer and a hash center generation network which are connected in sequence.

[0091] Specifically, the Swin Transformer network receives an input tensor of dimension bs×3×448×448 and outputs a matrix of dimension bs×768, where bs is the batch size set during training, preferably 64.

[0092] The hash map layer consists of three fully connected layers. The first layer's input is the bs×768 matrix output by the Swin Transformer network. This layer uses a 768×512-dimensional weight matrix, and its output is a bs×512-dimensional matrix. The second layer's input is the bs×512-dimensional matrix output by the first layer. This layer uses a 512×256-dimensional weight matrix, and its output is a bs×256-dimensional matrix. The third layer's input is the bs×256-dimensional matrix output by the second layer. This layer uses a 256×K-dimensional weight matrix, and its output is a bs×K-dimensional matrix. K represents the hash code length.

[0093] The hash center generation network consists of three stacked fully connected layers. The first layer takes as input a C×D-dimensional label word vector matrix, uses a D×512-dimensional weight matrix, and outputs a C×512-dimensional matrix. The second layer takes as input the C×512-dimensional matrix output from the first layer, uses a 512×256-dimensional weight matrix, and outputs a C×256-dimensional matrix of hash center points. The third layer takes as input a C×256-dimensional matrix output from the second layer, uses a 256×K-dimensional weight matrix, and outputs a C×K-dimensional matrix of hash center points, where K represents the hash code bit length, with K∈{16,32,48,64}. C represents the number of label categories in the image database, and D represents the dimension of each label word vector, with D=768.

[0094] Specifically, the retrieval model in this step is trained through the following steps:

[0095] (2-1) Get N train The training set consists of images, N testThe test set consists of images, and N db The training set image samples are composed of a matrix Each image sample x in the training set i Input into the Swin Transformer network to extract the feature vector of each image sample in the training set, and then input the feature vector of each image sample into the hash map layer to obtain the image sample x at training round t. i The corresponding continuous value hash code with a bit length of K The continuous value hash code corresponding to all image samples at training round t constitutes a matrix Samples per image x i The corresponding continuous value hash code is:

[0096]

[0097] where N train , N test and N db are all positive integers, F Swin represents the Swin Transformer network, represents the learnable parameters of the SwinTransformer network at training round t, f hash represents the hash map layer, represents the learnable parameters in the hash map layer at training epoch t.

[0098] If the VOC2012 dataset is used, N train is 4000, N test is 1000, N db is 6540; if the MS-COCO dataset is used, N train is 10000, N test is 5000, N db is 114217; if the NUS-WIDE dataset is used, N train is 10500, N test is 2100, N db It is 149685.

[0099] (2-2) Get each image sample x in the training set i The label word vector corresponding to each label is obtained using BERT word vector mapping technology (which is D-dimensional). All label word vectors constitute a C×D-dimensional label word vector matrix The label word vector matrix M label Input the hash center generation network to obtain a hash center point matrix with a dimension of C×K at training round t

[0100] Specifically, for the VOC2012, MS-COCO and NUS-WIDE datasets, the values ​​of C are 20, 80 and 21 respectively.

[0101] The calculation process of the hash center point matrix is:

[0102]

[0103] Among them, g represents the hash center generation network, represents the learnable parameters of the hash center generation network at training round t.

[0104] (2-3) For each image sample x in the training set i For the image sample x i Establishing tag affinity coefficients Its initial value (i.e., at t=0) is equal to t represents the training round, and t∈[0,50]:

[0105] Specifically, this step is to set The total number of elements in is C.

[0106] (2-4) For each image sample x in the training set i For example, the image sample x obtained according to step (2-3) i The label affinity coefficient and the hash center point matrix obtained in step (2-2) Get the image sample x i The corresponding hash centroid r i , the hash centroids corresponding to all image samples at training round t constitute the hash centroid set

[0107] Specifically, this step is calculated using the following formula:

[0108]

[0109] in, Indicates that at training round t, the hash center point matrix E t The jth element in , and j∈[1,C], Represents image sample x i Label affinity coefficient at training round t The jth element of .

[0110] The advantages of the above steps (2-1) to (2-4) are that a label affinity coefficient related to the object size is proposed to reflect the degree of influence of objects of different sizes on the overall feature vector of the image; and a deep learning method is used to generate hash center points based on label semantic information (i.e., label word vectors); in addition, a method for calculating the hash center of mass of each image sample is also defined.

[0111] (2-5) The continuous value hash code matrix at training round t obtained according to step (2-1) And the hash centroid set at training round t obtained in steps (2-4) Get the loss function.

[0112] This step is specifically as follows: first, according to the image sample x i Corresponding continuous value hash code and its corresponding hash centroid r i t Get the similarity loss value;

[0113] This process uses the following formula:

[0114]

[0115]

[0116] Among them, the superscript t represents the training round, Represents a continuous value hash code Its corresponding hash centroid r i t The distance between them, γ represents a scaling factor based on the Cauchy distribution, preferably γ = 0.15.

[0117] Then, according to the image sample x i The consecutive value hash code and its binary hash code The difference between them is used to obtain the quantized loss value;

[0118] This process uses the following formula:

[0119]

[0120] Then, according to the label word vector obtained in step (2-2) And the hash centroid set obtained in steps (2-4) at training round t Get the loss value based on KL divergence (i.e. Kullback-Leibler divergence):

[0121] This process uses the following formula:

[0122]

[0123]

[0124]

[0125] Among them, the superscript t represents the training round, and Respectively represent the hash center point matrix E obtained in step (2-2) t The mth and nth elements in d m and d n Respectively represent the label word vector matrix M obtained in step (2-2) label The mth and nth elements in , and m,n∈[1,C].

[0126] Then, according to the hash center point matrix obtained in step (2-2) Get the inter-class loss value;

[0127] This process uses the following formula:

[0128]

[0129] Finally, the first loss value is obtained based on the similarity loss value and the quantization loss value obtained above, and the second loss value is obtained based on the similarity loss value, the loss value based on KL divergence, and the inter-class loss value:

[0130] This process uses the following formula:

[0131] L1=L center +L q

[0132] L2=L center +L kl +L class

[0133] The advantages of this step are: First, a center similarity loss based on the hash code of each image sample and the corresponding hash centroid is designed, aiming to continuously narrow the distance between each hash code and the corresponding hash centroid. Second, in order to make the hash center contain the information in the label word vector, a loss based on KL divergence is designed to make the relative distance between hash center points consistent with the relative distance between label word vectors, so that the distribution of the two is consistent. Third, in order to continuously increase the relative distance between hash centers, the inter-class loss is designed to continuously increase the distance between these hash center points. Fourth, the center similarity loss measures the distance between each hash code and the corresponding hash centroid, which has lower computational overhead than the method based on pairwise hash loss.

[0134] (2-6) According to the loss functions L1 and L2 obtained in step (2-5), the learnable parameters of the Swin Transformer network at training round t-1 for L1 are and Find the gradients separately, that is and Regarding L2 Taking the derivative we get Then use the following formula to get the updated learnable parameters and

[0135] This step specifically uses the following formula:

[0136]

[0137]

[0138]

[0139] Among them, α and β represent different learning rates, preferably α = 2×10 -6 , β=5×10 -4 .

[0140] (2-7) The initialized image obtained in step (2-3) is compared with each image sample x i The corresponding tag affinity coefficient As the initial value, according to the label affinity coefficient at training round t-1 And obtain the label affinity coefficient at training round t through mapping operation and iterative optimization

[0141] Specifically, this step includes the following sub-steps:

[0142] (2-7-1) According to the loss function L1 obtained in step (2-5), obtain the intermediate coefficient at training round t The jth element in Then we get the intermediate coefficient at training round t

[0143] This step specifically uses the following formula:

[0144]

[0145]

[0146] in, Indicates the label affinity coefficient at training round t-1 The jth element in , η=5×10-4 .

[0147] (2-7-2) The label affinity coefficient of the tth training round All elements in are arranged in descending order to obtain the sorting result u=[u1,u2,...,u C ];

[0148] (2-7-3) Calculate the quantization factor based on the sorting result u obtained in step (2-7-2)

[0149]

[0150] (2-7-4) Obtain the normalized bias based on the quantization factor p obtained in step (2-7-3) And obtain the label affinity coefficient at training round t based on the calculated normalized bias q Each element in Then obtain the label affinity coefficient of the tth training round

[0151]

[0152] The advantage of this step is that it proposes a Euclidean mapping method to map the tag affinity coefficients to meet the constraints. Furthermore, a gradient descent method is used to update these tag affinity coefficients, which helps them better adapt to the data distribution and ultimately results in a retrieval model with better retrieval performance.

[0153] (2-8) Using the step (2-6) Update the hash centroid matrix in steps (2-4)

[0154] (2-9) Using the N obtained in step (2-1) test The test set consists of images and N db The database consists of images, the retrieval model at training round t is tested and its MAP value is obtained;

[0155] (2-10) Repeat the above steps (2-1) to (2-9) until the upper limit of the training rounds (i.e., 50 times), and select the retrieval model with the largest MAP value as the final trained retrieval model.

[0156] Experimental results

[0157] The experimental environment of this invention is as follows: CPU is 10 Intel Xeon (R) @ 2.4GHz, GPU is 2 NVIDIA TeslaK40m 12GB, memory is 64GB DDR4, hard disk capacity is 8TB, under Ubuntu 16.04 operating system, PyTorch programming is used to implement this algorithm. The specific parameter settings are as follows: batch size is 64, the initial learning rate of the hash code generation module is 2×10 -6 , the initial learning rate of the hash centroid generation module is 5×10 -4 .

[0158] To illustrate the effectiveness of the hash centroid generation module in the present invention, the present invention conducted relevant tests on the VOC2012 dataset, MS-COCO dataset, and NUS-WIDE dataset with a preset code length of 64 bits, and recorded the average distance between the hash center points of the model at different epochs. Figure 2 The test results are given. Figure 2 It can be seen that when the model training converges, the average value of the mutual distances between all hash center points is greater than half the value of the preset hash code length, which meets the requirements of relevant theories, indicating that the present invention is effective.

[0159] In order to illustrate the effectiveness of the label affinity coefficient in the present invention and the superiority of the retrieval results of the present invention, the present invention has conducted relevant tests on the MS-COCO dataset and the NUS-WIDE dataset under the condition of 64-bit code length. The present invention records the retrieval MAP (Mean average precision) index of the model under two conditions: using the priori equal-weighted label affinity coefficient and using the self-learning label affinity coefficient proposed by the present invention. "Prior" means using a fixed averaged label affinity coefficient, and "self-learning" means using the self-learning label affinity coefficient proposed by the present invention. Table 1 shows the test results, that is, the retrieval MAP values ​​of the present invention under different types of label affinity coefficients:

[0160] Table 1

[0161]

[0162] As shown in Table 1, the present invention achieves higher MAP values, and thus better retrieval performance, on both datasets using the self-learned tag affinity coefficient. These results demonstrate that the self-learned tag affinity coefficient plays a crucial role in calculating the hash centroid that incorporates object scale information. They also demonstrate that hash centroids that consider both tag semantics and object scale can improve the model's retrieval performance.

[0163] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-label image retrieval method based on object scale perception, characterized in that: The following steps are involved: (1) Obtain the image to be retrieved and multiple image samples in the image database, and use the PIL library to convert the image to be retrieved and each image sample into multidimensional tensors respectively. All multidimensional tensors constitute a multidimensional tensor set; (2) Inputting the multidimensional tensor set obtained in step (1) into a pre-trained retrieval model to obtain the hash code corresponding to the image to be retrieved and the hash code corresponding to each image sample; the retrieval model includes a SwinTransformer network, a hash mapping layer, and a hash center generation network connected in sequence; The Swin Transformer network receives input dimensions of The output dimension is The matrix, The batch data size set during training; The hash map layer is composed of three fully connected layers; the input of the first layer is the Swin Transformer network output dimension is Matrix, this layer uses dimensional weight matrix, the output of this layer is dimensional matrix; the input of the second layer is the output of the first layer dimensional matrix, this layer uses dimensional weight matrix, the output is dimensional matrix; the input of the third layer is the output of the second layer dimensional matrix, this layer uses dimensional weight matrix, the output is dimensional matrix; where Indicates the hash code length; The hash center generation network consists of three stacked fully connected layers; the first layer input is dimensional label word vector matrix, this layer uses dimensional weight matrix, the output of this layer is dimensional matrix; The second layer input is the output of the first layer dimensional matrix, using dimensional weight matrix, the output of this layer is dimensional hash center point matrix; the third layer input is the second layer output dimensional matrix, using dimensional weight matrix, the output of this layer is dimensional hash center point matrix, where K represents the hash code bit length, and , Represents the number of all label categories contained in the image database, Represents the dimension of each label word vector; (3) For each hash code corresponding to the image sample obtained in step (2), calculate the similarity between the hash code corresponding to the image to be retrieved obtained in step (2), and select multiple image samples that match the image to be retrieved and output them based on the multiple similarity calculation results obtained.

2. The multi-label image retrieval method based on object scale perception according to claim 1, characterized in that: The retrieval model is trained through the following steps: (2-1) Acquisition The training set consists of images. The test set consists of images, and The database set consists of images; the training set image samples consist of a matrix , each image sample in the training set Input into the Swin Transformer network to extract the feature vector of each image sample in the training set, and then input the feature vector of each image sample into the hash map layer to obtain the feature vector of each image sample in the training round. When the image sample The corresponding bit length is The consecutive value hash code , all image samples corresponding to the training round The continuous value hash code when composes the matrix , each image sample The corresponding continuous value hash code is: ， in , and are all positive integers, represents the Swin Transformer network, Indicates the number of training rounds of the Swin Transformer network The learnable parameters when represents the hash map layer, Indicates the number of training rounds in the hash map layer The learnable parameters when ; (2-2) Get each image sample in the training set The label word vector corresponding to each label is obtained using BERT word vector mapping technology. All label word vectors constitute a dimensional label word vector matrix , the label word vector matrix Input hash center generator network to get The time dimension is The hash center matrix ; (2-3) For each image sample in the training set For the image sample Establishing tag affinity coefficients , whose initial value is equal to , represents the training round, and there are : (2-4) For each image sample in the training set For example, the image sample obtained according to steps (2-3) The label affinity coefficient and the hash center point matrix obtained in step (2-2) , get the image sample Corresponding hash centroid , all image samples corresponding to the training round The hash centroids at the time constitute the hash centroid set ; (2-5) The training rounds obtained according to step (2-1) The continuous value hash code matrix when , and the training rounds obtained in steps (2-4) The hash centroid set at Get the loss function; (2-6) Loss function obtained according to step (2-5) and ,against About training rounds The learnable parameters of the Swin Transformer network when and Find the gradients separately, that is and ,against about Taking the derivative we get , and then use the following formula to get the updated learnable parameters , and : (2-7) The initialized image obtained in step (2-3) is compared with each image sample The corresponding tag affinity coefficient As the initial value, according to the training round The tag affinity coefficient at , and obtain training rounds through mapping operations and iterative optimization The tag affinity coefficient at : (2-8) Using the results from step (2-6) Update the hash centroid matrix in steps (2-4) ; (2-9) Using the result obtained in step (2-1) The test set consists of images and The database consists of images, for training rounds The retrieval model is tested and its MAP value is obtained; (2-10) Repeat the above steps (2-1) to (2-9) until the upper limit of the training round is reached, and select the retrieval model with the largest MAP value as the final trained retrieval model.

3. The multi-label image retrieval method based on object scale perception according to claim 2, characterized in that: In step (2-2), the calculation process of the hash center point matrix is: ， in, represents the hash center generation network, Indicates the hash center generation network in the training round The learnable parameters when ; Steps (2-3) are to set , The total number of elements in is ; Steps (2-4) are calculated using the following formula: ， in, Indicates the number of training rounds When the hash center matrix The elements, and there are , Represents an image sample In the training round Time tag affinity coefficient No. elements.

4. The multi-label image retrieval method based on object scale perception according to claim 3 is characterized in that: Steps (2-5) are as follows: first, according to the image sample Corresponding continuous value hash code , and its corresponding hash centroid Get the similarity loss value; This process uses the following formula: ， Among them, the superscript represents the training rounds, Represents a continuous value hash code Its corresponding hash centroid The distance between represents the scaling factor based on the Cauchy distribution; Then, according to the image sample The consecutive value hash code and its binary hash code The difference between them is used to obtain the quantized loss value; This process uses the following formula: ， Then, according to the label word vector obtained in step (2-2) And steps (2-4) get the training round The hash centroid set at Get the loss value based on KL divergence, namely Kullback-Leiblerdivergence: This process uses the following formula: ， Among them, the superscript represents the training rounds, and Represents the hash center point matrix obtained in step (2-2) The Hedi elements, and Represent the label word vector matrix obtained in step (2-2) The Hedi elements, and ; Then, according to the hash center point matrix obtained in step (2-2) Get the inter-class loss value; This process uses the following formula: ， Finally, the first loss value is obtained based on the similarity loss value and the quantization loss value obtained above, and the second loss value is obtained based on the similarity loss value, the loss value based on KL divergence, and the inter-class loss value: This process uses the following formula: 。 5. The multi-label image retrieval method based on object scale perception according to claim 4 is characterized in that: Steps (2-6) are specifically implemented using the following formula: ， in, and They represent different learning rates.

6. The object scale-aware multi-label image retrieval method according to claim 5, characterized in that: Steps (2-7) include the following sub-steps: (2-7-1) Loss function obtained according to step (2-5) , get the training round The intermediate coefficient The Elements , and then get the training round The intermediate coefficient ; This step specifically uses the following formula: ， in, Indicates the training round The tag affinity coefficient at The elements, ; (2-7-2) Label affinity coefficient for training rounds Arrange all elements in the array from largest to smallest to get the sorted result. ; (2-7-3) Sorting results obtained according to step (2-7-2) Calculate the quantization factor ； (2-7-4) Quantization factor obtained according to step (2-7-3) Get normalized bias , and according to the calculated normalized bias Get training rounds The tag affinity coefficient at Each element in , and then obtain the Label affinity coefficient for training rounds : 。 7. A multi-label image retrieval system based on object scale perception, characterized in that: include: The first module is used to obtain the image to be retrieved and multiple image samples in the image database, and use the PIL library to convert the image to be retrieved and each image sample into multidimensional tensors respectively, and all multidimensional tensors constitute a multidimensional tensor set; The second module is used to input the multidimensional tensor set obtained in the first module into a pre-trained retrieval model to obtain the hash code corresponding to the image to be retrieved and the hash code corresponding to each image sample; the retrieval model includes a Swin Transformer network, a hash mapping layer, and a hash center generation network connected in sequence; The Swin Transformer network receives input dimensions of The output dimension is The matrix, The batch data size set during training; The hash map layer is composed of three fully connected layers; the input of the first layer is the Swin Transformer network output dimension is Matrix, this layer uses dimensional weight matrix, the output of this layer is dimensional matrix; the input of the second layer is the output of the first layer dimensional matrix, this layer uses dimensional weight matrix, the output is dimensional matrix; the input of the third layer is the output of the second layer dimensional matrix, this layer uses dimensional weight matrix, the output is dimensional matrix; where Indicates the hash code length; The hash center generation network consists of three stacked fully connected layers; the first layer input is dimensional label word vector matrix, this layer uses dimensional weight matrix, the output of this layer is dimensional matrix; the second layer input is the output of the first layer dimensional matrix, using dimensional weight matrix, the output of this layer is dimensional hash center point matrix; the third layer input is the second layer output dimensional matrix, using dimensional weight matrix, the output of this layer is dimensional hash center point matrix, where K represents the hash code bit length, and , Represents the number of all label categories contained in the image database, Represents the dimension of each label word vector; The third module is used to calculate the similarity between the hash code corresponding to each image sample obtained by the second module and the hash code corresponding to the image to be retrieved obtained by the second module, and select multiple image samples that match the image to be retrieved for output based on the multiple similarity calculation results.

Images