A long tail data hash code generation method and system based on mutual information

Abstract

The application relates to the technical field of image data processing, and discloses a long-tail data hash code generation method and system based on mutual information, which comprises the following steps: acquiring image data and category labels in a long-tail data set; inputting the image data into a feature extraction network combined with feature enhancement to obtain image data features subjected to feature enhancement; inputting the image data features into a hash activation layer to generate corresponding binary hash codes; and constructing a loss function based on the maximization of mutual information between the category labels and the image data features, so as to optimize the feature extraction network and the hash activation layer, obtain a trained long-tail hash model, and generate long-tail data hash codes. The application integrates the theory of mutual information into a deep hash method, maximizes the mutual information between image features and labels, improves the correlation between the features and the labels, reduces the influence of the long-tail distribution, and can quickly and accurately search in image data subjected to the long-tail distribution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and more particularly, to a long-tail data hash code generation method and system based on mutual information. BACKGROUND

[0002] With the development of network and media, the data volume of images is growing explosively, and we can obtain a large amount of image data from network devices and media every day. These images are huge in quantity, and most of the time, users only need to use a part of the image data, which may be a certain type of image or a specific type of image. Therefore, how to efficiently and accurately retrieve specific content from such a large image database according to the user's demand is one of the challenging tasks of image retrieval.

[0003] As an important branch of image retrieval methods, deep hashing methods have received more and more attention. The deep hashing-based image retrieval method uses a deep neural network to extract features from the input image, converts it into a high-dimensional feature vector, and maps the vector to a low-dimensional binary hash code through a learned hash function. This hash-based method has very fast retrieval speed while ensuring the accuracy of the retrieval results. However, the image data in reality is often in a long-tail distribution, and the number of images in each category is unevenly distributed. In the long-tail distribution data, the category with a large number of images is called the head class, and the category with a small number of images is called the tail class. For the head class, because the number of samples is sufficient, the network trained has strong feature extraction ability for these class data, and the retrieval effect is good; while for the tail class, because the number of samples is insufficient, the network cannot extract good features, resulting in poor retrieval effect. On the whole data set, the different feature extraction capabilities of the head class and the tail class interact with each other, resulting in poor retrieval results of existing deep hashing methods on long-tail data sets. SUMMARY

[0004] In order to overcome the defect that the existing deep hashing method has poor retrieval results on the long-tail data set, the present application provides a long-tail data hash code generation method and system based on mutual information.

[0005] To solve the above technical problems, the technical solutions of the present application are as follows:

[0006] A long-tail data hash code generation method based on mutual information, comprising the following steps:

[0007] Obtain image data and its class label in the long-tail data set;

[0008] Input the image data into the feature extraction network combined with feature enhancement to obtain the image data features after feature enhancement;

[0009] inputting the image data features into a hash activation layer to generate corresponding binary hash codes;

[0010] constructing a loss function based on maximizing mutual information between the class labels and the image data features, for optimizing the feature extraction network and the hash activation layer, to obtain a long-tail hash model after training, for generating long-tail data hash codes.

[0011] Further, the application also proposes a long-tail data hash code generation system based on mutual information, which applies the long-tail data hash code generation method of the application. The system comprises:

[0012] a data acquisition module for acquiring image data and class labels thereof in a long-tail data set;

[0013] a feature extraction module for inputting the image data into a feature extraction network combined with feature enhancement, to obtain image data features after feature enhancement;

[0014] a hash activation module for inputting the image data features into a hash activation layer to generate corresponding binary hash codes;

[0015] an optimization module for constructing a loss function based on maximizing mutual information between the class labels and the image data features, for optimizing the feature extraction network and the hash activation layer, to obtain a long-tail hash model after training, for generating long-tail data hash codes.

[0016] Further, the application also proposes a device comprising a memory and a processor, the memory storing computer readable instructions, wherein the computer readable instructions are executed by the processor to perform all or part of the steps of the long-tail data hash code generation method of the application.

[0017] Further, the application also proposes a storage medium storing computer readable instructions, wherein the computer readable instructions are executed by a processor to implement all or part of the steps of the long-tail data hash code generation method of the application.

[0018] Compared with the prior art, the technical scheme of the application has the following beneficial effects:

[0019] The application integrates the theory of mutual information into the deep hash method, maximizes the mutual information between the image features and the labels, improves the correlation between the features and the labels, reduces the influence of the long-tail distribution, and can quickly and accurately search in the image data subject to the long-tail distribution; the user can overcome the influence of the uneven number of images of different categories and quickly and accurately search the images. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 Flow chart of the long-tail data hash code generation method according to an embodiment of the present application.

[0021] Figure 2 Flow chart of the long-tail data hash code generation method according to an embodiment of the present application.

[0022] Figure 3 Architecture diagram of the long-tail data hash code generation system according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The exemplary embodiments will be described in detail herein below with reference to the drawings. In the following description, the same drawings refer to the same or similar elements unless otherwise indicated. The following exemplary embodiments described in the following examples are not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0024] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0025] It should be understood that although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a particular order or hierarchy. These terms are used only to distinguish one from another. For example, a first information can be termed a second information, and similarly, a second information can be termed a first information, without departing from the scope of the present application. As used herein, the word "if' can be interpreted to mean "when" or "upon" or "in response to determining" taking into account the context in which the term is used.

[0026] The present application will be described in detail below with reference to the drawings and specific embodiments.

[0027] Embodiment 1

[0028] Referring to Figure 1 , 2 , the present embodiment proposes a long-tail data hash code generation method based on mutual information, which comprises the following steps:

[0029] S1, obtaining image data and its class label in a long-tail data set;

[0030] S2, input the image data into a feature extraction network combined with feature enhancement, to obtain image data features after feature enhancement;

[0031] S3, input the image data features into a hash activation layer to generate corresponding binary hash codes;

[0032] S4, construct a loss function based on maximizing the mutual information between the class labels and the image data features, for optimizing the feature extraction network and the hash activation layer, to obtain a trained long-tail hash model for generating long-tail data hash codes.

[0033] In this embodiment, the theory of mutual information is integrated into the deep hash method, by maximizing the mutual information between the image features and the labels, the correlation between the features and the labels is improved, the influence of the long-tail distribution is reduced, and fast and accurate retrieval can be performed in image data subject to long-tail distribution.

[0034] The trained long-tail hash model includes a trained feature extraction network and a hash activation layer. In actual application, after the image data to be processed is sequentially passed through the trained feature extraction network and the hash activation layer, the corresponding hash codes can be generated.

[0035] In an optional embodiment, the S1 step further includes the following steps:

[0036] The class labels corresponding to the images in the obtained long-tail data set are preprocessed to obtain a one-hot vector of the class labels, which is used to convert the class labels into a binary vector that can be understood by machine learning algorithms.

[0037] In an optional embodiment, when the image data is input into the feature extraction network combined with feature enhancement in the S2 step, the following steps are performed:

[0038] The image I is input into the feature extraction network to obtain the image features f(I) of the image I;

[0039] The image features f(I) are input into the feature enhancement module to obtain the image features f'(I) after feature enhancement.

[0040] Further, in an optional embodiment, the feature extraction network includes a pre-trained ResNet-50 architecture on the ImageNet data set.

[0041] The ResNet-50 is a deep convolutional neural network architecture with a network structure of 50 layers, including an input layer, a residual block, a global average pooling layer, and a fully connected layer. It can learn rich feature representations through multi-scale feature extraction and residual connection, and has strong generalization ability.

[0042] Further, in an optional embodiment, the feature enhancement module comprises a cross-attention feature enhancement (CAFE) module based on ACHNet or a dynamic meta-embedding (DME) module based on LTHNet.

[0043] ACHNet (Attention-guided Contrastive Hashing for Long-tailed Image Retrieval) is an attention-guided contrastive hashing method for long-tailed image retrieval, which enhances features through cross-attention mechanisms to improve the retrieval performance of the model under long-tailed data distribution.

[0044] LTHNet (Long-Tail Hashing Network) is a deep hashing network for processing long-tailed data distribution, which aims to enrich feature representation by combining direct features and memory features, and can effectively improve the recognition performance of the model under long-tailed data distribution.

[0045] In an optional embodiment, in S3, the image data features are input into a hash activation layer to generate corresponding binary hash codes, including the following steps:

[0046] The feature-enhanced image data features are input into a hash activation layer, and the image data features are reduced in dimension through several fully connected layers.

[0047] The reduced image data features are converted into binary hash codes through an activation function.

[0048] Among them, the hash activation layer contains multiple fully connected layers that map high-dimensional features into low-dimensional binary codes of fixed code length, and the features after dimension reduction are output as binary hash codes composed of -1 and +1 through an activation function.

[0049] Further optionally, the activation function includes a Sign function, a Tanh function, and a Sigmoid function.

[0050] For example, in this embodiment, a Tanh function is selected for binary hash code conversion.

[0051] In an optional embodiment, in S4, the loss function based on the mutual information between the class label and the image data features is constructed, and the loss function includes an InfoNCE loss; its expression is:

[0052]

[0053] wherein (f',y) represents a joint distribution between the feature-enhanced image data feature f' and its category label y; W is a trainable weight matrix for realizing a bilinear transformation of the image data feature f' to the category label y; f' j represents the jth feature-enhanced image data feature in the joint distribution (f',y);

[0054] The InfoNCE loss is used to maximize the mutual information between the category label and the image data feature, and the feature extraction network and the hash activation layer are optimized by using the mini-batch gradient descent method to obtain the trained long-tail hash model.

[0055] In this embodiment, the InfoNCE loss is used to maximize the mutual information, which has the advantages of being convenient and easy to use and excellent effect.

[0056] As an illustrative example, the InfoNCE loss is a loss function for self-supervised learning, which learns model parameters by comparing the similarity of positive and negative samples to improve the discriminability of features. It is defined as:

[0057]

[0058] wherein X represents a sample set, and f(x,y) is a similarity function for measuring the similarity between the positive sample x and the negative sample y.

[0059] In this embodiment, the feature-enhanced image data feature is taken as the x sample, the label is taken as the y sample, and By using the properties of logarithm, the following equation is obtained:

[0060]

[0061] In practical applications, the F(x,y) often uses a trainable weight matrix W to realize the bilinear transformation of the x sample to the y sample, that is:

[0062] F(x,y)=xWy

[0063] The expression of the InfoNCE loss function applied to maximize the mutual information between the category label and the image data feature can be obtained by combining the above equation. Using this loss function and optimizing the feature extraction network and the hash activation layer by using the mini-batch gradient descent method, a trained long-tail hash model can be obtained, which is used for image retrieval in a long-tail data set. In practical applications, users can overcome the influence of the imbalance of the number of images of different categories and quickly and accurately perform image retrieval.

[0064] Embodiment 2

[0065] Please refer to Figure 3The embodiment applies the long-tail data hash code generation method proposed in Embodiment 1, and proposes a long-tail data hash code generation system based on mutual information.

[0066] The long-tail data hash code generation system based on mutual information proposed in the embodiment comprises:

[0067] A data acquisition module is configured to acquire image data and class labels in a long-tail data set.

[0068] A feature extraction module is configured to input the image data into a feature extraction network combined with feature enhancement to obtain image data features after feature enhancement.

[0069] A hash activation module is configured to input the image data features into a hash activation layer to generate corresponding binary hash codes.

[0070] An optimization module is configured to construct a loss function based on maximizing mutual information between class labels and image data features, to optimize the feature extraction network and the hash activation layer, to obtain a trained long-tail hash model for generating long-tail data hash codes.

[0071] In an optional embodiment, the data acquisition module is further configured to preprocess the class labels corresponding to the images in the acquired long-tail data set to obtain one-hot vectors of the class labels.

[0072] In an optional embodiment, the feature extraction module is mounted with a ResNet-50 architecture, and the ResNet-50 architecture is pre-trained on an ImageNet data set.

[0073] In an optional embodiment, the feature extraction module further comprises an ACHNet-based cross-attention feature enhancement module or an LTHNet-based dynamic meta-embedding module, which is used as a feature enhancement module to enhance image features.

[0074] In an optional embodiment, when the feature extraction module inputs the image data into the feature extraction network combined with feature enhancement, the following steps are performed:

[0075] The image I is input into the feature extraction network to obtain image features f(I) of the image I.

[0076] The image features f(I) are input into the feature enhancement module to obtain image features f'(I) after feature enhancement.

[0077] In an optional embodiment, when the hash activation module inputs the image data features into the hash activation layer to generate corresponding binary hash codes, the following steps are performed:

[0078] The feature-enhanced image data features are input into a hash activation layer, and the image data features are reduced in dimension by several fully connected layers;

[0079] The reduced image data features are converted into binary hash codes by an activation function.

[0080] In an optional embodiment, the loss function based on maximizing the mutual information between the class labels and the image data features in the optimization module comprises an InfoNCE loss; the expression of which is:

[0081]

[0082] wherein (f′,y) represents the joint distribution between the feature-enhanced image data features f′ and the class labels y; W is a trainable weight matrix for realizing the bilinear transformation of the image data features f′ to the class labels y; f′ j represents the jth feature-enhanced image data feature in the joint distribution (f′,y);

[0083] The InfoNCE loss is used to maximize the mutual information between the class labels and the image data features, and the feature extraction network and the hash activation layer are optimized by using the small batch gradient descent method to obtain the trained long-tail hash model.

[0084] It can be understood that the system of the embodiment corresponds to the method of the above-mentioned embodiment 1, and the optional items in the above-mentioned embodiment 1 are also applicable to the embodiment, so the description is not repeated here.

[0085] Embodiment 3

[0086] The embodiment proposes a computer device, which comprises a memory and a processor, and the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to make the processor execute all or part of the steps of the long-tail data hash code generation method proposed in embodiment 1.

[0087] Embodiment 4

[0088] The embodiment proposes a storage medium, which stores computer readable instructions, wherein the computer readable instructions are executed by a processor to realize all or part of the steps of the long-tail data hash code generation method proposed in embodiment 1.

[0089] Exemplarily, the storage medium comprises but is not limited to a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0090] Exemplarily, the instructions, programs, code sets or instruction sets can be implemented in a conventional programming language.

[0091] Exemplarily, the processor includes but is not limited to a smart phone, a personal computer, a server, a network device, etc., and is used to execute all or part of the steps of the long tail data hash code generation method described in Embodiment 1.

[0092] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, for the device embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts can be referred to the part of the description of the method embodiments. The device embodiments described above are only exemplary, and the modules described as separate components can be or can not be physically separated, and the functions of the modules can be implemented in the same or multiple software and / or hardware when implementing the solutions of the present application. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0093] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the implementation manner of the present application. Any modification, equivalent replacement and improvement made on the basis of the above description for those skilled in the art should be included in the protection scope of the claims of the present application.

Claims

1. A method for generating long-tailed data hash codes based on mutual information, characterized in that, Includes the following steps: Obtain image data and its category labels from a long-tailed dataset; The image data is input into a feature extraction network that combines feature enhancement to obtain image data features that have been enhanced. The image data features are input into the hash activation layer to generate the corresponding binary hash code; A loss function is constructed based on maximizing the mutual information between category labels and image data features. This function is used to optimize the feature extraction network and the hash activation layer, resulting in a trained long-tail hash model used to generate long-tail data hash codes. The image data is input into a feature extraction network that incorporates feature enhancement, including the following steps: Image I is input into the feature extraction network to obtain the image features f(I) of image I; The image feature f(I) is input into the feature enhancement module to obtain the enhanced image feature f′(I); The feature enhancement module includes a cross-attention feature enhancement module based on ACHNet or a dynamic meta-embedding module based on LTHNet. The loss function, constructed based on maximizing the mutual information between category labels and image data features, includes the InfoNCE loss; its expression is: Where (f′,y) represents the joint distribution between the enhanced image data features f′ and their class labels y; W is a trainable weight matrix used to implement the bilinear transformation from image data features f′ to class labels y; f′ j Let f' represent the j-th image data feature that has undergone feature enhancement in the joint distribution (f', y); The mutual information between the category label and the image data features is maximized using InfoNCE loss, and the feature extraction network and the hash activation layer are optimized using mini-batch gradient descent to obtain a trained long-tail hash model.

2. The method for generating long-tailed data hash codes according to claim 1, characterized in that, The feature extraction network includes a ResNet-50 architecture pre-trained on the ImageNet dataset.

3. The method for generating long-tailed data hash codes according to claim 1, characterized in that, The feature enhancement module includes a cross-attention feature enhancement module based on ACHNet or a dynamic meta-embedding module based on LTHNet.

4. The method for generating long-tailed data hash codes according to claim 1, characterized in that, The image data features are input into the hash activation layer to generate the corresponding binary hash code, including the following steps: The enhanced image data features are input into a hash activation layer, and the image data features are then reduced in dimensionality through several fully connected layers. The dimensionality-reduced image data features are converted into binary hash codes using an activation function.

5. The method for generating long-tailed data hash codes according to any one of claims 1 to 4, characterized in that, The method further includes the following steps: preprocessing the category labels corresponding to the images in the acquired long-tail dataset to obtain the one-hot vectors of the category labels.

6. A long-tail data hash code generation system based on mutual information, employing the long-tail data hash code generation method according to any one of claims 1 to 5, characterized in that, The system includes: The data acquisition module is used to acquire image data and its category labels from long-tail datasets; The feature extraction module is used to input the image data into a feature extraction network that combines feature enhancement to obtain the enhanced image data features. The hash activation module is used to input the image data features into the hash activation layer to generate the corresponding binary hash code; An optimization module is used to construct a loss function based on maximizing the mutual information between the category label and the image data features, and to optimize the feature extraction network and the hash activation layer to obtain a trained long-tail hash model, which is used to generate long-tail data hash codes.

7. A device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that, When the computer-readable instructions are executed by the processor, the processor performs all or part of the steps of the long-tailed data hash code generation method as described in any one of claims 1 to 5.

8. A storage medium having computer-readable instructions stored thereon, characterized in that, When the computer-readable instructions are executed by a processor, they implement all or part of the steps of the long-tailed data hash code generation method as described in any one of claims 1 to 5.

Images