Method, device, electronic device and storage medium for knowledge migration of imbalanced data sets

Abstract

The present application belongs to a large-scale image retrieval method, and specifically relates to a knowledge migration method, device, electronic device and storage medium for an imbalanced data set. The method first randomly samples a balanced data set to obtain a long-tail data set that satisfies the Zipf law; the data therein are used to obtain the direct features of each image, and the feature vectors of each type of image are summed and averaged to obtain the class center feature vector; according to the class center feature vector and all feature vectors of each class, several edge feature vectors with the largest gap from the class center are obtained in each class; the class center vector and the class edge feature vector are aggregated together to obtain a memory module for the entire data set; the corresponding features are selected from the memory module to help the network migrate the knowledge learned from the head to the tail knowledge. The present disclosure does not need to resample the training set according to the degree of sample imbalance, or does not need to know the degree of sample imbalance when calculating the loss function.

Description

Technical Field

[0001] The present application relates to a large-scale image retrieval method, and specifically to a knowledge migration method, device, electronic device and storage medium for an imbalanced data set. Background Art

[0002] With the rapid development of deep learning technology, computer vision has made great progress in image classification, semantic segmentation and other aspects in recent years. Nowadays, more and more computer vision scenes are completed by deep neural networks. When there is a large amount of data, convolutional neural networks can extract the features of the image very well and complete the task very well. However, when the amount of data is insufficient, the convolutional network cannot play the expected role. In the real environment, image data is often unbalanced (long tail), that is, some classes have more data, while some classes have less data. Summary of the invention

[0003] The present application aims to solve the problems existing in the prior art to at least a certain extent. Based on the inventor's understanding and recognition of the following facts and problems, the main idea of ​​existing methods to solve the image retrieval problem of imbalanced data sets is: re-sample or re-weight the data according to the imbalance factor of the imbalanced data set during training, and transform the image retrieval problem of the imbalanced data set into the image retrieval problem of the balanced data set. This type of method requires the imbalance factor of the image. The present application can enable the neural network to transfer the knowledge learned from the class with large data volume in the imbalanced data set to the class with small data volume, and help the network to learn in the class with small data volume, and the main goal of transfer learning is to solve this type of problem. Therefore, the research on solving the problem of computer vision knowledge transfer under long-tail data sets is of great significance and can bring huge economic and social benefits.

[0004] In view of this, the purpose of the present invention is to propose a knowledge migration method, device, electronic device and storage medium for unbalanced data sets, and to realize a knowledge migration method for unbalanced data sets based on a memory mechanism. The method obtains direct feature vectors of all images of a long-tail data set based on a convolutional neural network, uses the feature vector to calculate the class center vector of each class of data, and then uses the determinant point process algorithm to obtain the feature memory module, thereby obtaining an extended dynamic meta-embedding module, thereby solving the important technical problem of knowledge migration of long-tail distribution data sets.

[0005] According to a first aspect of the present disclosure, a knowledge migration method for an imbalanced data set is proposed, comprising:

[0006] Step 1, construct an unbalanced image dataset;

[0007] Step 2, from each class of images in the imbalanced image data set, obtain the direct feature vector and class center vector of each image in a class of images, and calculate the class edge feature vector of a class of images;

[0008] Step 3, constructing a memory module according to the class center vector and the class edge feature vector;

[0009] Step 4: Use the self-attention method to obtain the knowledge transfer vector and complete the knowledge transfer of the imbalanced data set.

[0010] The present invention is a method for solving the image retrieval problem of large-scale datasets. Compared with existing methods, the present invention has the advantage of being insensitive to the degree of sample imbalance. This is mainly reflected in the fact that during the network training process, the present invention does not need to resample the training set according to the degree of sample imbalance, or does not need to know the degree of sample imbalance when calculating the loss function.

[0011] Optionally, constructing an unbalanced image dataset includes:

[0012] (1) Obtain a balanced image dataset with N types of images from the Internet; (balanced image data can be images of animals, plants, vehicles, buildings, etc. Balanced means that the number of images of each type in the dataset is the same)

[0013] (2) Setting multiple imbalance factors μ;

[0014] (3) Using Zipf's law, according to the imbalance factor μ, an imbalanced image dataset X is randomly extracted from the existing balanced image dataset.

[0015] Where n is x n is a set of images of a class in the imbalanced image dataset X that conforms to the Zipf law, y n For the image set x n The corresponding label.

[0016] Optionally, the method for obtaining the direct feature vector and class center vector of each image in the class of images is: inputting the unbalanced image dataset X into a Resnet34 network, and the Resnet34 network outputs the direct feature vector v of each image in the unbalanced image dataset X. direct , sum up the feature vectors of the same type of images and take the average value to get the class center vector v center .

[0017] Optionally, the method for calculating the class edge feature vector of each class of images according to the direct feature vector of each image in each class is:

[0018] Using the determinant point process (DPP) algorithm, all direct feature vectors v of a class of images are direct Find m vectors with the same class center in The m vectors with the longest distance are used as the class edge feature vectors v of this class of images. marginal .

[0019] Optionally, a memory module is constructed according to the class center vector and the class edge feature vector, and the specific process is as follows:

[0020] The class center vector v center Set as the initial memory module, the class edge feature vector v marginal After being spliced ​​to the initial memory module, the memory module M is obtained.

[0021] Optionally, the self-attention method is used to perform self-attention calculation on the memory module M to obtain a self-attention vector; the self-attention vector is concatenated with the direct feature vector to obtain a knowledge transfer vector v trans ,include:

[0022] (1) using a self-attention method to perform self-attention calculation on the memory module M to obtain a self-attention vector;

[0023] (2) After splicing the self-attention vector to the direct feature vector, the knowledge transfer vector v is obtained trans .

[0024] According to a second aspect of the present disclosure, a knowledge migration device for an imbalanced data set is provided, comprising:

[0025] Dataset construction module, used to construct an imbalanced image dataset;

[0026] A calculation module, used to obtain the direct feature vector and the class center vector of each image in a class of images from each class of images in the unbalanced image data set, and calculate the class edge feature vector of the class of images;

[0027] Memory building block, used to build a memory module;

[0028] The knowledge transfer module is used to obtain the knowledge transfer vector using the self-attention method to complete the knowledge transfer of the imbalanced data set.

[0029] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0030] A memory for storing instructions executable by the processor;

[0031] A processor configured to execute:

[0032] Step 1, construct an unbalanced image dataset;

[0033] Step 2, from each class of images in the imbalanced image data set, obtain the direct feature vector and class center vector of each image in a class of images, and calculate the class edge feature vector of a class of images;

[0034] Step 3, constructing a memory module according to the class center vector and the class edge feature vector;

[0035] Step 4: Use the self-attention method to obtain the knowledge transfer vector and complete the knowledge transfer of the imbalanced data set.

[0036] According to a fourth aspect of the present disclosure, a computer readable storage medium is provided, wherein a computer program is stored on the computer readable storage medium, and the computer program is used to cause the computer to execute:

[0037] Construct an imbalanced image dataset;

[0038] From each class of images in the imbalanced image data set, obtain a direct feature vector and a class center vector of each image in a class of images, and calculate a class edge feature vector of a class of images;

[0039] Build memory modules;

[0040] The self-attention method is used to obtain the knowledge transfer vector and complete the knowledge transfer of imbalanced data sets.

[0041] According to the embodiments of the present disclosure, it is not sensitive to the degree of image imbalance. The class center vector calculated by image processing is first used to represent the class. Considering that the class center vector of a small number of samples cannot accurately represent the class, the present disclosure further uses the DPP algorithm to extract the class edge feature vector, obtains a relatively complete description of each class in the data set and stores the description in the memory module, and then uses the self-attention mechanism to process the memory module to find the association and difference between the features of different classes, thereby realizing the migration of knowledge from classes with large data volume to classes with small data volume, thereby completing the image retrieval problem of large-scale imbalanced data sets.

[0042] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A logical block diagram of a knowledge migration method for an imbalanced data set is shown according to an embodiment of the present disclosure.

[0045] Figure 2 A structural diagram of a knowledge transfer method for an unbalanced data set based on an attention mechanism according to an embodiment of the present disclosure

[0046] Figure 3 A structural block diagram of a knowledge migration device for an imbalanced data set is shown according to an embodiment of the present disclosure.

[0047] Figure 4 Log-log curves of each data set at different IF values.

[0048] Figure 5 Hash retrieval results for long-tail datasets with different mini-batch values.

[0049] Figure 6 Comparison of results between different methods on a long-tailed dataset based on cifar100.

[0050] Figure 7 Comparison of results between different methods on a long-tail dataset built on imagenet100. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the implementation described is only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0052] Figure 1 1 is a logic block diagram of a knowledge migration method for an unbalanced data set according to an embodiment of the present disclosure. The knowledge migration method for an unbalanced data set shown in this embodiment can be applied to user devices, such as mobile phones, tablet computers, etc.

[0053] like Figure 2 As shown, the knowledge migration method of the imbalanced data set may include the following steps:

[0054] In step 1, an imbalanced image dataset is constructed;

[0055] In one embodiment, constructing an unbalanced image dataset includes:

[0056] (1) Obtain a balanced image dataset with N types of images from the Internet; (balanced image data can be images of animals, plants, vehicles, buildings, etc., and balanced means that the number of images of each type in the dataset is the same).

[0057] (2) Setting multiple imbalance factors μ;

[0058] (3) Using Zipf's law, according to the imbalance factor μ, an imbalanced image dataset X is randomly extracted from the balanced image dataset.

[0059] Among them, n is the serial number of the image category, x n is a set of images of a class in the imbalanced image dataset X that conforms to the Zipf law, y n For the image set x n The corresponding label.

[0060] By setting different imbalance factors μ, multiple long-tail distribution data sets can be constructed. Let n(i) represent the number of samples in the i-th category, i = (1, 2, ..., N), and generally assume that n(1) ≥ n(2) ≥ ... ≥ n(N). Based on the above content, the rules that the constructed long-tail data set needs to satisfy are as follows:

[0061] n(i)=n(1)×i -μ

[0062] where μ can be defined by n(1) / n(N).

[0063] In step 2, from each class of images in the imbalanced image data set, a direct feature vector and a class center vector of each image in a class of images are obtained, and a class edge feature vector of a class of images is calculated.

[0064] In one embodiment, the method for obtaining the direct feature vector and class center vector of each image in the class of images is: inputting the unbalanced image dataset X into the Resnet34 network, and the Resnet34 network outputs the direct feature vector v of each image in the unbalanced image dataset X. direct , sum up the feature vectors of the same type of images and take the average value to get the class center vector v center .

[0065] The class center vector is composed of the direct feature vector v of each image in each class direct The sum is then averaged.direct It is the feature vector obtained by the pre-trained Resnet34 network.

[0066] Get v direct Then, the class center vector of each class of images is for:

[0067]

[0068] In one embodiment, the method for calculating the class edge feature vector of each class of images according to the direct feature vector of each image in each class is:

[0069] Using the determinant point process (DPP) method, all direct feature vectors v of a class of images are direct Find m vectors with the same class center in The m vectors with the longest distance are used as the class edge feature vectors v of this class of images. marginal .

[0070] In step 3, the memory module is constructed.

[0071] In one embodiment, the specific process of constructing a memory module is as follows:

[0072] The class center vector v center Set as the initial memory module, the class edge feature vector v marginal After being spliced ​​to the initial memory module, the memory module M is obtained.

[0073] In step 4, the self-attention method is used to obtain the knowledge transfer vector and complete the knowledge transfer of the imbalanced data set.

[0074] In one embodiment, the method of using the self-attention method to obtain the knowledge transfer vector and complete the knowledge transfer of the imbalanced data set includes:

[0075] (1) using a self-attention method to perform self-attention calculation on the memory module M to obtain a self-attention vector;

[0076] (2) After splicing the self-attention vector to the direct feature vector, the knowledge transfer vector v is obtained trans .

[0077] The present invention is a method for solving the image retrieval problem of large-scale datasets. Compared with existing methods, the present invention has the advantage of being insensitive to the degree of sample imbalance. This is mainly reflected in the fact that during the network training process, the present invention does not need to resample the training set according to the degree of sample imbalance, or does not need to know the degree of sample imbalance when calculating the loss function.

[0078] Corresponding to the above-mentioned knowledge migration method for an unbalanced data set, the present disclosure also proposes a knowledge migration device for an unbalanced data set.

[0079] Figure 3 FIG. 1 is a schematic block diagram of a knowledge migration device for an unbalanced data set proposed in an embodiment of the present disclosure. Figure 3 As shown, the knowledge transfer device of the imbalanced data set includes:

[0080] Dataset construction module, used to construct an imbalanced image dataset;

[0081] A calculation module, used to obtain the direct feature vector and the class center vector of each image in a class of images from each class of images in the unbalanced image data set, and calculate the class edge feature vector of the class of images;

[0082] Memory building block, used to build a memory module;

[0083] The knowledge transfer module is used to obtain the knowledge transfer vector using the self-attention method to complete the knowledge transfer of the imbalanced data set.

[0084] The embodiment of the present disclosure further provides an electronic device, including:

[0085] A memory for storing instructions executable by the processor;

[0086] A processor, the processor being configured to execute:

[0087] Construct an imbalanced image dataset;

[0088] From each class of images in the imbalanced image data set, obtain a direct feature vector and a class center vector of each image in a class of images, and calculate a class edge feature vector of a class of images;

[0089] Build memory modules;

[0090] The self-attention method is used to obtain the knowledge transfer vector and complete the knowledge transfer of imbalanced data sets.

[0091] The present disclosure also provides a computer storable medium, on which a computer program is stored, wherein the computer program is used to cause the computer to execute:

[0092] Construct an imbalanced image dataset;

[0093] From each class of images in the imbalanced image data set, obtain a direct feature vector and a class center vector of each image in a class of images, and calculate a class edge feature vector of a class of images;

[0094] Build memory modules;

[0095] The self-attention method is used to obtain the knowledge transfer vector and complete the knowledge transfer of imbalanced data sets.

[0096] The present disclosure is not sensitive to the degree of image imbalance. It first uses the class center vector calculated by image processing to represent the class. Considering that the class center vector of a small number of samples cannot accurately represent the class, the present disclosure further uses the DPP algorithm to extract the class edge feature vector, obtains a relatively complete description of each class in the data set and stores the description in the memory module, and then uses the self-attention mechanism to process the memory module to find the association and difference between the features of different classes, thereby realizing the migration of knowledge from classes with large data volume to classes with small data volume, thereby solving the image retrieval problem of large-scale imbalanced data sets.

[0097] It should be noted that in the embodiments of the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The memory may be used to store the computer program and / or module, and the processor implements various functions of the automobile parts picture data set making device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Sma rt Med ia Car rd, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. If the module / unit of the construction device of the stable operation domain of the wind power system is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present disclosure, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0098] The following is a detailed description of the knowledge migration method for imbalanced data sets disclosed in the present invention through specific embodiments. Figure 1 As shown, it mainly includes:

[0099] (1) The hash retrieval problem of CIFAR100 and imagenet100 datasets is explained. According to artificial settings, three imbalance factors are randomly extracted from the two datasets using Zipf's law to construct new datasets, which are IF=1, IF=50, and IF=100;

[0100] (2) The dataset is used as the input of a trained convolutional neural network (Resnet34), which outputs a 512-dimensional feature vector. A 2000-dimensional direct feature vector is obtained through a fully connected layer. The class center vector of each class is calculated using the direct feature vector, and then three diverse features are extracted from each class using the determinant point process to obtain the class edge vector.

[0101] (4) concatenate the class center vector and the class edge vector together to form a memory module for each class in the image;

[0102] (5) The self-attention mechanism is used to process the data stored in the memory module, and then it is concatenated with the class edge vector to obtain a knowledge transfer vector with knowledge transfer capabilities.

[0103] (6) Based on the knowledge transfer vector learning data hash code, the binary code of the database and query set can be obtained.

[0104] The present invention verifies the two-stage deep hash retrieval method under the long-tail distribution data set with data sets of different IF factors.

[0105] Two datasets were selected: Cifar100 and ImageNet100.

[0106] According to formula (1), three data sets with different imbalance factors (IF) are selected from the above two data sets, including: IF=1, IF=50, IF=100, where IF=1 represents a balanced data set, IF=50 represents a long-tail data set with a medium deviation, and IF=100 represents a long-tail data set with a very large deviation;

[0107] Several existing data sets are set for comparison, including shallow algorithms: LSH, PCAH, ITQ, KNNH, SDH, COSDISH, FastHash, FSSH, SCDH, and deep algorithms: DPSH, HashNet, DSDH, LTH sq (k=0), LTH sq (k=3), LTH(k=0), LTH(k=3), the results are as follows Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown;

[0108] According to step 2, the feature memory module is calculated, and according to steps 3 and 4, the knowledge transfer vector is calculated. The knowledge transfer vector is used as the input of a binary neural network to obtain the binary encoding and prediction label of the data set. After testing the database and query set, the average accuracy of learning under different data sets can be obtained.

Claims

1. A knowledge transfer method for an unbalanced data set, characterized in that: include: Construct an imbalanced image dataset; From each class of images in the imbalanced image data set, obtain a direct feature vector and a class center vector of each image in a class of images, and calculate a class edge feature vector of a class of images; Build memory modules; The self-attention method is used to obtain the knowledge transfer vector and complete the knowledge transfer of the imbalanced data set; The constructing of an imbalanced image dataset comprises: (1) Obtain a balanced image dataset with N types of images from the network; (2) Setting multiple imbalance factors μ; (3) Using Zipf's law, according to the imbalance factor μ, an imbalanced image dataset X is randomly extracted from the balanced image dataset. , Among them, n is the serial number of the image category, is a set of images in the imbalanced image dataset X that conforms to the Zipf law. For image collection The corresponding label; The method for obtaining the direct feature vector and class center vector of each image in the class of images is as follows: the unbalanced image dataset X is input into the Resnet34 network, and the Resnet34 network outputs the direct feature vector of each image in the unbalanced image dataset X. , sum up the feature vectors of the same type of images and take the average value to get the class center vector ; The method for calculating the class edge feature vector of each class of images according to the direct feature vector of each image in each class is: Using the determinant point process method, all direct feature vectors of a class of images are Find m vectors with the same class center in The m vectors with the longest distance are used as the class edge feature vectors of this class of images. .

2. The knowledge migration method of the imbalanced data set according to claim 1, characterized in that: The specific process of constructing a memory module is as follows: The class center vector Set as the initial memory module, the class edge feature vector After being spliced ​​to the initial memory module, the memory module M is obtained.

3. The knowledge migration method of the imbalanced data set according to claim 1, characterized in that: The self-attention method is used to obtain the knowledge transfer vector to complete the knowledge transfer of the imbalanced data set, including: (1) using a self-attention method to perform self-attention calculation on the memory module M to obtain a self-attention vector; (2) After splicing the self-attention vector to the direct feature vector, a knowledge transfer vector vtrans is obtained.

4. A knowledge migration device for an unbalanced data set, characterized in that: The knowledge migration device is configured to execute the knowledge migration method for an unbalanced data set according to any one of claims 1 to 3, and the knowledge migration device comprises: Dataset construction module, used to construct an imbalanced image dataset; A calculation module, used to obtain the direct feature vector and the class center vector of each image in a class of images from each class of images in the unbalanced image data set, and calculate the class edge feature vector of the class of images; Memory building block, used to build a memory module; The knowledge transfer module is used to obtain the knowledge transfer vector using the self-attention method to complete the knowledge transfer of the imbalanced data set.

5. An electronic device, characterized in that: include: A memory for storing instructions executable by the processor; A processor, wherein the processor is configured to execute the knowledge migration method for an imbalanced data set as described in any one of claims 1-3.

6. A computer storable medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to enable the computer to execute the knowledge migration method for an unbalanced data set according to any one of claims 1 to 3.

Images