Method and system for representative and diverse landmark profile image selection

Abstract

The application provides a label distribution image selection method and system based on representation and diversity, comprising: acquiring an image dataset and performing image feature extraction; training an initial label distribution learning model according to a small part of labeled dataset; calculating a dissimilarity matrix and a diversity matrix by using the features of the unlabeled image sample set and the label distribution of the unlabeled image sample predicted according to the label distribution learning model; and selecting valuable image samples according to the dissimilarity matrix and the diversity matrix for query. The application actively selects the most valuable images as training samples from the two aspects of representation and diversity, constructs a label distribution model of active learning, and has great significance for improving the performance of the label distribution model under the condition of insufficient image label data.

Description

TECHNICAL FIELD

[0001] The present application relates to a digital image retrieval technology, in particular to a label distribution image selection method and system based on representativeness and diversity. BACKGROUND

[0002] Compared with single label learning (SLL) and multi-label learning (MLL) which can only solve the 0 and 1 problem, i.e. whether each label is related to an example, in the label distribution learning (LDL) paradigm, the relevance of the label to the example is given by an exact value between 0 and 1, and each label has a different description degree for the example. For example, there are three labels of "sea", "sky" and "beach" in a picture, and the output of MLL can only know that the three labels are related to the image, but cannot know which label is most relevant to the image.

[0003] Although label distribution learning has been successfully applied in recent years, to obtain a model with better performance, it is often necessary to train and learn on a large number of samples, i.e. sufficient labeled data is needed. However, in the label distribution learning framework, the relevant categories of samples change from single to multiple, and the output space of learning is sharply expanded, making it more difficult to train a model, and therefore, in order to improve the performance of the model, more training samples are often needed. In addition, when using label distribution instead of logical label to annotate examples, the importance of each label needs to be considered, which means a higher annotation cost than traditional multi-label learning.

[0004] However, in real applications, the labeled data set is limited, and it is often easier to collect a large amount of unlabeled data, but it is time-consuming and laborious to annotate these data. For example, when performing facial expression discrimination, a large number of facial expression images can be obtained from the real world, but annotating these images not only requires discriminating the expressions contained in each picture, but also requires discriminating the description degree of each expression for the face image, which requires a lot of manpower and time. The problem of lack of labeled data is widespread in label distribution learning for image classification, which largely limits the development of label distribution learning. The active learning method in machine learning can discover the most valuable supervised information for improving the performance of the current label distribution model. Most of the existing active learning methods are for multi-label learning tasks and cannot be directly applied to the image selection problem under the label distribution representation. SUMMARY

[0005] The present application aims to provide a label distribution image selection method and system based on representativeness and diversity, to solve the problems of lack of image label data and high data annotation cost in label distribution learning when training modeling.

[0006] Technical scheme: In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is:

[0007] A label distribution image selection method based on representativeness and diversity, comprising the following steps:

[0008] (1) obtaining a label distribution image dataset composed of a labeled image dataset L and an unlabeled image dataset U, and performing image feature extraction;

[0009] (2) obtaining a label distribution model according to the labeled image dataset L;

[0010] (3) predicting the labels of the image samples in the unlabeled image dataset U using the label distribution model trained in step (2);

[0011] (4) calculating the dissimilarity of the image samples in the unlabeled image dataset U in the feature space to obtain a dissimilarity matrix D, so as to measure the degree to which an image sample can represent other unlabeled image samples;

[0012] (5) calculating the distance between the label distribution sequences of the image samples in the unlabeled dataset U and the image samples in the labeled dataset L according to the predicted labels of the unlabeled image samples obtained in step (3) to obtain a diversity matrix C, so as to measure the diversity of the unlabeled image samples;

[0013] (6) considering the representativeness and diversity of the image samples to select subset samples that can represent other samples in the unlabeled dataset, and these subset samples have different label distributions in the label space with the samples in the labeled dataset as the target, and selecting the most valuable image samples to form a query sample set Q;

[0014] (7) labeling the image samples in the query sample set Q obtained in step (6) and removing them from U and putting them into L;

[0015] (8) repeating steps (2) to (7) until the maximum number of cycles is reached or the performance of the retrained label distribution model meets the requirements.

[0016] Further, the dissimilarity matrix D is obtained according to the following method in step (4):

[0017] D = (d ij ) n×n

[0018] wherein 1≤i≤n, 1≤j≤n, n represents the number of unlabeled image samples;

[0019] Dissimilarity measure

[0020] wherein x i = [xi1 x i2 ,...x is ] and x j = [x j1 x j2 ,...x js ] respectively represent the feature matrix of the i-th image sample and the j-th image sample, and s is the feature dimension.

[0021] Further, the diversity matrix C is obtained according to the following formula in step (5):

[0022] C = diag(c(x1),...,c(x n ))

[0023] The diversity score of the i-th image sample in the unlabeled dataset U:

[0024]

[0025] where σ > 1, i, k are the image sample indexes in the unlabeled dataset U, j is the image sample index in the labeled dataset L, and min j∈L (KT) ji represents the minimum value of the Kendall Tau distance between the i-th unlabeled image sample label distribution and the labeled image sample label distribution; the label of the unlabeled image sample is the predicted label obtained in step (3).

[0026] Further, the Kendall Tau distance between the image sample label distributions is calculated according to the following formula:

[0027] (KT) ij = ∑ p＜q R pq [(L ip <L iq ∧L jp >L jq )or(L ip >L iq ∧L jp <L jq )]

[0028] where L ip and L iq respectively represent the ranking of the p-th label and the q-th label in the i-th image sample label distribution y i in the unlabeled dataset U, and L jp and L jq respectively represent the ranking of the p-th label and the q-th label in the j-th image sample label distribution y j in the unlabeled dataset U;

[0029] Rpq ||y .p -y .q ||2 represents the similarity between the pth label and the qth label of the image sample in the labeled dataset L, y .p = [y 1p , y 2p ,...y mp ] and y ·q = [y 1q , y 2q ,...y mq ] are vectors composed of the values of the pth label and the qth label in the label distribution of all image samples in the labeled dataset L, respectively, and m is the number of labeled image samples.

[0030] Further, step (6) selects valuable image samples by optimizing the following formula:

[0031]

[0032] s.t. Z ≥ 0, 1 T Z = 1 T

[0033] where λ is a balance parameter balancing the importance of the two terms, Z = (z ij ) n×n , z ij ∈ [0, 1] represents the probability that the ith image sample can represent the jth image sample, and n represents the number of unlabeled image samples, and the larger z ij , the larger the probability, and vice versa; ||·||2 2,1 represents the 2,1 norm; d ij z ij represents the encoding loss generated by using the ith image sample to represent the jth image sample; d ij represents the dissimilarity between the ith image sample and the jth image sample; 1 T Z = 1 T represents 1 T is a 1 × n all-1 matrix.

[0034] Further, the step (6) forms the query sample set Q according to the following manner:

[0035] The optimal solution of the model parameter Z solved according to the above steps selects valuable image samples, and when the image sample i can represent other unlabeled image samples, there is a non-zero element in the ith row of the matrix Z, so the unlabeled dataset U has several valuable image samples, that is, corresponds to several non-zero rows in the matrix Z.

[0036] Further, the step (7) updates the marked set according to the following manner:

[0037] The selected image samples in the query sample set Q are labeled by experts, and the labeling result of each image is the label distribution of the relevant labels of the image. After labeling, the image samples are removed from the unmarked data set U and placed in the marked data set L.

[0038] Further, the step (8) determines whether to terminate the image sample selection according to the following manner:

[0039] When the label distribution model training reaches the maximum number of cycles or the prediction result of the label distribution model performance on the image test data set meets the accuracy requirement, the image sample selection is terminated.

[0040] Based on the same inventive concept, the application provides a label distribution image selection system based on representativeness and diversity, comprising: a preprocessing unit for obtaining an image data set composed of a marked image data set L and an unmarked image data set U, and performing image feature extraction;

[0041] A label distribution model training unit is configured to obtain a label distribution model according to the marked image data set L.

[0042] An image sample selection unit is configured to predict the predicted labels of the image samples in the unmarked image data set U according to the trained label distribution model, calculate the dissimilarity of the image samples in the unmarked image data set U in the feature space to obtain a dissimilarity matrix D, and measure the degree of representativeness of the image samples to other unmarked image samples; calculate the distance between the label distribution sequences of the image samples in the unmarked data set U and the image samples in the marked data set L according to the predicted labels of the unmarked image samples obtained by the currently trained label distribution model, to obtain a diversity matrix C, and measure the diversity of the unmarked image samples; considering the representativeness and diversity of the image samples, the most valuable image samples are selected to form a query sample set Q, which can represent the subset samples of other samples in the unmarked data set and have different label distributions in the label space with the samples in the marked data set.

[0043] An image labeling unit is configured to label the image samples in the obtained query sample set Q and remove them from U and place them in L.

[0044] A termination determination unit is configured to determine whether to continue the image sample selection, and stop when the maximum number of cycles is met or the performance of the retrained label distribution model meets the requirement.

[0045] Based on the same inventive concept, the application provides a computer system, comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the computer program, when loaded into the processor, implements the representative and diverse label distribution image selection method.

[0046] Advantages: Compared with the prior art, the application has the following advantages: the application applies the active learning technology to the label distribution learning, and can significantly improve the performance of the label distribution learner by actively selecting the most valuable images as the training samples. In the selection of the representative images, the distance of the image samples in the feature space is calculated, and the difference between the image samples is measured by the Euclidean distance between the image samples, so as to measure whether the image sample can represent other image samples. Meanwhile, in the image sample selection process, the diversity of the image samples is considered, and it is hoped that the image sample selected next has as different label distribution as possible from the labeled image samples, so as to explore the image information most valuable for improving the current label distribution model. The application in the label distribution image classification problem has great significance for improving the performance of the label distribution model in the case of insufficient data. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a method flowchart of an embodiment of the application.

[0048] Figure 2 is a flowchart of an image sample selection mechanism. DETAILED DESCRIPTION

[0049] The specific embodiments of the application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] As shown in Figure 1 , an embodiment of the application discloses a representative and diverse label distribution image selection method, comprising the following steps:

[0051] Step 1: Obtain a label distribution image dataset composed of a small amount of labeled image dataset L and a large amount of unlabeled image dataset U, and perform image feature extraction, wherein the HOG feature of the image is used in the embodiment.

[0052] Step 2: Obtain a label distribution learning model according to the small amount of labeled image dataset L, wherein the AA-kNN label distribution learning algorithm is used in the embodiment.

[0053] Step 3: Use the label distribution model trained in step 2 to predict the unlabeled image dataset U, and obtain the predicted label of the image sample in U l is the dimension of the label, and i is the unlabeled image sample in U.

[0054] Step 4: Calculate the dissimilarity of the image samples in the unlabeled dataset U in the feature space and obtain the dissimilarity matrix D, which is used to measure the degree to which the image samples can represent other unlabeled image samples.

[0055] The degree of dissimilarity of image samples in the feature space can measure the degree to which the image sample can reflect other image samples. The smaller the dissimilarity, the greater the possibility that it can represent other image samples.

[0056] First, define the dissimilarity d between each pair of image samples ij , 1≤i≤n,1≤j≤n, n represents the number of unlabeled image samples. ij The smaller it is, the better the effect of sample i on representing sample j is.

[0057] For the dissimilarity measure:

[0058] Among them, x i =[x i1 ,x i2 ,...x is ] and x j =[x j1 ,x j2 ,...x js ] represent the feature matrices of the i-th image sample and the j-th image sample respectively, and s is the feature dimension.

[0059] According to the difference d between image samples ij , and obtain the dissimilarity matrix D, the matrix form is as follows:

[0060]

[0061] Step 5: Based on the predicted labels of the unlabeled image samples obtained in step 3, calculate the distance between the label distribution sequences of the image samples in the unlabeled dataset U and the image samples in the labeled dataset L to obtain the diversity matrix C, which is used to measure the diversity of the unlabeled image samples.

[0062] First, based on the label space of image samples in the labeled dataset L, the label correlation R is extracted pq , R pq Represents the similarity between the p-th token and the q-th token:

[0063] R pq =||y ·p -y ·q ||2

[0064] Among them, y i =[y i1 ,y i2 ,...y ily ·p 1p 2p mp ·q 1q 2q mq are vectors consisting of the values of the p-th label and the q-th label in the label distributions of all samples in the labeled dataset L, respectively, and m is the number of labeled image samples.

[0065] Secondly, Kendall Tau distance is used to represent the difference between the label distribution sequences of image samples:

[0066] (KT) ij i j =∑ p＜q R pq [(L ip <L iq ∧L jp >L jq )or(L ip >L iq ∧L jp <L jq )]

[0067] where L i and L j are sequences representing the order relations of the labels of y i and y j , respectively, L ip and L iq correspond to the rankings of the p-th label and the q-th label in the label distribution y i of the i-th image sample in the unlabeled dataset U, respectively, and L jp and L jq correspond to the rankings of the p-th label and the q-th label in the label distribution y j of the j-th image sample in the unlabeled dataset U, respectively.

[0068] Finally, the diversity score c(x i ) of each image sample in the unlabeled dataset is calculated:

[0069]

[0070] where i, k are image samples in the unlabeled dataset U, and j is an image sample in the labeled dataset L. min j∈L (KT) ji ​​​​​​​​​denotes the minimum distance between the i-th unlabeled image sample label distribution and the labeled data set image sample label distribution. For unlabeled image sample i, k, use step (3)

[0071] the predicted label obtained in step (4) is brought into the calculation.

[0072] According to the diversity score c(x i ), the diversity matrix C = diag(c(x1),..., c(x n )) is obtained.

[0073] Step 6: Considering the representativeness and diversity of image samples, select a subset of samples that can represent other samples in the unlabeled data set, and these subset samples have different label distributions in the label space with the samples in the labeled data set as the goal, select a small number of most valuable image samples to form the query sample set Q;

[0074] Based on the dissimilarity matrix D, introduce a variable z ij related to d ij , where z ij ∈ [0, 1] represents the probability that image sample i represents image sample j, and Z is initialized as the unit matrix I, which has the following matrix form:

[0075]

[0076] When image sample i is the representative of other image samples, z i ≠ 0, that is, there is a non-zero element in the i-th row of the Z matrix. Therefore, the convex function related to the number of selected representative images is defined as:

[0077]

[0078] where z i represents the i-th row of matrix Z. Further considering the diversity of image samples, the above convex function is further optimized:

[0079]

[0080] where c i is the diversity score value of the unlabeled image sample i.

[0081] For the unlabeled image sample i with a smaller diversity score c(x i ), it means that selecting the unlabeled image sample i can promote the diversity of images in the labeled data set L, and the optimization function imposes less punishment on the non-zero element of the i-th Z. Conversely, for the unlabeled image sample i with a larger diversity score c(x i ), it means that selecting the unlabeled image sample i can promote the diversity of images in the labeled data set L, and the optimization function imposes less punishment on the non-zero element of the i-th Z. Conversely, for the unlabeled image sample i with a larger diversity score c(x), the optimization function will impose more penalty on the non-zero elements in the i-th row of matrix Z to obtain unlabelled images that are as different as possible from the labelled images.

[0082] Meanwhile, the image sample i is selected to represent the encoding cost of sample j as d ij z ij , d ij z ij ∈ [0, d ij ], the total cost of encoding all other image samples using image sample i is:

[0083]

[0084] The final optimization objective is represented as:

[0085]

[0086] s.t.Z≥0,1 T Z=1 T

[0087] The optimization problem is a smooth convex optimization problem, and the ADMM algorithm is used for optimization in this embodiment. When image sample i can represent other unlabelled image samples, there are non-zero elements in the i-th row of matrix Z, so the unlabelled data set U has several valuable image samples, i.e. corresponding to several non-zero rows in matrix Z.

[0088] Step 7: The image samples in the query sample set Q obtained in step 6 are given to an expert for labelling, and the selected image samples are removed from U and placed in L;

[0089] Step 8: Repeat steps 2 to 7 until the maximum number of cycles is reached or the performance of the retrained labelled distribution model meets the requirements.

[0090] Based on the same inventive concept, the embodiment of the present application discloses a representative and diverse label distribution image selection system, comprising: a preprocessing unit, configured to acquire an image dataset, the dataset being composed of a small amount of labeled image dataset L and a large amount of unlabeled image dataset U, and perform image feature extraction; a label distribution model training unit, configured to obtain a label distribution model according to the small amount of labeled image dataset L; an image sample selection unit, configured to predict the labels of the image samples in the unlabeled image dataset U according to the obtained label distribution model, and obtain the predicted labels of the image samples in the unlabeled image dataset U; calculate the dissimilarity of the image samples in the unlabeled image dataset U in the feature space to obtain a dissimilarity matrix D, so as to measure the degree to which the image samples can represent other unlabeled image samples; calculate the distance between the label distribution sequences of the image samples in the unlabeled dataset U and the image samples in the labeled dataset L according to the predicted labels of the unlabeled image samples obtained by the currently trained label distribution model to obtain a diversity matrix C, so as to measure the diversity of the unlabeled image samples; comprehensively consider the representativeness and diversity of the image samples, and select a small amount of most valuable image samples to form a query sample set Q; an image labeling unit, configured to label the image samples in the obtained query sample set Q, and remove the image samples from the U and put them into the L; and a termination judgment unit, configured to judge whether to continue the image sample selection, and stop when the maximum number of cycles is met or the performance of the retrained label distribution model meets the requirements.

[0091] Based on the same inventive concept, the embodiment of the present application discloses a computer system, comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, when the computer program is loaded into the processor, each step of the above-mentioned representative and diverse label distribution image selection method is implemented.

Claims

1. A method for selecting a marked profile image based on representativeness and diversity, characterized in that, The method comprises the following steps: (1) obtaining a labeled distribution image dataset composed of a labeled image dataset L and an unlabeled image dataset U, and performing image feature extraction; (2) obtaining a labeled distribution model according to the labeled image dataset L; (3) predicting the predicted labels of the image samples in the unlabeled image dataset U by using the labeled distribution model trained in step (2); (4) calculating the dissimilarity of the image samples in the unlabeled image dataset U in the feature space to obtain a dissimilarity matrix D, so as to measure the degree to which the image samples can represent other unlabeled image samples; (5) calculating the distance between the label distribution sequences of the image samples in the unlabeled dataset U and the image samples in the labeled dataset L according to the predicted labels of the unlabeled image samples obtained in step (3) to obtain a diversity matrix C, so as to measure the diversity of the unlabeled image samples; (6) comprehensively considering the representativeness and diversity of the image samples, selecting a subset of samples that can represent other samples in the unlabeled dataset, and selecting the most valuable image samples to form a query sample set Q, with the goal of having different label distributions in the label space between the subset samples and the samples in the labeled dataset; (7) labeling the image samples in the query sample set Q obtained in step (6) and removing them from U and placing them in L; (8) repeating steps (2) to (7) until the maximum number of iterations is reached or the performance of the retrained labeled distribution model meets the requirements.

2. The representative and diverse landmark distribution image selection method according to claim 1, wherein, The dissimilarity matrix D is obtained according to the following method: D = (d ij ) n×n wherein, 1≤i≤n, 1≤j≤n, n represents the number of unlabeled image samples; Dissimilarity measure where x i = [x i1 , x i2 ,...x is ] and x j = [x j1 , x j2 ,...x js ] represent the feature matrix of the i-th image sample and the j-th image sample, respectively, and s is the feature dimension.

3. The representative and diverse landmark distribution image selection method according to claim 1, wherein, The diversity matrix C is obtained according to the following formula: C = diag(c(x1),...,c(x n )) wherein, 1≤i≤n, n represents the number of unlabeled image samples; The diversity score of the i-th image sample in the unlabeled dataset U is: where σ > 1, i, k are image sample indices in the unlabelled dataset U, j is an image sample index in the labelled dataset L, min j∈L (KT) ji denotes the minimum Kendall Tau distance between the i-th unlabelled image sample label distribution and the labelled image sample label distribution; the label of the unlabelled image sample is the predicted label obtained in step (3).

4. The representative and diverse landmark distribution image selection method according to claim 3, wherein, The Kendall Tau distance between the label distributions of the image samples is calculated according to the following formula: (KT) ij =∑ p＜q R pq [(L ip <L iq ∧L jp >L jq )or(L ip >L iq ∧L jp <L jq ] wherein L ip and L iq respectively denote the rank of the p-th label and the q-th label in the label distribution y i of the i-th image sample in the unlabelled dataset U, L jp and L jq respectively denote the rank of the p-th label and the q-th label in the label distribution y j of the j-th image sample in the unlabelled dataset U. R pq =||y ·p -y ·q ||2 represents the similarity between the pth label and the qth label of the image samples in the labeled dataset L, y ·p = [y 1p , y 2p ,...y mp ] and y ·q = [y 1q , y 2q ,...y mq ] are vectors composed of the values of the pth label and the qth label in the label distribution of all image samples in the labeled dataset L, respectively, and m is the number of labeled image samples.

5. The representative and diverse landmark distribution image selection method according to claim 1, wherein, The valuable image samples are selected by optimizing the following formula in step (6): s.t. Z ≥ 0,1 T Z = 1 T where λ is a balance parameter balancing the importance between two terms, Z = (z ij ) n×n , z ij ∈ [0, 1] represents the probability that the ith image sample can represent the jth image sample, n represents the number of unlabeled image samples, z ij The larger z 2,1 is, the larger the probability is, and vice versa; ||·||2,1 d ij z ij represents the encoding loss generated by using the ith image sample to represent the jth image sample; d ij represents the dissimilarity between the ith image sample and the jth image sample; 1 T Z = 1 T represents 1 T is a 1 x n all-1 matrix.

6. The representative and diverse landmark distribution image selection method according to claim 5, wherein, The selected labeled distribution image samples have several valuable image samples corresponding to several non-zero rows in the matrix Z.

7. The representative and diverse landmark distribution image selection method according to claim 1, wherein, Step (7) comprises the following steps: labeling the image samples in the selected query sample set Q by experts, and the labeling result of each image is the label distribution of the related labels of the image, and after labeling, the image samples are removed from the unlabeled dataset U and placed in the labeled dataset L.

8. A representative and diverse landmark distribution image selection system, comprising: It comprises: a preprocessing unit for obtaining an image dataset composed of a labeled image dataset L and an unlabeled image dataset U, and performing image feature extraction; a labeled distribution model training unit for obtaining a labeled distribution model according to the labeled image dataset L; an image sample selection unit for predicting the predicted labels of the image samples in the unlabeled image dataset U by using the labeled distribution model trained. calculating the dissimilarity of the image samples in the unlabeled image dataset U in the feature space to obtain a dissimilarity matrix D, so as to measure the degree that the image samples can represent other unlabeled image samples; calculating the distance between the label distribution sequences of the image samples in the unlabeled dataset U and the image samples in the labeled dataset L according to the predicted labels of the unlabeled image samples obtained by the currently trained label distribution model, to obtain a diversity matrix C, so as to measure the diversity of the unlabeled image samples; comprehensively considering the representativeness and diversity of the image samples, to select the subset samples that can represent other samples in the unlabeled dataset, and the subset samples have different label distribution with the samples in the labeled dataset in the label space as the target to select the most valuable image samples to constitute the query sample set Q; an image labeling unit, configured to label the image samples in the obtained query sample set Q and remove the image samples from U to L; and a termination judging unit, configured to judge whether to continue the selection of the image samples, and stop when the maximum number of cycles is met or the performance of the retrained label distribution model reaches the requirement.

9. The representative and diverse landmark distribution image selection system of claim 8, wherein, In the image sample selecting unit, the valuable image samples are selected by optimizing the following formula: s.t. Z ≥ 0,1 T Z = 1 T where λ is a balance parameter balancing the importance between two terms, Z = (z ij ) n×n , z ij e [0, 1] represents the probability that the ith image sample can represent the jth image sample, n represents the number of unlabeled image samples, z ij The larger z 2,1 represents the larger probability, and vice versa; ||·||2,1 represents the 2,1 norm; d ij z ij represents the encoding loss generated by using the ith image sample to represent the jth image sample; d ij represents the dissimilarity between the ith image sample and the jth image sample; 1 T Z = 1 T represents 1 T is a 1 x n all-1 matrix.

10. A computer system, characterized by The computer program is loaded into the processor to implement the steps of the method of any one of claims 1-7. The computer program is loaded into the processor to implement the steps of the method of any one of claims 1-7.

Images