An image labeling method capable of processing new labels and missing labels

Abstract

The application provides an image labeling method capable of processing new labels and missing labels, and relates to the technical field of image classification. The application obtains a multi-image data set by processing a real original image data set, maps each image in the multi-image package from an image space to a high-dimensional feature space to obtain a graph kernel-based graph representation of each image, models the relationship between the graph and the label by using a weight matrix to semantically describe the category label, constructs a minimization objective function to learn the new label and the missing label of the graph, optimizes and updates the minimization objective function by using an accelerated proximal gradient method and a proximal gradient method to obtain an updated weight matrix, and further constructs a classifier capable of modeling the new label and the missing label at the same time, constructs a graph-level classifier according to the weight matrix to predict the label of the graph in the test package, realizes the label prediction of the unknown multi-image package, and completes the labeling of the image data.

Description

Technical Field

[0001] This invention belongs to the field of image classification technology, specifically relating to an image annotation method capable of handling new and missing labels. Background Technology

[0002] With the rapid development and widespread application of internet and cloud storage technologies, the amount of multimedia data, such as images, is growing exponentially. Each image in this data typically involves multiple themes and has a complex structure. Furthermore, each image is often associated with multiple semantic tags, making it difficult to capture complete tags. How to effectively represent this data and develop comprehensive classification tags that fully describe its content has become a challenge.

[0003] Existing image annotation methods extract features from each research object, such as an image, describing the object using one or more feature vectors (examples), and then associate the examples with corresponding category labels. While this representation is widely used in real-world applications, it still suffers from incompatibility with complex learning tasks. In real-world applications, much research data has a complex structure, and feature vectors cannot represent such complex data, let alone effectively capture the relationships between the various parts of an object.

[0004] Multi-graph multi-label classification methods have been applied to complex learning tasks. In this framework, each research object is represented by a graph bag containing multiple graphs. The graphs in the bag can capture the dependencies between the various parts of the object and associate multiple class labels, describing the ambiguity of labels in real life. The learning task of multi-graph multi-label classification is to build a classifier that predicts multiple relevant labels for an unknown multi-graph bag.

[0005] In practical applications, experts only focus on relevant labels during annotation. However, some new labels may exist outside the fixed label set, which are often overlooked. For example, some rare birds in bird images may be present but not labeled. Furthermore, due to cost or noise interference, obtaining complete label information is difficult, resulting in missing labels; for instance, background objects in animal images are often ignored. The presence of both new and missing labels leads to incomplete label information, thus limiting the model's ability to annotate image data. However, some existing multi-image multi-label classification methods assume that the label set of multi-image data is complete, failing to address the issue of potentially incomplete label information in images, thus limiting the scope of multi-image multi-label learning. Therefore, the discovery of new labels and the filling of missing labels are highly significant. Currently, researchers have proposed some methods for handling new and missing labels, but these methods are designed for feature vector-based objects. Multi-image structures are more complex and cannot be directly applied to multi-image multi-label learning environments. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides an image annotation method capable of handling both new and missing markers. It aims to overcome the limitations of existing multi-image, multi-marker methods in terms of applicable scenarios, effectively process image data with both new and missing markers, and accurately annotate the image data.

[0007] The technical solution of the present invention is: an image annotation method capable of handling new and missing markers, comprising the following steps:

[0008] Step 1: Obtain the real original image dataset and perform data preprocessing on the real original image dataset to obtain a multi-image dataset, which includes a multi-image bag structure representation and a known category label set. Divide the multi-image dataset into a training set and a test set.

[0009] Each image is divided into multiple regions based on the objects within it, with each region corresponding to a marker. The SLIC superpixel segmentation algorithm is used to segment each region, resulting in multiple superpixels. Each superpixel is considered a node, and adjacent superpixels are considered to have edges. The feature of each node is the vector sum of its RGB color histogram and its directional gradient histogram. At this point, each region of the image is transformed into a graph, and an image can be represented as a graph bag. The original image dataset is preprocessed to obtain a multi-graph dataset. The multi-graph bag during training is represented as... Where n represents the number of multi-image packages, and a multi-image package B is... i B i ={g i1 ,g i2 ,…,g ini}, where g ij Let n represent the j-th graph in the i-th bag. i This represents the number of graphs in the i-th bag, where 1 ≤ i ≤ n and 1 ≤ j ≤ n. i The known set of labels corresponding to the multi-map package is represented as Y = {y1, y2, ..., y...} n}∈{0,1} n×c Among them, package B i The symbol is represented as y i ={y i1 ,y i2 ,…,y ic}, where c is the known number of tags, and when y ik =1 indicates that the i-th package belongs to the k-th class tag, and when y = 1, it means that the i-th package belongs to the k-th class tag. ik =0 indicates that the i-th packet does not belong to the k-th class tag or the current value is missing. Let q be the number of new tags that have not been mined, and l = c + q be the total number of tags. Then the complete tag set should be Y = {y1, y2, ..., y}.n}∈{0,1} n×l .

[0010] Step 2: Map each graph in the multi-graph bag from the graph space to the high-dimensional feature space, and use the graph kernel method to calculate the similarity between each graph in the training set and all graphs in the multi-graph bag, thereby obtaining a graph kernel-based graph representation for each graph;

[0011] For any graph g, calculate the similarity between graph g and all graphs in the multi-graph bag to obtain the graph representation based on the graph kernel. The specific formula is as follows:

[0012] x g =f(g)=[K(g,g1),…,K(g, ... d (1)

[0013] Among them, g i Let be the i-th graph in the dataset, d be the number of all graphs, and K(·,·) denote the graph kernel function;

[0014] Step 3: Based on the graph representations of all graphs, semantically describe the category labels using the weight matrix W, model the relationship between the graphs and the labels, and construct the confidence matrix of the graph labels.

[0015] To learn incomplete labeling information from a graph, a confidence matrix for unknown graph labels is defined. The relationship between the graph representation and the label is modeled using the following formula:

[0016]

[0017] Where W is the weight matrix consisting of the weight vectors corresponding to all labels. It is the confidence matrix of the graph labels, which consists of the confidence vectors of all labels in the graph, including known labels and new labels, X. G ={x g1 ,…,x gN} is a graph representation matrix consisting of graph labels of all graphs, where α is the regularization parameter and ||·||1 is the l1 norm;

[0018] Step 4: Construct two-view regularization terms, projection similarity regularization terms, and bag correlation regularization terms to assess the confidence matrix of the graph labels. To learn;

[0019] Step 4.1: To learn the confidence matrix of the graph labels We construct a two-perspective regularization term, making two hypotheses from the perspectives of graph correlation and label correlation. From the perspective of graph correlation, we hypothesize that graphs that are close to each other in the graph space are more likely to be associated with similar labels, i.e., we decompose the graph similarity matrix S to obtain... From the perspective of label relevance, multiple labels are usually related. Assuming that the labels of a graph can be inferred from similar labels, that is, by decomposing the label similarity matrix C... Simultaneously, a two-view regularization term is obtained by making assumptions from these two perspectives, which affects the confidence matrix of the graph labeling. The recovery formula is as follows:

[0020]

[0021] Where S is the similarity matrix of the graph calculated by the graph kernel method, and C is the similarity matrix of the learnable labels, ||·|| F It is the Frobenius norm, and l = c + q is the total number of labels;

[0022] Step 4.2: Calculate the similarity of known labels. Using the projection matrix P, multiply the label similarity matrix C by both the left and right sides to return the label similarity of the top-left corner c×c. Construct a projection similarity regularization term to ensure that the partial similarity of known labels matches the true label similarity. The specific formula is as follows:

[0023]

[0024] Where, P∈{0,1} c×l It is a projection matrix, with a c×c identity matrix on the left and zeros on the other side. It is a similarity matrix of known labels associated with the training set, calculated using cosine similarity.

[0025] Step 4.3: Construct package-related regularization terms to further learn the confidence matrix of graph labels. Now consider the relationship between packet labels and graph labels: if a packet is associated with a label, then there is at least one graph in the packet associated with this label. The confidence matrix controls the graph labels. Truth matrix of graph labels The difference between them, D, is Using the indicator matrix Q and projection matrix P, which can indicate which graphs exist in the package, respectively... Perform left and right multiplications to return the top c columns of the number of graphs associated with each label in the bag. Then, after a clever transformation of this and the bag label Y, design the relationship between the bag label and the graph label. The formula for calculating the bag-related regularization term is as follows:

[0026]

[0027] Where, Q∈{0,1} n×N D is the indicator matrix, representing the existence of the bag-in-the-box graph; D is the difference matrix, the confidence matrix controlling the graph labeling. Truth matrix of graph labels The differences between them; It is a non-negative matrix. It represents the Hadamardi (or Hadama) stack;

[0028] Step 5: Based on the three regularization terms constructed in Step 4, namely formulas (3)-(5), a unified minimization objective function is formed to learn the new and missing labels of the graph. The specific formulas are as follows:

[0029]

[0030] Where λ1, λ2, λ3, α, β, γ, and η are hyperparameters, and the range of values ​​for the hyperparameters is set to {10}. -3 10 -2 ,…,10 2 10 3};

[0031] Step 6: Optimize and update the minimized objective function from Step 5 using accelerated proximal gradient method and proximal gradient method to obtain the updated weight matrix W. Then, construct a classifier that can simultaneously model new labels and missing labels, specifically including the following steps:

[0032] Step 6.1: Initialize all parameters: weight matrix W, confidence matrix Similarity matrix C, difference matrix D, nonnegative matrix Z, and additional parameters ω0 and ω1 for the 0th and 1st iterations;

[0033] Step 6.2: Using the graph kernel method, calculate the similarity between each graph and all other graphs to obtain a graph kernel-based graph representation of all graphs, and form the graph representation matrix X. G ;

[0034] Step 6.3: Update the weight matrix W using the accelerated proximal gradient method:

[0035]

[0036]

[0037] Among them, W t This is the t-th iteration of W, W (t) It is the combination of the (t-1)th and tth iterations of w, ω t It is the t-th iteration with additional parameter ω. It is the gradient of the objective function on W. It is the Lipschitz constant with respect to W. It is a soft threshold operation in

[0038] Step 6.4: Update the labeled similarity matrix C using the proximal gradient method:

[0039]

[0040] in, It is the gradient of the objective function on C, where C is the gradient of the objective function on C. t This is the t-th iteration of C. It is the Lipschitz constant with respect to C. It is a singular value thresholding operation;

[0041] Step 6.5: Apply the proximal gradient method to the confidence matrix of the graph labels. Update:

[0042]

[0043] in, and yes The t and t+1 iterations, Is the objective function in gradient on, It is about The Lipschitz constant;

[0044] Step 6.6: Update the difference matrix D using the proximal gradient method:

[0045]

[0046] Among them, D t It is the t-th iteration of D. It is the gradient of the objective function on D; It is the Lipschitz constant with respect to D;

[0047] Step 6.7: At this point, the objective function has a closed-form solution, so we directly update the non-negative matrix Z:

[0048]

[0049] Step 6.8: Update the additional parameter ω:

[0050]

[0051] Step 6.9: Repeat steps 6.3 to 6.8 until the preset number of iterations is reached, then end the training process.

[0052] Step 7: Given the test set A test package Based on the weight matrix W learned in step 6, a graph classifier is constructed to predict the labels of the graphs in the test package. By utilizing the relationship between the label of the package and the label of the graph, the label prediction of unknown multi-graph packages is achieved, thus completing the annotation of image data.

[0053] When new image packs are available To perform label prediction, we first need to obtain the graph representations of the graphs in the new graph package. Based on the weight matrix W obtained during the training phase, a graph-level classifier is constructed. Calculate the predicted value of the test map package For each graph, take the maximum value of all predicted values ​​associated with it. The label corresponding to this maximum value is used as the label of the graph in the bag. Set the label position to 1 and the other positions to 0. Based on the relationship between the bag label and the graph label: if the graph in the bag has a label, then the bag also has a corresponding label, and the predicted label of the bag is obtained.

[0054] Compared with the prior art, the technical solution proposed in this invention has the following beneficial effects:

[0055] 1. This invention proposes for the first time an image annotation method capable of handling both new and missing labels. It integrates the setting of new and missing labels into the existing multi-image multi-label learning framework, overcoming the limitations of existing multi-image multi-label methods in terms of usage scenarios. It can effectively process image data with both new and missing labels for accurate annotation.

[0056] 2. The present invention obtains a graph kernel-based graph representation for each graph using the graph kernel method. It only needs to calculate the similarity between each graph and other graphs, without explicitly converting the graph into a binary feature vector, thus preserving rich structural information.

[0057] 3. Under the new multi-image multi-label learning framework with new and missing labels, this invention constructs a two-view regularization term, a projection similarity regularization term, and a package correlation regularization term, and forms a unified objective function to process the incomplete label information of image data. The objective function is then optimized and updated to obtain a classifier that processes incomplete labels, thereby enabling the prediction of labels for unknown multi-image data.

[0058] 4. The method of the present invention converts images into multi-image, multi-label data, represents the data in a reasonable manner, and processes new and missing labels, which can improve the classification accuracy of known labels in image data, and at the same time has the ability to fill in missing labels and discover new labels. Attached Figure Description

[0059] To more clearly illustrate the specific methods in the embodiments of the present invention, the relevant drawings involved in the embodiments will be briefly described below. The drawings below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative changes.

[0060] Figure 1 This is a flowchart illustrating the image annotation method of this embodiment that can handle new and missing markers;

[0061] Figure 2 This is a flowchart illustrating the training process for optimizing and updating the objective function in this embodiment.

[0062] Figure 3 This is a schematic diagram comparing the index results of this implementation method with five existing comparison methods;

[0063] Figure (a) shows a comparison of the Hamming loss metrics obtained when both are applied to MSRCv2 with 50% missing labels; Figure (b) shows a comparison of the Hamming loss metrics obtained when both are applied to MSRCv2 with 50% new labels; and Figure (c) shows a comparison of the Hamming loss metrics obtained when both are applied to MSRCv2 with 20% new labels and missing labels. Detailed Implementation

[0064] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0065] The purpose of this invention is to provide an image annotation method capable of handling both new and missing labels. This method integrates the settings for new and missing labels into a multi-graph multi-label learning framework to accurately annotate image data. To preserve graph structure information, the method utilizes a graph kernel function to calculate the similarity between each graph and all graphs, resulting in an effective graph representation. Since image data contains incomplete labeling information, including ignored new labels outside the known label set and missing labels due to cost burdens or noise interference, a two-view regularization term based on graph correlation and label correlation, a projection similarity regularization term constraining the label similarity matrix, and a bag correlation regularization term considering the complex relationship between bag-level and graph-level labels are constructed to form a unified objective function. By accelerating the proximal gradient method and optimizing the objective function, a classifier that handles new and missing label information is obtained, thereby enabling the prediction of labels for unknown image data. This invention overcomes the limitations of existing multi-graph multi-label methods in terms of applicable scenarios, effectively processing image data with both new and missing labels, thus enabling accurate image data annotation.

[0066] Figure 1 This is a flowchart illustrating the image annotation method of this embodiment, which can handle both new and missing markers. Figure 1 As shown, the image annotation method capable of handling new and missing markers includes the following steps:

[0067] Step 1: Obtain the real original image dataset and perform data preprocessing on the real original image dataset to obtain a multi-image dataset, which includes a multi-image bag structure representation and a known category label set. Divide the multi-image dataset into a training set and a test set.

[0068] In this embodiment, the real image dataset is a subset of Microsoft Research Cambridge, MSRCv2. MSRCv2 consists of 591 images of 23 types of objects (buildings, grass, trees, cows, horses, sheep, sky, mountains, airplanes, water, faces, cars, bicycles, flowers, road signs, birds, books, chairs, roads, cats, dogs, human bodies, and boats). Each image is divided into multiple regions according to the objects in the image, and each region corresponds to a label. The superpixel segmentation algorithm SLIC is used to cut each region to obtain multiple superpixels. Each superpixel is regarded as a node, and adjacent superpixels are considered to have edges. The feature of each node is the vector sum of the RGB color histogram and the directional gradient histogram. At this time, each region of the image is transformed into a graph, and an image can be represented as a graph bag. The original image dataset MSRCv2 is preprocessed, that is, the images are processed to obtain a multi-graph dataset. In this embodiment, 2 / 3 to 4 / 5 of the multi-graph dataset is selected as the training set, and the rest is used as the test set for annotation and testing. The multi-graph bag during training is represented as follows. Where n represents the number of multi-image packages, and a multi-image package B is... i B i ={g i1 ,g i2 ,…,g ini}, where g ij Let n represent the j-th graph in the i-th bag. i This represents the number of graphs in the i-th bag, where 1 ≤ i ≤ n and 1 ≤ j ≤ n. i The known set of labels corresponding to the multi-map package is represented as Y = {y1, y2, ..., y...} n}∈{0,1} n×c Among them, package B i The symbol is represented as y o ={y i1 ,y i2 ,…,y ic}, where c is the known number of tags, and when y ik =1 indicates that the i-th package belongs to the k-th class tag, and when y = 1, it means that the i-th package belongs to the k-th class tag. ik =0 indicates that the i-th packet does not belong to the k-th class tag or the current value is missing. Let q be the number of new tags that have not been mined, and l = c + q be the total number of tags. Then the complete tag set should be Y = {y1, y2, ..., y}. n}∈{0,1} n×l .

[0069] Step 2: Map each graph in the multi-graph bag from the graph space to the high-dimensional feature space, and use the graph kernel method to calculate the similarity between each graph in the training set and all graphs in the multi-graph bag, thereby obtaining a graph kernel-based graph representation for each graph;

[0070] For any graph g, calculate the similarity between graph g and all graphs in the multi-graph bag to obtain the graph representation based on the graph kernel. The specific formula is as follows:

[0071] x g =f(g)=[K(g,g1),…,K(g, ... d (1)

[0072] Among them, g i Let be the i-th graph in the dataset, d be the number of all graphs, and K(·,·) denote the graph kernel function. This function is represented as an inner product in Hilbert space. The graph kernel method maps each graph from the graph space to an infinite-dimensional Hilbert space, converting it into a feature vector without feature extraction. The inner product of the feature vectors of two graphs after mapping is calculated, which is then used as the similarity between the two graphs. To ensure the generality of the method, different graph kernel functions are used for different classification problems. For node label graphs in text datasets, the Weisfeiler-Lehman (WL) kernel is used, and for node attribute graphs in image datasets, the Graph Hopper kernel is used.

[0073] Step 3: Based on the graph representations of all graphs, semantic descriptions of the category labels are performed using the weight matrix W, a linear classification model in the low-dimensional label space is learned, the relationship between the graph and the labels is modeled, and a confidence matrix for the graph labels is constructed.

[0074] To learn incomplete labeling information from a graph, a confidence matrix for unknown graph labels is defined. The relationship between the graph representation and the label is modeled using the following formula:

[0075]

[0076] Where W is the weight matrix consisting of the weight vectors corresponding to all labels. It is the confidence matrix of the graph labels, which consists of the confidence vectors of all labels in the graph, including known labels and new labels, X. G ={x g1 ,…,x gN} is a graph representation matrix consisting of graph labels of all graphs, where α is the regularization parameter and ||·||1 is the l1 norm;

[0077] Step 4: Construct two-view regularization terms, projection similarity regularization terms, and bag correlation regularization terms to assess the confidence matrix of the graph labels. To learn;

[0078] Step 4.1: To learn the confidence matrix of the graph labels We construct a two-perspective regularization term, making two hypotheses from the perspectives of graph correlation and label correlation. From the perspective of graph correlation, we hypothesize that graphs that are close to each other in the graph space are more likely to be associated with similar labels, i.e., we decompose the graph similarity matrix S to obtain... From the perspective of label relevance, multiple labels are usually related. Assuming that the labels of a graph can be inferred from similar labels, that is, by decomposing the label similarity matrix C... Simultaneously, a two-view regularization term is obtained by making assumptions from these two perspectives, which affects the confidence matrix of the graph labeling. The recovery formula is as follows:

[0079]

[0080] Where S is the similarity matrix of the graph calculated by the graph kernel method, and C is the similarity matrix of the learnable labels, ||·|| F It is the Frobenius norm, and l = c + q is the total number of labels;

[0081] Step 4.2: Calculate the similarity of known labels. Using the projection matrix P, multiply the label similarity matrix C by both the left and right sides to return the label similarity of the top-left corner c×c. Construct a projection similarity regularization term to ensure that the partial similarity of known labels matches the true label similarity. The specific formula is as follows:

[0082]

[0083] Where, P∈{0,1} c×l It is a projection matrix, with a c×c identity matrix on the left and zeros on the other side. It is a similarity matrix of known labels associated with the training set, calculated using cosine similarity.

[0084] Step 4.3: Construct package-related regularization terms to further learn the confidence matrix of graph labels. Now consider the relationship between packet labels and graph labels: if a packet is associated with a label, then there is at least one graph in the packet associated with this label. The confidence matrix controls the graph labels. Truth matrix of graph labels The difference between them, D, is Using the indicator matrix Q and projection matrix P, which can indicate which graphs exist in the package, respectively... Perform left and right multiplications to return the top c columns of the number of graphs associated with each label in the bag. Then, after a clever transformation of this and the bag label Y, design the relationship between the bag label and the graph label. The formula for calculating the bag-related regularization term is as follows:

[0085]

[0086] Where, Q∈{0,1} n×N D is the indicator matrix, representing the existence of the bag-in-the-box graph; D is the difference matrix, the confidence matrix controlling the graph labeling. Truth matrix of graph labels The differences between them; It is a non-negative matrix, and ° denotes the Hadamard product;

[0087] Step 5: Based on the three regularization terms constructed in Step 4, namely formulas (3)-(5), a unified minimization objective function is formed to learn the new and missing labels of the graph. The specific formulas are as follows:

[0088]

[0089] Where λ1, λ2, λ3, α, β, γ, and η are hyperparameters, and the range of values ​​for the hyperparameters is set to {10}. -3 10 -2 ,…,10 2 10 3};

[0090] Step 6: Optimize and update the minimization objective function from Step 5 using accelerated proximal gradient method and proximal gradient method, such as... Figure 2 As shown, the updated weight matrix W is obtained, and then a classifier capable of simultaneously modeling new and missing labels is constructed, specifically including the following steps:

[0091] Step 6.1: Initialize all parameters: weight matrix W, confidence matrix Similarity matrix C, difference matrix D, nonnegative matrix Z, and additional parameters ω0 and ω1 for the 0th and 1st iterations;

[0092] Step 6.2: Using the graph kernel method, calculate the similarity between each graph and all other graphs to obtain a graph kernel-based graph representation of all graphs, and form the graph representation matrix X. G ;

[0093] Step 6.3: Update the weight matrix W using the accelerated proximal gradient method:

[0094]

[0095]

[0096] Among them, W t This is the t-th iteration of W, W (t) It is the combination of the (t-1)th and tth iterations of W, ω t It is the t-th iteration with additional parameter ω. It is the gradient of the objective function on W. It is the Lipschitz constant with respect to W. It is a soft threshold operation in

[0097] Step 6.4: Update the labeled similarity matrix C using the proximal gradient method:

[0098]

[0099] in, It is the gradient of the objective function on C, where C is the gradient of the objective function on C. t This is the t-th iteration of C. It is the Lipschitz constant with respect to C. It is a singular value thresholding operation;

[0100] Step 6.5: Apply the proximal gradient method to the confidence matrix of the graph labels. Update:

[0101]

[0102] in, and yes The t and t+1 iterations, Is the objective function in gradient on, It is about The Lipschitz constant;

[0103] Step 6.6: Update the difference matrix D using the proximal gradient method:

[0104]

[0105] Among them, D t It is the t-th iteration of D. It is the gradient of the objective function on D. It is the Lipschitz constant with respect to D;

[0106] Step 6.7: At this point, the objective function has a closed-form solution, so we directly update the non-negative matrix Z:

[0107]

[0108] Step 6.8: Update the additional parameter ω:

[0109]

[0110] Step 6.9: Repeat steps 6.3 to 6.8 until the preset number of iterations is reached, then end the training process.

[0111] Step 7: Given the test set A test package Based on the weight matrix W learned in step 6, a graph classifier is constructed to predict the labels of the graphs in the test package. By utilizing the relationship between the label of the package and the label of the graph, the label prediction of unknown multi-graph packages is achieved, thus completing the annotation of image data.

[0112] When new image packs are available To perform label prediction, we first need to obtain the graph representations of the graphs in the new graph package. Based on the weight matrix W obtained during the training phase, a graph-level classifier is constructed. Calculate the predicted value of the test map package For each graph, take the maximum value of all predicted values ​​associated with it. The label corresponding to this maximum value is used as the label of the graph in the bag. Set the label position to 1 and the other positions to 0. Based on the relationship between the bag label and the graph label: if the graph in the bag has a label, then the bag also has a corresponding label, and the predicted label of the bag is obtained.

[0113] This embodiment applies the method of the present invention and five existing methods to the real image dataset MSRCv2, and compares them on the Hamming loss metric. A smaller Hamming loss metric indicates better performance of the corresponding method. The five existing methods include cfMGML (Multi-graph Multi-label Learning with Dual-granularity Labeling) proposed by Zhao et al., DMNL (Discover Multiple Novel Labels in Multi-Instance Multi-Label Learning) proposed by Zhu et al., MIML-NC (Multi-instance Multi-label Learning in the Presence of Novel ClassInstances) proposed by Pham et al., MIMLwel (Multi-Instance Multi-Label Learning with Weak Label) proposed by Yang et al., and MIMLfast (Fast Multi-Instance Multi-Label Learning) proposed by Huang et al. Figure 3The results show the comparison between the proposed method and five other methods on the MSRCv2 image dataset in terms of the Hamming loss metric, considering missing labels, new labels, and a mixture of missing and new labels. Figure 3 As can be seen in (a), in the case of 50% missing markers, the method of this invention has superior classification performance compared to the five comparative methods. Figure 3 As can be seen in (b), in the case of 50% new tags, since cfMGML and MIMLwel cannot process new tags, the method of the present invention is significantly superior to MIML-NC and MIMLfast, and slightly superior to DMNL, when compared only with DMNL, MIML-NC, and MIMLfast. Figure 3 (c) It can be seen that, with 20% new and missing markers, the method of the present invention is superior to the other three comparative methods by 0.8% to 6.4%.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; therefore, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. An image annotation method capable of handling newly labeled and missing labels, characterized by, The method comprises the following steps: Step 1: obtaining a real original image dataset, and performing data preprocessing on the real original image dataset to obtain a multi-image dataset, wherein the multi-image dataset comprises a multi-image package structure representation and a known category label set, and the multi-image dataset is divided into a training set and a test set; Step 2: mapping each image in the multi-image package from an image space to a high-dimensional feature space, and calculating the similarity between each image in the training set and all images in the multi-image package by using a graph kernel method, so as to obtain a graph kernel-based image representation about each image; Step 3: Based on the graph representation of all graphs, utilize the weight matrix semantically describe the class labels, model the relationship between graphs and labels; Step 4: Constructing two-view regularization term, projection similarity regularization term and bag correlation regularization term to the confidence matrix of the graph label learning; The step 4 specifically comprises the following steps: Step 4.1: To learn the confidence matrix of the graph labels We construct a two-perspective regularization term, making two hypotheses from the perspectives of graph correlation and label correlation. From the perspective of graph correlation, we hypothesize that graphs that are close to each other in the graph space are more likely to be associated with similar labels, i.e., based on the graph similarity matrix. Decompose to obtain From the perspective of label relevance, multiple labels are usually related. It is assumed that the labels of a graph can be inferred from similar labels, i.e., from the label similarity matrix. Decompose to obtain Simultaneously, a two-view regularization term is obtained by making assumptions from these two perspectives, which affects the confidence matrix of the graph labeling. The recovery formula is as follows: （3）； wherein, is a similarity matrix of the graph computed by the graph kernel method, is a learnable labeled similarity matrix, is the Frobenius norm, is the number of all labels; Step 4.2: Compute similarity of known markers, using projection matrix , the similarity matrix of markers , left and right multiplication to return , the upper left corner of , the similarity of markers, construct the projection similarity regularization term, make the known marker part similarity consistent with the true marker similarity, the specific formula is as follows: （4）； wherein, is a projection matrix whose left side is an identity matrix and the rest are 0, is a similarity matrix of known labels of the training set correlation calculated by cosine similarity. Step 4.3: Constructing the package-related regularization term to further learn the confidence matrix of graph labels At this time, the relationship between the package label and the graph label is considered: if a package is associated with a label, at least one graph in the package is associated with this label, and the difference between the confidence matrix of the graph label and the true value matrix of the graph label , that is , using the indicator matrix that can indicate which graphs exist in the package and the projection matrix , respectively left and right multiply to return the first c columns of the number of graphs associated with each label in the package, and then after a clever transformation of the sum of the package label , the relationship between the label of the package and the label of the graph is designed, and the formula of the package-related regularization term is as follows:​ （5）； wherein, is an indicator matrix indicating the existence of a bag in the graph; is a difference matrix controlling the difference between the confidence matrix of the graph label and the true value matrix of the graph label ; is a non-negative matrix, denotes the Hadamard product; Step 5: forming a minimization objective function based on the three regularization terms, i.e., formulas (3)-(5), constructed in step 4, to learn new labels and missing labels of the images; Step 6: Update the weight matrix by optimizing the minimization objective function in step 5 through accelerated proximal gradient method and proximal gradient method , and further construct a classifier capable of simultaneously modeling new markers and missing markers; Step 7: Given a test set one test bag in the test set , according to the weight matrix learned in step 6 , construct a graph-level classifier, make label prediction for the graphs in the test bag, and use the relationship between the label of the bag and the label of the graph to realize the label prediction of the unknown multi-graph bag, and complete the annotation of the image data.

2. The image annotation method capable of handling newly labeled and missing labels according to claim 1, wherein, Step 1 specifically involves dividing each image into multiple regions based on the objects within the image. Each region corresponds to a marker. The Superpixel Segmentation Algorithm (SLIC) is used to segment each region, resulting in multiple superpixels. Each superpixel is considered a node, and adjacent superpixels are considered to have edges. The feature of each node is the vector sum of its RGB color histogram and its directional gradient histogram. At this point, each region of the image is transformed into a graph, and an image can be represented as a graph bag. The original image dataset is preprocessed, resulting in a multi-graph dataset. The multi-graph bag during training is represented as follows: Where n represents the number of multi-image packages, and a multi-image package is... express ,in Indicates the first The first of the packages One image, Indicates the first The number of images in each package The known set of tags corresponding to the multi-map package is represented as: Among them, package The mark is represented as , Given the number of known markers, when Time indicates the first The package belongs to the first Each class tag, when Time indicates the first The package does not belong to the first If a class marker or current value is missing, a setting is provided. A new marker has not been discovered. If the total number of tags is , then the complete tag set should be . .

3. The image annotation method capable of handling newly labeled and missing labels according to claim 1, wherein, The step 2 is specifically, for any one graph , calculate the graph Similarity with all graphs in the multi-graph package, get the graph representation based on the graph kernel, the specific formula is as follows: （1）； wherein, is the number of graphs in the dataset, is the number of graphs in the dataset, is the number of graphs in the dataset, denotes the graph kernel function.

4. The image annotation method capable of handling newly labeled and missing labels according to claim 1, wherein, The step 3 is specifically to define a confidence matrix of unknown graph labels for learning incomplete label information of the graph Modeling the relationship between the graph representation and the label, and the specific formula is as follows: （2）； wherein, is a weight matrix composed of weight vectors corresponding to all the labels, is a confidence matrix of the graph labels composed of confidence vectors of all the graph labels, including known labels and new labels, is a graph representation matrix composed of all the graph labels of the graph, is a regularization parameter, is norm.

5. The image annotation method capable of handling newly labeled and missing labels according to claim 1, wherein, The minimization objective function in step 5 has the following specific formula: （6）； wherein , , , , , and are hyperparameters, for which the value domain is set to .

6. The image annotation method capable of handling newly labeled and missing labels according to claim 1, wherein, The step 6 specifically comprises the following steps: Step 6.1 : Initialization of all parameters: weight matrix , confidence matrix , similarity matrix , difference matrix , non-negative matrix , additional parameters 0th and 1st iteration and ; Step 6.2: Compute the similarity between each graph and all the graphs by using the graph kernel method, obtain the graph representation based on graph kernel about all the graphs, and compose the graph representation matrix ; Step 6.3: Update the weight matrix using the accelerated proximal gradient method with the update: （7）； （8）； wherein is the tth iteration of is the joint of the t-1th and tth iteration of is an additional parameter the tth iteration of is the gradient of the objective function on , is a Lipschitz constant with respect to , is a soft thresholding operation wherein ; Step 6.4: Using a Proximity Gradient Method on the labeled similarity matrix Update: （9）； wherein, is the gradient of the objective function at , is the tth iteration of , is the Lipschitz constant with respect to , is a singular value thresholding operation; Step 6.5: Using the proximal gradient method on the confidence matrix of the icon Update: （10）； wherein and are t and t + 1 iterations of is the gradient of the objective function at is the gradient of the objective function at is a Lipschitz constant with respect to is a Lipschitz constant with respect to Step 6.6: Update the difference matrix using the proximal gradient method Update: （11）； wherein is the tth iteration of is the gradient of the objective function at ; is the Lipschitz constant with respect to ; Step 6.7: At this point the objective function has a closed-form solution, and the non-negative matrix is updated directly: （12）； Step 6.8: Update on additional parameters Update: （13）； Step 6.9: repeatedly iteratively running steps 6.3 to 6.8 until a preset iteration number is reached, and then ending the training process.

7. The image annotation method capable of handling newly labeled and missing labels according to claim 1, wherein, The step 7, when there is a new graph package To make label prediction, first get the graph representation of the graph in the new graph package According to the weight matrix obtained in the training stage Build a graph-level classifier Calculate the prediction value of the test graph package Take the maximum value of the prediction value of all labels associated with each graph, and the label corresponding to this maximum value is the label of the graph in the package. Set the label position to 1 and other positions to 0. According to the relationship between the label of the package and the label of the graph: if the graph in the package has a label, the package also corresponds to this label, and the predicted label of the package is obtained.

Images