Multi-label image recognition method, device and equipment based on hierarchical prompt learning

Abstract

The application relates to a multi-label image recognition method, device and equipment based on hierarchical prompt learning, which comprises the following steps: grouping and compiling the labels of at least one image to be classified to obtain group features and an inter-group relationship prior matrix of each group; based on the generated inter-group dependency graph, a graph convolution network is used to capture the inter-group dependency relationship of the group features to obtain corrected group features; based on a group attention meta network, the group features corresponding to each label are processed to obtain corresponding prompt vectors and encode the prompt vectors to obtain label features and an inter-label relationship prior matrix, generate an inter-label dependency graph, use a graph convolution network to capture the inter-label dependency relationship of the label features of each label to obtain corrected label features, use an image encoder to obtain image features of each image to be classified, and combine the corrected label features to obtain a classification result of the image to be classified. Thus, the problem of label difficulty caused by ignoring the semantic relationship of labels in the data set is solved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a multi-label image recognition method, apparatus and device based on hierarchical cue learning. Background Technology

[0002] With the rapid development of deep learning and the emergence of large-scale multi-label datasets, significant progress has been made in multi-label classification tasks under different settings. Among these, various improvements to loss functions and backbone networks have also achieved remarkable success in the fully labeled setting. Furthermore, many methods have been proposed for partial label settings, such as label matrix completion and semantic distribution transfer.

[0003] In related technologies, image recognition typically employs either a single-label strategy or a full-label strategy. However, under a single-label strategy, the limitation of only one positive label per image makes most methods under full-label and partial-label settings ineffective, often resulting in insufficient label supervision information. To mitigate this, most methods have designed various more robust loss objective functions for single-label settings. For example, firstly, Hill loss and SPLC (Self-Paced Loss Correction) are used to alleviate the influence of false negative labels; secondly, to mitigate the model's memory effect on false negative samples during training under single-label settings, samples with particularly large loss functions are corrected or rejected; and thirdly, a loss maximizing the entropy of unlabeled samples is designed to utilize the unique gradient regions of unlabeled samples during optimization. Besides more robust loss function designs, different training procedures have also been proposed to increase the annotation information of false labels. For example, firstly, unlabeled samples estimated as positive labels during training are ignored; and secondly, asymmetric false label strategies are proposed to generate more accurate supervision.

[0004] However, considering that related objects often co-occur in an image, many methods in both fully and partially labeled settings actively explore dependencies between labels to discover more existing labels. But due to limited annotation information, transferring these methods to single-label settings is severely restricted. For example, explicitly modeling label dependencies using conditional probability matrices requires sufficient labeled data; similarly, implicitly capturing label co-occurrence dependencies through CNN-RNN architectures and transformer-decoder structures requires ample label supervision for training. Therefore, although previous work achieved significant performance improvements under single-label settings, the rich relationships between labels were ignored, resulting in these methods failing to adequately address the lack of label supervision information under single-label strategies, a problem that urgently needs to be solved.

[0005] Application content

[0006] This application provides a multi-label image recognition method, apparatus, and device based on hierarchical cue learning to solve the problems of label supervision information loss caused by ignoring the semantic relationship between labels in the dataset under single labeling, which makes it impossible to mine other unobserved labels from the observed labels, thus leading to label labeling difficulties and high manpower costs.

[0007] The first aspect of this application provides a multi-label image recognition method based on hierarchical cue learning, including the following steps:

[0008] At least one image to be classified is obtained, and the images are grouped according to their labels. Each group is then compiled using a pre-defined text compiler to obtain the group features and prior matrix of inter-group relationships for each group.

[0009] Based on the prior matrix of inter-group relationships, an inter-group dependency graph is generated. Based on the inter-group dependency graph, a preset graph convolutional network is used to capture the inter-group dependencies of the group features of each group, thereby obtaining the corrected group features of each group.

[0010] Based on a pre-defined group attention network, the group features corresponding to each label are processed to obtain a cue vector for each label. A pre-defined text encoder is then used to encode the cue vectors for each label, resulting in label features and a priori matrix of inter-label relationships for each label. An inter-label dependency graph is generated based on the priori matrix of inter-label relationships. Furthermore, based on this dependency graph, a pre-defined graph convolutional network is used to capture the inter-label dependencies in the label features of each label, resulting in corrected label features for each label.

[0011] The image features of each image to be classified are obtained using a preset image encoder, and the predicted probability between the at least one label of the image to be classified is calculated based on the image features of each image to be classified and the label features of each corrected label, so as to perform classification based on the predicted probability.

[0012] According to one embodiment of this application, the at least one image to be classified is grouped according to its label, and each group in the grouping result is compiled by a preset text compiler to obtain the group features and inter-group relationship prior matrix of each group, including:

[0013] Based on the grouping results of at least one image label to be classified, the mapping relationship between the label and the group in the at least one image to be classified is obtained;

[0014] The set of labels corresponding to each group in the at least one image to be classified is obtained according to the mapping relationship;

[0015] The set of tags is compiled using the preset text compiler to obtain the group features and prior matrix of inter-group relationships for each group.

[0016] According to one embodiment of this application, generating an inter-group dependency graph based on the inter-group relationship prior matrix includes:

[0017] Based on the prior matrix of inter-group relationships, at least one image label to be classified is selected from each row of the prior matrix of inter-group relationships that meets the preset correlation criteria. After weighting the label and the labels adjacent to the label, the inter-group dependency graph of each group in the prior matrix of inter-group relationships is obtained.

[0018] According to one embodiment of this application, based on a preset group attention meta-network, the group features corresponding to each label are processed to obtain a cue vector for each label, and a preset text encoder is used to encode the cue vector for each label to obtain the label features and the prior matrix of the relationship between labels for each label, including:

[0019] Based on the corrected group features of each group, the group features corresponding to each label are processed through the preset group attention meta-network to obtain the cue vector of each label.

[0020] Based on the cue vector of each tag, the cue vector of each tag is encoded by the preset text encoder to obtain the tag features and the prior matrix of relationships between tags for each tag.

[0021] According to one embodiment of this application, generating an inter-label dependency graph based on the prior matrix of inter-label relationships includes:

[0022] Based on the prior matrix of inter-label relationships, at least one image label to be classified is selected from each row of the prior matrix of inter-label relationships that meets the preset correlation criteria. After weighting the label and the labels adjacent to the label, the inter-label dependency graph of each label in the prior matrix of inter-label relationships is obtained.

[0023] According to one embodiment of this application, calculating the predicted probability between at least one label of the images to be classified based on the image features of each image to be classified and the label features of each corrected label, and classifying based on the predicted probability, includes:

[0024] The image features of each image to be classified are obtained using a preset image encoder. The prediction probability between the labels of at least one image to be classified is calculated based on the image features of each image to be classified and the label features of each corrected label.

[0025] The at least one image label to be classified is classified according to the predicted probability to obtain a multi-label image recognition model.

[0026] The multi-label image recognition model is optimized based on a preset multi-label classification loss function to obtain an optimized multi-label image recognition model.

[0027] According to the multi-label image recognition method based on hierarchical cue learning in the embodiments of this application, the labels of at least one image to be classified are grouped and compiled to obtain the group features and inter-group relationship prior matrix of each group. Based on the generated inter-group dependency graph, a graph convolutional network is used to capture the inter-group dependency relationship of the group features to obtain the corrected group features. Based on the group attention meta-network, the group features corresponding to each label are processed to obtain the corresponding cue vector and encoded to obtain the label features and inter-label relationship prior matrix. An inter-label dependency graph is generated. A graph convolutional network is used to capture the inter-label dependency relationship of the label features of each label to obtain the corrected label features. An image encoder is used to obtain the image features of each image to be classified, and the classification result of the image to be classified is obtained by combining the corrected label features. Therefore, this paper addresses the problem of missing label supervision information caused by ignoring the semantic relationships between labels in a dataset under single labeling, which makes it impossible to discover other unobserved labels from the observed labels, leading to difficulties in labeling and high manpower consumption. The paper effectively transfers cross-modal image and text features to multi-label classification tasks through a hierarchical cue learning method, and explores the dependencies between labels and groups through a hierarchical graph convolutional network, thereby improving the performance of multi-label image classification tasks under single labeling.

[0028] A second aspect of this application provides a multi-label image recognition device based on hierarchical cue learning, comprising:

[0029] The acquisition module is used to acquire at least one image to be classified, group the images according to their labels, and compile each group in the grouping results using a preset text compiler to obtain the group features and inter-group relationship prior matrix of each group.

[0030] The first capturing module is used to generate an inter-group dependency graph based on the prior matrix of inter-group relationships, and capture the inter-group dependency relationships of the group features of each group using a preset graph convolutional network based on the inter-group dependency graph, so as to obtain the corrected group features of each group.

[0031] The second capture module is used to process the group features corresponding to each label based on a preset group attention meta-network to obtain a cue vector for each label, and to encode the cue vector of each label using a preset text encoder to obtain the label features and a prior matrix of inter-label relationships for each label. Based on the prior matrix of inter-label relationships, an inter-label dependency graph is generated, and based on the inter-label dependency graph, the preset graph convolutional network is used to capture the inter-label dependencies of the label features of each label to obtain the corrected label features of each label; and

[0032] The calculation module is used to acquire the image features of each image to be classified using a preset image encoder, and to calculate the predicted probability between the at least one label of the image to be classified based on the image features of each image to be classified and the label features of each corrected label, so as to perform classification based on the predicted probability.

[0033] According to one embodiment of this application, the acquisition module is specifically used for:

[0034] Based on the grouping results of at least one image label to be classified, the mapping relationship between the label and the group in the at least one image to be classified is obtained;

[0035] The set of labels corresponding to each group in the at least one image to be classified is obtained according to the mapping relationship;

[0036] The set of tags is compiled using the preset text compiler to obtain the group features and prior matrix of inter-group relationships for each group.

[0037] According to one embodiment of this application, the first capturing module is specifically used for:

[0038] Based on the prior matrix of inter-group relationships, at least one image label to be classified is selected from each row of the prior matrix of inter-group relationships that meets the preset correlation criteria. After weighting the label and the labels adjacent to the label, the inter-group dependency graph of each group in the prior matrix of inter-group relationships is obtained.

[0039] According to one embodiment of this application, the second capturing module is specifically used for:

[0040] Based on the corrected group features of each group, the group features corresponding to each label are processed through the preset group attention meta-network to obtain the cue vector of each label.

[0041] Based on the cue vector of each tag, the cue vector of each tag is encoded by the preset text encoder to obtain the tag features and the prior matrix of relationships between tags for each tag.

[0042] According to one embodiment of this application, the second capturing module is specifically used for:

[0043] Based on the prior matrix of inter-label relationships, at least one image label to be classified is selected from each row of the prior matrix of inter-label relationships that meets the preset correlation criteria. After weighting the label and the labels adjacent to the label, the inter-label dependency graph of each label in the prior matrix of inter-label relationships is obtained.

[0044] According to one embodiment of this application, the computing module is specifically used for:

[0045] The image features of each image to be classified are obtained using a preset image encoder. The prediction probability between the labels of at least one image to be classified is calculated based on the image features of each image to be classified and the label features of each corrected label.

[0046] The at least one image label to be classified is classified according to the predicted probability to obtain a multi-label image recognition model.

[0047] The multi-label image recognition model is optimized based on a preset multi-label classification loss function to obtain an optimized multi-label image recognition model.

[0048] According to the embodiments of this application, the multi-label image recognition device based on hierarchical cue learning groups and compiles the labels of at least one image to be classified to obtain the group features and inter-group relationship prior matrix of each group. Based on the generated inter-group dependency graph, a graph convolutional network is used to capture the inter-group dependency relationship of the group features to obtain the corrected group features. Based on the group attention meta-network, the corresponding cue vector is processed for each label to obtain the corresponding cue vector and encoded to obtain the label features and inter-label relationship prior matrix. An inter-label dependency graph is generated. A graph convolutional network is used to capture the inter-label dependency relationship of the label features of each label to obtain the corrected label features. An image encoder is used to obtain the image features of each image to be classified, and the classification result of the image to be classified is obtained by combining the corrected label features. Therefore, this paper addresses the problem of missing label supervision information caused by ignoring the semantic relationships between labels in a dataset under single labeling, which makes it impossible to discover other unobserved labels from the observed labels, leading to difficulties in labeling and high manpower consumption. The paper effectively transfers cross-modal image and text features to multi-label classification tasks through a hierarchical cue learning method, and explores the dependencies between labels and groups through a hierarchical graph convolutional network, thereby improving the performance of multi-label image classification tasks under single labeling.

[0049] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-label image recognition method based on hierarchical cue learning as described in the above embodiments.

[0050] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the multi-label image recognition method based on hierarchical cue learning as described in the above embodiments.

[0051] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0052] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0053] Figure 1 This is a flowchart of a multi-label image recognition method based on hierarchical prompting learning according to an embodiment of this application;

[0054] Figure 2 This is a flowchart of a model based on hierarchical prompting learning according to an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the network framework of a multi-label recognition method based on hierarchical prompting learning according to an embodiment of this application;

[0056] Figure 4 This is a schematic diagram illustrating the hierarchical dependency relationship between tags according to an embodiment of this application;

[0057] Figure 5 This is a block diagram of a multi-label image recognition device based on hierarchical prompting learning according to an embodiment of this application;

[0058] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0059] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0060] The following describes a multi-label image recognition method, apparatus, and device based on hierarchical cue learning according to embodiments of this application, with reference to the accompanying drawings. Addressing the issues mentioned in the background art, where single labeling leads to a lack of label supervision information due to ignoring semantic relationships between labels in the dataset, thus preventing the discovery of unobserved labels from observed labels and resulting in difficult and labor-intensive labeling, this application provides a multi-label image recognition method based on hierarchical cue learning. In this method, the labels of at least one image to be classified are grouped and compiled to obtain group features and a prior matrix of inter-group relationships for each group. Based on the generated inter-group dependency graph, a graph convolutional network is used to capture the inter-group dependencies of group features, resulting in corrected group features. Based on a group attention meta-network, the group features corresponding to each label are processed to obtain a corresponding cue vector, which is then encoded to obtain label features and a prior matrix of inter-label relationships, generating an inter-label dependency graph. A graph convolutional network is used to capture the inter-label dependencies of the label features of each label, resulting in corrected label features. An image encoder is used to obtain the image features of each image to be classified, and the classification result of the image to be classified is obtained by combining the corrected label features. Therefore, this paper addresses the problem of missing label supervision information caused by ignoring the semantic relationships between labels in a dataset under single labeling, which makes it impossible to discover other unobserved labels from the observed labels, leading to difficulties in labeling and high manpower consumption. The paper effectively transfers cross-modal image and text features to multi-label classification tasks through a hierarchical cue learning method, and explores the dependencies between labels and groups through a hierarchical graph convolutional network, thereby improving the performance of multi-label image classification tasks under single labeling.

[0061] Specifically, Figure 1 This is a flowchart illustrating a multi-label image recognition method based on hierarchical prompting learning, provided in an embodiment of this application.

[0062] like Figure 1 As shown, this multi-label image recognition method based on hierarchical cue learning includes the following steps:

[0063] In step S101, at least one image to be classified is obtained, and the images are grouped according to the labels of the at least one image to be classified. Each group in the grouping result is compiled by a preset text compiler to obtain the group features and the prior matrix of inter-group relationships for each group.

[0064] Furthermore, in some embodiments, the images are grouped according to the labels of at least one image to be classified, and each group in the grouping results is compiled by a preset text compiler to obtain the group features and inter-group relationship prior matrix of each group. This includes: obtaining the mapping relationship between the labels and groups in at least one image to be classified based on the grouping results of the labels of at least one image to be classified; obtaining the set of labels corresponding to each group in at least one image to be classified according to the mapping relationship; and compiling the set of labels using a preset text compiler to obtain the group features and inter-group relationship prior matrix of each group.

[0065] The preset text compiler can be any text compiler selected by those skilled in the art based on actual compilation needs, and no specific limitations are made here.

[0066] Specifically, such as Figure 2 and Figure 3 As shown, to address the problem of missing supervision information between labels and groups under a single labeling strategy, this embodiment employs hierarchical cue learning to leverage the rich hierarchical dependencies between labels to mine more existing labels in the image. Therefore, this embodiment first obtains at least one image to be classified, and then, based on the labels Y = {y0, y1, ..., y...} of the at least one image to be classified, it... n Grouping them, we get group Z = {z1, z2, ..., z} m Secondly, based on the grouping results of at least one image label to be classified, the mapping relationship Φ: Y→Z between the label and the group in at least one image to be classified is obtained, and the group z in each of the at least one image to be classified is obtained through the mapping relationship between the label and the group. j The corresponding set of tags h j ={y∈Y|Φ(y)=z j Finally, for each group z j The prompt "a photo of a [CLS]" is used to process each group z using a pre-defined text compiler. j The set of included tags is compiled to obtain the group feature g for each group. j Thus, based on the group feature g j Construct the prior matrix A of intergroup relationships Z =(a ij ) m×m , where a ij =sim(g i g j ).

[0067] In step S102, an inter-group dependency graph is generated based on the prior matrix of inter-group relationships. Based on the inter-group dependency graph, a preset graph convolutional network is used to capture the inter-group dependencies of each group's group features, thereby obtaining the corrected group features of each group.

[0068] Furthermore, in some embodiments, generating an inter-group dependency graph based on the inter-group relationship prior matrix includes: selecting at least one image label to be classified that meets a preset correlation standard in each row of the inter-group relationship prior matrix based on the inter-group relationship prior matrix, and weighting the label and the labels adjacent to the label to obtain an inter-group dependency graph for each group in the inter-group relationship prior matrix.

[0069] The preset graph convolutional networks and preset correlation standards can be selected by those skilled in the art according to actual usage needs, and no specific limitations are made here.

[0070] Specifically, in this embodiment, after obtaining the prior matrix of inter-group relationships for at least one image to be classified, in order to remove additional noise in the prior relationships, at least one image label that meets a preset correlation criterion is selected from each row of the prior matrix of inter-group relationships. For example, the K elements with the highest correlation among the labels are selected and retained, and the remaining elements are set to 0, thereby obtaining a sparsified matrix. In order to alleviate To address the oversmoothing problem, embodiments of this application can set hyperparameters to adjust the weights of the label itself and its nearest neighbor labels in at least one image to be classified. After weighted processing, the final adjacency matrix is ​​obtained, denoted as the inter-group dependency graph. The processing procedure can be represented as follows:

[0071]

[0072]

[0073] Furthermore, based on inter-group dependency graphs In this embodiment of the application, a preset graph convolutional network can be used to process the group features G = {g1, g2, ..., g...} for each group. m Adjustments are made to capture inter-group dependencies. The input features are... The l-th graph convolutional layer is updated to... Therefore, after convolution of the l-layer graph, the group features of each group are obtained after correction through the residual structure. The update process can be represented by the following formula:

[0074]

[0075] In step S103, based on a preset group attention network, the group features corresponding to each label are processed to obtain the cue vector of each label. The cue vector of each label is encoded using a preset text encoder to obtain the label features and the prior matrix of inter-label relationships for each label. An inter-label dependency graph is generated based on the prior matrix of inter-label relationships. Based on the inter-label dependency graph, a preset graph convolutional network is used to capture the inter-label dependencies of the label features of each label to obtain the corrected label features of each label.

[0076] Furthermore, in some embodiments, based on a preset group attention network, the group features corresponding to each label are processed to obtain a cue vector for each label, and a preset text encoder is used to encode the cue vector for each label to obtain the label features and the prior matrix of relationships between labels for each label. This includes: based on the corrected group features of each group, processing the group features corresponding to each label through a preset group attention network to obtain a cue vector for each label; and based on the cue vector for each label, encoding the cue vector for each label through a preset text encoder to obtain the label features and the prior matrix of relationships between labels for each label.

[0077] Furthermore, in some embodiments, generating an inter-label dependency graph based on the inter-label relationship prior matrix includes: selecting at least one image label to be classified that meets a preset association criterion in each row of the inter-label relationship prior matrix based on the inter-label relationship prior matrix, and obtaining an inter-label dependency graph for each label in the inter-label relationship prior matrix after weighting the label and the labels adjacent to the label.

[0078] Specifically, in order to capture the relationship between labels and groups, embodiments of this application further design a group attention meta-network Meta, based on the corrected group features G of each group. * For each label y i Provide hint vectors.

[0079] Specifically, in this application embodiment, the corresponding group feature G is first... * After a pre-defined group attention meta-network is used for each label y i Corresponding group features G * The process is performed to obtain the cue vector s for each label. i , where s i ={v i1 v i2 , ..., v iN}=Meta(g jThus, all labels belonging to the same group can share the same group features. Therefore, it encourages the aggregation of label features belonging to the same group in the same manifold space, making the group attention features of the label more compact and more conducive to the matching of the label with at least one image to be classified.

[0080] For example, such as Figure 4 As shown, each label can be associated with a group based on certain metrics, such as semantic similarity. For example, "cat" and "dog" belong to the "animals" group, and "table" and "chair" belong to the "furniture" group. Therefore, labels within the same group are easier to infer from a given set of labels than labels in other groups that have a weaker semantic relationship with the labeled labels. Thus, by utilizing the correspondence between labels and groups, we can uncover unobserved related labels within the same group, while focusing more on labels with weaker relationships between different groups.

[0081] Secondly, based on the cue vectors s for each label i The hint vector s for each tag is generated by the preset text compiler. i Compile the code to obtain the tag feature t for each tag. i And the prior matrix of relationships between labels.

[0082] Finally, in each row of the prior matrix of relationships between labels, at least one image label to be classified is selected that meets the preset correlation criteria. For example, the K elements with the highest correlation among the labels are selected and retained, and the remaining elements are set to 0. After obtaining the sparsified matrix, the labels and the labels adjacent to them are weighted to obtain the final adjacency matrix, which is denoted as the label dependency graph.

[0083] Furthermore, based on the tag dependency graph This application embodiment can utilize a preset graph convolutional network to process the label features T = {t1, t2, ..., t} of each label. m Adjustments are made to capture dependencies between labels. The input features are... The l-th graph convolutional layer is updated to... Therefore, after convolution of the l-layer graph, the label features of each label are obtained through a residual structure after correction. The update process can be represented by the following formula:

[0084]

[0085] In step S104, the image features of each image to be classified are obtained using a preset image encoder, and the predicted probability between at least one label of the image to be classified is calculated based on the image features of each image to be classified and the corrected label features of each label, so as to classify the image based on the predicted probability.

[0086] Furthermore, in some embodiments, calculating at least one predicted probability between at least one label of an image to be classified based on the image features of each image to be classified and the corrected label features of each label, and classifying based on the predicted probability, includes: acquiring the image features of each image to be classified using a preset image encoder; calculating at least one predicted probability between at least one label of an image to be classified based on the image features of each image to be classified and the corrected label features of each label; classifying at least one label of an image to be classified based on the predicted probability to obtain a multi-label image recognition model; and optimizing the multi-label image recognition model based on a preset multi-label classification loss function to obtain an optimized multi-label image recognition model.

[0087] The preset image encoder can be selected by those skilled in the art according to actual usage needs, and no specific limitations are made here.

[0088] Specifically, in this embodiment, in at least one image to be classified, image features f of each image to be classified are obtained using a preset image encoder, and then the image features f of each image to be classified and the corrected label features T of each label are used to classify the image features f. * The predicted probability p(y) between at least one image label to be classified is calculated using cosine similarity. i |x), where, The model classifies at least one image label based on the predicted probability to obtain a multi-label image recognition model. During the training process of the multi-label image recognition model, it is optimized based on a preset multi-label classification loss function L to obtain the optimized multi-label image recognition model. The preset multi-label classification loss function L can be expressed as:

[0089]

[0090] in, And p i =p(y i |x).

[0091] In summary, based on the above detailed discussion of the embodiments of this application, the following beneficial effects can be achieved:

[0092] (1) The hierarchical prompting strategy can effectively capture the dependency relationship between labels and groups, while the hierarchical graph convolutional network can further explore the dependency relationship between labels and groups, which is beneficial to the semantic matching between images and labels and effectively alleviates the problem of insufficient supervision information under the single annotation strategy.

[0093] (2) The hierarchical cue learning multi-label image recognition method proposed in the embodiments of this application improves the performance of related techniques by 2.5%, 1.4%, 6.6% and 3.9% respectively on the COCO, VOC2012, NUSWIDE and CUB (Caltech-UCSD Birds-200-2011, Caltech Birds Dataset) datasets based on a single positive label setting. Among them, COCO and VG-200 also achieved the state-of-the-art recognition performance achievable in current multi-label image recognition with improvements of 1.5% and 3.4% respectively under partial label settings.

[0094] According to the multi-label image recognition method based on hierarchical cue learning in the embodiments of this application, the labels of at least one image to be classified are grouped and compiled to obtain the group features and inter-group relationship prior matrix of each group. Based on the generated inter-group dependency graph, a graph convolutional network is used to capture the inter-group dependency relationship of the group features to obtain the corrected group features. Based on the group attention meta-network, the group features corresponding to each label are processed to obtain the corresponding cue vector and encoded to obtain the label features and inter-label relationship prior matrix. An inter-label dependency graph is generated. A graph convolutional network is used to capture the inter-label dependency relationship of the label features of each label to obtain the corrected label features. An image encoder is used to obtain the image features of each image to be classified, and the classification result of the image to be classified is obtained by combining the corrected label features. Therefore, this paper addresses the problem of missing label supervision information caused by ignoring the semantic relationships between labels in a dataset under single labeling, which makes it impossible to discover other unobserved labels from the observed labels, leading to difficulties in labeling and high manpower consumption. The paper effectively transfers cross-modal image and text features to multi-label classification tasks through a hierarchical cue learning method, and explores the dependencies between labels and groups through a hierarchical graph convolutional network, thereby improving the performance of multi-label image classification tasks under single labeling.

[0095] Next, referring to the accompanying drawings, a multi-label image recognition device based on hierarchical prompting learning, according to an embodiment of this application, is described.

[0096] Figure 5 This is a block diagram of a multi-label image recognition device based on hierarchical prompting learning according to an embodiment of this application.

[0097] like Figure 5 As shown, the multi-label image recognition device 10 based on hierarchical prompting learning includes: an acquisition module 100, a first capture module 200, a second capture module 300, and a calculation module 400.

[0098] The acquisition module 100 is used to acquire at least one image to be classified, group them according to the label of the at least one image to be classified, and compile each group in the grouping result through a preset text compiler to obtain the group features and inter-group relationship prior matrix of each group.

[0099] The first capturing module 200 is used to generate an inter-group dependency graph based on the inter-group relationship prior matrix, and based on the inter-group dependency graph, use a preset graph convolutional network to capture the inter-group dependency relationship of each group's group features to obtain the corrected group features of each group.

[0100] The second capture module 300 is used to process the group features corresponding to each label based on a preset group attention meta-network to obtain the cue vector of each label, and to encode the cue vector of each label using a preset text encoder to obtain the label features and the prior matrix of inter-label relationships for each label. Based on the prior matrix of inter-label relationships, an inter-label dependency graph is generated, and based on the inter-label dependency graph, a preset graph convolutional network is used to capture the inter-label dependencies of the label features of each label to obtain the corrected label features of each label; and

[0101] The calculation module 400 is used to acquire the image features of each image to be classified using a preset image encoder, and to calculate at least one predicted probability between the labels of the images to be classified based on the image features of each image to be classified and the corrected label features of each label, so as to classify the images based on the predicted probabilities.

[0102] Furthermore, in some embodiments, the acquisition module 100 is specifically used for:

[0103] Based on the grouping results of at least one image label to be classified, obtain at least one mapping relationship between the label and the group in the image to be classified;

[0104] Based on the mapping relationship, obtain at least one set of labels corresponding to each group in the image to be classified;

[0105] The set of tags is compiled using a pre-defined text compiler to obtain the group features and prior matrix of inter-group relationships for each group.

[0106] Furthermore, in some embodiments, the first capture module 200 is specifically used for:

[0107] Based on the inter-group relationship prior matrix, at least one image label to be classified is selected from each row of the inter-group relationship prior matrix that meets the preset correlation criteria. After weighting the labels and the labels adjacent to the labels, the inter-group dependency graph of each group in the inter-group relationship prior matrix is ​​obtained.

[0108] Furthermore, in some embodiments, the second capture module is specifically used for:

[0109] Based on the corrected group features of each group, the group features corresponding to each label are processed through a pre-defined group attention meta-network to obtain the cue vector for each label.

[0110] Based on the cue vector of each tag, the cue vector of each tag is encoded by a pre-set text encoder to obtain the tag features of each tag and the prior matrix of relationships between tags.

[0111] Furthermore, in some embodiments, the second capture module 300 is specifically used for:

[0112] Based on the prior matrix of inter-label relationships, at least one image label to be classified is selected from each row of the prior matrix of inter-label relationships that meets the preset correlation criteria. After weighting the labels and the labels adjacent to the labels, the inter-label dependency graph of each label in the prior matrix of inter-label relationships is obtained.

[0113] Furthermore, in some embodiments, the computing module 400 is specifically used for:

[0114] The image features of each image to be classified are obtained using a preset image encoder. Based on the image features of each image to be classified and the corrected label features of each label, at least one predicted probability between the labels of the images to be classified is calculated.

[0115] A multi-label image recognition model is obtained by classifying at least one image label based on the predicted probability.

[0116] The multi-label image recognition model is optimized based on a preset multi-label classification loss function to obtain the optimized multi-label image recognition model.

[0117] According to the embodiments of this application, the multi-label image recognition device based on hierarchical cue learning groups and compiles the labels of at least one image to be classified to obtain the group features and inter-group relationship prior matrix of each group. Based on the generated inter-group dependency graph, a graph convolutional network is used to capture the inter-group dependency relationship of the group features to obtain the corrected group features. Based on the group attention meta-network, the corresponding cue vector is processed for each label to obtain the corresponding cue vector and encoded to obtain the label features and inter-label relationship prior matrix. An inter-label dependency graph is generated. A graph convolutional network is used to capture the inter-label dependency relationship of the label features of each label to obtain the corrected label features. An image encoder is used to obtain the image features of each image to be classified, and the classification result of the image to be classified is obtained by combining the corrected label features. Therefore, this paper addresses the problem of missing label supervision information caused by ignoring the semantic relationships between labels in a dataset under single labeling, which makes it impossible to discover other unobserved labels from the observed labels, leading to difficulties in labeling and high manpower consumption. The paper effectively transfers cross-modal image and text features to multi-label classification tasks through a hierarchical cue learning method, and explores the dependencies between labels and groups through a hierarchical graph convolutional network, thereby improving the performance of multi-label image classification tasks under single labeling.

[0118] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0119] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0120] When the processor 602 executes the program, it implements the multi-label image recognition method based on hierarchical prompting learning provided in the above embodiments.

[0121] Furthermore, electronic devices also include:

[0122] Communication interface 603 is used for communication between memory 601 and processor 602.

[0123] The memory 601 is used to store computer programs that can run on the processor 602.

[0124] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0125] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0126] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0127] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0128] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-label image recognition method based on hierarchical cue learning as described above.

[0129] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0130] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0131] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A multi-label image recognition method based on hierarchical cue learning, characterized in that, Includes the following steps: At least one image to be classified is obtained, and the images are grouped according to their labels. Each group is then compiled using a pre-defined text compiler to obtain the group features and prior matrix of inter-group relationships for each group. Based on the prior matrix of inter-group relationships, an inter-group dependency graph is generated. Based on the inter-group dependency graph, a preset graph convolutional network is used to capture the inter-group dependencies of the group features of each group, thereby obtaining the corrected group features of each group. Based on a pre-defined group attention network, the group features corresponding to each label are processed to obtain the cue vector of each label. The cue vector of each label is then encoded using a pre-defined text encoder to obtain the label features and the prior matrix of inter-label relationships for each label. An inter-label dependency graph is generated based on the prior matrix of inter-label relationships. Based on the inter-label dependency graph, the pre-defined graph convolutional network is used to capture the inter-label dependencies of the label features of each label to obtain the corrected label features of each label. as well as The image features of each image to be classified are obtained using a preset image encoder, and the predicted probability between the at least one label of the image to be classified is calculated based on the image features of each image to be classified and the label features of each corrected label, so as to perform classification based on the predicted probability.

2. The method according to claim 1, characterized in that, The images are grouped according to their labels, and each group is compiled using a pre-defined text compiler to obtain the group features and prior matrix of inter-group relationships for each group, including: Based on the grouping results of at least one image label to be classified, the mapping relationship between the label and the group in the at least one image to be classified is obtained; The set of labels corresponding to each group in the at least one image to be classified is obtained according to the mapping relationship; The set of tags is compiled using the preset text compiler to obtain the group features and prior matrix of inter-group relationships for each group.

3. The method according to claim 1, characterized in that, The step of generating an inter-group dependency graph based on the prior matrix of inter-group relationships includes: Based on the prior matrix of inter-group relationships, at least one image label to be classified is selected from each row of the prior matrix of inter-group relationships that meets the preset correlation criteria. After weighting the label and the labels adjacent to the label, the inter-group dependency graph of each group in the prior matrix of inter-group relationships is obtained.

4. The method according to claim 1, characterized in that, Based on a pre-defined group attention meta-network, the group features corresponding to each label are processed to obtain a cue vector for each label. A pre-defined text encoder is then used to encode the cue vector for each label to obtain the label features and prior matrix of relationships between labels for each label, including: Based on the corrected group features of each group, the group features corresponding to each label are processed through the preset group attention meta-network to obtain the cue vector of each label. Based on the cue vector of each tag, the cue vector of each tag is encoded by the preset text encoder to obtain the tag features and the prior matrix of relationships between tags for each tag.

5. The method according to claim 1, characterized in that, The step of generating an inter-label dependency graph based on the prior matrix of inter-label relationships includes: Based on the prior matrix of inter-label relationships, at least one image label to be classified is selected from each row of the prior matrix of inter-label relationships that meets the preset correlation criteria. After weighting the label and the labels adjacent to the label, the inter-label dependency graph of each label in the prior matrix of inter-label relationships is obtained.

6. The method according to claim 1, characterized in that, Calculate the predicted probability between the at least one label of the images to be classified based on the image features of each image to be classified and the label features of each corrected label, and perform classification based on the predicted probability, including: The image features of each image to be classified are obtained using a preset image encoder. The prediction probability between the labels of at least one image to be classified is calculated based on the image features of each image to be classified and the label features of each corrected label. The at least one image label to be classified is classified according to the predicted probability to obtain a multi-label image recognition model. The multi-label image recognition model is optimized based on a preset multi-label classification loss function to obtain an optimized multi-label image recognition model.

7. A multi-label image recognition device based on hierarchical cue learning, characterized in that, include: The acquisition module is used to acquire at least one image to be classified, group the images according to their labels, and compile each group in the grouping results using a preset text compiler to obtain the group features and inter-group relationship prior matrix of each group. The first capturing module is used to generate an inter-group dependency graph based on the prior matrix of inter-group relationships, and capture the inter-group dependency relationships of the group features of each group using a preset graph convolutional network based on the inter-group dependency graph, so as to obtain the corrected group features of each group. The second capture module is used to process the group features corresponding to each label based on a preset group attention network to obtain the cue vector of each label, and to encode the cue vector of each label using a preset text encoder to obtain the label features and the prior matrix of the relationship between labels for each label. The module generates a dependency graph between labels based on the prior matrix of the relationship between labels, and captures the dependency relationship between labels based on the dependency graph using the preset graph convolutional network to obtain the corrected label features of each label. as well as The calculation module is used to acquire the image features of each image to be classified using a preset image encoder, and to calculate the predicted probability between the at least one label of the image to be classified based on the image features of each image to be classified and the label features of each corrected label, so as to perform classification based on the predicted probability.

8. The apparatus according to claim 7, characterized in that, The acquisition module is specifically used for: Based on the grouping results of at least one image label to be classified, the mapping relationship between the label and the group in the at least one image to be classified is obtained; The set of labels corresponding to each group in the at least one image to be classified is obtained according to the mapping relationship; The set of tags is compiled using the preset text compiler to obtain the group features and prior matrix of inter-group relationships for each group.

9. The apparatus according to claim 7, characterized in that, The first capture module is specifically used for: Based on the prior matrix of inter-group relationships, at least one image label to be classified is selected from each row of the prior matrix of inter-group relationships that meets the preset correlation criteria. After weighting the label and the labels adjacent to the label, the inter-group dependency graph of each group in the prior matrix of inter-group relationships is obtained.

10. The apparatus according to claim 7, characterized in that, The second capture module is specifically used for: Based on the corrected group features of each group, the group features corresponding to each label are processed through the preset group attention meta-network to obtain the cue vector of each label. Based on the cue vector of each tag, the cue vector of each tag is encoded by the preset text encoder to obtain the tag features and the prior matrix of relationships between tags for each tag.

11. The apparatus according to claim 7, characterized in that, The second capture module is specifically used for: Based on the prior matrix of inter-label relationships, at least one image label to be classified is selected from each row of the prior matrix of inter-label relationships that meets the preset correlation criteria. After weighting the label and the labels adjacent to the label, the inter-label dependency graph of each label in the prior matrix of inter-label relationships is obtained.

12. The apparatus according to claim 7, characterized in that, The computing module is specifically used for: The image features of each image to be classified are obtained using a preset image encoder. The prediction probability between the labels of at least one image to be classified is calculated based on the image features of each image to be classified and the label features of each corrected label. The at least one image label to be classified is classified according to the predicted probability to obtain a multi-label image recognition model. The multi-label image recognition model is optimized based on a preset multi-label classification loss function to obtain an optimized multi-label image recognition model.

13. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the multi-label image recognition method based on hierarchical cue learning as described in any one of claims 1-6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the multi-label image recognition method based on hierarchical cue learning as described in any one of claims 1-6.

Images