Method for training multi-label classification model, multi-label classification method and product

Abstract

The application discloses a multi-label classification model training method, a multi-label classification method and a product. The training method comprises the following steps: inputting a training image into a backbone network for feature extraction, so as to output an initial feature map, wherein the training image has a category label; inputting the initial feature map into a first convolution branch and a second convolution branch of the classification network for processing respectively, so as to output a first spatial feature map and a second spatial feature map respectively; performing a weighting operation on the first spatial feature map based on the second spatial feature map, so as to obtain a class-specific spatial feature map; outputting a category score of the training image based on the class-specific spatial feature map; and training the multi-label classification model based on the category score and the category label. According to the training method, the trained multi-label classification model has better classification performance.

Description

Technical Field

[0001] This application generally relates to the field of image processing technology. More specifically, this application relates to a training method for a multi-label classification model, a method for multi-label classification based on images, a device for multi-label classification, and a computer-readable storage medium. Background Technology

[0002] Multi-label classification refers to the task of outputting multiple classification labels contained in an input image. In medical imaging, multi-label classification can be used to analyze the probabilities of multiple disease labels contained in a medical image, thereby assisting doctors in diagnosis and treatment. Currently, the mainstream method for multi-label classification of medical images is based on convolutional neural networks (CNNs). This involves processing the input medical image through multiple layers of convolution, pooling, and fully connected layers to obtain a feature vector, which is then passed through a multi-output classifier to output the probability of each disease label.

[0003] However, traditional CNN methods have some drawbacks. Typically, CNNs use global average pooling for feature dimensionality reduction before fully connected layers. This operation loses spatial information about the features and fails to effectively capture the different spatial regions occupied by objects of different categories. For disease labels with clear local lesion features, this loss of local information leads to a decline in classification performance.

[0004] Therefore, there is an urgent need to provide an image classification scheme that can effectively capture spatial information of features in order to improve the classification performance of multi-label classification tasks. Summary of the Invention

[0005] To address at least one or more of the technical problems mentioned above, this application proposes a training method for a multi-label classification model, a method for image-based multi-label classification, a device for multi-label classification, and a computer-readable storage medium in several aspects.

[0006] In a first aspect, this application provides a training method for a multi-label classification model, the multi-label classification model including a backbone network and a classification network, the classification network including multiple convolutional branches, the training method including: inputting a training image into the backbone network for feature extraction to output an initial feature map, wherein the training image has a class label; inputting the initial feature map into a first convolutional branch and a second convolutional branch of the classification network for processing to output a first spatial feature map and a second spatial feature map respectively; performing a weighted operation on the first spatial feature map based on the second spatial feature map to obtain a class-specific spatial feature map; outputting a class score of the training image based on the class-specific spatial feature map; and training the multi-label classification model based on the class score and the class label.

[0007] In some embodiments, the first convolutional branch is used to output first spatial feature maps of multiple categories, and the second convolutional branch is used to output second spatial feature maps of multiple categories; the weighting operation includes: normalizing the second spatial feature map of each category to obtain a spatial attention score for each category; and using the spatial attention score of each category as a weight to perform a weighting operation on the first spatial feature map of the corresponding category to obtain a class-specific spatial feature map for each category.

[0008] In other embodiments, the first spatial feature map and the second spatial feature map are obtained through the following processing: z 1c =X T w 1c ;z 2c =X T w 2c Among them, z 1c The first spatial feature map representing category c, z 2c Let X represent the second spatial feature map of category c, and w represent the initial feature map. 1c w represents the first convolution weight for class c. 2c This represents the second convolution weight for class c.

[0009] In some other embodiments, the class-specific spatial feature map is obtained through the following weighting operation: v c =z 1c *σ(z 2c ); where v c The class-specific spatial feature map representing category c, z 1c The first spatial feature map representing category c, z 2c The second spatial feature map represents category c, σ represents the Sigmoid function, and * indicates element-wise multiplication of the matrices.

[0010] In some embodiments, outputting the class score of the training image based on the class-specific spatial feature map includes: performing global average pooling on the class-specific spatial feature map; and normalizing the result of the global average pooling to obtain the class score.

[0011] In other embodiments, the training method further includes: acquiring lesion location information related to the category label in the training image; and using the lesion location information as a supervision signal for the second spatial feature map to train the multi-label classification model.

[0012] In some other embodiments, the training method further includes: processing the initial feature map using a self-attention mechanism to obtain a self-attention feature map; and outputting a class score of the training image based on the class-specific spatial feature map and the self-attention feature map.

[0013] In some embodiments, the self-attention feature map is obtained by the following formula: K c =X T W Kc Q c =X T W Qc ; Where Kc represents the K-map of class c, Qc represents the Q-map of class c, X represents the initial feature map, and W represents the initial feature map. Kc W represents the weight matrix used to calculate Kc. Qc z represents the weight matrix used to calculate Qc. 3c Let M represent the self-attention feature map, and M represent the number of channels in the initial feature map.

[0014] In other embodiments, outputting the class score of the training image based on the class-specific spatial feature map and the self-attention feature map includes: performing global average pooling on the result of multiplying the class-specific spatial feature map and the self-attention feature map; and normalizing the result of the global average pooling to obtain the class score.

[0015] In some other embodiments, the category score for each category is obtained by the following formula: s c =σ(GAP(z) 1c *σ(z 2c )*z 3c )); where s c z represents the category score for category c. 1c The first spatial feature map representing category c, z 2c The second spatial feature map representing category c, z 3c This represents the self-attention feature map, σ represents the Sigmoid function, * represents element-wise multiplication of matrices, and GAP represents global average pooling.

[0016] In some embodiments, the training images include fundus images.

[0017] In a second aspect, this application provides a method for multi-label classification based on images, comprising: inputting an image to be classified into a multi-label classification model trained according to any of the training methods described in the first aspect of this application; and using the multi-label classification model to perform a classification operation on the image to be classified, and outputting a classification result.

[0018] In a third aspect, this application provides an apparatus for multi-label classification, comprising: a processor for executing program instructions; and a memory storing the program instructions, which, when loaded and executed by the processor, cause the processor to perform a training method according to any of the first aspects of this application or to perform a method according to the second aspect of this application.

[0019] In a fourth aspect, this application provides a computer-readable storage medium, characterized in that it stores computer-readable instructions thereon, which, when executed by one or more processors, implement the training method as described in any one of the first aspects of this application or the method as described in the second aspect of this application.

[0020] By employing the training scheme for the multi-label classification model and the image-based multi-label classification scheme provided above, this application embodiment effectively captures and preserves the spatial information of features by setting multiple convolutional branches in the classification network of the multi-label classification model and performing weighted operations on the spatial feature maps output by each of the multiple convolutional branches. This results in a multi-label classification model with better classification performance after training. Furthermore, in some embodiments, by incorporating a self-attention mechanism into the classification network, the multi-label classification model can learn the interactions between features at different spatial locations. Consequently, the trained multi-label classification model will have a better ability to capture long-range features (i.e., features with relatively large spatial distances) and feature correlations. Attached Figure Description

[0021] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein:

[0022] Figure 1 An exemplary flowchart of a training method for a multi-label classification model according to an embodiment of this application is shown;

[0023] Figure 2 A schematic diagram showing the processing of training images via a backbone network according to an embodiment of this application is illustrated;

[0024] Figure 3 A schematic flowchart of a training method for a multi-label classification model according to another embodiment of this application is shown;

[0025] Figure 4 An exemplary flowchart of a training method utilizing a self-attention mechanism according to an embodiment of this application is shown;

[0026] Figure 5A schematic flowchart of a training method for a multi-label classification model including a self-attention mechanism according to another embodiment of this application is shown;

[0027] Figure 6 A flowchart illustrating a method for multi-label classification based on images according to an embodiment of this application is shown; and

[0028] Figure 7 This is a schematic block diagram illustrating a system for multi-label classification according to an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] It should be understood that the terms "comprising" and "including" used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0031] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0032] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0033] The inventors discovered that, to better adapt to multi-label classification tasks, researchers developed a simple yet effective multi-label image recognition module called Class-Specific Residual Attention (CSRA), which can effectively capture the different spatial regions occupied by objects of different categories. CSRA has achieved advanced results in multi-label recognition while being much simpler than other traditional methods. However, CSRA also has limitations. It only has one set of convolution operations, and by normalizing the result of this set of convolution operations and then multiplying it by the result itself, features with larger convolution operation results receive greater weight. While this can emphasize some clear local lesion features, it also leads to more false positives.

[0034] In other words, CSRA makes strong features (i.e., positive features) even stronger, while features with weak or negative contributions (or inverse features) become almost ineffective. Furthermore, overemphasizing local positive features is unsuitable for disease labels that require a focus on global features. In some applications, certain disease labels may be positively correlated with a certain lesion feature or negatively correlated with another lesion feature. For example, the presence of a certain lesion feature in a medical image suggests the patient may have that disease (such a lesion feature is called a positive feature); while the presence of another lesion feature in the medical image may suggest the patient does not have that disease (such a lesion feature is called an inverse feature). In this application scenario, CSRA will render inverse features ineffective, thus affecting the classification results of multi-label classification models.

[0035] Based on this, the inventors propose a novel solution that captures spatial features at different levels based on categories by setting multiple convolutional branches. This enables the trained multi-label classification model to not only identify spatial features at different spatial locations related to the category, but also avoids the limitations caused by overemphasizing positive features. Consequently, the trained multi-label classification model can be adapted to more application scenarios and output more accurate classification results. The specific implementation of this application is described in detail below with reference to the accompanying drawings.

[0036] Figure 1 An exemplary flowchart of a training method for a multi-label classification model according to an embodiment of this application is shown. The multi-label classification model according to an embodiment of this application may include a backbone network and a classification network, wherein the classification network may include multiple convolutional branches. Figure 1 As shown, training method 100 may include: in step 101, a training image may be input into a backbone network for feature extraction to output an initial feature map, wherein the training image has a class label.

[0037] In some embodiments, the backbone network may include multiple convolutional layers, non-linear activation layers, pooling layers, and other operations for feature extraction from the image to obtain an initial feature map. In other embodiments, the backbone network may employ a network structure such as EfficientNet-B4 or a CNN backbone network. In some embodiments, the training images may include medical images. In other embodiments, the training images may include fundus images.

[0038] Figure 2 A schematic diagram illustrating the processing of training images via a backbone network according to an embodiment of this application is shown. Figure 2 As shown, after the training image 201 is input into the backbone network 202, the feature extraction operation of the backbone network 202 can output an initial feature map. This initial feature map can be, for example, an M×P matrix, where M and P are both positive integers, where M represents the number of channels in the initial feature map and P represents the number of spatial locations in the initial feature map. Figure 2 The initial feature map shown includes k*k spatial locations. It can be understood that each element at each spatial location in the initial feature map corresponds to a region in the training image 201, thus preserving spatially specific information. It can also be understood that... Figure 2 The initial feature map shown is exemplary and not limiting. For example, the initial feature map may not be limited to the square feature map shown in the figure, but may also be a rectangular feature map as needed (e.g., P = k * h, where k and h are both positive integers and k and h are not equal).

[0039] Return below Figure 1 Continue the description. For example... Figure 1 As shown, in step 102, the initial feature map can be input into the first and second convolutional branches of the classification network for processing, respectively, to output a first spatial feature map and a second spatial feature map. The classification network may include at least one first convolutional branch and at least one second convolutional branch. The first and second convolutional branches can be two parallel convolutional branches, meaning they can have the same input data.

[0040] In some embodiments, the first convolutional branch can perform a 1×1 convolution operation on the initial feature map, and the second convolutional branch can perform a 1×1 convolution operation on the initial feature map. In other embodiments, each convolutional weight in the first and second convolutional branches can be class-related and used to perform feature-dimensional to class-dimensional operations on the initial feature map, i.e., the convolutional kernel dimension can be the same as the number of classes, thereby outputting a first spatial feature map and a second spatial feature map based on the classes, respectively. In some embodiments, the first convolutional branch can be used to output a first spatial feature map of multiple classes, and the second convolutional branch can be used to output a second spatial feature map of multiple classes.

[0041] For example, assuming the initial feature map contains 128 dimensions, and the multi-label classification task is a 10-class task (i.e., including 10 classes), the dimensionality of the initial classification map can be reduced through the operations of the first and second convolutional branches to obtain a first spatial feature map and a second spatial feature map with 10 class dimensions. As another example, assuming the initial feature map contains 128 dimensions, and the multi-label classification task is a 200-class classification task, the dimensionality of the initial classification map can be increased through the operations of the first and second convolutional branches to obtain a first spatial feature map and a second spatial feature map with 200 class dimensions. The "classes" mentioned in this paper can be understood as the label categories or classification categories of the multi-label classification task.

[0042] Next, in step 103, a weighted operation can be performed on the first spatial feature map based on the second spatial feature map to obtain a class-specific spatial feature map. In some embodiments, the weights of the weighting operation can be determined based on the second spatial feature map to perform the weighting operation on the first spatial feature map. By performing a weighted operation on the first spatial feature map of the corresponding category based on the second spatial feature map of each category, class-specific features can be generated for each category, thereby generating a class-specific spatial feature map based on the category.

[0043] Since the first spatial feature map and the second spatial feature map are obtained from two different convolutional branches, by training the parameters of the first and second convolutional branches, the first spatial feature map and the second spatial feature map can retain or capture features at different levels respectively (for example, the first spatial feature map can be used to retain all feature information, and the second spatial feature map can be used to capture local feature information). Therefore, when performing weighted operations between the two, local positive features will not be overemphasized while local negative features will be completely ignored. Thus, the obtained class-specific spatial feature map can take into account both local and global features, which will be better adapted to more classification scenarios and will also help improve the accuracy of the classification results of multi-label classification models.

[0044] The process can then proceed to step 104, where class scores for the training images can be output based on class-specific spatial feature maps. In some embodiments, class scores for each class can be output based on class-specific spatial feature maps for each class. In other embodiments, step 104 may include: performing global average pooling on the class-specific spatial feature maps; and normalizing the result of the global average pooling to obtain class scores. Global average pooling can be performed on the class-specific spatial feature maps for each class, and the result of the global average pooling for each class can be normalized to obtain class scores for each class in a multi-label classification task.

[0045] like Figure 1 As further illustrated, in step 105, the multi-label classification model can be trained based on class scores and class labels. In some embodiments, the class labels may be annotation information of disease categories associated with the training images. In other embodiments, the multi-label classification model can be trained based on a first loss of both class scores and class labels. In still other embodiments, the first loss can be calculated using, for example, a cross-entropy loss function. In some embodiments, the training images used for training may include multiple images, and the class labels of the multiple training images may cover the categories required for the classification task of the multi-label classification model.

[0046] The above combination Figure 1 The training method of the multi-label classification model according to the embodiments of this application has been described exemplarily. It is understood that, unlike traditional CNNs which directly perform global average pooling and normalization operations after obtaining the initial feature map to obtain class scores, and unlike CSRA which uses the same set of convolutional operations to enhance local positive features, the training method of the embodiments of this application processes the initial feature map by setting multiple convolutional branches. This not only captures class-specific spatial features but also avoids the problem of over-enhancing local positive features, resulting in a multi-label classification model with stronger multi-label classification capabilities after training. It is also understood that the above description is exemplary and not restrictive. The following will combine... Figure 3 A training method according to another embodiment of this application will be described.

[0047] Figure 3 A schematic flowchart of a training method for a multi-label classification model according to another embodiment of this application is shown. As will be seen from the following description, Figure 3 The training method 300 shown can be a combination of the methods described above. Figure 1 The training method 100 described is a specific manifestation of this, therefore, the preceding text combines... Figure 1 The description of training method 100 can also be applied to the following description of training method 300.

[0048] like Figure 3 As shown, in training method 300, the training image 201 can first be input into the backbone network 202 of the multi-label classification model for feature extraction to obtain an initial feature map X. Then, the initial feature map X can be input into a first convolutional branch for processing to obtain a first spatial feature map z1; and the initial feature map X can be input into a second convolutional branch for processing to obtain a second spatial feature map z2. In some embodiments, the first spatial feature map z1 and the second spatial feature map z2 can be calculated using the following formulas 1 and 2, respectively:

[0049] z1c =X T w 1c (Formula 1);

[0050] z 2c =X T w 2c (Formula 2);

[0051] Among them, z 1c The first spatial feature map representing category c, z 2c Let X represent the second spatial feature map of category c, and w represent the initial feature map. 1c w represents the first convolution weight for class c. 2c This represents the second convolution weight for category c. The first and second convolution weights can be determined through model training. Both the first and second convolution branches can perform 1×1 convolution operations. The dimension of the convolution kernel can be determined based on the number of categories N required for the classification task of the multi-label classification model. The value of c can be a positive integer from 1 to N.

[0052] Next, after obtaining the first spatial feature map z1 and the second spatial feature map z2, a weighting operation can be performed on the first spatial feature map z1 based on the second spatial feature map z2 to obtain a class-specific spatial feature map V. In some embodiments, the weighting operation may include: normalizing the second spatial feature map z2 for each class to obtain a spatial attention score for each class; and using the spatial attention score of each class as a weight to perform a weighting operation on the first spatial feature map z1 of the corresponding class to obtain a class-specific spatial feature map for each class. In other embodiments, the normalization operation may include Sigmoid function calculation. Here, the corresponding class refers to the class of the spatial attention score used as the weight (i.e., the class of the second spatial feature map) corresponding to the class of the first spatial feature map to which the weighting operation is performed.

[0053] For example, in one specific embodiment, a class-specific spatial feature map V can be obtained through the following weighting operation:

[0054] v c =z 1c *σ(z 2c ) (Formula 3);

[0055] Among them, v c The class-specific spatial feature map representing category c, z 1c The first spatial feature map representing category c, z 2cLet σ represent the second spatial feature map of category c, where σ represents the Sigmoid function, and * denotes element-wise multiplication of matrices. Compared to the Softmax function, normalization using the Sigmoid function ensures that features at different spatial locations in the feature map act independently rather than competitively. This allows for a more flexible spatial attention score, adapting to a wider range of situations.

[0056] The feature at each spatial location in a class-specific spatial feature map can be represented as:

[0057] v cj =z 1cj σ(z 2cj ) (Formula 4);

[0058] Where j represents the j-th spatial location in the feature map, and there are a total of P spatial locations, v cj z represents the feature at the j-th spatial location in the class-specific spatial feature map of category c. 1c z represents the feature at the j-th spatial location in the first spatial feature map of category c. 2c Let σ represent the feature at the j-th spatial location in the second spatial feature map of category c, where σ represents the Sigmoid function.

[0059] Furthermore, after obtaining the class-specific spatial feature map V, the class score S of the training image 201 can be output based on the class-specific spatial feature map V. In some embodiments, global average pooling can be performed on the class-specific spatial feature map V; and the result of global average pooling can be normalized to obtain the class score. For example, this can be achieved by the following formula:

[0060] s c =σ(GAP(v) c )) (Formula 5);

[0061] Where, s c This represents the category score for category c, σ represents the Sigmoid function, GAP represents global average pooling, and v c This represents the class-specific spatial feature map of category c.

[0062] Furthermore, a multi-label classification model can be trained based on the distance between the category score S and the category label of the training image 201.

[0063] like Figure 3As further illustrated, in some embodiments, the training method 300 may further include: acquiring lesion location information 301 related to the category label in the training image 201; and using the lesion location information 301 as a supervision signal for the second spatial feature map Z2 to train the multi-label classification model. In other embodiments, lesion detection and / or lesion segmentation operations may be performed on the training image 201 to obtain the lesion location information 301. Based on the correlation between lesions and diseases, the lesion location information 301 labeled in each training image 201 can be related to the disease category included in the category label of the current training image 201, thereby establishing the association between disease categories and lesion information.

[0064] Using the lesion location information 301 as a supervision signal for the second spatial feature map Z2 can also be understood as using the lesion location information 301 as a supervision signal for the spatial attention score. In one embodiment, the lesion location information 301 can be used as annotation information for training images to train a multi-label classification model. In another specific implementation, the lesion location information 301 can be converted into a heatmap and used as a supervision signal for the second spatial feature map. In still other embodiments, the multi-label classification model can be trained based on a second loss between the lesion location information and the second spatial feature map. In some embodiments, the second loss can be calculated using, for example, a cross-entropy loss function.

[0065] For example, in some applications, taking retinal images as an example, an important indicator for glaucoma diagnosis is the cup-to-disc ratio. Therefore, the position of the optic disc can be used as a supervisory signal for the second spatial feature map to guide the learning of spatial attention scores for glaucoma categories. In other applications, the heatmap of hemorrhage location information in fundus images can be mapped to the corresponding positions in the second spatial feature map to facilitate the learning of spatial attention scores for diabetic retinopathy categories based on the heatmap of fundus hemorrhage coverage.

[0066] The above combination Figure 3 The training method according to embodiments of this application has been described exemplarily. It is understood that the above description is exemplary and not restrictive. For example, the classification network of the multi-label classification model according to embodiments of this application may not be limited to only including the first convolutional branch and the second convolutional branch, and more convolutional branches may be set as needed. It is also understood that by introducing lesion location information as a supervision signal, it is beneficial to enhance the extraction capability of local lesion information in the second spatial feature map, so that the multi-label classification model can obtain a stronger ability to capture the association between category labels and local information.

[0067] Furthermore, the inventors have discovered that the relationship between some disease labels and lesions is not a one-to-one absolute one. For example, retinal hemorrhages not only indicate the possibility of diabetic retinopathy but may also indicate the possibility of hypertension. Therefore, the judgment of these disease labels often requires the combined effect of other evidence, rather than simply looking at a single lesion feature.

[0068] The inventors also noted that some methods have introduced attention mechanisms into the field of visual algorithms, such as SENET and CBAM. The introduction of Transformers into visual algorithms led to the development of ViT and Swin. Attention mechanisms can capture the interactions between features at different spatial locations. The aforementioned methods all incorporate various attention structures into the backbone network, improving the feature extraction capabilities of traditional CNN methods. However, this also results in a huge number of trainable parameters, posing a significant challenge to the richness of the training data. Furthermore, these methods are all class-independent, thus offering no particular advantage for multi-label classification tasks.

[0069] Based on the above findings, the inventors have further optimized the training method of the multi-label classification model provided in this application. The following will combine... Figure 4 and Figure 5 Provide a detailed description.

[0070] Figure 4 An exemplary flowchart of a training method utilizing a self-attention mechanism according to an embodiment of this application is shown. Figure 4 As shown, training method 400 may include: in step 401, the training image can be input into the backbone network of a multi-label classification model for feature extraction to output an initial feature map, wherein the training image has class labels. Next, in step 402, the initial feature map can be input into the first and second convolutional branches of the classification network of the multi-label classification model for processing to output a first spatial feature map and a second spatial feature map, respectively. Then, the process can proceed to step 403, where a weighted operation can be performed on the first spatial feature map based on the second spatial feature map to obtain a class-specific spatial feature map. Steps 401, 402, and 403 have been combined in the preceding text. Figures 1-3 Each embodiment has been described in detail and will not be repeated here.

[0071] like Figure 4As further illustrated, in step 404, a self-attention mechanism can be used to process the initial feature map to obtain a self-attention feature map. In some embodiments, the features at each spatial location in the initial feature map can be processed through a fully connected layer to obtain K (“Key”) vectors and Q (“Query”) vectors, respectively, and then self-attention operations can be performed on them to obtain the self-attention feature map. In classification networks, by training the category-based weight parameters in the fully connected layers, the self-attention feature map can be associated with each category in a multi-label classification task, and it has the characteristics of simple structure and easy training. Therefore, the application of this self-attention mechanism can be applied to multi-label classification tasks.

[0072] Further, in step 405, the class score of the training image can be output based on the class-specific spatial feature map and the self-attention feature map. In some embodiments, the class score of each class can be output based on the class-specific spatial feature map and the self-attention feature map of each class. In other embodiments, step 405 may include: performing global average pooling on the multiplication result of the class-specific spatial feature map and the self-attention feature map; and normalizing the result of the global average pooling to obtain the class score.

[0073] Then, in step 406, the multi-label classification model can be trained based on the class scores and class labels. Step 406 is combined with the previous steps. Figure 1 The steps described are the same as or similar to steps 105, and will not be repeated here.

[0074] The above combination Figure 4 A training method for a multi-label classification model according to another embodiment of this application has been exemplarily described. It is understood that, compared to adding an attention mechanism to the backbone network, adding a self-attention mechanism to the classification network (i.e., processing the output of the backbone network with self-attention) according to this embodiment of the application can train weight parameters related to each category of the multi-label task, thereby improving the multi-label classification model's ability to capture the correlation between local features based on each category. By combining class-specific spatial feature maps and self-attention feature maps, the multi-label classification model takes into account local features, long-range features, and the ability to capture feature correlations at different spatial locations, thus possessing stronger multi-label classification capabilities.

[0075] It is also understood that the above description is exemplary and not restrictive. For example, in some embodiments, in Figure 4The training method 400 shown may further include using lesion location information as a supervision signal for the second spatial feature map. With this setting, the relationship between local lesion features in the second feature map and the disease category label can be enhanced under the supervision of lesion location information, while eliminating the interference of lesion information on disease category judgment under non-absolute relationship. This makes the multi-label classification model not only have the ability to highlight local lesion features, but also the ability to associate the relationship between these features, which is conducive to further improving the classification effect of the multi-label classification model.

[0076] Figure 5 A schematic flowchart illustrating a training method for a multi-label classification model including a self-attention mechanism according to another embodiment of this application is shown. As will be seen from the following description, Figure 5 The training method 500 shown can be a combination of the methods mentioned above. Figure 4 The training method 400 described is a specific manifestation of this, therefore, the previous text combined with Figure 4 The description of training method 400 can also be applied to the description of training method 500 below.

[0077] like Figure 5 As shown, in training method 500, the training image 201 is first input into the backbone network 202 of the multi-label classification model for feature extraction to obtain an initial feature map X. Then, the initial feature map X is input into a first convolutional branch for processing to obtain a first spatial feature map z1; and the initial feature map X is input into a second convolutional branch for processing to obtain a second spatial feature map z2. Next, after obtaining the first spatial feature map z1 and the second spatial feature map z2, a weighted operation is performed on the first spatial feature map z1 based on the second spatial feature map z2 to obtain a class-specific spatial feature map V.

[0078] In some embodiments, the classification network may further include a third convolutional branch and a fourth convolutional branch for implementing a self-attention mechanism. The third and fourth convolutional branches may be multiple convolutional branches parallel to the first and second convolutional branches. Specifically, the third and fourth convolutional branches may be used to process the initial feature map X respectively to generate feature map K (or K-map) and feature map Q (or Q-map). Performing self-attention operations on feature map K and feature map Q yields a self-attention feature map z3. In some embodiments, both the third and fourth convolutional branches may perform 1×1 convolution operations, and the convolutional kernel dimension may be M×M.

[0079] In other embodiments, the self-attention feature map z3 is a class-specific self-attention feature map that is class-related and can be obtained by the following formula:

[0080] Kc =X T W Kc (Formula 6);

[0081] Q c =X T W Qc (Formula 7);

[0082]

[0083] Where Kc represents the K-map of class c, Qc represents the Q-map of class c, X represents the initial feature map, and W represents the initial feature map. Kc and W Qc All of these are weight parameters in the autonomous force mechanism, W Kc W represents the weight matrix used to calculate Kc. Qc z represents the weight matrix used to calculate Qc. 3c The self-attention feature map represents the category, and M represents the number of channels in the initial feature map. This represents the transpose of Kc. W Kc and W Qc It can be determined through training.

[0084] like Figure 5 As further illustrated, after obtaining the self-attention feature map z3 and the class-specific spatial feature map V, the feature map Z (or fused feature map) can be obtained by combining the two. In some embodiments, the feature map Z can be obtained by multiplying the self-attention feature map z3 and the class-specific spatial feature map V. Based on the feature map Z, the class score S can be output.

[0085] In one specific embodiment, the category score S for each category can be obtained by the following formula:

[0086] s c =σ(GAP(z) c ))=σ(GAP(z 1c *σ(z 2c )*z 3c )) (Formula 9);

[0087] Among them, s c z represents the category score for category c. 1c The first spatial feature map representing category c, z 2c The second spatial feature map representing category c, z 3c This represents the self-attention feature map for class c, where σ represents the Sigmoid function, * indicates element-wise matrix multiplication, GAP indicates global average pooling, and z c This represents the fusion feature of category c.

[0088] Based on this, assuming the initial feature map X is an M×P feature map, then the first spatial feature map z1 and the second spatial feature map z2 for each category obtained after the first and second convolutional branch operations can both be P×1 feature maps, and the resulting class-specific spatial feature map V for each category is also a P×1 feature map. The K-map and Q-map obtained after the initial feature map X undergoes the third and fourth convolutional branch operations can both be P×M feature maps, and the resulting self-attention feature map z3 can be a P×1 feature map. The feature map Z for each category obtained by multiplying the self-attention feature map z3 and the class-specific spatial feature map V is also a P×1 feature map. M represents the number of channels in the initial feature map X, and P represents the number of spatial positions in the initial feature map X.

[0089] The training method of the present application embodiment has been described above with reference to several accompanying drawings. In order to facilitate understanding of the technical effects of the training method of the present application embodiment, several experimental examples will be used for illustration below.

[0090] By constructing multi-label classification models with different structures, and then applying training sets containing multiple fundus images to different multi-label classification models for training in a fundus multi-disease classification task, this task includes 80 label categories. Most of these categories have an imbalanced positive-negative sample ratio with fewer positive samples. Therefore, in the experimental examples below, mAP (Mean Average Precision) is used as the evaluation metric. Furthermore, the backbone network used in the experimental examples can be EfficientNet-B4. The experimental results are shown in Table 1 below.

[0091] Table 1:

[0092]

[0093] In Table 1, "backbone network + GAP" refers to the traditional CNN structure. Class-specific spatial attention refers to the process of obtaining class-specific spatial feature maps after processing through the first and second convolutional branches. Class-specific self-attention refers to the use of features such as those described above in the classification network. Figure 4 or Figure 5 The process of obtaining a self-attention feature map using the self-attention mechanism described in the text refers to the process of using lesion location information to supervise the second spatial feature map.

[0094] As can be seen from the results in Table 1, compared to the traditional CNN structure used for multi-label classification tasks (i.e., Experiment 1), the multi-label classification models of Experiments 3 to 6 trained according to the training method of this application embodiment all have better classification performance. Compared to using the CSRA module for multi-label classification (i.e., Experiment 2), the multi-label classification models of Experiments 3, 5, and 6 trained according to the training method of this application embodiment all have better classification performance. Furthermore, by combining class-specific spatial attention with class-specific self-attention, the performance of the trained multi-label classification model can be significantly improved. On this basis, combining the lesion location information supervision scheme will further improve the model performance of the multi-label classification model.

[0095] It is understandable that after training, various parameters in the multi-label classification model can be determined, such as the weights of the first convolution, the weights of the second convolution, and the weight parameters in the autonomous mechanism. After training the multi-label classification model, it can be used for inference or prediction to achieve the desired function, such as image recognition or image classification. In the inference phase, the parameters determined during training can be used directly. Therefore, this application also provides, in another aspect, a method for multi-label classification based on images, i.e., an inference method or detection method for a multi-label classification model. The following will combine... Figure 6 Please provide an explanation.

[0096] Figure 6 A flowchart illustrating a method for multi-label classification based on images according to an embodiment of this application is shown. Figure 6 As shown, method 600 may include: in step 601, the image to be classified may be input into the above-mentioned combination Figures 1-5 In any of the embodiments described above, a multi-label classification model is trained using the training method described above. Next, in step 602, the multi-label classification model can be used to classify the image to be classified and output the classification result.

[0097] In some embodiments, the image to be classified may include medical images, etc. In other embodiments, the image to be classified may include fundus images, etc. The image to be classified is input into a multi-label classification model trained according to the training method of the embodiments of this application. The backbone network in the multi-label classification model can extract features of the image to be classified to generate an initial feature map. Based on this, the initial feature map can be classified using the parameters determined in the classification network of the trained multi-label classification model to output the category score (i.e., classification result) of each category in the multiple categories included in the multi-label classification task. The category score can be a probability value.

[0098] The above-described solutions in the embodiments of this application can be implemented using program instructions. Therefore, this application also provides a device for multi-label classification, comprising: a processor for executing program instructions; and a memory storing program instructions, which, when loaded and executed by the processor, cause the processor to execute the aforementioned combined... Figures 1-5 The training method described in any embodiment or in combination with the preceding text Figure 6 The method described is a multi-label classification method based on images.

[0099] Figure 7 This is a schematic block diagram illustrating a system for multi-label classification according to an embodiment of this application. The system 700 may include a device 701 according to an embodiment of this application, as well as its peripheral devices and an external network. The device 701 is used for training a multi-label classification model or for classifying images to be classified, etc., to achieve the aforementioned combination. Figures 1-6 The technical solutions of any of the embodiments described in this application.

[0100] like Figure 7 As shown, device 701 may include a CPU 7011, which may be a general-purpose CPU, a dedicated CPU, or other information processing and program execution unit. Furthermore, device 701 may also include a mass storage device 7012 and a read-only memory (ROM) 7013. The mass storage device 7012 can be configured to store various types of data, including training images, lesion location information, weight parameters, and classification results, as well as various programs required to run the neural network. The ROM 7013 can be configured to store data required for power-on self-test of device 701, initialization of various functional modules in the system, drivers for basic input / output of the system, and data required to boot the operating system.

[0101] Furthermore, device 700 also includes other hardware platforms or components, such as the TPU 7014, GPU 7015, FPGA 7016, and MLU 7017 shown. It is understood that although various hardware platforms or components are shown in device 700, they are merely exemplary and not limiting, and those skilled in the art can add or remove corresponding hardware as needed. For example, device 701 may include only a CPU as a known hardware platform and another hardware platform as the test hardware platform of this invention.

[0102] The device 701 of this application also includes a communication interface 7018, through which it can connect to a local area network / wireless local area network (LAN / WLAN) 705, and further connect to a local server 706 or the Internet 707 via the LAN / WLAN. Alternatively or additionally, the device 701 of this application can also directly connect to the Internet or cellular network via the communication interface 7018 based on wireless communication technology, such as third-generation ("3G"), fourth-generation ("4G"), or fifth-generation ("5G") wireless communication technology. In some application scenarios, the device 701 of this application can also access an external network server 708 and a possible database 709 as needed to obtain various known training data, and can remotely store various parameters or intermediate data.

[0103] Peripheral devices of device 701 may include a display device 702, an input device 703, and a data transmission interface 704. In one embodiment, the display device 702 may include, for example, one or more speakers and / or one or more visual displays, configured to provide voice prompts and / or display images and videos regarding the computation process or classification results of the device. The input device 703 may include, for example, a keyboard, mouse, microphone, gesture capture camera, or other input buttons or controls, configured to receive input of training data or user instructions. The data transmission interface 704 may include, for example, a serial interface, parallel interface, or Universal Serial Bus interface (“USB”), Small Computer System Interface (“SCSI”), Serial ATA, FireWire (“FireWire”), PCI Express, and High Definition Multimedia Interface (“HDMI”), configured for data transmission and interaction with other devices or systems. According to the present application, the data transmission interface 704 can receive training images or lesion location information, and transmit various types of data and results to device 701.

[0104] The CPU 7011, mass storage 7012, read-only memory ROM 7013, TPU 7014, GPU 7015, FPGA 7016, MLU 7017, and communication interface 7018 of the device 701 of this application can be interconnected via bus 7019, and can interact with peripheral devices through this bus. In one embodiment, the CPU 7011 can control other hardware components in the device 701 and its peripheral devices through this bus 7019.

[0105] In operation, the processor CPU 7011 of the device 701 of this application can receive training images or images to be classified through the input device 703 or the data transmission interface 704, and retrieve computer program instructions or code (e.g., code involving neural networks) stored in the memory 7012 to train the received training data or classify the images to be classified, so as to obtain the weight parameters or classification results of the trained multi-label classification model. After the CPU 7011 determines the classification result by executing the program instructions, it can display the result on the display device 702 or output the classification result through voice prompts. In addition, the device 701 can also upload the classification result to a network, such as a remote database 709, through the communication interface 7018.

[0106] It should also be understood that any module, unit, component, server, computer, terminal, or device of the executable instructions in this application may include or otherwise access computer-readable media, such as storage media, computer storage media, or data storage devices (removable) and / or non-removable) such as disks, optical discs, or magnetic tapes. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0107] Based on the foregoing, this application also provides a computer-readable storage medium storing computer-readable instructions thereon, which, when executed by one or more processors, implement the above-described combination of instructions. Figures 1-5 The training method described in any of the embodiments or in combination with the foregoing Figure 6 The method for multi-label classification based on images is described above.

[0108] Computer-readable storage media can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise retained by such a computer-readable medium.

[0109] Based on the above description of the training method and multiple embodiments of the multi-label classification model of this application, those skilled in the art will understand that the training method of this application sets multiple convolutional branches in the classification network of the multi-label classification model to process the initial feature map respectively, thereby avoiding the defect of overemphasizing local positive features in the weighting operation, so that the trained multi-label classification model has better classification performance.

[0110] Furthermore, in the training method of this embodiment, by introducing lesion location information as a supervision signal for the second spatial feature map, the second convolutional branch can learn local lesion features. Thus, the output second spatial feature map can highlight local features at different spatial locations related to the category label, strengthening the multi-label classification model's ability to capture the relationship between local features, lesion features, and disease categories. Even further, by incorporating a self-attention mechanism into the classification network, the multi-label model's ability to capture the correlation between features at different spatial locations can be significantly improved, thereby significantly enhancing the model performance of the multi-label classification model.

[0111] While numerous embodiments of this application have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will arise for those skilled in the art without departing from the spirit and intent of this application. It should be understood that various alternatives to the embodiments of this application described herein may be employed in the practice of this application. The appended claims are intended to define the scope of protection of this application and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A training method for a multi-label classification model, characterized in that, The multi-label classification model includes a backbone network and a classification network, wherein the classification network includes multiple convolutional branches, and the training method includes: The training images are input into the backbone network for feature extraction to output an initial feature map, wherein the training images have category labels and include fundus images; The initial feature map is input into the first convolutional branch and the second convolutional branch of the classification network for processing, so as to output the first spatial feature map and the second spatial feature map respectively, wherein the first convolutional branch and the second convolutional branch are two parallel convolutional branches; Based on the second spatial feature map, a weighted operation is performed on the first spatial feature map to obtain a class-specific spatial feature map; Based on the class-specific spatial feature map, output the class score of the training image; and The multi-label classification model is trained based on the category scores and category labels. This also includes: Obtain lesion location information in the training images that is related to the category label; and The lesion location information is used as a supervision signal for the second spatial feature map to train the multi-label classification model.

2. The training method according to claim 1, characterized in that, The first convolutional branch is used to output first spatial feature maps of multiple categories, and the second convolutional branch is used to output second spatial feature maps of multiple categories; The weighting operation includes: Normalize the second spatial feature map for each category to obtain the spatial attention score for each category; and Using the spatial attention score of each category as a weight, a weighted operation is performed on the first spatial feature map of the corresponding category to obtain the class-specific spatial feature map of each category.

3. The training method according to claim 1 or 2, characterized in that, The first spatial feature map and the second spatial feature map are obtained through the following processes: in, The first spatial feature map representing category c. Let X represent the second spatial feature map of category c, and let X represent the initial feature map. This represents the first convolution weight for class c. This represents the second convolution weight for class c.

4. The training method according to claim 3, characterized in that, The class-specific spatial feature map is obtained through the following weighting operation: in, Represents the class-specific spatial feature map of category c. The first spatial feature map representing category c. The second spatial feature map representing category c. Represents the Sigmoid function. This indicates element-wise multiplication of matrices.

5. The training method according to claim 1, characterized in that, Based on the class-specific spatial feature map, the class score of the training image is output, including: Global average pooling is performed on the class-specific spatial feature map; and The result of the global average pooling is normalized to obtain the category score.

6. The training method according to claim 1, characterized in that, Also includes: The initial feature map is processed using a self-attention mechanism to obtain a self-attention feature map; as well as Based on the class-specific spatial feature map and the self-attention feature map, the class score of the training image is output.

7. The training method according to claim 6, characterized in that, The self-attention feature map is obtained by the following formula: Where Kc represents the K-map of class c, Qc represents the Q-map of class c, and X represents the initial feature map. This represents the weight matrix used to calculate Kc. This represents the weight matrix used to calculate Qc. Let M represent the self-attention feature map, and M represent the number of channels in the initial feature map.

8. The training method according to claim 6, characterized in that, Based on the class-specific spatial feature map and the self-attention feature map, the class score of the training image is output, including: Global average pooling is performed on the product of the class-specific spatial feature map and the self-attention feature map; and The result of the global average pooling is normalized to obtain the category score.

9. The training method according to claim 8, characterized in that, The category score for each category is obtained using the following formula: in, This represents the category score for category c. The first spatial feature map representing category c. The second spatial feature map representing category c. Represents the self-attention feature map. Represents the Sigmoid function. This indicates element-wise multiplication of matrices. This indicates global average pooling.

10. A method for multi-label classification based on images, characterized in that, include: The image to be classified is input into the multi-label classification model trained by the training method according to any one of claims 1-9; as well as The multi-label classification model is used to classify the image to be classified, and the classification result is output.

11. A device for multi-label classification, characterized in that, include: A processor is used to execute program instructions; as well as A memory storing the program instructions, which, when loaded and executed by the processor, cause the processor to perform the training method according to any one of claims 1-9 or the method according to claim 10.

12. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by one or more processors, implement the training method as described in any one of claims 1-9 or the method as described in claim 10.

Images