Multi-label classification method and device for images

Abstract

The invention provides a multi-label classification method and device for an image. The method comprises the following steps: acquiring a to-be-processed image; extracting features of the to-be-processed image by using the trained first network to obtain first features; according to the first feature and a preset second feature, obtaining a prediction score of each label; and obtaining a classification label of the to-be-processed image according to the prediction score of each label and a preset score threshold. According to the method, features of an image are extracted through a trained first network, the features extracted by the first network are processed according to preset second features used for representing semantic features of all labels and correlation information of the semantic features, and therefore prediction scores of all the labels are obtained. The method not only considers the features of the image, but also considers the relevance between the labels, thereby achieving the deep fusion of the image features and the label topology, and enabling the multi-label classification result to be more accurate.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-label classification method and device for images. Background Art

[0002] With the breakthrough development of medical imaging technologies such as X-rays and computed tomography (CT), the application value of multimodal imaging data in the entire cycle of clinical diagnosis and treatment has become increasingly prominent. According to statistics, global medical imaging data is expanding at a compound annual growth rate of 30%, which has put forward an urgent need for intelligent analysis of massive imaging data. Especially in multi-label classification scenarios, a single image may contain multiple pathological features (such as coexisting pneumonia, lung nodules, and pneumothorax in chest X-rays), and it is urgent to establish an accurate multi-label joint recognition system to achieve comprehensive diagnosis. The development of efficient and reliable multi-label classification algorithms for medical images has become a core topic in the field of smart healthcare.

[0003] Traditional image analysis methods rely on the manual observation and experience of radiologists, which have two inherent defects: first, the efficiency of expert interpretation is limited by the manual processing speed (an average of 6 to 8 minutes per case), which makes it difficult to cope with the exponentially growing diagnostic needs; second, there are significant differences in the diagnostic consistency of physicians with different qualifications (studies have shown that the difference in lung nodule detection between junior and senior physicians is as high as 15%-20%), and the risk of misdiagnosis caused by subjective factors is prominent. Although the introduction of deep learning technology has partially solved the above problems, it still faces key technical bottlenecks in the field of multi-label classification: (1) Existing algorithms mostly adopt independent binary classification strategies, ignoring the strong correlation between pathological features in clinical practice, such as the correlation between pulmonary edema and abnormal cardiac function; (2) The long-tail distribution of labels leads to a decrease in the recognition accuracy of the model for low-frequency diseases, such as the recognition rate of rare diseases is generally less than 65%; (3) There is significant heterogeneity in cross-center data, and the difference in Hounsfield Unit (HU) between different CT devices can reach 10%-15%, and the model generalization ability is insufficient.

[0004] The current mainstream solution is to alleviate the category imbalance problem by improving the loss function (such as weighted cross entropy loss, focal loss, etc.). A typical example is the Heuristic Stacking algorithm, which improves the F1-score of rare diseases by nearly 10 percentage points through a dynamic sample weighting strategy; but this cannot solve the problem of multi-classification. Summary of the invention

[0005] In view of the above defects in the prior art, the present invention provides a multi-label classification method and device for images to solve the technical problem that the correlation between labels is ignored in the independent binary classification strategy.

[0006] To achieve the above-mentioned purpose and other related purposes, the present invention provides a multi-label classification method for images, comprising: obtaining an image to be processed; using a trained first network to extract features of the image to be processed to obtain a first feature, wherein the first network is a convolutional neural network, and the first feature is used to characterize the visual features of the image to be processed; according to the first feature and a preset second feature, a prediction score of each label is obtained, wherein the second feature is used to characterize the semantic features of each label and its correlation information; according to the prediction score of each label and a preset score threshold, a classification label of the image to be processed is obtained.

[0007] In one embodiment of the present invention, the first network includes a densely connected convolutional network integrated with wavelet transform enhanced convolution.

[0008] In one embodiment of the present invention, the first network includes an initial visual layer, a wavelet transform dense block, a transition layer and a global average pooling layer; the features of the image to be processed are extracted using the trained first network to obtain the first feature, including: using the initial visual layer to extract the low-level features of the image to be processed; using a number of the wavelet transform dense blocks to extract the multi-level features of the low-level features; using the transition layer set between two of the wavelet transform dense blocks to compress and reduce the dimension of the features output by the previous wavelet transform dense block; using the global average pooling layer to compress the spatial dimension of the features output by the last wavelet transform dense block to one, to obtain the first feature.

[0009] In one embodiment of the present invention, there are four wavelet transform dense blocks, and the four wavelet transform dense blocks include 6, 12, 24, and 16 nonlinear combination layers respectively, and each of the nonlinear combination layers includes batch normalization, an activation function, and a wavelet transform enhanced convolution.

[0010] In one embodiment of the present invention, the second feature is obtained by the following steps: based on the training set used to train the first network, a training set label file and a label word vocabulary are obtained; based on the training set label file, a correlation matrix is ​​obtained, and the correlation matrix is ​​used to characterize the correlation between labels; based on the label word vocabulary, word vectors of all labels are obtained; based on the correlation matrix and the word vectors of all labels, the second feature is obtained.

[0011] In one embodiment of the present invention, a correlation matrix is ​​obtained according to the training set label file, and the correlation matrix is ​​used to characterize the correlation between labels, including: according to the training set label file, counting the number of times each pair of labels appear at the same time to obtain a first matrix; preprocessing the first matrix to obtain a second matrix to reduce noise and redundant information; using a trained second network to process the second matrix to obtain a third matrix, and the second network dynamically models the relationship between label nodes through a self-attention mechanism; optimizing the third matrix into a matrix suitable for use in a graph convolutional network to obtain the correlation matrix.

[0012] In one embodiment of the present invention, the first matrix is ​​preprocessed to obtain a second matrix, including: normalizing the first matrix to obtain a conditional probability matrix; filtering the conditional probability matrix according to a preset noise threshold to obtain a binary matrix; and reweighting the binary matrix according to a preset hyperparameter to obtain the second matrix.

[0013] In one embodiment of the present invention, the third matrix is ​​optimized into a matrix suitable for use in a graph convolutional network to obtain the correlation matrix, including: removing the first dimension of the third matrix to obtain a fourth matrix; adding a unit matrix to the fourth matrix to obtain a fifth matrix; calculating the degree matrix of the fifth matrix, taking the inverse matrix of the square root of the degree matrix, and then using the inverse matrix to normalize the fifth matrix to obtain the correlation matrix.

[0014] In one embodiment of the present invention, the second feature is obtained based on the correlation matrix and the word vectors of all labels, including: using a trained third network to process the correlation matrix and the word vectors of all labels to obtain the second feature, wherein the third network is composed of two layers of graph convolutional networks stacked together.

[0015] To achieve the above-mentioned purpose and other related purposes, the present invention also provides a multi-label classification device for images, comprising: a data acquisition unit, used to acquire an image to be processed; a feature extraction unit, used to extract features of the image to be processed using a trained first network to obtain a first feature, wherein the first network is a convolutional neural network, and the first feature is used to characterize the visual features of the image to be processed; a score calculation unit, used to obtain a predicted score for each label based on the first feature and a preset second feature, wherein the second feature is used to characterize the semantic features of each label and its correlation information; and a label acquisition unit, used to obtain a classification label for the image to be processed based on the predicted score of each label and a preset score threshold.

[0016] Beneficial effects of the present invention: The present invention proposes a multi-label classification method and device for images. The method extracts features of the image through a trained first network, and processes the features extracted by the first network according to a preset second feature for characterizing the semantic features of each label and its correlation information, thereby obtaining prediction scores for all labels. When predicting, the prediction score not only considers the features of the image itself, but also considers the correlation between the labels, thereby achieving a deep fusion of image features and label topology, making the multi-label classification results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of a multi-label classification method provided by an embodiment of the present invention; Figure 2 A structural diagram of a first network provided in one embodiment of the present invention; Figure 3 A detailed flow chart of step S200 provided in one embodiment of the present invention; Figure 4 A flowchart of a second feature acquisition step provided in an embodiment of the present invention; Figure 5 A detailed flow chart of step S320 provided for one embodiment of the present invention; Figure 6 An overall network structure diagram provided for an embodiment of the present invention; Figure 7 A schematic diagram of a multi-label classification device provided by an embodiment of the present invention.

[0019] Explanation of the reference numerals: 101, data acquisition unit; 102, feature extraction unit; 103, score calculation unit; 104, label acquisition unit. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention through specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. It should be noted that the following embodiments and the features in the embodiments can be combined with each other without conflict. In addition to the specific methods, equipment, and materials used in the embodiments, according to the technical personnel in this technical field's grasp of the prior art and the records of the present invention, any methods, equipment, and materials of the prior art that are similar or equivalent to the methods, equipment, and materials described in the embodiments of the present invention can also be used to implement the present invention.

[0021] It should be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those generally understood by those skilled in the art.

[0022] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In some of the embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0023] The flowchart and block diagram in the accompanying drawings illustrate the system architecture, functions and operations that may be implemented according to the methods and computer program products of various embodiments disclosed in the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0024] See also Figure 1 , Figure 1 A multi-label classification method for an image provided by an embodiment of the present invention includes steps S100 to S400.

[0025] Step S100: Acquire an image to be processed. The image to be processed may be, for example, a medical image taken by an X-ray or computed tomography (CT) device.

[0026] Step S200: extract features of the image to be processed using the trained first network to obtain a first feature, wherein the first network is a convolutional neural network, and the first feature is used to characterize visual features of the image to be processed.

[0027] In this step, it is understandable that the image to be processed needs to be preprocessed to adapt to the input size of the first network. Convolutional Neural Network (CNN) is a deep learning model specifically used to process grid-structured data (such as images, videos, audio, etc.). Its core idea is to extract local features of data through convolution operations, reduce feature dimensions through pooling operations, and finally perform classification or regression through a fully connected layer. In the present invention, the features of the image to be processed are extracted by a convolutional neural network, and the convolutional neural network can be, for example, a network structure such as DenseNet, VGG, or ResNet.

[0028] In a specific embodiment of the present invention, the first network includes a densely connected convolutional network that incorporates wavelet transform enhanced convolution. Densely connected convolutional network (DenseNet) is an efficient convolutional neural network structure. Its core idea is to directly connect the output of each layer with the input of all subsequent layers through dense connections, thereby enhancing feature reuse and gradient flow. This design reduces the number of parameters, alleviates the gradient vanishing problem, and improves model performance. Wavelet transform is a multi-scale signal analysis tool that can simultaneously capture the time domain and frequency domain information of the signal. Wavelet transform is introduced into convolutional neural networks to replace or supplement traditional convolution operations to extract richer multi-scale features.

[0029] This embodiment incorporates wavelet transform technology. Through wavelet decomposition, the first network can emphasize low-frequency components in medical images while operating on multiple frequencies using compact convolution kernels. This design enables the first network to better adapt to noise and inhomogeneity in medical images, thereby significantly improving the robustness of classification. The introduction of wavelet transform technology not only enhances the accuracy of feature extraction, but also provides a more efficient and reliable solution for medical image analysis.

[0030] See also Figure 2 In a specific embodiment of the present invention, the first network includes an initial visual layer, a wavelet transform dense block, a transition layer, and a global average pooling layer. Step S200 includes steps S210 to S240.

[0031] Step S210: extract low-level features of the image to be processed using the initial visual layer. The initial visual layer is mainly used to extract low-level features, and includes, for example, a convolutional layer and a maximum pooling layer.

[0032] Step S220, using a plurality of wavelet transform dense blocks to extract multi-level features of low-level features. Wavelet transform dense block is a core module designed in the present invention, mainly used for feature extraction of medical images, which extracts features through multiple nonlinear combination layers (including batch normalization, ReLU activation function and wavelet transform enhanced convolution), and splices the output of each layer with the input feature map in the channel dimension. This design can retain multi-level feature information and enhance the expression ability of the model.

[0033] Step S230, using the transition layer set between two wavelet transform dense blocks, compress and reduce the features output by the previous wavelet transform dense block. The transition layer is a transition module between wavelet transform dense blocks, which is mainly used for compression and dimensionality reduction of feature maps. By setting the transition layer between wavelet transform dense blocks, this alternating structure enables the first network to gradually extract more abstract and advanced features while maintaining efficient calculation, and the first network can better adapt to the complexity and diversity of medical imaging data, thereby improving the accuracy and generalization ability of classification.

[0034] Step S240: Use a global average pooling layer to compress the spatial dimension of the feature output by the last wavelet transform dense block to one, and obtain a first feature. The global average pooling layer is mainly used to adjust the output feature size of the first network.

[0035] See also Figure 3 In a specific embodiment of the present invention, four wavelet transform dense blocks are provided, and the four wavelet transform dense blocks include 6, 12, 24, and 16 nonlinear combination layers respectively, and each nonlinear combination layer includes batch normalization, activation function, and wavelet transform enhanced convolution. The first network is described in detail below in conjunction with this structure.

[0036] (1) As mentioned above, the image to be processed needs to be preprocessed before being input into the first network. In this embodiment, for example, the image to be processed is resized to 448×448 and then enters the initial visual layer.

[0037] (2) The initial visual layer first processes the input image through a normal convolution layer (convolution kernel size is 7×7, stride is 2) to obtain a 224×224×96 feature map, and then passes through batch normalization, ReLU activation function, and maximum pooling layer (convolution kernel size is 3×3, stride is 2) to obtain low-level features of size 112×112×96.

[0038] (3) The low-level features enter the first wavelet transform dense block, which processes the low-level features through 6 nonlinear combination layers in turn, and connects the output of each combination layer in a cross-channel form. The output and input features of the wavelet transform enhanced convolution are concatenated in the channel dimension, and finally a feature map of size 112×112×(96+6×48) is output. The wavelet transform dense blocks in the subsequent steps are repetitions of this structure.

[0039] (4) The transition layer includes a convolution layer (the convolution kernel size is 1×1) and an average pooling layer (the convolution kernel size is 2×2). After the features in the previous step pass through the first transition layer, a feature map of size 56×56×(96+3×48) is output. All transition layers in the first network sample the same structure.

[0040] (5) After the features in the previous step pass through the 12 nonlinear combination layers in the second wavelet transform dense block, the output feature size is 56×56×(96+3×48+12×48).

[0041] (6) After the features in the previous step pass through the second transition layer, a feature map of size 28×28×(96+3×48+6×48) is output.

[0042] (7) After the features in the previous step pass through the 24 nonlinear combination layers in the third wavelet transform dense block, the output feature size is 28×28×(96+3×48+6×48+24×48).

[0043] (8) After the features in the previous step pass through the third transition layer, a feature map of size 14×14×(96+3×48+6×48+12×48) is output.

[0044] (9) After the features in the previous step pass through the 16 nonlinear combination layers in the fourth wavelet transform dense block, the output feature size is 14×14×(96+3×48+6×48+12×48+16×48).

[0045] (10) The feature map in the previous step finally passes through the global average pooling layer to compress the spatial dimension of the feature map to 1, and obtain the one-dimensional first feature , where D represents the size of the first feature.

[0046] Step S300: According to the first feature and the preset second feature, the prediction score of each label is obtained, wherein the second feature is used to characterize the semantic features of each label and its correlation information. The second feature is mainly used to introduce the semantic features of each label and its correlation information. It is the core output of the dynamic graph fusion mechanism. The second feature not only captures the dynamic correlation between labels, but also realizes the joint optimization of label semantics and image visual features through fusion with the image feature matrix, thereby significantly improving the accuracy and robustness of multi-label classification.

[0047] For a certain usage scenario, such as multi-label classification of chest medical images, the labels used have been determined and the first network has been trained. At this time, the semantic features and correlation information between the labels have been determined. Therefore, the second feature can be saved as a preset value. When predicting the image to be processed, calculations can be performed directly based on the preset second feature.

[0048] See also Figure 4 In a specific embodiment of the present invention, the second feature is obtained through steps S310~S340.

[0049] Step S310: According to the training set used to train the first network, a training set label file and a label word vocabulary are obtained. Since the training set is required when training the first network, in this step, in order to ensure the dynamic correlation between the captured labels and the correlation between the image visual features, the second feature is obtained step by step from the training set.

[0050] The present invention can use some existing data sets, the samples in the data sets are composed of images and their corresponding labels, and the data sets are divided into training sets and test sets to complete the training and testing of the first network. In step S310, only the training set is used, and the training set label file mentioned above stores the mapping relationship between the encoding of each image and its label, and the label word vocabulary is a vocabulary containing all label words in the training set.

[0051] Step S320: Obtain a correlation matrix based on the training set label file, where the correlation matrix is ​​used to characterize the correlation between labels.

[0052] See also Figure 5 In a specific embodiment of the present invention, step S320 includes steps S321 to S324.

[0053] Step S321: According to the training set label file, the number of times each pair of labels appear at the same time is counted to obtain a first matrix. In specific implementation, for example, each row of data in the training set label file can be traversed through a multi-layer nested loop statement, and the first matrix finally obtained can be recorded as M. , C represents the total number of labels. In this matrix, M ij It represents the number of times the ith label and the jth label appear simultaneously in multiple labels corresponding to the same image. Since each label does not count itself, the diagonal element values ​​of the first matrix M are all 0.

[0054] Step S322: preprocess the first matrix to obtain a second matrix to reduce noise and redundant information. The preprocessing may include normalization, denoising, weighting, etc.

[0055] In a specific implementation of the present invention, step S322 specifically includes the following operations.

[0056] (1) Normalize the first matrix to obtain the conditional probability matrix. In this step, the label correlation dependency is modeled in the form of conditional probability to obtain the conditional probability matrix M'. The conditional probability matrix M' is obtained by normalizing the first matrix M, and its elements M ij ' represents the conditional probability of label j appearing when label i appears.

[0057] (2) Filter the conditional probability matrix according to the preset noise threshold to obtain a binary matrix. This step is to avoid the noise and other problems existing in the conditional probability matrix M'. Therefore, the threshold τ can be used to filter the noisy edges to obtain the binary matrix A (A ij is the element value of the i-th row and j-th column of matrix A): .

[0058] (3) Reweight the binary matrix according to the preset hyperparameters to obtain the second matrix. This step is mainly to solve the over-smoothing problem caused by the binary correlation matrix, so the reweighting strategy can be used to obtain the second matrix N (N ij is the element value of the i-th row and j-th column of matrix N): , Among them, p is a hyperparameter.

[0059] Step S323: Process the second matrix using the trained second network to obtain a third matrix. The second network dynamically models the relationship between label nodes through a self-attention mechanism. The second network can be, for example, a GraphTransformer or GraphSAGE network. The following takes the Graph Transformer network as an example to explain step S323 in detail.

[0060] In order to obtain more abundant related structural information between label nodes, the Graph Transformer network is used to transform the obtained second matrix N into a new graph structure. The second matrix N obtained in step S322 is used as the input of the second network, and different Q i , K i 、V i : , , And through Q i , K i 、V i Calculate the attention matrix: , For the h-head attention layer, we can get a subgraph G containing label node information: , For all sub-graphs G obtained from different attention layers, the third matrix P can be obtained by matrix multiplication: , In the above formulas, are all learnable parameters. Dh represents the dimension of each attention, which is a hyperparameter. It's K i The shape of the third matrix P can be recorded as (1, num_classes, num_classes).

[0061] Step S324, optimizing the third matrix to a matrix suitable for use in a graph convolutional network to obtain a correlation matrix. In a specific embodiment of the present invention, step S324 specifically includes the following steps.

[0062] (1) Remove the first dimension of the third matrix to obtain the fourth matrix. The shape of the fourth matrix is ​​(num_classes, num_classes). This step is to remove the redundant batch dimensions and simplify the matrix structure for subsequent processing.

[0063] (2) Add the unit matrix to the fourth matrix to obtain the fifth matrix. The unit matrix is ​​a matrix with diagonal elements of 1. The purpose of this step is to ensure that each node is connected to itself. In a graph convolutional network, the characteristics of a node depend not only on its neighboring nodes, but also on its own characteristics. Adding the unit matrix can ensure that the node retains its own information during the information propagation process. If there is no self-loop, the node may lose its original characteristics after multiple graph convolution operations.

[0064] (3) Calculate the degree matrix of the fifth matrix, take the inverse matrix of the square root of the degree matrix, and then use the inverse matrix to normalize the fifth matrix to obtain the correlation matrix. In this step, first calculate the degree matrix of the fifth matrix, that is, for each node, calculate its degree (that is, the sum of each row of the adjacency matrix), and then take the inverse of its square root to obtain the degree matrix D; then take the inverse matrix of the square root of the degree matrix, which can be recorded as D -1 / 2 ; Finally, the fifth matrix is ​​normalized using the inverse matrix to obtain the final correlation matrix P'.

[0065] Step S330: Obtain word vectors for all tags according to the tag word vocabulary. Tags are generally composed of multiple words or characters, and the word vector of each tag can be directly composed of the sum of the word vectors of the words or characters contained in the tag. For example, the word vector of the "lungopacity" category is composed of the sum of the word vector of "lung" and the word vector of "opacity". The word vector can be trained, for example, by Glove, or FastText, or GoogleNews method.

[0066] Step S340: Obtain the second feature according to the correlation matrix and the word vectors of all labels.

[0067] In a specific embodiment of the present invention, the correlation matrix and the word vectors of all labels are processed using a trained third network to obtain the second feature, wherein the third network is composed of two layers of graph convolutional networks stacked together. A graph convolutional network (GCN) is a neural network specifically used to process graph structured data, which updates node features by aggregating information of neighboring nodes. In this embodiment, the main purpose of using a two-layer graph convolutional network is to gradually propagate and aggregate label correlation information through multi-layer graph convolution operations, thereby more effectively modeling the relationship between label nodes.

[0068] The operation of graph convolutional network can be expressed as: ; in, , which is the correlation matrix P' (obtained in step S320) with self-loops added, yes The degree matrix of It is l The node characteristics of the layer, It is l The learnable weight matrix of the layer, σ is the activation function (such as ReLU).

[0069] For the first graph convolutional network, l=0, and the formula This corresponds to the label word vector obtained in step S330. The output of the first graph convolutional network is ; For the second graph convolutional network, l=1, and its final output is , that is, the second feature W, the second feature , where n is the number of classification labels.

[0070] According to the first feature ( ), and the preset second feature ( ), in the step of obtaining the prediction score of each label, the prediction score can be directly According to the size of the first feature and the second feature, we can know that the multiplication results in , where each element value corresponds to the prediction score of a label.

[0071] Step S400: Obtain the classification label of the image to be processed according to the prediction score of each label and the preset score threshold. After the prediction score is obtained, it can be compared with the preset score threshold. For example, labels with scores exceeding 0.8 are labels of the image to be processed.

[0072] In the above embodiment, the second feature is used as a preset value because after the training of each network model is completed, the second feature is fixed and does not need to be recalculated. When the usage scenario changes or the label changes, the second feature needs to be recalculated.

[0073] Although the first network, the second network, and the third network are described separately in the above steps, in actual applications, the above three models together constitute an end-to-end model framework, whose structure is as follows: Figure 6 As shown (due to image size, some network structures are simply schematic and do not represent the actual number, for example Figure 6 In the first network, only two wavelet transform dense blocks are shown), which are used for multi-label classification tasks of images. They need to be trained together. The following briefly introduces their training process. (1) Forward propagation: The first network is used to extract the features of the image and obtain the first feature x; the second network is used to generate the correlation matrix P'; the third network uses the correlation matrix P' and the label word vector to propagate the label correlation information and obtain the second feature W. Finally, the first feature x and the second feature W are fused to obtain the prediction score of multi-label classification. (2) Calculate the loss: Calculate the loss function based on the true label and prediction result of the multi-label classification task. (3) Back propagation: Through the back propagation algorithm, calculate the gradient of the loss function to the learnable parameters of the first network, the second network and the third network. (4) Parameter update: Use the optimization algorithm to update the learnable parameters of all networks to minimize the loss function.

[0074] It should be noted that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the protection scope of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this patent.

[0075] See also Figure 7 , Figure 7 A multi-label classification device for images provided in one embodiment of the present invention includes a data acquisition unit 101, a feature extraction unit 102, a score calculation unit 103 and a label acquisition unit 104. The data acquisition unit 101 is used to acquire an image to be processed; the feature extraction unit 102 is used to extract features of the image to be processed using a trained first network to obtain a first feature, wherein the first network is a convolutional neural network, and the first feature is used to characterize the visual features of the image to be processed; the score calculation unit 103 is used to obtain a predicted score of each label according to the first feature and a preset second feature, wherein the second feature is used to characterize the semantic features of each label and its correlation information; the label acquisition unit 104 is used to obtain a classification label of the image to be processed according to the predicted score of each label and a preset score threshold.

[0076] It should be noted that the multi-label classification device of this embodiment is a device corresponding to the multi-label classification method described above, and the functional modules in the multi-label classification device may correspond to the corresponding steps in the multi-label classification method. The multi-label classification device of this embodiment can be implemented in conjunction with the multi-label classification method, that is, in the absence of conflict, the relevant technical details mentioned in the multi-label classification method of the above embodiment can also be applied to the multi-label classification device of this embodiment.

[0077] In general, the present invention creatively proposes: (1) constructing a dynamic label correlation matrix and establishing a dynamic representation of disease association through a graph attention mechanism; (2) designing a wavelet transform dense block fused with a wavelet transform enhanced convolution kernel, whose multi-scale feature fusion capability significantly improves the accuracy of fine-grained feature extraction; (3) constructing a dual-branch heterogeneous network architecture, in which the first network (first branch) realizes image feature extraction, and the third network (second branch) initializes the label semantic space through the Glove word vector, and generates a learnable adjacency matrix through the second network, and finally realizes the deep fusion of image features and label topology through the feature space alignment strategy, thereby further optimizing the model performance.

[0078] The present invention has a wide range of applications, mainly targeting multi-label classification tasks in the field of medical imaging, and is suitable for the analysis and diagnosis of various medical imaging data such as chest X-ray images and CT images. In addition, it can also be used for other multi-label classification scenarios. Through the technology of the present invention, more accurate multi-label classification results can be provided, which is of great significance for medical image analysis. In actual medical images, an image may often have multiple disease manifestations at the same time, and the present invention can more accurately capture the correlation between different disease labels through a method based on a correlation matrix, thereby significantly improving the accuracy of classification. In addition, the present invention integrates a new convolutional neural network structure, enhances the model's ability in feature extraction, helps to extract richer and more abstract image features, and further improves classification performance. Therefore, by integrating the label correlation matrix, the present invention can effectively improve the accuracy of multi-label image classification and provide strong support for the accurate diagnosis of medical images.

[0079] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A multi-label classification method for images, characterized in that: include: Get the image to be processed; Extracting features of the image to be processed using a trained first network to obtain a first feature, wherein the first network is a convolutional neural network, and the first feature is used to characterize visual features of the image to be processed; Obtaining a prediction score for each tag according to the first feature and a preset second feature, wherein the second feature is used to characterize the semantic features of each tag and its correlation information; According to the predicted score of each label and a preset score threshold, the classification label of the image to be processed is obtained.

2. The multi-label classification method for images according to claim 1, characterized in that: The first network includes a densely connected convolutional network fused with wavelet transform enhanced convolution.

3. The multi-label classification method for images according to claim 2, characterized in that: The first network includes an initial visual layer, a wavelet transform dense block, a transition layer, and a global average pooling layer; The features of the image to be processed are extracted using the trained first network to obtain the first features, including: Extracting low-level features of the image to be processed using the initial visual layer; Extracting multi-level features of the low-level features using a plurality of the wavelet transform dense blocks; Using a transition layer disposed between two wavelet transform dense blocks, the features output by the previous wavelet transform dense block are compressed and dimensionally reduced; The global average pooling layer is used to compress the spatial dimension of the feature output by the last wavelet transform dense block to one, so as to obtain the first feature.

4. The multi-label classification method for images according to claim 3, characterized in that: There are four wavelet transform dense blocks, and the four wavelet transform dense blocks include 6, 12, 24, and 16 nonlinear combination layers respectively, and each of the nonlinear combination layers includes batch normalization, an activation function, and a wavelet transform enhanced convolution.

5. The multi-label classification method for images according to claim 1, characterized in that: The second feature is obtained by the following steps: According to the training set used to train the first network, obtain a training set label file and a label word vocabulary; According to the training set label file, a correlation matrix is ​​obtained, wherein the correlation matrix is ​​used to characterize the correlation between labels; According to the label word vocabulary, word vectors of all labels are obtained; The second feature is obtained according to the correlation matrix and the word vectors of all labels.

6. The multi-label classification method for images according to claim 5, characterized in that: According to the training set label file, a correlation matrix is ​​obtained, and the correlation matrix is ​​used to characterize the correlation between labels, including: According to the training set label file, the number of times each pair of labels appear at the same time is counted to obtain a first matrix; Preprocessing the first matrix to obtain a second matrix to reduce noise and redundant information; Processing the second matrix using the trained second network to obtain a third matrix, wherein the second network dynamically models the relationship between label nodes through a self-attention mechanism; The third matrix is ​​optimized to be a matrix suitable for use in a graph convolutional network to obtain the correlation matrix.

7. The multi-label classification method for images according to claim 6, characterized in that: Preprocessing the first matrix to obtain a second matrix includes: Normalizing the first matrix to obtain a conditional probability matrix; Filtering the conditional probability matrix according to a preset noise threshold to obtain a binary matrix; The binary matrix is ​​reweighted according to a preset hyperparameter to obtain the second matrix.

8. The multi-label classification method for images according to claim 6, characterized in that: Optimizing the third matrix to a matrix suitable for use in a graph convolutional network to obtain the correlation matrix includes: Removing the first dimension of the third matrix to obtain a fourth matrix; Adding a unit matrix to the fourth matrix to obtain a fifth matrix; The degree matrix of the fifth matrix is ​​calculated, and the inverse matrix of the square root of the degree matrix is ​​taken, and then the fifth matrix is ​​normalized by using the inverse matrix to obtain the correlation matrix.

9. The multi-label classification method for images according to claim 5, characterized in that: The second feature is obtained according to the correlation matrix and the word vectors of all labels, including: The correlation matrix and word vectors of all labels are processed using a trained third network to obtain the second feature, wherein the third network is formed by stacking two layers of graph convolutional networks.

10. A multi-label classification device for images, characterized in that: include: A data acquisition unit, used for acquiring an image to be processed; A feature extraction unit, configured to extract features of the image to be processed using a trained first network to obtain a first feature, wherein the first network is a convolutional neural network, and the first feature is used to characterize visual features of the image to be processed; a score calculation unit, configured to obtain a prediction score for each tag according to the first feature and a preset second feature, wherein the second feature is used to characterize a semantic feature of each tag and its correlation information; and The label acquisition unit is used to obtain the classification label of the image to be processed according to the predicted score of each label and a preset score threshold.

Images