Application of unclassified feature memory module on classification based on information similarity

Abstract

The invention provides an unclassified feature memory module based on information similarity. According to the module, feature representation is enhanced by utilizing non-category memory and a non-category memory attention enhancement mechanism, the module does not depend on specific category information of a data set, and the model can still keep good performance when facing new data. A network integrating the module can be used for various downstream tasks, such as classification, target detection, segmentation and the like. A comprehensive experiment shows that when the module is added into the most advanced neural network model, indexes such as classification accuracy and F1-scores on a medical data set and other general data sets are greatly improved.

Description

Technical Field

[0001] The present invention relates to the application of artificial intelligence (AI), machine learning (ML), deep learning (DL), convolutional neural networks (CNN), information similarity (IBS) statistical analysis theory, attention mechanism and digital image processing (Digital Image Process) in classification. Background Art

[0002] In deep learning, feature extraction is a crucial step because it allows the model to learn useful information from the input data and use it for subsequent tasks such as classification, detection or segmentation. With the development of deep learning technology, researchers have been exploring how to improve the model's feature extraction capabilities. Among them, the attention mechanism has become an important research direction because it allows the model to focus on the most important part of the input data. This mechanism has achieved great success in natural language processing (NLP) and has gradually been introduced into the field of computer vision.

[0003] In the field of deep learning, using memory banks can help models store useful feature representations, which can be used to enhance the model's understanding and processing capabilities of new data. Categorical memory usually refers to memory under a specific category or label. Its main difference from non-category memory is that it relies on category information. In neural networks, categorical memory can provide models with prior knowledge of specific categories, which helps the model's performance when processing specific tasks. However, categorical memory also has its limitations. For example, it may limit the generalization ability of the model, causing the model to perform poorly when dealing with new categories that have not been seen. In addition, if the category information is inaccurate or biased, then categorical memory may lead the model to learn the wrong features. In contrast, non-category memory focuses more on extracting common features from the data and does not rely on specific category labels, which helps the model maintain better generalization capabilities when facing diverse data.

[0004] Compared with categorical memory, category-free memory has the following advantages: 1. Better generalization ability: category-free memory does not rely on specified labels, so it can better generalize to new or unseen data. This is especially important for processing diverse data sets or deploying models in new environments; 2: Reduce category dependence: categorical memory may overemphasize the characteristics of certain categories and ignore the characteristics of other categories. Category-free memory avoids this dependence because it does not rely on category information, but focuses on the overall characteristics of the data; 3: Capture of data set rules: Category-free memory libraries are usually built based on the feature domain of the entire data set. They can capture the rules of the data set itself, rather than relying on the rules of specific categories.

[0005] The present invention proposes a category-free feature memory module based on information similarity. Its function is to embed a feature attention module in the feature extraction layer of the backbone network, and enhance the feature representation of the image input through the category-free memory attention enhancement mechanism. Specifically, based on information similarity, the module extracts a category-free memory that is most similar to the feature map information of this layer from the trained category-free memory library, and then uses the category-free memory to enhance the feature map. It is called category-free memory because the memory library is based on the feature domain of all categories of input images in this layer. It does not extract category information, but focuses on extracting the rules of the data set itself. Summary of the invention

[0006] The purpose of the present invention is mainly to solve several key problems in the feature extraction process: 1. Enhanced feature representation: The category-free feature memory module based on information similarity enhances the model's ability to represent features by introducing a category-free memory library. This enhancement is achieved by comparing the feature map with similar features in the memory library and selecting the most similar memory, so that the model can better capture the inherent laws of the data; 2. Improved model generalization ability: Since the category-free memory library does not rely on specific category information, it can help the model maintain good performance when facing new and unseen data, thereby improving the generalization ability of the model; 3: Introducing intermediate layer memory: The intermediate layer memory ensures that the features extracted in each layer will not be completely lost in the nonlinear transformation of the subsequent layers. This is very important for retaining multi-scale information at the bottom layer (such as edges, textures) and the top layer (such as semantic information).

[0007] Traditional attention mechanisms may rely too much on category labels to determine the focus, while the category-free memory attention mechanism can capture more semantic similarities by comparing with the category-free memory in the memory library without relying on category labels. In this way, the model can focus on key patterns in the image or data instead of being restricted by categories. The category-free memory attention enhancement mechanism combines the category-free memory library and attention weighting methods to enhance feature representation. This mechanism uses category-free memory as a query, and the model is able to dynamically select the memory that is most similar to the current feature map. This dynamic selection mechanism makes the calculation of attention weights more accurate and relevant. Not only the similarity between feature maps and memories is considered, but also the interaction between them, which helps the model understand the data more comprehensively.

[0008] In order to realize a neural network that can be used for classification tasks, the present invention realizes a classification network integrating multiple unclassified feature memory modules based on information similarity, such as Figure 1 As shown in Figure 2, the network consists of residual blocks and information similarity-based unclassified feature memory modules alternating with each other, and has a higher accuracy in classification tasks compared to the classic network.

[0009] In summary, the main contents of the present invention are as follows:

[0010] (1) The present invention proposes a category-free memory attention enhancement mechanism, which is beneficial to enhancing the feature representation of images.

[0011] (2) Based on this mechanism, the present invention designs a category-free feature memory module based on information similarity, which can be easily integrated into the existing neural network architecture and has greatly improved the indicators in classification tasks compared with the original network. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly and easily explain the embodiments of the present invention or the technical solutions in the prior art, I will now briefly introduce the drawings required to be used in the embodiments or the description of the prior art.

[0013] Figure 1 A detailed structural diagram of the information similarity-based non-classified memory module and an overall model architecture diagram of the network provided by an embodiment of the present invention;

[0014] Figure 2 A flow chart of a category-free memory attention enhancement mechanism provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is clearly and detailed described below with reference to the accompanying drawings and specific examples.

[0016] like Figure 1As shown in the figure, the application of the residual network integrated with the information similarity-based unclassified feature memory module in the classification task. The whole process mainly includes three stages. First, we use a series of processing methods to convert the liver cancer DICOM format dataset into an input that the network can process; second, the output classification prediction is obtained through the backbone network composed of residual blocks and unclassified feature memory modules; finally, the gradient backpropagation training network is realized by designing the loss function.

[0017] First, data preprocessing.

[0018] First, the liver cancer dataset contains three categories, namely hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma (ICC) and fatty liver (FAT). The original data of the CT data image is in DICOM format. Set the appropriate window width and window position to convert the DICOM file into a JPEG format image. The data of a patient contains a complete image sequence. The slices are taken out at equal intervals and segmented using the corresponding software, and only the liver part of the slice image is retained. The dataset is divided into a training set and a test set in a 4:1 ratio by shuffling. Due to the small amount of medical data, data enhancement is needed to increase the diversity of the data. Therefore, the experiment uses random flipping, random rotation, brightness adjustment and other methods to enhance each training data to alleviate the overfitting of the network.

[0019] Second, network structure.

[0020] The network structure consists of multiple residual blocks and classless feature memory modules alternating with each other.

[0021] (1) Residual module: The residual block in the network is the basic building block of the residual network (ResNet). It solves the gradient vanishing problem in the neural network by introducing skip connections. Each block contains two convolutional layers, each followed by a batch normalization layer (BatchNormalization) and a ReLU activation function. There is a 1*1 convolutional layer on the skip connection path to match the input and output.

[0022] (2) Classless feature memory module: feature map x i First, GAP (global average pooling) is used to obtain x i_channel :

[0023]

[0024] Then pass through the Encoder encoder to get x f :

[0025]

[0026] The memory bank is randomly initialized by the Embedding layer, and each memory can be represented as a vector of fixed dimension:

[0027] M=Embedding(m_size)

[0028] The encoder extracts the feature representation of the memory:

[0029] M f =Encoder(M)

[0030] The obtained x f , M f After normalization, the IBS distance between the input features and the feature representation is calculated:

[0031] IBS X_M =IBS(x f , M f )

[0032] Select the memory in the memory bank that is closest to the input features:

[0033]

[0034] Use the most similar memory to enhance the input feature map through the attention mechanism:

[0035] x i_enhance =Atten(x i , M closest )

[0036] Atten represents a classless memory enhancement mechanism, and its process is as follows Figure 2 As shown, the process is as follows:

[0037] 1: Input feature map (Input): The initial feature map of the input data, with dimensions B×C×H×W, where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively.

[0038] 2: Global Average Pooling (GAP): Perform global average pooling on the input feature map to generate channel features with a dimension of B×C.

[0039] x i_channel =GAP(x i )

[0040] 3: Memory-closest: Select the memory that is most similar to the current feature map from the memory bank.

[0041] 4: One-dimensional convolution (ConvlD): Perform a one-dimensional convolution operation on the channel features to generate Key (K), and perform a one-dimensional convolution operation on the most similar memory to generate Query (Q).

[0042]

[0043] 5: Spatial flattening operation (Flatten): Perform spatial flattening operation on the input feature map to generate Value (V) with dimension B×C×(HW).

[0044] V = Flatten(x i )

[0045] 6: Attention calculation: Calculate the transposed matrix multiplication of Key and Query to get the attention score (Attn_scores), and use the Softmax function to normalize the attention score.

[0046] Atten scores =Softmax(K T Q)

[0047] 7: Feature enhancement: Multiply the normalized attention score by Value to obtain the enhanced feature map.

[0048] x i_enhance =Atten(x i , M closest )

[0049] Third, the loss function.

[0050] In the classification task, the possible loss function consists of three parts, including softmax loss , IBS loss , IFM_loss. The specific expression is:

[0051] loss = α * softmax loss +IBS loss +β*IFM_loss

[0052] α and β are used to control the size of the loss function to avoid the loss function being too small or too large.

[0053] 1. Among them, softmax loss Represents the cross entropy loss between output and label:

[0054] softmax loss =crossEntryLoss(outputs, targets)

[0055] 2. IBS lossThe loss function expression used to calculate the information bottleneck score can be expressed as follows:

[0056] loss=-(∑labels*log(IBSh sub +ε)+decay*(1-labels)*log(IBSh plus +ε))

[0057] labels is an indicator variable used to distinguish between positive and negative classes.

[0058] ·IBSh sub Indicates the similarity between the current sample and the negative sample in memory, IBSh plus Indicates the similarity between the current sample and the positive sample in memory.

[0059] ε is a small constant used to ensure that the parameters in the logarithmic function are not zero, and decay represents the decay factor used to provide additional regularization.

[0060] The goal of this loss function is to minimize the difference between the model output and the true label. In this way, the model is encouraged to place positive samples close to the positive samples in memory and negative samples close to the negative samples in memory.

[0061] 3. IFM_loss encourages the model to learn to distinguish feature representations of different categories, so that samples of the same category are closer in the feature space and samples of different categories are farther away.

[0062]

Claims

1. An application of a non-class feature memory module based on information similarity in classification, characterized in that: The following steps are involved: Step 1, processing the liver cancer data set collected by the hospital, processing the DICOM format data set through window width and window position, converting the data set into jpeg image format, and segmenting the liver part through software; Step 2: divide the obtained liver cancer dataset into two parts, with most of the images used as network training sets and a small number of images used as network validation sets; Step 3: Preprocess the images of the training set and the validation set, including random cropping, rotation angle and other preprocessing methods; Step 4, establishing a liver cancer image feature extraction module based on convolutional neural network; Step 5, establishing a category-free feature memory module based on information similarity; Step 6: Establish a feature memory network based on information similarity, and train and fine-tune the feature memory module; Step 7: Construct a classifier to determine the type of liver cancer in the input image. The classifier is composed of MLP layers. Step 8: Complete the training and fine-tuning of the entire module, and perform an evaluation test on the test set. After the test meets the standards, the network module is applied to more category judgments of liver cancer images.

2. The application of a feature memory network based on information similarity in classification according to claim 1 is characterized in that: In step 1, the DIC0M format images obtained from the hospital were converted into jpeg format images using the window width and window position specified by the software, and the liver part was manually segmented and framed.

3. The application of a feature memory network based on information similarity in classification according to claim 1 is characterized in that: In step 2, the liver cancer dataset is divided to keep the ratio of training set to test set at about 4:

1.

4. The application of a feature memory network based on information similarity in classification according to claim 1 is characterized in that: After the liver cancer dataset is segmented in step 3, the image training set is augmented using some data augmentation methods, and all images are normalized before being input into the network.

5. The application of a feature memory network based on information similarity in classification according to claim 1 is characterized in that: In step 4, a CNN-based network, such as Resnet, is used to extract features from the image. Residual blocks are used to extract features, and all parameters are pre-trained for ImageNet classification.

6. The application of a feature memory network based on information similarity in classification according to claim 1 is characterized in that: The unclassified feature memory module based on information similarity in step 5 embeds a feature attention module in the feature extraction layer of the backbone network. Specifically, based on information similarity, this module extracts a unclassified memory that is most similar to the feature map information of this layer from the trained unclassified memory library, and then uses the unclassified memory to enhance the feature map. The specific expression is: x i_enhance =Atten(x i ,M closest ) 7. According to the application of a feature memory network based on information similarity in classification as described in claim 1, its characteristic is that in step 6, in order to better extract the rules of the data set itself, a feature attention module is embedded in the feature extraction layer of the backbone layer network, and in order to better train this module, a loss function is designed to help it better learn features through gradient backpropagation.

8. The application of a feature memory network based on information similarity in classification according to claim 1 is characterized in that: In step 7, MLP is used as the classifier, and the ReLU activation function is selected as the activation function between the MLP layers. The expression predicted by the classification process is: where f cls Represents the classifier, pred cls Represents the prediction of the label for the input image. At the same time, the loss function of the classifier is: When the image is of the i-th category, enter the label label i When it is 1, the image is not of the i-th category, enter the label label i is 0.

9. The application of a feature memory network based on information similarity in classification according to claim 1, characterized in that: The loss function of the overall network in step 8 is: loss=ɑ*softmax loss +IBS loss +β*IFM_loss Among them, softmax loss represents the classification loss, and represents the cross entropy loss between output and label. IBS loss The purpose of the loss function is to minimize the difference between the model output and the true label. In this way, the model is encouraged to move positive samples close to the positive samples in memory and negative samples close to the negative samples in memory. IFM_loss encourages the model to learn feature representations that distinguish different categories, so that samples of the same category are closer in feature space and samples of different categories are farther away.

Images