A multi-task image feature extraction and classification system based on deep learning

Abstract

The present invention discloses a deep learning-based image multi-task feature extraction and classification system, including: an image preprocessing module, a general feature extraction module, a specific task feature extraction module, and a classification module; the image preprocessing module is used to preprocess the medical image to be identified; the general feature extraction module is used to extract general features; the specific task feature extraction module is used to extract specific features; the classification module is used to classify the image to be identified according to the general features and the specific features. The present invention is designed as a general framework that supports a wide range of image analysis tasks, such as image type recognition, abnormal area detection in images, and automatic evaluation of image features, so as to achieve knowledge sharing and performance improvement in different tasks.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and in particular to an image multi-task feature extraction and classification system based on deep learning. Background Art

[0002] The deep learning model directly learns features from the original image, simplifying the image preprocessing process. The following deep learning models have been reported:

[0003] (1) Convolutional Neural Network (CNN)-based models: CNN is widely used to automatically extract image features, including local patterns such as shape, texture, and edge. Although these models have shown effectiveness in image classification tasks, they mainly focus on local features and may sometimes ignore the global information of the image, limiting their performance in processing complex medical images.

[0004] (2) Models based on attention mechanisms: such as the Transformer model, which attempts to improve the accuracy of image classification by emphasizing key areas in the image. This method optimizes the classification results by focusing on important areas of the image that are relevant to the task. Although the attention mechanism improves the model's sensitivity to key features, its high computational cost and sensitivity to non-standardized images are its main limitations.

[0005] (3) Models based on convolutional-attention modules: This type of model aims to combine the advantages of convolutional neural networks and attention mechanisms, extracting local features through convolutional layers and using attention mechanisms to focus on global information to provide more comprehensive image analysis. This hybrid model works better when processing complex image data, such as more accurate classification when processing complex medical images. However, most existing models focus on a single classification task and fail to fully utilize the potential of multi-task learning, thus limiting the generalization ability and application scope of the model.

[0006] The main problems of existing technologies include dependence on a single task, the need for a large amount of high-quality training data for deep learning models, and the limitation of model generalization ability. In order to solve these problems, it is necessary to develop an image multi-task feature extraction and classification system based on deep learning. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes an image multi-task feature extraction and classification system based on deep learning to solve the problems existing in the above-mentioned prior art.

[0008] To achieve the above objectives, the present invention provides an image multi-task feature extraction and classification system based on deep learning, comprising:

[0009] Image preprocessing module, general feature extraction module, task-specific feature extraction module, and classification module;

[0010] The image preprocessing module is used to preprocess the image to be identified;

[0011] The general feature extraction module is used to extract general features;

[0012] The specific task feature extraction module is used to extract specific features;

[0013] The classification module is used to classify the image to be identified according to the general features and the specific features.

[0014] Optionally, the image preprocessing module performs normalization processing within a preset window range after standardizing, denoising, and adjusting the contrast of the image to be identified.

[0015] Optionally, the general feature extraction module includes an inverse residual block, a compression and excitation block;

[0016] The reverse residual block is used to extract the original features of the image to be recognized;

[0017] The compression and excitation block is used to perform compression and excitation operations on the original features to obtain universal features.

[0018] Optionally, the reverse residual block includes an expansion layer, a deep convolution layer and a projection layer;

[0019] The expansion layer includes a 1*1 point convolution, a batch normalization block and a ReLU activation function;

[0020] The deep convolution layer includes a 3*3 depth convolution, a batch normalization block and a ReLU activation function;

[0021] The projection layer includes a 1*1 point convolution and a batch normalization block.

[0022] Optionally, the specific task feature extraction module includes a reverse residual block and an attention encoder module; the original features are extracted through the reverse residual block; and specific features are extracted from the original features based on the attention encoder module.

[0023] Optionally, the attention encoder module includes a multi-head attention module and a multi-layer perceptron module, and the multi-layer perceptron module includes two linear transformations with a nonlinear activation function between the linear transformations.

[0024] Optionally, the classification module is used to classify the image to be recognized based on the general features and the specific features using a multi-task learning strategy.

[0025] Compared with the prior art, the present invention has the following advantages and technical effects:

[0026] The present invention adopts a deep learning architecture that integrates an inverse residual block, a compression and excitation module, and an attention encoder to perform comprehensive feature extraction and classification analysis on complex images through a multi-task learning strategy. The present invention is designed as a general framework that supports a wide range of medical image analysis tasks, such as image type recognition, abnormal area detection in images, and automatic evaluation of image features, thereby achieving knowledge sharing and performance improvement in different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0028] Figure 1 is an original image of lung CT according to an embodiment of the present invention, wherein: 1. nodule;

[0029] Figure 2 The technical roadmap of the embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] Embodiment 1

[0033] like Figure 1-2 As shown, this embodiment provides an image multi-task feature extraction and classification system based on deep learning, including:

[0034] Image preprocessing module: This module is responsible for the preliminary processing of medical image data, including image standardization, denoising, contrast adjustment, etc., to ensure the quality and consistency of input data. This module uses the B-spline interpolation algorithm to adjust the image resolution and normalize the CT value within the window range of [-1400HU, 400HU] to provide accurate input data for subsequent feature extraction and classification.

[0035] General Feature Extraction Module: With the Inverted Residual Block and Squeeze-and-Excitation (SE) as the basic architecture, this module can extract a wide range of general features from medical images, such as shape, texture, and edge. These features lay the foundation for further analysis and classification of medical images.

[0036] In the reverse residual block, the input features will first pass through an expansion layer, which consists of a 1*1 point convolution, a batch normalization block and a ReLU activation function. This expansion layer allows the features to be dimensionally upgraded. Next, the features will pass through a depthwise convolution layer, which consists of a 3*3 depthwise convolution, a batch normalization block and a ReLU activation function. Information in the features is extracted through this depthwise convolution layer. Finally, the features will pass through a projection layer, which consists of a 1*1 point convolution and a batch normalization block, and the features are output through this layer.

[0037] Among them, the convolution operation formula is:

[0038]

[0039] In this formula, represents the neuron activation value at position (x, y) on the i-th layer and j-th feature map; b ij is the bias term of the jth feature map in the i-th layer; P i and Q i Defines the size of the receptive field or convolution kernel applied to the feature map; represents the weight of the connection from the m-th feature map of the (i-1)-th layer to the j-th feature map of the i-th layer at the filter position (p,q); It represents the activation value of the neuron at position (x, y) with position offset (p, q) on the m-th feature map of the previous layer (i-1), which represents the input of the current neuron at a specific position of the feature map of the previous layer.

[0040] The calculation formula of batch normalization is:

[0041]

[0042] In this formula, represents the normalized value of the kth feature in a given batch. Batch normalization processes the input in each mini-batch layer by layer and feature by feature; (k) represents the original value of the kth feature of a given input in the batch; E[x (k)] represents the expected value or average value of the kth feature value in the batch. In batch normalization, the average value of each feature over all instances of the mini-batch is calculated; Represents the square root of the variance of the k-th eigenvalue in the batch.

[0043] The calculation formula of the ReLU activation function is:

[0044] f(x) = max(0, x)

[0045] In the SE block, the features will be squeezed and excited successively. After the compression operation, the image becomes a 1*1*C feature vector. After the excitation operation, the dimension of the feature vector does not change, but the vector value becomes a new value. These values ​​will be calculated together with the original features to obtain the output features. The dimension of the output features is the same as the dimension of the input features.

[0046] The compression calculation formula is:

[0047]

[0048] In this formula, z c , F sq (u c ) is equivalent to, which means the feature map u of the c-th channel c The global information obtained after compression is a scalar representation of the channel; u c (i, j) represents the feature response value of position (i, j) on the cth channel; W×H represents the size of the feature map, where W is the width and H is the height.

[0049] The calculation formula for excitation is:

[0050] s=F ex (z, W)=σ(g(z, W))=σ(g(W 2 δ(W 1 z)))

[0051] In this formula, s = F ex (z,W) represents the activation operation on the compressed global information z to obtain the weight coefficient s of each channel; σ(·) represents the activation function; g(·) represents a learned transformation, which is usually composed of two fully connected layers, namely W 1 and W 2 , connected by the ReLU activation function in the middle; W 1 and W 2 Represents the weight of the fully connected layer, which is used to learn the nonlinear mapping from global information z to channel weight coefficient s.

[0052] The final output integrates the results of the excitation operation and the original features. The operation formula is:

[0053]

[0054] In this formula, Indicates the weight coefficient s obtained by the excitation operation c With the original feature map u c Perform element-by-element product operation to obtain the recalibrated feature map s c is the weight coefficient of the cth channel; u c is the original feature map.

[0055] Image-specific task feature extraction module: Combining the inverse residual block and the Attention Encoder (AE), this module extracts key features for specific medical image analysis tasks. By using the multi-head attention mechanism to focus on important areas in the image, this module can improve the recognition accuracy and efficiency of specific tasks.

[0056] In the AE block, features are passed through a multi-head attention module and a multi-layer perceptron module. In multi-head attention, the input consists of a query (Q), a key (K), and a value (V), which are projections of the input embedding. The attention mechanism is applied multiple times in parallel, with each "head" learning a different aspect of the data. The outputs of the various heads are then concatenated and linearly transformed to the final output. The multi-layer perceptron module is a fully connected feed-forward neural network. The block consists of two linear transformations with a non-linear activation function (ReLU) in between.

[0057] Among them, the calculation formula of the multi-head attention module is:

[0058] MultiHead(Q,K,V)=Concat(head 1 , ..., head h )W O

[0059] where head i =Attention(QW i Q , KW i K , VW i V )

[0060]

[0061] In this formula, MultiHead(Q, K, V) is a multi-head attention function, which accepts three parameters Q (query), K (key), V (value); Concat (head1 , ..., head h ) represents the connection operation, which connects all the attention heads (head i ) are connected. Each attention head i is the weight matrix W corresponding to a specific i Q , W i K , W i V The output of the attention function; W O is the output weight matrix, which is used to convert the connection results of all attention heads into the final output; head i The result of the calculation of the i-th attention head, which is the output of the attention function corresponding to the i-th set of weight matrices; Attention(Q, K, V) is an attention function that takes a query (Q), a key (K), and a value (V), calculates the dot product attention score of the query and the key, then applies the softmax function to obtain the weight distribution, and finally this weight distribution is used to weight the value (V); QW i Q , KW i K , VW i V It is the input Q, K, V and their respective weight matrices W i Q , W i K , W i V These weight matrices are learnable parameters and are different for different heads, allowing the model to capture information in different subspaces. is the dot product of the query and the key, divided by d k The square root of k represents the dimension of the key. This scaling helps stabilize the gradient, especially when the dimension is high. The softmax function is used to convert the attention score (i.e., the dot product of the query and the key) into a probability distribution.

[0062] The calculation formula of the multi-layer perceptron module is:

[0063] MLP(x)=W 2 ReLU(W 1 ·x+b 1 )+b 2

[0064] In this formula, MLP(x) represents the result of the multilayer perceptron operating on the input x. 1is the weight matrix of the first layer, which defines the linear transformation from the input layer to the hidden layer. x is the input vector of the multilayer perceptron. b 1 is the bias term of the first layer. W 2 is the weight matrix of the second layer. b 2 is the bias term of the second layer.

[0065] The loss function of the model is the weighted sum of multiple task loss functions, namely:

[0066]

[0067] In this formula, L(θ) is the total loss function of multi-task learning. i (θ i ) is the loss function of the i-th task. i is the weight coefficient of the i-th task, which is used to adjust the relative importance of each task loss in the total loss.

[0068] Image classification module: Based on the general and specific task features extracted above, this module uses a multi-task learning strategy to classify images. By designing loss functions suitable for a variety of image analysis tasks, this module can simultaneously handle multiple classification tasks, such as image type recognition and anomaly detection, to improve the generalization ability and classification accuracy of the model.

[0069] Through these four modules, the algorithm of the present invention can effectively extract key features from images and accurately perform multi-task classification. The algorithm not only improves the efficiency and accuracy of image processing, but also has a wide range of application potentials, including but not limited to automatic recognition of image content, evaluation of image quality, etc.

[0070] Example 1:

[0071] The following is a process of using the system of this embodiment to perform multi-task classification on images containing nodules. The steps are as follows:

[0072] 1. CT images of 772 lung cancer patients were collected. The age of the patients ranged from 33 to 90 years old, with an average age of 66.1 years old.

[0073] 2. Use the image preprocessing module to process the original image.

[0074] 3. Figure 1 It is an original CT image containing nodules. After preprocessing, the CT image is cropped according to the specified nodule location, and 4632 CT images containing nodules are obtained. These images are randomly assigned to the training set, validation set, and test set in a ratio of 8:1:1. The training set has 3702 images, the validation set has 467 images, and the test set has 463 images.

[0075] 4. Divide the cropped images into training, validation, and test sets, and repeat the following steps to train the model:

[0076] 1) Use the medical image general feature extraction module to extract features that can be used in downstream tasks;

[0077] 2) Use the medical image specific task feature extraction module to extract features for specific classification tasks;

[0078] 3) Use the medical image classification module to output the final classification result of the image;

[0079] 4) Evaluate the accuracy of the classification results;

[0080] 5. Adjust the model training hyperparameter settings on the validation set to correct the prediction model and obtain an ideal model with relatively accurate results. Finally, evaluate the performance of the model on the test set.

[0081] The batch size of the model training is 64, the loss function is cross entropy loss, the optimizer is stochastic gradient descent optimizer, the learning rate is 0.001, and the learning rate is updated to 10% of the original learning rate after every 30 training cycles. The total training cycle is 100, and the model with the highest accuracy on the validation set during the training cycle is saved.

[0082] After training, the model had an accuracy of 91.82%, a sensitivity of 88.33%, a specificity of 93.85%, and an AUC of 0.9674 in the morphological angle classification task. The model had an accuracy of 92.44%, a sensitivity of 91.00%, a specificity of 93.54%, and an AUC of 0.9710 in the disease progression angle classification task.

[0083] The comprehensive model architecture proposed in this embodiment optimizes the feature extraction and classification processing flow, so that it can effectively extract key information from various complex images, not only limited to lung CT images containing nodules. The flexibility and versatility of the model architecture enable it to be applicable to a wide range of medical image types and other complex images, such as MRI, X-ray, etc., and supports a variety of image analysis tasks, including but not limited to image content classification, anomaly detection, and image quality assessment.

[0084] After rigorous experimental verification, the model of the present invention has demonstrated excellent performance in multiple image classification tasks. The model of the present invention has shown excellent generalization ability during the training process, ensuring that high accuracy can be maintained on different data sets, proving its reliability and effectiveness in practical applications.

[0085] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A deep learning-based image multi-task feature extraction and classification system, characterized in that: include: Image preprocessing module, general feature extraction module, task-specific feature extraction module, and classification module; The image preprocessing module is used to preprocess the image to be identified; The image preprocessing module performs normalization processing within a preset window range after standardizing, denoising and adjusting the contrast of the image to be identified; The general feature extraction module is used to extract general features; The general feature extraction module includes an inverse residual block, a compression and excitation block; The reverse residual block is used to extract the original features of the image to be recognized; The compression and excitation block is used to perform compression and excitation operations on the original features to obtain universal features; The reverse residual block includes an expansion layer, a deep convolution layer and a projection layer; The expansion layer includes a 1*1 point convolution, a batch normalization block and a ReLU activation function; The deep convolution layer includes a 3*3 depth convolution, a batch normalization block and a ReLU activation function; The projection layer includes a 1*1 point convolution and a batch normalization block; The specific task feature extraction module is used to extract specific features; The task-specific feature extraction module includes a reverse residual block and an attention encoder module; the original features are extracted through the reverse residual block; Extracting specific features from the original features based on an attention encoder module; The attention encoder module includes a multi-head attention module and a multi-layer perceptron module, wherein the multi-layer perceptron module includes two linear transformations, and a nonlinear activation function is provided between the linear transformations; The classification module is used to classify the image to be identified according to the general features and the specific features.

2. The image multi-task feature extraction and classification system based on deep learning according to claim 1, characterized in that: The classification module is used to classify the image to be recognized by adopting a multi-task learning strategy according to the general features and the specific features.

Images