Tumor rapid identification system based on multi-scale lightweight neural network

Abstract

The invention discloses a tumor rapid identification system based on a multi-scale lightweight neural network, and relates to the technical field of medical image processing, and the system comprises a region-of-interest block generation module which is used for carrying out the binarization processing of a pathological image, then screening and calculating the weighted weight of a block based on a weight accumulation method, and carrying out the calculation of the weighted weight of the block; the region of interest in the pathological image is extracted based on the weighted weights of the blocks, so that the recognition workload can be greatly reduced, and the recognition speed is improved; the multi-scale lightweight fast neural network is used for carrying out fast classification on regions of interest in the pathological image to obtain a final classification result, and the network can be conveniently deployed on lightweight equipment; and the image expression module is used for generating a tumor probability thermodynamic diagram based on the final classification result. The attention loss function used for training the multi-scale lightweight fast neural network is convenient for controlling the weight of the positive sample and the negative sample, so that the classification performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a tumor rapid recognition system based on a multi-scale lightweight neural network. Background Art

[0002] A digital pathology slide is a high-resolution digital image that shows a complete microscope slide containing a tissue sample or other specimen. Digital pathology slides are captured by specialized scanners that can digitize slides at very high magnifications to create a digital representation of the slide. Digital pathology slides have many advantages and are becoming increasingly popular in the field of pathology as they can assist in the diagnosis and treatment of various diseases, such as cancer. By viewing digital pathology slides, pathologists can examine different areas of the slide in detail on a computer screen. This allows for a more comprehensive and detailed analysis of tissue samples, allowing pathologists to more accurately diagnose conditions and provide more appropriate treatment options for patients. In addition, digital pathology slides can be accessed remotely, allowing pathologists and researchers to collaborate more effectively across geographic locations. This convenience also allows researchers to analyze large numbers of tissue samples quickly and efficiently, accelerating the development of new drugs and treatments.

[0003] Although digital pathology slides have many advantages in the field of pathology and medicine, there are also several challenges:

[0004] (1) Digital pathology slides have ultra-high resolution, and the size of a digital pathology slide is usually about 1-5GB. However, about 70% of the area is useless background area. When pathologists spend a lot of time on these boring and meaningless background areas, it is easy to cause common problems such as misdiagnosis or missed diagnosis.

[0005] (2) Each digital pathology slide has about one billion pixels, but the lesion area usually has only hundreds of pixels. It is very difficult to use this extremely unbalanced dataset of positive and negative samples to train an excellent deep neural network.

[0006] (3) Due to the ultra-high resolution of digital pathology images, if pathologists need to examine every pixel, the diagnosis time will be very long, which will pose a potential risk to patients. How to assist pathologists in quickly diagnosing digital pathology sections and improve the accuracy and efficiency of diagnosis is an important and difficult problem to solve. Summary of the invention

[0007] The present invention provides a tumor rapid recognition system based on a multi-scale lightweight neural network to assist pathologists in quickly locating suspicious positions of digital pathology sections.

[0008] In order to solve the above problems, the technical solution adopted by the present invention is as follows:

[0009] The present invention provides a tumor rapid identification system based on a multi-scale lightweight neural network, comprising:

[0010] The region of interest block generation module is used to perform binarization processing on the pathological image, and then screen and calculate the weighted weights of the blocks based on the weight accumulation method, and extract the region of interest in the pathological image based on the weighted weights of the blocks;

[0011] Pre-trained multi-scale lightweight fast neural network is used to quickly classify the regions of interest in pathological images and obtain the final classification results;

[0012] An image expression module is used to generate a tumor probability heat map based on the final classification results;

[0013] The attention loss function AttLoss used to train multi-scale lightweight fast neural networks is:

[0014]

[0015] Among them, p i represents the probability that the i-th sample is predicted as a positive class, y i is the label of the ith sample, and α and γ are hyperparameters.

[0016] In a preferred embodiment of the present invention, the method for binarizing the pathological image is: using the Otsu algorithm, binarizing the pathological image under the RGB color model and the HSV color model respectively.

[0017] In a preferred embodiment of the present invention, the weight accumulation formula is:

[0018]

[0019] Among them, W i,j represents the weighted weight of the tile (i, j) in the i-th row and j-th column, Represents the pixel value of the mth row and nth column in tile (i, j);

[0020] The filtering rules for the area of ​​interest are:

[0021]

[0022] If Patch i,j =1, then the block (i, j) is the final region of interest, otherwise the block (i, j) is the region to be discarded.

[0023] In a preferred embodiment of the present invention, the multi-scale lightweight fast neural network includes a multi-scale module and a multi-branch module with jump connections and alternating combinations. Both the multi-scale module and the multi-branch module use ReLU6 as the nonlinear activation function. At the same time, depthwise separable convolution is used in the multi-scale module and the multi-branch module to reduce the number of parameters and computational overhead.

[0024] In a preferred embodiment of the present invention, the multi-scale lightweight fast neural network first performs initial feature extraction before performing multi-scale feature extraction and multi-branch feature extraction alternately, and the extraction formula is:

[0025] I low =Conv(I input ,K initial )

[0026] Among them, I input is the input image, i.e., the region of interest in the pathological image, K initial is the convolution kernel of the initial convolution layer, I low is a low-level feature map.

[0027] In a preferred embodiment of the present invention, the nonlinear activation function is:

[0028] F1 = ReLU(BN(W1*I+b1))

[0029] Among them, W1 is the initialization convolution kernel, b1 is the bias, * represents the convolution operation, BN represents batch normalization, and ReLU is the activation function.

[0030] In a preferred embodiment of the present invention, the multi-scale feature extraction formula is:

[0031] I multi =Concat(DSConv(I low ,K multi,i ))

[0032] Among them, DSConv is the depth-wise separable convolution, K multi,i is the convolution kernel of the i-th scale, Concat is the connection operation, I multi is the multi-scale feature extracted.

[0033] In a preferred embodiment of the present invention, the multi-branch feature extraction formula is:

[0034] I branch =Concat(DSConv(I multi ,K branch,j ))

[0035] Among them, K branch,j is the convolution kernel of the jth branch, Ibranch is the extracted multi-branch feature.

[0036] The output of each branch is expressed as:

[0037] F bi =ReLU(BN(W bi *F prev +b bi ))

[0038] Among them, F prev is the output of the previous layer, W bi and b bi are the weight and bias of the ith branch.

[0039] In a preferred embodiment of the present invention, the multi-scale lightweight fast neural network uses global average pooling to aggregate the spatial dimensions of each feature channel, expressed as:

[0040]

[0041] Where W and H are the width and height of the feature map, respectively.

[0042] The multi-scale lightweight fast neural network uses a fully connected layer for final classification, expressed as:

[0043] O=σ(W f ·F GAP +b f )

[0044] Among them, W f is the weight of the fully connected layer, b f is the bias, σ is the sigmoid activation function, which makes the output fall between [0,1], indicating the probability of binary classification.

[0045] In a preferred embodiment of the present invention, during the training of a multi-scale lightweight and fast neural network, the hyperparameters α and γ are set to 3 and 2 respectively, the multi-scale lightweight and fast neural network is trained for 100 iterations using a training set, and the optimal model is screened out using a test set.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1) The region of interest block generation module uses the weight accumulation method to screen and calculate the weighted weights of the blocks, and extracts the regions of interest in the pathological images based on the weighted weights of the blocks, so that the multi-scale lightweight fast neural network can focus on the regions of interest, thereby greatly reducing the workload, improving the recognition speed, significantly reducing the required computing resources, achieving higher energy efficiency, and effectively extending the service life of the equipment to which this system belongs and reducing energy consumption.

[0048] 2) A novel attention loss function is designed for datasets with imbalanced positive and negative samples. On the basis of the cross entropy loss function, a hyperparameter α is added to increase the loss weight of positive samples. In order to further improve the training accuracy of the model, a hyperparameter γ is also introduced to make the model pay more attention to samples that are difficult to distinguish and pay less attention to samples that are easy to distinguish. The loss function uses factors and 1-(1-p i ) γ To control the weights of positive and negative samples, thereby improving classification performance.

[0049] 3) A lightweight and fast neural network was designed for image classification and recognition, which is easy to deploy on lightweight devices, thereby achieving the lightweight of the entire network while maintaining excellent performance and accelerating the diagnosis of pathological images.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the embodiments of the present invention are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a process flow chart of the rapid tumor identification system;

[0053] Figure 2 It is a flowchart of region of interest extraction tiles of digital pathology images;

[0054] Figure 3 It is the structural diagram of the multi-scale module;

[0055] Figure 4 It is a structural diagram of a multi-branch module;

[0056] Figure 5 This is the structural diagram of the lightweight neural network FastNet. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0058] Combination Figure 1 As shown, the present invention provides a tumor rapid identification system based on a multi-scale lightweight neural network, including a region of interest block generation module, a multi-scale lightweight fast neural network and an image expression module.

[0059] 1. Region of Interest Block Generation Module

[0060] The region of interest block generation module is used to perform binarization processing on the pathological image to obtain a preliminary region of interest, and then screen and calculate the weighted weight of the block based on the weight accumulation method, and extract the region of interest in the pathological image based on the weighted weight of the block, see Figure 2 shown.

[0061] Specifically, a weight accumulation method was used to reduce the computational cost. In order to obtain the region of interest of the tissue mask and remove the background area in the image, the Otsu algorithm was applied to binarize the pathological image under the RGB color model and the HSV color model, respectively. Then, the threshold tissue mask was obtained by adjusting the threshold of the RGB color value. Finally, the RGB tissue mask, the HSV tissue mask and the threshold tissue mask were merged to obtain the preliminary region of interest. In this way, the neural network can focus on the preliminary region of interest, thereby greatly reducing the workload and improving the recognition speed.

[0062] In the present invention, the binary image mask is divided into 256×256 pixel blocks, where a pixel value of 1 represents a valid area and a pixel value of 0 represents an invalid area. The following weight accumulation formula is then used to calculate the weight of the blocks:

[0063]

[0064] Among them, W i,j represents the weighted weight of the patch mask (i, j) at the i-th row and j-th column, Represents the pixel value of the mth row and nth column in the tile mask (i, j);

[0065] According to the tile mask (i, j), the tiles that meet the training data are filtered. The filtering rules are as follows:

[0066]

[0067] when When i,j is the final region of interest; otherwise, 256, then the patch i,j The data to be discarded.

[0068] 2. Multi-scale lightweight and fast neural network

[0069] The present invention designs a multi-scale lightweight fast neural network FastNet to quickly classify tumor blocks, which is easy to deploy on lightweight devices and accelerates the diagnosis of pathological images.

[0070] FastNet is constructed by alternating multi-scale modules and multi-branch modules. The multi-scale module is designed to extract features of different scales, while the multi-branch module is used to extract more details from the previous layer. When building deeper networks, both multi-scale blocks and multi-branch blocks add jump connections to solve the gradient vanishing problem. Based on its robustness, the multi-scale module and the multi-branch module use ReLU6 as the nonlinear activation function. At the same time, the use of depthwise separable convolution in the multi-scale module and the multi-branch module can reduce the number of parameters and computational overhead.

[0071] The multi-scale module is an efficient feature extraction mechanism in deep learning. It can enhance the network’s receptive field by extracting features at different spatial resolutions and capturing local and global information of the image. The proposed multi-scale module structure is shown in Figure 3 As shown. The size of the feature map of this module is (W scale ,W scale ,M scale ), where W scale Represents the length and width of the input feature map, M scale Represents the number of input channels. First, the module uses a standard 1×1 convolution kernel to achieve channel expansion. Then, 1×1, 3×3, and two consecutive 3×3 simulated 5×5 depthwise separable convolutions are used to capture multi-scale features. Then, the multi-scale features are fused and reduced in dimension. Finally, a residual connection is added to the output.

[0072] The multi-branch module is designed to achieve the fusion of multiple feature expressions by introducing different depth-separable convolution branches to capture information at multiple scales. The proposed multi-branch module structure is shown in Figure 4 The core idea is to extract features of various scales by using multiple depth-separable convolution branches and then combine them to enhance the feature expression ability of the model. The size of the feature map of this module is (W branch ,W branch ,M branch ), where W branch Represents the length and width of the input feature map, M branch Represents the number of input channels. First, the module uses a standard 1×1 convolution kernel to achieve channel expansion. Then, B 3×3 depth-separable convolution branches are used to extract features. Then, multi-scale features are fused and dimensionally reduced. Finally, a residual connection is added to the output.

[0073] FastNet was constructed by using the proposed multi-scale module and multi-branch module, and taking into account the number of parameters and the amount of computation of the multi-scale module and the multi-branch module. The network architecture is designed to achieve efficient feature extraction while reducing the number of parameters and the amount of computation. The detailed network architecture of FastNet is as follows Figure 5 The detailed information is shown in Table 1.

[0074] Table 1 FastNet network implementation details

[0075]

[0076] 1) Initial feature extraction: FastNet first uses a convolution layer to capture the low-level features of the original input image (i.e., the region of interest in the pathological image). Because the low-level features of the image (such as edges and colors) are crucial for subsequent high-level feature extraction. The convolution step size of 2×2 can effectively reduce the image size while reducing the amount of calculation, but retaining important spatial information. The initial feature extraction formula is:

[0077] I low =Conv(I input ,K initial )

[0078] Among them, I input is the input image, i.e., the region of interest in the pathological image, K initial is the convolution kernel of the initial convolution layer, I low is a low-level feature map.

[0079] The F1 function is as follows:

[0080] F1 = ReLU(BN(W1*I+b1))

[0081] Among them, W1 is the initialization convolution kernel, I is the input feature map, b1 is the bias, * represents the convolution operation, BN represents batch normalization, and ReLU is the activation function.

[0082] 2) Multi-scale feature extraction: Using the multi-scale module, FastNet can capture image features at different spatial scales. Each scale captures different feature details, from micro to macro. Specifically, larger scales can capture broader contextual information, while smaller scales can capture detailed information. It is expressed as follows:

[0083] I multi =Concat(DSConv(I low ,K multi,i ))

[0084] Among them, DSConv is the depth-wise separable convolution, I lowis a low-level feature map, K multi,i is the convolution kernel of the i-th scale, Concat is the connection operation, I multi is the multi-scale feature extracted.

[0085] 3) Multi-branch feature extraction: In the multi-branch module, FastNet further extracts detailed features through multiple parallel depth-wise separable convolutions. This not only provides a richer feature set, but also achieves computational efficiency through multi-branch parallel processing. It is expressed as follows:

[0086] I branch =Concat(DSConv(I multi ,K branch,j ))

[0087] Among them, K branch,j is the convolution kernel of the jth branch, I multi is the multi-scale feature extracted, I branch is the extracted multi-branch feature.

[0088] 4) Multi-scale multi-branch module alternation: In FastNet, multi-scale and multi-branch modules are designed to be stacked alternately, providing feature extraction at different scales and rich detail representation. Considering each multi-scale block, there are M input channels and N expansion factors. For B branches, the output of each branch is represented as:

[0089] F bi =ReLU(BN(W bi *F prev +b bi ))

[0090] Among them, F prev is the output of the previous layer, W bi and b bi are the weight and bias of the ith branch.

[0091] 5) Global spatial feature aggregation: In order to reduce the number of parameters and concentrate features, FastNet uses global average pooling (GAP). GAP aggregates the spatial dimensions of each feature channel and retains channel information. It is expressed as:

[0092]

[0093] Where W and H are the width and height of the feature map, respectively.

[0094] 6) Classification output: FastNet uses a fully connected layer for final classification, expressed as:

[0095] O=σ(Wf ·F GAP +b f )

[0096] Among them, W f is the weight of the fully connected layer, b f is the bias, and σ is the sigmoid activation function, which makes the output fall between [0,1], indicating the probability of binary classification.

[0097] FastNet ensures good feature extraction capabilities at all scales through this structure, and the multi-branch structure provides richer feature representation. When designing FastNet, the computational complexity and performance were balanced so that it can capture detailed information while maintaining computational efficiency.

[0098] For extremely unbalanced classification problems, neural networks tend to predict negative samples, resulting in extremely high prediction probabilities and large gradients for negative samples. Usually, the tumor region only occupies a very small part of the tissue pathology image. Therefore, only a small number of positive samples can be extracted from the tumor region. In the tumor prediction set, the number of negative samples far exceeds the number of positive samples, which results in a very low recall rate for tumor prediction.

[0099] Therefore, the present invention provides a novel attention loss function (Attention Loss, AttLoss) for training multi-scale lightweight fast neural networks to increase the recognition ability of positive samples and difficult-to-distinguish samples. That is, by adding a hyperparameter α to the loss function to increase the loss weight of positive samples, and at the same time, introducing a hyperparameter γ to make the model pay more attention to those difficult-to-distinguish samples and reduce the attention to easy-to-distinguish samples. The definition of the attention loss function is as follows:

[0100]

[0101] Among them, p i represents the probability that the i-th sample is predicted as a positive class, y i is the label of the ith sample (0 or 1), and N represents the total number of samples. α and γ are hyperparameters. By setting a larger α (α>1), the weight of positive samples can be increased, and by setting a smaller α (0<α<1), the weight of positive samples can be reduced. In order to reduce the weight of samples that are easy to distinguish, the loss function uses the factors and 1-(1-p i ) γ To control the weights of positive and negative samples.

[0102] In the process of training the multi-scale lightweight and fast neural network, the hyperparameters α and γ were set to 3 and 2 respectively, and the model's ability to recognize positive examples reached the optimal level. The multi-scale lightweight and fast neural network was trained for 100 iterations using the training set, and the optimal model was screened out through the test set.

[0103] Compared with existing lightweight neural networks (MobileNet V2, ShuffleNet V2 and EfficientNet), the proposed FastNet achieves the highest accuracy (0.9734), recall rate (0.9694) and F1 score (0.9733) with the least training parameters (0.22M) and the lowest computational cost (210M FLOPs).

[0104] 3. Image expression module

[0105] The image expression module is used to generate a tumor probability heat map based on the final classification results output by FastNet. That is, all tumor prediction probabilities are embedded into the tissue pathology image and arranged in the order of image blocks to form a complete tumor probability heat map. In the heat map, the red area represents the high probability tumor area, and the green area represents the normal tissue area.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A tumor rapid identification system based on a multi-scale lightweight neural network, characterized in that: include: The region of interest block generation module is used to perform binarization processing on the pathological image, and then screen and calculate the weighted weights of the blocks based on the weight accumulation method, and extract the region of interest in the pathological image based on the weighted weights of the blocks; Pre-trained multi-scale lightweight fast neural network is used to quickly classify the regions of interest in pathological images and obtain the final classification results; An image expression module is used to generate a tumor probability heat map based on the final classification results; The attention loss function AttLoss used to train multi-scale lightweight fast neural networks is: Among them, p i represents the probability that the i-th sample is predicted as a positive class, y i is the label of the i-th sample, α and γ are hyperparameters, and N is the total number of training samples.

2. The tumor rapid identification system based on multi-scale lightweight neural network according to claim 1 is characterized in that: The method for binarizing the pathological image is: using the Otsu algorithm, the pathological image is binarized under the RGB color model and the HSV color model respectively.

3. The tumor rapid identification system based on multi-scale lightweight neural network according to claim 1 is characterized in that: The weight accumulation formula is: Among them, W i,j represents the weighted weight of the patch mask (i, j) at the i-th row and j-th column, Represents the pixel value of the mth row and nth column in the tile mask (i, j); The filtering rules for the area of ​​interest are: If Patch i,j =1, then the block (i, j) is the final region of interest, otherwise the block (i, j) is the region to be discarded.

4. The tumor rapid identification system based on multi-scale lightweight neural network according to claim 1 is characterized in that: The multi-scale lightweight fast neural network includes multi-scale modules and multi-branch modules with jump connections and alternating combinations. Both the multi-scale modules and the multi-branch modules use ReLU6 as the nonlinear activation function. At the same time, depthwise separable convolution is used in the multi-scale modules and the multi-branch modules to reduce the number of parameters and computational overhead.

5. The tumor rapid identification system based on multi-scale lightweight neural network according to claim 4 is characterized in that: The multi-scale lightweight fast neural network performs initial feature extraction before alternately performing multi-scale feature extraction and multi-branch feature extraction, and the extraction formula is: I low =Conv(I input ,K initial ) Among them, I input is the input image, i.e., the region of interest in the pathological image, K initial is the convolution kernel of the initial convolution layer, I low is a low-level feature map.

6. The tumor rapid identification system based on multi-scale lightweight neural network according to claim 5 is characterized in that: The nonlinear activation function is: F1 = ReLU(BN(W1*I+b1)) Among them, W1 is the initialization convolution kernel, I is the input feature map, b1 is the bias, * represents the convolution operation, BN represents batch normalization, and ReLU is the activation function.

7. The tumor rapid identification system based on multi-scale lightweight neural network according to claim 6 is characterized in that: The multi-scale feature extraction formula is: I multi =Concat(DSConv(I low ,K multi,i )) Among them, DSConv is the depth-wise separable convolution, K multi,i is the convolution kernel of the i-th scale, Concat is the connection operation, I low is a low-level feature map, I multi is the multi-scale feature extracted.

8. The tumor rapid identification system based on multi-scale lightweight neural network according to claim 7 is characterized in that: The multi-branch feature extraction formula is: I branch =Concat(DSConv(I multi ,K branch,j )) Among them, K branch,j is the convolution kernel of the jth branch, I branch is the extracted multi-branch feature. The output of each branch is expressed as: F bi =ReLU(BN(W bi *F prev +b bi )) Among them, F prev is the output of the previous layer, W bi and b bi are the weight and bias of the ith branch.

9. The tumor rapid identification system based on multi-scale lightweight neural network according to claim 8 is characterized in that: The multi-scale lightweight fast neural network uses global average pooling to aggregate the spatial dimensions of each feature channel, expressed as: Where W and H are the width and height of the feature map, respectively. The multi-scale lightweight fast neural network uses a fully connected layer for final classification, expressed as: O=σ(W f ·F GAP +b f ) Among them, W f is the weight of the fully connected layer, b f is the bias, σ is the sigmoid activation function, which makes the output fall between [0,1], indicating the probability of binary classification.

10. The tumor rapid identification system based on multi-scale lightweight neural network according to claim 4, characterized in that: In the process of training the multi-scale lightweight fast neural network, the hyperparameters α and γ were set to 3 and 2 respectively. The multi-scale lightweight fast neural network was trained for 100 iterations using the training set, and the optimal model was screened out using the test set.

Images