HER2 image classification method and system based on lightweight neural network

Abstract

This invention proposes a HER2 image classification method and system based on a lightweight neural network, relating to the field of image processing technology. The method comprises the following steps: constructing a lightweight neural network model, inputting the HER2 image to be classified, performing preliminary feature extraction on the image to be classified to obtain a preliminary image feature map; repeatedly performing lightweight feature extraction on the preliminary image feature map to obtain multi-scale features of the HER2 image; inputting the multi-scale features of the HER2 image into a classifier, and outputting the classification results. The invention designs a lightweight, highly modular network architecture, enabling this method to achieve lower computational costs and higher recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a HER2 image classification method and system based on a lightweight neural network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Breast cancer is the second most common cancer after lung cancer and the leading cause of cancer death in women. Human epidermal growth factor receptor 2 (HER2) is a protein that is highly aggressive in breast cancer. HER2 has been shown to be overexpressed through gene amplification, thereby enhancing the proliferation of cancer cells.

[0004] The expression level of HER2 can be determined by immunohistochemistry (IHC) or fluorescence in situ hybridization (FISH). Due to the low cost and simple operation of IHC testing, IHC is generally used first to detect HER2 in breast cancer specimens. HER2 positivity is closely related to the use of targeted drugs. When HER2 is negative (0 or 1+), targeted drugs are not used. When the HER2 test is weakly positive (2+), a more accurate FISH test is performed to determine the use of targeted drugs. If HER2 is strongly positive (3+), targeted drugs are used. Therefore, the correct classification of the IHC image score of HER2 has positive significance for the treatment of breast cancer.

[0005] HER2 immunohistochemical staining provides a simple and reliable method for clinical diagnosis and prognosis. However, reviewing large numbers of IHC images is labor-intensive and time-consuming. Furthermore, discordance often exists between different observers due to personal experience and knowledge. Therefore, modern developments have led to the use of automated computer-assisted assessment to provide consistent and objective evaluation of HER2 expression and reduce the burden on clinical pathologists.

[0006] With the rapid development of information technology, the use of artificial intelligence (AI) methods can help automate HER2 assessment. However, the inventors have discovered that existing AI methods for HER2 assessment are computationally complex, computationally expensive, or exhibit poor performance, failing to strike a balance between computational cost and performance. Summary of the Invention

[0007] To overcome the shortcomings of the above-mentioned prior art, the present invention provides a HER2 image classification method and system based on a lightweight neural network, and designs a lightweight, highly modular network architecture. The network is repeatedly composed of a downsampling module and a feature extraction module. By using group convolution, void convolution and channel shuffling methods, the downsampling module and the feature extraction module are designed to be a multi-branch structure with different inputs and different scale feature extraction. Based on deep learning, the method does not require excessive manual intervention, and the lightweight multi-branch structure module enables the method to have lower computational cost and higher recognition accuracy.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of the present invention provides a HER2 image classification method based on a lightweight neural network.

[0010] The HER2 image classification method based on lightweight neural network includes the following steps:

[0011] A lightweight neural network model is constructed, and the HER2 image to be classified is input. Preliminary feature extraction is performed on the HER2 image to be classified, and a preliminary feature map of the image is obtained.

[0012] Repeated lightweight feature extraction of the preliminary feature map of the image to obtain multi-scale features of the HER2 image;

[0013] The multi-scale features of the HER2 image are input into the classifier and the classification results are output.

[0014] A second aspect of the present invention provides a HER2 image classification system based on a lightweight neural network.

[0015] HER2 image classification system based on lightweight neural network, including:

[0016] The preliminary feature processing module is configured to: construct a lightweight neural network model, input the HER2 image to be classified, perform preliminary feature extraction on the HER2 image to be classified, and obtain a preliminary feature map of the image;

[0017] The multi-scale feature extraction module is configured to: repeatedly perform lightweight feature extraction on the preliminary feature map of the image to obtain multi-scale features of the HER2 image;

[0018] The classification result acquisition module is configured to: input the multi-scale features of the HER2 image into the classifier and output the classification result.

[0019] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the HER2 image classification method based on a lightweight neural network as described in the first aspect of the present invention.

[0020] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the HER2 image classification method based on a lightweight neural network as described in the first aspect of the present invention are implemented.

[0021] One or more of the above technical solutions have the following beneficial effects:

[0022] 1. Based on deep learning methods, this invention improves the automation level of HER2 score discrimination. Relying on the lightweight design of the model framework, lightweight feature extraction is repeated on the initial feature map of the image, greatly reducing the computational cost during model operation. At the same time, the lightweight network model used in this invention has fewer than 2 million training parameters and only 0.8 GFLOPs, both of which are lower than similar methods. While the computational cost is low, the recognition accuracy of this invention for HER2 scores can reach 96.92%.

[0023] 2. When performing lightweight feature extraction, the present invention uses the maximum pooling layer and the average pooling layer in parallel. The maximum pooling layer focuses on extracting texture information, and the average pooling layer focuses on extracting background information, thereby making the obtained abstract features richer and allowing the network to freely select better features.

[0024] 3. When performing lightweight feature extraction, the present invention reduces the amount of calculation through grouped convolution and channel shuffling. The use of grouped convolution can reduce the number of operations on the feature map. The use of one channel shuffle replaces the use of a 1×1 convolution kernel, reduces parameters, and avoids the isolation of feature extraction caused by grouped convolution of different pooling layers.

[0025] 4. The present invention uses dilated convolution to replace the traditional convolution layer. Compared with the traditional convolution layer, the superposition of a small number of dilated convolutions with different expansion rates can obtain a larger local receptive field. The use of dilated convolution not only reduces the parameters of the model, but also improves the performance of the model.

[0026] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0028] Figure 1 A flow chart of the method of the first embodiment;

[0029] Figure 2 The original IHC images of the four different HER2 score categories in Example 1;

[0030] Figure 3 HER2 image blocks before and after image preprocessing in Example 1;

[0031] Figure 4 2 is a structural diagram of the lightweight neural network model (LWNet) of Example 1;

[0032] Figure 5 This is a schematic diagram of the lightweight feature extraction process of Example 1;

[0033] Figure 6 Schematic diagram of the verification results of the LWNet model before and after lightweighting in Example 1;

[0034] Figure 7 Schematic diagram of the verification accuracy of the LWNet model of Example 1 and other lightweight models;

[0035] Figure 8 Schematic diagram of verification loss of the LWNet model of Example 1 and other lightweight models;

[0036] Figure 9 This is a system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0037] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0038] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0039] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0040] The overall idea proposed by the present invention is:

[0041] In order to address the shortcomings of the existing methods, the present invention provides a HER2 image classification method based on a lightweight neural network. The lightweight neural network of the method designs a lightweight and highly modular network architecture, namely, a lightweight feature processing module. The lightweight feature processing module is repeatedly composed of a downsampling part and a multi-scale feature extraction part, and by using group convolution, void convolution and channel shuffling methods, the downsampling part and the multi-scale feature extraction part are designed to be a multi-branch structure with different inputs and different scale feature extractions.

[0042] The lightweight feature processing module (LW) is repeatedly used to process image features. The first half of the LW structure is the downsampling part. In the downsampling part, the input feature maps are first grouped. Then, different pooling layers are used in parallel to abstract the feature maps of different groups to highlight the strong feature information of the image and eliminate weak feature information. At the output, the feature maps of different groups are shuffled and then grouped to avoid the isolation of feature information. The second half of the LW structure is the multi-scale feature extraction part, which uses mixed void convolutions with different receptive fields in parallel. Each branch receives a group of feature maps, fuses the input image feature information separately, and then extracts different scale information of the image separately. At the output, the feature maps of different groups are shuffled and input to the next layer.

[0043] The grouping operation and channel shuffling operation reduce the reuse of feature maps, greatly reducing the computational cost of the model; based on deep learning, this method does not require excessive human intervention, and the lightweight multi-branch structure module gives this method lower computational cost and higher recognition accuracy.

[0044] Explanation of terms:

[0045] FLOPs: Floating point operations (s stands for plural), meaning the number of floating-point operations performed, or the amount of computation. It can be used to measure the complexity of an algorithm or model.

[0046] GFLOPs = 1 × 10 9 FLOPs.

[0047] Example 1

[0048] This embodiment discloses a HER2 image classification method based on a lightweight neural network.

[0049] like Figure 1 As shown, the HER2 image classification method based on lightweight neural network includes the following steps:

[0050] A lightweight neural network model is constructed, and the HER2 image to be classified is input. Preliminary feature extraction is performed on the HER2 image to be classified, and a preliminary feature map of the image is obtained.

[0051] Repeated lightweight feature extraction of the preliminary feature map of the image to obtain multi-scale features of the HER2 image;

[0052] The multi-scale features of the HER2 image are input into the classifier and the classification results are output.

[0053] Furthermore, the method also includes preprocessing the HER2 IHC image, specifically, cropping, denoising and feature enhancement processing of the HER2 IHC image.

[0054] Furthermore, preliminary feature extraction is performed on the HER2 images to be classified, specifically:

[0055] The HER2 image to be classified is input into a convolutional layer with a size of 3×3, a stride of 2, and a convolution kernel of 64 for preliminary feature extraction to obtain a preliminary feature map of the image.

[0056] Furthermore, the preliminary feature map of the image is repeatedly subjected to lightweight feature extraction to obtain the multi-scale features of the HER2 image, specifically:

[0057] The preliminary feature maps of the image are initially grouped and input into different pooling layers for feature abstraction. The output feature maps of different groups are subjected to a channel shuffle and grouped twice. The feature maps after the second grouping are input in parallel into the mixed convolution layer and the mixed void convolution layer with different receptive fields. The feature maps output by the mixed convolution layer and the mixed void convolution layer are subjected to a second channel shuffle. The second channel shuffled feature maps are spliced ​​together in the channel dimension to obtain the multi-scale features of the HER2 image.

[0058] Furthermore, the preliminary feature maps of the image are grouped and input into different pooling layers for feature abstraction, specifically:

[0059] The preliminary feature maps of the image are initially grouped into two groups and input into the maximum pooling layer and the average pooling layer respectively, with a pooling size of 3×3 and a stride of 2.

[0060] Furthermore, the feature maps after secondary grouping are input in parallel into the mixed convolution layer and the mixed hole convolution layer with different receptive fields, specifically:

[0061] The secondary grouping divides the feature maps into 4 groups, which are input in parallel into the mixed convolutional layers and mixed hole convolutional layers with 4 different receptive fields. The local receptive fields of the mixed convolutional layers and mixed hole convolutional layers are in an increasing form.

[0062] Furthermore, the multi-scale features of the HER2 image are input into the classifier, and the classification results are output, specifically:

[0063] The multi-scale features of the HER2 image are input into an average pooling layer, and then a fully connected layer is used to receive and process the information of each node in the previous step. A softmax layer is used after the fully connected layer to output the classification result.

[0064] Specifically, when using the lightweight neural network model of the present invention, users can perform targeted weight training based on their own HER2 images, or directly load the weight data trained by technicians. After the network model loads the weight data, it can directly perform HER2 image recognition. The workflow is shown in Figure 1 .

[0065] For ease of understanding, the weight training principle is briefly summarized as follows: Assume that the input of the neuron is the vector The weight vector is The activation function is the f function, then its output y:

[0066]

[0067] Assume that t is the actual result, g function is the loss function, x i is a vector The i-th item, w i Is the same as input c i The corresponding weight term, β is the learning rate, Δw i Is the weight update value, then:

[0068] Δw i =β·g(t,y)·x i (2)

[0069] w i (new)=w i (old)+Δw i (3)

[0070] Loss can be understood as the absolute value of the "difference" between the true sample value and the output value of the neural network. The purpose of network model training is to minimize the loss, or even to zero. In an ideal state, the loss is approximately 0, that is, g(t, y) is approximately equal to 0, and the weights are basically not updated at this time. However, due to the differences between images, the loss cannot be equal to 0 in reality, but it will oscillate within a range close to 0. In this case, it can be understood that the network model training has achieved the best result it can achieve. In addition to using the training set, a validation set is also needed to observe the training status and avoid overfitting (overfitting will make the trained weights only valid for the training set). The following are the relevant settings for the experiment:

[0071] 1) Experimental environment

[0072] All models were trained in a cluster environment using a single GPU. The GPU was a GeForce RTX2080 Ti with a compute capability of 7.5 and 12GB of graphics memory. All experiments used the Keras deep learning library within the TensorFlow framework. FLOPs for all models were calculated using the Keras-flops package.

[0073] 2) Optimization and iteration

[0074] All models used the Cross Loss (Adam) optimizer. The initial learning rate and learning rate decay were adjusted based on the training experience of each model, while other parameters were set to default values. Due to inconsistent convergence rates across models, all models were trained 100 times, retaining the neuron weights corresponding to the iteration with the highest validation accuracy. The training batch size for all models was set to 32.

[0075] The specific representation of the cross entropy loss function is shown in (1) and (2). x represents the input sample, represents the predicted probability distribution of the model, y represents the true probability distribution, and n represents the total number of categories. represents the predicted probability of the i-th category, y i represents the true probability of the i-th class.

[0076]

[0077]

[0078] like Figure 2 As shown in the dataset, HER2 images with a score of 0 are from the Stanford Tissue Microarray Database (resolution 1504×1440), while the other three types of HER2 images are from real data in the project (resolution 1584×1584). Due to the large image resolution, the images were segmented into non-overlapping patches (each 360×360 in size). Finally, valid patches were selected as the experimental dataset. The dataset contains 5200 images, 1300 per category. 300 images per category were selected as the test dataset, and the remaining 1000 images were divided into training and validation datasets in an 8:2 ratio.

[0079] Image preprocessing primarily involves cropping, denoising, and enhancing features of the original image. Since neural network models automatically scale images to the model's size limits upon input, cropping the image to an appropriate size first can help prevent information loss caused by scaling.

[0080] Noise reduction and feature enhancement processing are key parts of the image preprocessing module. Reasonable noise reduction of the image is conducive to eliminating interference factors during image feature extraction, and feature enhancement is helpful for the accuracy of image feature extraction in subsequent processes. It can effectively improve the accuracy of HER2 score classification. Since the HER2 score depends on the degree of cell membrane staining, and the staining color presented is brownish yellow. Therefore, the noise reduction processing method of this scheme is: first, transfer the HER2 image from the RGB color space to the HSV color space, and then adjust the range values ​​of the H, S, and V components to maintain the desired color. After experimental verification, it is determined that the following component ranges are appropriate: 0≤H≤36,28≤S≤255,0≤V≤255. Unmatched pixel components are set to 0. Since there are differences between HER2 images that are not from the same database, technicians can adjust the component range according to their actual situation. Finally, the image is transferred back to the RGB color space. This process effectively removes the effects of color, light, and saturation, and enhances the brownish yellow color features. As Figure 3 As shown, this process effectively removes the effects of color, light, and saturation, leaving only the desired sepia-toned characteristics.

[0081] The initial feature extraction module, lightweight feature processing module and classification module constitute the entire deep network framework, that is, the lightweight neural network model (LWNet) in the present invention. The specific structure of the LWNet model is shown in Figure 4 The model consists of a 3×3 convolutional kernel, five LW structures, a 4×4 average pooling layer, and a fully connected layer. The number of fully connected nodes is 1024, and the dropout rate is set to 0.3. ReLU is used as the activation function, and a softmax layer is used at the end of the model.

[0082] Multi-branch networks can simultaneously extract features at different scales, making them suitable for HER2 classification. In HER2 IHC images, because each cell has a different degree of membrane staining and cells with different staining levels are unevenly distributed throughout the image, extracting features at a single scale can overlook significant detail. Multi-branch networks, however, utilize multiple branches with different receptive fields in parallel, allowing them to extract features at different scales within the same layer, improving the robustness of automated HER2 evaluation.

[0083] The lightweight feature processing module corresponds to the multiple reused LW structures in the neural network model of the present invention. Figure 4 As shown in Figure 1, the LW architecture consists of a downsampling component in the first half and a multi-scale feature extraction component in the second half. The downsampling component has two branches: the maximum pooling layer and the average pooling layer. The multi-scale feature extraction component has four branches, each with an increasing local receptive field. The combination of these two modules increases the complexity of the model, giving it a variety of receptive fields.

[0084] In the downsampling phase, the maximum pooling layer and average pooling layer are used in parallel, and grouped convolution and channel shuffling are used to reduce computational complexity. The pooling size is 3×3, and the stride is 2. The maximum pooling layer focuses on extracting texture information, while the average pooling layer focuses on extracting background information. The simultaneous use of different downsampling layers enriches the abstract features, allowing the network to freely select better features. Grouped convolution reduces the number of feature map operations. Channel shuffling replaces the 1×1 convolution kernel, reducing parameters and avoiding the isolation of feature extraction in each branch caused by grouped convolution.

[0085] The multi-scale feature extraction component consists of four branches with different local receptive fields. Branch 1 has a single 3×3 convolution kernel, providing a local receptive field of 3. Branch 2 consists of two stacked 3×3 convolution kernels, providing a receptive field of 5. Branch 3 consists of two dilated convolutions with dilation rates of 1 and 2, respectively, providing a receptive field of 7. Branch 4 consists of three dilated convolutions with dilation rates of 1, 2, and 3, providing a receptive field of 13. All convolution kernels have a stride of 1 and are zero-padded to ensure the consistency of the output feature map size. Due to the uneven distribution of cells and different cell sizes in HER2 IHC images, the parallel use of branch structures with different receptive fields is beneficial for fully extracting image feature information and improving the performance of HER2 automatic evaluation.

[0086] The multi-scale feature extraction component uses grouped convolution, dilated convolution, and channel shuffling to reduce computational complexity. Grouped convolution reduces the number of feature map operations, while channel shuffling replaces the 1×1 convolution kernel, reducing parameters and avoiding the isolation of feature extraction between branches caused by grouped convolution. Dilated convolution replaces traditional convolutional layers. Compared to traditional convolutional layers, stacking a small number of dilated convolutions with different dilation rates can achieve a larger local receptive field. Using dilated convolution not only reduces model parameters but also improves performance.

[0087] like Figure 5 The figure shows the details of the grouped convolution, channel shuffle, and dilated convolution in the LW structure. The input feature map of the downsampling part is divided into two equal groups (grouped convolution) and input to two branches of different pooling layers. The output feature maps of the two branches are then shuffled into four groups (channel shuffle), concatenated, and input to the multi-scale feature extraction part.

[0088] The input feature map of the multi-scale feature extraction part is divided into four equal groups (group convolution) and then input into four different branches. The output feature maps of the four branches are then shuffled into two groups (channel shuffling). Finally, all feature maps are spliced ​​together in the channel dimension and input into the next structure. Using a single hole convolution will produce a checkerboard effect, resulting in the loss of feature information. Therefore, the present invention adopts hybrid hole convolution (HDC). The expansion rates of the stacked dilated convolution are r1 = [1, 2] and r2 = [1, 2, 3], respectively, which meet the conditions of HDC.

[0089] In this embodiment, preliminary feature extraction of the input image provides multiple feature maps for the subsequent lightweight feature processing module, ensuring that the lightweight feature processing module has sufficient feature maps for grouping operations upon image input. This preliminary feature extraction module also helps reduce model depth and computational cost. The module details use a convolutional layer with a size of 3×3, a stride of 2, and a convolution kernel of 64.

[0090] In this embodiment, the lightweight feature extraction process is performed using a lightweight feature processing module. Specifically, in the downsampling part of the lightweight feature processing module, the preliminary feature maps of the image are initially grouped and respectively input into different pooling layers for feature abstraction, and the output feature maps of different groups are subjected to a channel shuffle and a secondary grouping; in the multi-scale feature extraction part, the feature maps after the secondary grouping are input in parallel into the mixed convolution layer and the mixed void convolution layer with different receptive fields, the feature maps output by the mixed convolution layer and the mixed void convolution layer are subjected to a secondary channel shuffle, and the secondary channel shuffled feature maps are spliced ​​together in the channel dimension to obtain the multi-scale features of the HER2 image.

[0091] Finally, the multi-scale features of the HER2 image are classified. The classification module fits the features extracted in the previous step and then determines the category of the input HER2 IHC image. The classification process is as follows: first, a 4×4 average pooling layer is used to directly integrate the information of each feature map into a node. Then, a fully connected layer with 1024 neurons is used to receive and process the information of the previous nodes. During runtime, the neurons in the fully connected layer are randomly dropped, with a random dropout rate of 0.3. After the fully connected layer, a softmax layer, a normalized exponential function, is used. It maps the outputs of multiple neurons to the range (0, 1), which can be interpreted as probabilities, thereby performing multi-classification.

[0092] like Figure 6As shown in Table 1, comparative experiments analyze the impact of the lightweighting method of the present invention on model performance and computational cost. Resnet50 is a deep network framework with a 50-layer structure; non-lightweight refers to a network model that is consistent with the model framework of the present invention but does not use group convolution, void convolution, and channel shuffling methods.

[0093] Comparing the results with those of the present invention, the non-lightweight model using the framework of the present invention and the method of the present invention achieved higher accuracy than the 50-layer ResNet50. This demonstrates that the framework of the present invention is beneficial for HER2 classification. Furthermore, comparing the present invention with the non-lightweight model, the present invention significantly reduces the model's computational cost without compromising performance. This demonstrates the feasibility of the lightweighting method of the present invention.

[0094] Table 1 Comparison of LWNet model before and after lightweighting

[0095] Accuracy (%) Training parameters GFLOPs Resnet50 93.08 23,516,228 8.20 Non-lightweight model 96.83 36,802,644 14.70 The present invention model LWNet 96.91 1,934,572 0.80

[0096] The classification of HER2 scores was compared using the method of the present invention and other lightweight model methods. All models were run in the same environment and used the same preprocessed dataset.

[0097] In the automatic evaluation of HER2, ResNeXt50 is used as a regular network to highlight the difference between lightweight networks and regular networks. Figure 7 、 Figure 8 As shown in Table 2, the computational cost (training parameters and GFLOPs) of the conventional network, ResNeXt50, is clearly an order of magnitude higher than that of the lightweight network. However, conventional networks have an advantage over lightweight networks in image recognition accuracy. ResNeXt50 significantly outperforms EfficientNetB0 and MobileNetV3 in image recognition. However, our method, LWNet, combines both recognition performance and low computational cost when classifying HER2 images. As a lightweight network, LWNet's recognition performance significantly outperforms the conventional network, ResNeXt50.

[0098] like Figure 7 、 Figure 8 As shown in Table 2, LWNet has better performance and similar computational cost compared with EfficientNetB0 and MobileNetV3 lightweight networks. EfficientNetB0 and MobileNetV3 are efficient lightweight networks that have emerged in recent years. However, Figure 7 、 Figure 8 Table 2 shows that LWNet outperforms these network models. This shows that the proposed method has both performance and computational cost for HER2 score classification.

[0099] Table 2 Comparison of LWNet and other lightweight models

[0100]

[0101] This paper compares recent deep learning methods for HER2 classification. All models are run in the same environment and use the same pre-processed dataset (image patch size is 360×360 pixels).

[0102] Among all HER2 scoring and classification methods, the model proposed in this paper has higher accuracy and lower computational cost than other methods (see Table 3). The second method is to segment the HER2 images. Due to the lack of segmented images processed by professionals, the accuracy of the second method cannot be measured. Compared with the WPNet method, LWNet has higher accuracy. We speculate that this is because LWPet uses dilated convolution, which increases the receptive field of the model and reduces the training parameters. Due to grouped convolution and channel shuffling, LWNet has much lower training parameters and GFLOPs than other algorithms. LWNet combines grouped convolution, channel shuffling, and dilated convolution to significantly reduce computational cost while ensuring model performance.

[0103] Table 3 Comparison of LWNet and existing methods for automatic HER2 assessment

[0104]

[0105] Example 2

[0106] This embodiment discloses a HER2 image classification system based on a lightweight neural network.

[0107] like Figure 9 As shown in the figure, the HER2 image classification system based on lightweight neural network includes:

[0108] The preliminary feature processing module is configured to: construct a lightweight neural network model, input the HER2 image to be classified, perform preliminary feature extraction on the HER2 image to be classified, and obtain a preliminary feature map of the image;

[0109] The multi-scale feature extraction module is configured to: repeatedly perform lightweight feature extraction on the preliminary feature map of the image to obtain multi-scale features of the HER2 image;

[0110] The classification result acquisition module is configured to: input the multi-scale features of the HER2 image into the classifier and output the classification result.

[0111] Example 3

[0112] The purpose of this embodiment is to provide a computer-readable storage medium.

[0113] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the HER2 image classification method based on a lightweight neural network as described in Example 1 of the present disclosure.

[0114] Example 4

[0115] The purpose of this embodiment is to provide an electronic device.

[0116] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the HER2 image classification method based on a lightweight neural network as described in Example 1 of the present disclosure are implemented.

[0117] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0118] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0119] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A HER2 image classification method based on a lightweight neural network, characterized in that: The following steps are involved: A lightweight neural network model is constructed, and the HER2 image to be classified is input. Preliminary feature extraction is performed on the HER2 image to be classified, and a preliminary feature map of the image is obtained. Repeated lightweight feature extraction of the preliminary feature map of the image to obtain multi-scale features of the HER2 image; Input the multi-scale features of the HER2 image into the classifier and output the classification results; The lightweight feature extraction specifically comprises the following steps: grouping the preliminary feature maps of the image for the first time and inputting them into different pooling layers for feature abstraction, performing a channel shuffling on the output feature maps of different groups and grouping them twice, inputting the feature maps after the secondary grouping in parallel into a mixed convolution layer and a mixed void convolution layer with different receptive fields, performing a secondary channel shuffling on the feature maps output by the mixed convolution layer and the mixed void convolution layer, and splicing the secondary channel shuffled feature maps together in the channel dimension to obtain multi-scale features of the HER2 image.

2. The HER2 image classification method based on a lightweight neural network according to claim 1, wherein: The method also includes preprocessing of the HER2 IHC image, specifically, cropping, denoising and feature enhancement processing of the HER2 IHC image.

3. The HER2 image classification method based on a lightweight neural network according to claim 1, wherein: Perform preliminary feature extraction on the HER2 images to be classified, specifically: The HER2 image to be classified is input into a convolutional layer with a size of 3×3, a stride of 2, and a convolution kernel of 64 for preliminary feature extraction to obtain a preliminary feature map of the image.

4. The HER2 image classification method based on a lightweight neural network according to claim 1, wherein: The image preliminary feature maps are initially grouped and input into different pooling layers for feature abstraction, specifically: The preliminary feature maps of the image are initially grouped into two groups and input into the maximum pooling layer and the average pooling layer respectively, with a pooling size of 3×3 and a stride of 2.

5. The HER2 image classification method based on lightweight neural network according to claim 1, characterized in that: The feature maps after secondary grouping are input in parallel into the mixed convolution layer and the mixed hole convolution layer with different receptive fields, specifically: The feature maps after secondary grouping into 4 groups are input in parallel into the hybrid convolutional layer and the hybrid void convolutional layer with 4 different receptive fields. The local receptive fields of the hybrid convolutional layer and the hybrid void convolutional layer are in an increasing form.

6. The HER2 image classification method based on lightweight neural network according to claim 1, characterized in that: The multi-scale features of the HER2 image are input into the classifier, and the classification results are output, specifically: The multi-scale features of the HER2 image are input into an average pooling layer, and then a fully connected layer is used to receive and process the information of each node in the previous step. A softmax layer is used after the fully connected layer to output the classification result.

7. A HER2 image classification system based on a lightweight neural network, characterized by: include: The preliminary feature processing module is configured to: construct a lightweight neural network model, input the HER2 image to be classified, perform preliminary feature extraction on the HER2 image to be classified, and obtain a preliminary feature map of the image; The multi-scale feature extraction module is configured to: repeatedly perform lightweight feature extraction on the preliminary feature map of the image to obtain multi-scale features of the HER2 image; The classification result acquisition module is configured to: input the multi-scale features of the HER2 image into the classifier and output the classification result; The lightweight feature extraction specifically comprises the following steps: grouping the preliminary feature maps of the image for the first time and inputting them into different pooling layers for feature abstraction, performing a channel shuffling on the output feature maps of different groups and grouping them twice, inputting the feature maps after the secondary grouping in parallel into a mixed convolution layer and a mixed void convolution layer with different receptive fields, performing a secondary channel shuffling on the feature maps output by the mixed convolution layer and the mixed void convolution layer, and splicing the secondary channel shuffled feature maps together in the channel dimension to obtain multi-scale features of the HER2 image.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the HER2 image classification method based on a lightweight neural network as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the HER2 image classification method based on a lightweight neural network are implemented as described in any one of claims 1 to 6.

Images