A breast image classification method and device based on classification auxiliary information

Abstract

The present invention relates to a breast image classification method and device based on molecular typing auxiliary information, which is used to determine the histological grade of an image to be classified. The method comprises the following steps: obtaining an image to be classified and preprocessing the image to be classified; splitting the preprocessed image to be classified into multiple sequence data, each of which serves as input to different classification models, and fusing the classification results of each classification model to obtain a final classification label; wherein the classification model includes a main network and an auxiliary supervision branch based on molecular typing auxiliary information vectors, the main network includes a multi-scale feature extraction layer, the auxiliary supervision branch adjusts and processes the intermediate output features of the different scale feature extraction layers, and the output results of the main network and the auxiliary supervision branch are weighted and fused to form the classification result of the classification model. Compared with the existing technology, the present invention has the advantages of high accuracy and high efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing automation, and relates to a medical image classification method, in particular to a breast image classification method and device based on classification auxiliary information. Background Art

[0002] Even with the rapid development of modern medical technology, breast cancer has surpassed lung cancer to become the world's leading cancer. Pathological examination, the gold standard for breast cancer diagnosis, lays the foundation for precise diagnosis and treatment, as well as the development of personalized treatment plans. Histological grading, which provides a morphological assessment of the tumor, is crucial and serves as an independent prognostic factor that helps estimate patient prognosis and predict the risk of recurrence.

[0003] Currently, the internationally accepted Scarff-Bloom-Richardson breast cancer histological grading system calculates the total score of three parameters: the proportion of glandular ducts, nuclear pleomorphism, and mitotic count. Breast cancer histological grading is divided into three levels: Grade I, 3–5 points; Grade II, 6–7 points; and Grade III, 8–9 points. Numerous studies have shown that breast cancer histological grading, based on histomorphological classification, correlates with molecular subtypes, which reflect gene expression status. Compared with grades I and II, high-grade breast cancer is less differentiated, has a greater risk of malignant metastasis, a poorer prognosis, and a lower proportion of the luminal epithelial subtype. Therefore, it is necessary to further explore the underlying relationships between breast cancer histological grading and molecular subtypes to improve the consistency between pathological diagnosis and clinical decision-making.

[0004] Currently, pathologists manually identify the histological grade of breast cancer based on morphological information such as cell structure observed in hematoxylin-eosin-stained sections. Due to human subjectivity and the complexity and tediousness of pathological image analysis, it is difficult to ensure the accuracy and timeliness of identification. With the development of digital pathology and the support of computer-aided diagnosis, high-performance artificial intelligence models have powered the histological grading of breast cancer, including imaging omics algorithms based on feature engineering and highly parallelized deep learning algorithms. Even so, the analysis of pathological sections before applying these methods requires a puncture biopsy. In addition, operational steps such as paraffin sectioning and hematoxylin-eosin staining further increase the research burden. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a breast image classification method and device based on classification auxiliary information with high accuracy and efficiency.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A breast image classification method based on classification auxiliary information is used to determine the histological grade of the image to be classified. The method comprises the following steps:

[0008] Acquire an image to be classified, and preprocess the image to be classified;

[0009] The pre-processed image to be classified is split into multiple sequence data, which are used as inputs of different classification models respectively. The classification results of each classification model are fused to obtain the final classification label.

[0010] The classification model includes a main network and an auxiliary supervision branch based on the molecular typing auxiliary information vector. The main network includes a multi-scale feature extraction layer. The auxiliary supervision branch adjusts and processes the intermediate output features of the feature extraction layers of different scales. The output results of the main network and the auxiliary supervision branch are weighted and fused to form the classification result of the classification model.

[0011] Furthermore, the training data set used in training the classification model includes sample images and histological grade and molecular typing label information corresponding to the sample images, and the molecular typing auxiliary information vector is constructed based on the training data set.

[0012] Furthermore, the molecular typing auxiliary information vector is constructed by the following steps:

[0013] Based on the molecular typing label information of each sample in the training data set, a two-node information vector satisfying the Gaussian distribution is constructed to achieve information initialization;

[0014] Random Gaussian noise is added to the dual-node information vector, and the node values ​​are normalized to form the molecular typing auxiliary information vector.

[0015] Furthermore, the molecular typing label information includes luminal epithelial type and non-luminal epithelial type.

[0016] Furthermore, the preprocessing includes histogram equalization, cropping and intensity normalization.

[0017] Furthermore, the main network is built based on a two-dimensional convolutional neural network model, including a post-convolution activation block based on octave convolution, multiple main modules with both SEnet and SKnet excitations, and Dense-ASP containing dilated convolution layers with different dilation rates. 3 Module, multiple main modules form the multi-scale feature extraction layer.

[0018] Furthermore, the auxiliary supervision branch includes multiple GAP global average pooling layers and MS attention modules that are respectively connected to feature extraction layers of different scales.

[0019] Furthermore, when training the classification model, the loss function adopted by the main network is a cost-sensitive loss function based on the category F1-score, and the category F1-score is a model performance evaluation indicator composed of the harmonic mean of precision and recall; the loss function adopted by the auxiliary supervision branch is the Kullback-Leibler divergence minimization loss function.

[0020] Furthermore, when the classification results of each classification model are integrated, the weights of different classification models are determined based on the prediction accuracy of each classification model.

[0021] The present invention also provides an electronic device comprising one or more processors, a memory and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the breast image classification method based on typing auxiliary information as described above.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention designs an auxiliary supervision branch that incorporates molecular typing to promote the correlation characteristics between model learning and molecular typing estimation, effectively improving the performance of the model, improving image classification accuracy, and high efficiency.

[0024] 2. In the design of network model structure, the present invention proposes a method named IOS 2 -DA net two-dimensional convolutional neural network model. Among them, the octave convolution that replaces the traditional convolution layer weakens the model's attention to low-frequency redundant information and enhances the recognition rate of pathological grades with similar imaging features. The dual-core compression excitation module of SEnet and SKnet imitates the human visual encoding characteristics and realizes the extraction of information that is conducive to the final decision. In addition, the Dense-ASP of this network 3 The module makes full use of dense multi-scale features to enhance the learning ability of the model.

[0025] 3. For DCE-MRI images of different time series, this paper proposes a cost-sensitive loss function based on the category F1 score. This loss function fully utilizes the model's balance between precision and sensitivity for different categories to automatically weight the samples. Compared with traditional cross-entropy loss and focal loss, this loss function significantly improves the classification recall rate while maintaining specificity, resulting in higher robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the process of the present invention;

[0027] Figure 2A two-dimensional convolutional neural network 10S in the embodiment 2 -Schematic diagram of DA net structure;

[0028] Figure 3 Schematic diagram of the structure of the post-convolution activation block based on octave convolution in the embodiment;

[0029] Figure 4 Schematic diagram of the structure of the main module SE_Inception_SK in the embodiment;

[0030] Figure 5 Dense-ASP in the embodiment 3 Schematic diagram of the module structure;

[0031] Figure 6 A schematic diagram of the overall structure of a network with a classification auxiliary supervision branch in the embodiment DETAILED DESCRIPTION

[0032] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0033] like Figure 1 As shown, this embodiment provides a breast image classification method based on classification auxiliary information, which is used to determine the histological grade of the image to be classified. The method includes the following steps:

[0034] Acquire an image to be classified, and perform preprocessing on the image to be classified, wherein the preprocessing includes histogram equalization, cropping, and intensity normalization;

[0035] The preprocessed image to be classified is split into multiple sequence data, such as DCE-MRI TPs 1, TPs 2, and TPs 3 sequences, which are used as inputs of different classification models respectively. The classification results of each classification model are fused to obtain the final classification label.

[0036] like Figure 6 As shown, the classification model used in this embodiment is a network with an auxiliary supervision branch for typing, including a main network and an auxiliary supervision branch based on a molecular typing auxiliary information vector. The main network includes a multi-scale feature extraction layer, and the auxiliary supervision branch adjusts and processes the intermediate output features of the feature extraction layers of different scales. The output results of the main network and the auxiliary supervision branch are weighted and fused to form the classification result of the classification model.

[0037] like Figure 2 As shown, in this embodiment, the main network is built based on a two-dimensional convolutional neural network model and is named IOS 2-DAnet, including post-convolution activation blocks based on octave convolution, multiple main modules with both SEnet and SKnet excitations, and Dense-ASP with dilated convolution layers with different dilation rates 3 Module, multiple main modules form the multi-scale feature extraction layer. Figure 3 As shown in the figure, octave convolution is used to replace the traditional convolution layer to construct the post-convolution activation block. The hyperparameter α is used to control the ratio of high-frequency and low-frequency channels in the convolution layer, thereby weakening the model's attention to low-frequency redundant information. While enhancing the recognition rate of pathological grades with similar imaging features, it also reduces the consumption of memory and computing resources. Figure 4 As shown, the dual-core compression excitation module of SEnet and SKnet is inserted into Inception to form IOS 2 -DA net's main module SE_Inception_SK can adaptively select and fuse the appropriate receptive field sizes on different branches and the weight parameters of different convolution channels to extract key information that is beneficial to the final decision. In this embodiment, there are three main modules. Figure 5 As shown, combining the characteristics of multi-degree information and dilated convolution, the embodiment of the present invention also proposes Dense-ASP 3 The module densely connects dilated convolutional layers with different dilation rates (r=1, r=2) to generate more densely distributed multi-scale features and enhance learning capabilities.

[0038] like Figure 6 As shown in this embodiment, the auxiliary supervision branch includes multiple GAP global average pooling layers and MS attention modules that are connected to feature extraction layers of different scales. The GAP global average pooling layer takes the features from IOS 2 -DA net three multi-scale IOS 2 The output features of the layers are concatenated, and the Kullback-Leibler (KL) divergence minimization loss function is used to adjust the output features of the above intermediate layers, so as to obtain the prediction value that is most relevant to the typing information vector.

[0039] The training dataset used by the above classification model during the model training phase includes sample images and the histological grade and molecular typing label information corresponding to the sample images. The molecular typing auxiliary information vector is constructed based on the training dataset. The molecular typing auxiliary information vector used by the auxiliary supervision branch is obtained by the following steps:

[0040] (1) Information initialization. Based on the molecular typing markers of each sample in the breast cancer histological grading dataset, a two-node information vector that satisfies the Gaussian distribution is constructed. The calculation formula is:

[0041]

[0042] Here, x corresponds to a node in the information vector, and s represents the molecular subtype (0 for non-luminal epithelial type and 1 for luminal epithelial type).

[0043] (2) Add random noise. Add random Gaussian noise to the initialized classification information vector to simulate the different classification diagnoses of the same lesion by radiologists with different reading experience. This is shown in the following formula:

[0044]

[0045] I' s (x)=I s (x)+r s (x) (3)

[0046] (3) Normalize node values. As shown in the following formula:

[0047]

[0048] The prediction values ​​obtained by the auxiliary supervision branch are consistent with IOS 2 -The DA net's initial prediction results are weighted to obtain the classification results of a classification model:

[0049] p=(1-γ)·p1+γ·p2,0<γ<0.5 (5)

[0050] The prediction results based on DCE-MRI TPs 1, TPs 2, and TPs 3 are integrated and fused by weighted averaging to obtain the final classification label. The fusion process is as follows:

[0051] (1) Take the prediction accuracy of each base model as the classification performance weight:

[0052]

[0053] (2) The comprehensive prediction probability of the sample is obtained by weighted summation of the prediction results of each base learner:

[0054]

[0055] (3) Determine the prediction label of the integrated model based on the P value:

[0056]

[0057] In this embodiment, the training data set of the above-mentioned classification model in the training phase is constructed in the following way. In this embodiment, a total of 381 cases were enrolled in the group and diagnosed with breast cancer by pathological biopsy, and all of them had undergone at least one preoperative MRI imaging examination before diagnosis. Finally, 256 cases that met the inclusion criteria of the study were obtained. The pathologists gave the histological grade scores of the above-mentioned cases with reference to the Nottingham histological grading system, with a total of 9 cases of grade I; 113 cases of grade II; and 133 cases of grade III. The cases were labeled according to the immunohistochemical characteristics of the molecular typing, including luminal epithelial type and non-luminal epithelial type. Taking into account the imbalance between the sample size of low-grade histological grades and the medium and high grades, in this embodiment, the above-mentioned breast cancer histological grading method was applied to the histological grade prediction study of grades I & II and III.

[0058] After intercepting the lesion area in the breast DCE-MRI image, image preprocessing was performed to establish a breast cancer histological grading dataset. The preprocessing methods are as follows:

[0059] (1) After determining the image sequence segment containing the lesion area, the image block of the region of interest was intercepted at multiple scales with the lesion location as the center, and the bilinear interpolation algorithm was used to resize the square image block to 64*64 pixels.

[0060] (2) Use histogram equalization to enhance image contrast.

[0061] (3) Intensity normalization is used to scale the pixel values ​​of the image blocks to [0, 1] to facilitate network processing and analysis.

[0062] (4) After dividing the breast cancer histological grading dataset by ten-fold cross-validation, the data enhancement method provided by the Python deep learning library Keras was used to perform real-time data enhancement operations (rotation, mirroring, scaling, etc.) only on the training set, and the validation set was not processed.

[0063] Before training the classification model, set the batch size to 64 and the initial learning rate to l r =0.002, and l2 weight regularization is added to each convolution kernel with a coefficient of 0.005. During model training, the RMSprop optimizer is used to accelerate the optimization of model parameters. The cost-sensitive loss function CFSL based on the category F1-score is used to increase the model's attention to difficult samples. The expression is:

[0064]

[0065] Among them, F1-score is a model performance evaluation indicator composed of the harmonic mean of precision and recall, and the modulation term D(x) = x ββ is a weight factor that measures the impact of F1-score.

[0066] The initial number of training epochs is 120. If the CFSL does not decrease after 10 epochs of training, the learning rate is reduced to 0.2. If the CFSL still does not converge after 30 epochs of fine-tuning, the model training is terminated. The model weights that achieve the highest accuracy and lowest loss on the validation set are saved.

[0067] During the model evaluation phase, the present invention first trained a base learner model based on DCE-MRI image sequences TPs 1, TPs 2, and TPs 3. The hyperparameter α was adjusted to determine the optimal ratio of high- and low-frequency channels within the network model structure. The training loss function was fixed to the focal loss (α = 1, β = 2). The experimental results are shown in Table 1.

[0068] Table 1 The impact of different ratios of high and low frequency channels in octave convolution on prediction results

[0069]

[0070] As shown in Table 1, when the hyperparameter α of the base learner model based on the sequence images of TPs 1, TPs 2, and TPs 3 is selected as 0.5, 0.375, and 0.5 respectively, the model achieves a good balance between recognition accuracy and video memory usage, and the classification accuracy is as high as 86.6%.

[0071] Secondly, based on the above experiments, the embodiment of the present invention tested the impact of different loss functions on model prediction, and the results are shown in Table 2.

[0072] Table 2 Evaluation of model performance by different loss functions

[0073]

[0074]

[0075] As can be seen from Table 2, when the loss functions are selected as the classic cross entropy loss, focal loss and CFSL respectively, a prediction model with a relatively balanced Sen and Spec should be selected as much as possible under the premise of ensuring that the model accuracy is high enough. When the CFSL with a weight factor β = 0.3 is set, the base learner model based on the DCE-MRI TPs 1 sequence image has the best prediction performance, with an AUC of up to 0.902 and an F1-score of 0.906. Similarly, in the base learner model based on the DCE-MRI TPs 3 sequence image, the embodiment of the present invention selects the training weight under the CFSL loss with a weight factor β = 2.0 as the optimal weight. For the base learner model based on the DCE-MRI TPs 2 sequence image, although the model performance when trained with the optimal weight factor (β = 2.0) of CFSL is slightly inferior to the focal loss and cross entropy loss with the optimal weight factor (β = 1.0), the difference between the model's Sen and Spec is the smallest, and the performance is more stable. Therefore, the embodiment of the present invention still considers that the CFSL loss with a weight factor β=2.0 is suitable for the base learner model based on the DCE-MRI TPs 2 sequence image.

[0076] Finally, this example uses molecular typing auxiliary information to construct a model-assisted supervision branch to reduce false positives in the prediction results. The experimental results are shown in Table 3.

[0077] Table 3 The impact of the proportion of auxiliary information on the final prediction results

[0078]

[0079]

[0080] As shown in Table 3, after the molecular typing auxiliary supervision branch is introduced, the model can reduce false positives while maintaining good AUC and F1-score performance. That is, in actual application, the model has a lower probability of misclassifying grade I & II as grade III, which can effectively assist clinicians in formulating more accurate treatment plans.

[0081] The above evaluation shows that the embodiment of the present invention can achieve the purpose of accurately predicting pathological grade by automatically extracting features related to various histological grades in breast cancer DCE-MRI imaging through a two-dimensional convolutional neural network. Compared with the method of using only pathological images for analysis, this method reduces tedious steps such as puncture biopsy sampling, and has higher recognition efficiency and accuracy. At the same time, the introduction of molecular typing auxiliary information further improves the predictive performance of the model.

[0082] If the above-mentioned image classification method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0083] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A breast image classification method based on classification auxiliary information, characterized in that: For determining the histological grade of an image to be classified, the method comprises the following steps: Acquire an image to be classified, and preprocess the image to be classified; The pre-processed image to be classified is split into multiple sequence data, which are used as inputs of different classification models respectively. The classification results of each classification model are fused to obtain the final classification label. The classification model includes a main network and an auxiliary supervision branch based on the molecular typing auxiliary information vector. The main network includes a multi-scale feature extraction layer. The auxiliary supervision branch adjusts and processes the intermediate output features of the different scale feature extraction layers. The output results of the main network and the auxiliary supervision branch are weighted and fused to form the classification result of the classification model. The main network is built based on a two-dimensional convolutional neural network model, including a post-convolution activation block based on octave convolution, multiple main modules with both SEnet and SKnet excitations, and Dense-ASP with dilated convolution layers with different dilation rates. 3 Module, multiple main modules form the multi-scale feature extraction layer; The auxiliary supervision branch includes multiple GAP global average pooling layers and MSattention modules that are respectively connected to feature extraction layers of different scales.

2. The breast image classification method based on typing auxiliary information according to claim 1, characterized in that: The training data set used in the classification model training includes sample images and histological grade and molecular typing label information corresponding to the sample images. The molecular typing auxiliary information vector is constructed based on the training data set.

3. The breast image classification method based on typing auxiliary information according to claim 2, characterized in that: The molecular typing auxiliary information vector is constructed by the following steps: Based on the molecular typing label information of each sample in the training data set, a two-node information vector satisfying the Gaussian distribution is constructed to achieve information initialization; Random Gaussian noise is added to the dual-node information vector, and the node values ​​are normalized to form the molecular typing auxiliary information vector.

4. The breast image classification method based on typing auxiliary information according to claim 2, characterized in that: The molecular typing label information includes luminal epithelial type and non-luminal epithelial type.

5. The breast image classification method based on typing auxiliary information according to claim 1, characterized in that: The preprocessing includes histogram equalization, clipping and intensity normalization.

6. The breast image classification method based on typing auxiliary information according to claim 1, characterized in that: When training the classification model, the loss function adopted by the main network is a cost-sensitive loss function based on the category F1-score. The category F1-score is a model performance evaluation indicator composed of the harmonic mean of precision and recall; the loss function adopted by the auxiliary supervision branch is the Kullback-Leibler divergence minimization loss function.

7. The breast image classification method based on typing auxiliary information according to claim 1, characterized in that: When the classification results of each classification model are integrated, the weights of different classification models are determined based on the prediction accuracy of each classification model.

8. An electronic device, characterized in that: The system comprises one or more processors, a memory and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the breast image classification method based on typing auxiliary information as claimed in any one of claims 1 to 7.

Images