Method for constructing a deep learning model for cell detection in multi-center pathology images
By constructing a deep learning model for multi-center pathological image cell detection, and using random feature enhancement modules and domain invariant information consistency learning, the problem of model performance degradation caused by pathological image heterogeneity is solved, and the generalization performance of the model is improved and clinical deployment is simplified.
Patent Information
- Application Number
- CN202310410092.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-17
- Publication Date
- 2025-08-22
- Estimated Expiration
- 2043-04-17
AI Technical Summary
The application of existing deep learning models in multicenter pathological images is affected by pathological image heterogeneity, resulting in reduced performance and difficult to deploy in clinical practice.
A deep learning model for cell detection of multicenter pathological image is adopted, the original image features are extracted through a shared encoder, and the features are enhanced by a random feature enhancement module. Combined with the domain-invariant information consistency learning strategy, a cell detection model is constructed.
Improves the generalization performance of the model, simplifies network design, reduces computational costs, and is easy to deploy in clinical practice.
Smart Images
Figure CN116524323B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image analysis technology, and in particular, to a method for constructing a deep learning model for multi-center pathological image cell detection. Background Art
[0002] Cancer is one of the leading causes of death worldwide. Pathology, which relies on pathologists observing cell morphology under a microscope, is considered the "gold standard" for treatment planning. However, there is a significant shortage of pathologists, especially experienced pathologists. This shortage and the growing demand for pathology diagnosis have led to an immense workload for pathologists, resulting in unavoidable missed or misdiagnoses. Therefore, the use of computer technology to assist pathologists in pathology diagnosis has become a trend.
[0003] The development of deep learning technology has demonstrated significant advantages over traditional machine learning methods in assisting pathology diagnosis. Immunohistochemical markers such as Ki67, Her2, and ER are crucial diagnostic indicators for pathologists, requiring them to count different cell types, a significant workload. By feeding pathology images into a trained deep learning model, the model can automatically locate and classify cells, significantly reducing the workload for pathologists and improving the efficiency of pathology diagnosis.
[0004] However, the deployment of deep learning models for cell detection in clinical practice has been challenging. The heterogeneity of pathology images has hindered their clinical application. The pathology diagnostic process primarily consists of four stages: sampling, slide preparation, slide reading, and reporting. Notably, the concentration of the stain used, as well as the model and parameter settings of the digital instrument used to scan the slides, can affect the quality of the final slides. This leads to heterogeneity in pathology images across multiple centers, or institutions, manifesting as variations in cell staining tones, such as red, purple, or dark red. Furthermore, differences in morphology and size between the same cell type and disease types across different centers exacerbate this heterogeneity. This heterogeneity in pathology images results in a decline in the performance of deep learning cell detection models trained on data from a single center when applied to pathology data from other centers, making them difficult to deploy in the clinic. Summary of the Invention
[0005] In order to overcome at least one deficiency in the prior art, the present application provides a method for constructing a deep learning model for multi-center pathological image cell detection.
[0006] In a first aspect, a method for constructing a deep learning model for cell detection in multi-center pathology images is provided, comprising:
[0007] Obtaining a model construction dataset, which includes multiple immunohistochemical pathology images from multiple centers;
[0008] Constructing a cell detection deep learning model and a random feature enhancement module; the cell detection deep learning model includes a shared encoder, a first decoder, and a second decoder; the shared encoder is used to extract original image features; the random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple feature-enhanced image features; the first decoder is used to determine cell positioning results based on the original image features and the multiple feature-enhanced image features; the second decoder is used to determine cell classification results based on the original image features and the multiple feature-enhanced image features; the cell positioning results and the cell classification results constitute the cell detection results;
[0009] The cell detection deep learning model is trained based on the model construction data set to obtain a trained cell detection deep learning model.
[0010] In one embodiment, the random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple enhanced image features, including:
[0011] Perform multiple random feature enhancement operations on the original image features; each random feature enhancement operation uses the following method:
[0012] The original image features are convolved with a randomly determined convolution kernel to obtain the disturbed features;
[0013] The perturbed features are fused with the original image features using a randomly determined mixing factor to obtain feature-enhanced image features.
[0014] In one embodiment, the perturbed features are fused with the original image features using a randomly determined mixing factor to obtain feature-enhanced image features using the following formula:
[0015]
[0016] in, is the image feature after feature enhancement, β is a randomly determined mixing factor, f rc is the feature after perturbation, and f is the original image feature.
[0017] In one embodiment, a cell detection deep learning model is trained based on a model-built dataset, and the loss function used is:
[0018]
[0019] Among them, L consistis the loss function, n is the sample number, N is the number of samples in the model construction data set, To calculate the cross entropy loss between the cell detection results and the cell labeling results, j is the label of the image feature after feature enhancement, J is the number of image features after feature enhancement, is the cell detection result corresponding to the image feature after the j-th feature enhancement of the n-th sample, y n is the cell labeling result of the nth sample, μ is the balance factor, is the cell detection result corresponding to the original image feature of the nth sample.
[0020] In one embodiment, the first decoder includes multiple convolutional layers and pooling layers; the second decoder includes multiple convolutional layers, pooling layers and Sigmoid layers.
[0021] In a second aspect, a device for constructing a deep learning model for multi-center pathology image cell detection is provided, comprising:
[0022] A data set acquisition module is used to acquire a model construction data set, which includes multiple immunohistochemical pathology images from multiple centers;
[0023] A model construction module is used to construct a cell detection deep learning model and a random feature enhancement module; the cell detection deep learning model includes a shared encoder, a first decoder, and a second decoder; the shared encoder is used to extract original image features; the random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple feature-enhanced image features; the first decoder is used to determine cell positioning results based on the original image features and the multiple feature-enhanced image features; the second decoder is used to determine cell classification results based on the original image features and the multiple feature-enhanced image features; the cell positioning results and the cell classification results constitute the cell detection results;
[0024] The model training module is used to train the cell detection deep learning model based on the model construction data set to obtain the trained cell detection deep learning model.
[0025] In one embodiment, the random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple enhanced image features, including:
[0026] Perform multiple random feature enhancement operations on the original image features; each random feature enhancement operation uses the following method:
[0027] The original image features are convolved with a randomly determined convolution kernel to obtain the disturbed features;
[0028] The perturbed features are fused with the original image features using a randomly determined mixing factor to obtain feature-enhanced image features.
[0029] In one embodiment, the perturbed features are fused with the original image features using a randomly determined mixing factor to obtain feature-enhanced image features using the following formula:
[0030]
[0031] in, is the image feature after feature enhancement, β is a randomly determined mixing factor, f rc is the feature after perturbation, and f is the original image feature.
[0032] In one embodiment, a cell detection deep learning model is trained based on a model-built dataset, and the loss function used is:
[0033]
[0034] Among them, L consist is the loss function, n is the sample number, N is the number of samples in the model construction data set, To calculate the cross entropy loss between the cell detection results and the cell labeling results, j is the label of the image feature after feature enhancement, J is the number of image features after feature enhancement, is the cell detection result corresponding to the image feature after the j-th feature enhancement of the n-th sample, y n is the cell labeling result of the nth sample, μ is the balance factor, is the cell detection result corresponding to the original image feature of the nth sample.
[0035] In one embodiment, the first decoder includes multiple convolutional layers and pooling layers; the second decoder includes multiple convolutional layers, pooling layers and Sigmoid layers.
[0036] Compared with the prior art, this application has the following beneficial effects:
[0037] (1) Immunohistochemical pathology images are affected by factors such as dye concentration, scanning equipment type and parameters, and morphological differences of similar cells in different diseases. Immunohistochemical pathology images have the characteristics of heterogeneity. This application adopts a random feature enhancement module to increase the breadth of source domain data in the feature space and a domain-invariant information consistency learning strategy to enable the cell detection model to learn more generalized feature representations, thereby improving the generalization performance of the pathology image cell detection model.
[0038] (2) This application adopts a random feature enhancement module, which can be plug-and-play. Compared with existing methods, it does not require sophisticated design, has low computational cost, and is easy to implement.
[0039] (3) The random feature enhancement module used in this application only works in the model training phase and is not called in the inference phase. It does not increase the computational complexity of cell detection model inference, which is beneficial for clinical deployment. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The present application may be better understood by referring to the following description in conjunction with the accompanying drawings, which together with the following detailed description are incorporated into and form a part of this specification. In the drawings:
[0041] Figure 1 A flowchart of a method for constructing a deep learning model for multi-center pathology image cell detection according to an embodiment of the present application is shown;
[0042] Figure 2 shows a flowchart of model training;
[0043] Figure 3 A structural block diagram of a multi-center pathology image cell detection deep learning model construction device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0044] Exemplary embodiments of the present application are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of actual embodiments are described in this specification. However, it should be understood that in the process of developing any such actual embodiment, many implementation-specific decisions may be made to achieve the developer's specific goals, and these decisions may vary from one implementation to another.
[0045] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show the device structure closely related to the solution according to the present application, while other details that are not closely related to the present application are omitted.
[0046] It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the accompanying drawings. In this document, where feasible, the embodiments may be combined with each other, features between different embodiments may be replaced or borrowed, and one or more features may be omitted in one embodiment.
[0047] In response to the problems of different distributions of multi-center immunohistochemical pathology images due to differences in dye concentration control and imaging instruments, and the difficulty in deploying deep learning models in clinical practice, the existing technology uses source domain data to generate pseudo samples at the image level for data enhancement. This method requires careful design of the data generation network and has high computational costs. The present application provides a method for constructing a deep learning model for cell detection in multi-center pathology images. This method can improve the generalization performance of the deep learning model for cell detection in multi-center pathology images. It adopts a plug-in feature enhancement module that is plug-and-play, simplifies the design of the network model, and only works during the model training process, which is convenient for deployment in clinical practice.
[0048] Figure 1 A flowchart of a method for constructing a deep learning model for multi-center pathology image cell detection according to an embodiment of the present application is shown. Figure 1 , methods include:
[0049] Step S1, obtaining a model construction dataset, where the model construction dataset includes multiple immunohistochemical pathology images from multiple centers;
[0050] In this step, full-field digital immunohistochemistry slides are collected from multiple centers. Single-center data refers to pathology slides acquired at the same magnification using the same equipment at the same hospital, which can be from different patients. Image blocks captured from the full-field digital slides at 20x field of view form the dataset. A pathologist then uses point annotation to identify positive tumor cells, negative tumor cells, and other cell types in each image block. The annotated data is then divided into n sub-datasets based on the data source. One domain is randomly selected as the test set, and the remaining n-1 domains are used as the training set. Here, the multiple centers can be different hospitals or medical institutions.
[0051] Step S2, constructing a cell detection deep learning model and a random feature enhancement module; the cell detection deep learning model includes a shared encoder, a first decoder, and a second decoder; the shared encoder is used to extract original image features; the random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple feature-enhanced image features; the first decoder is used to determine a cell localization result based on the original image features and the multiple feature-enhanced image features; the second decoder is used to determine a cell classification result based on the original image features and the multiple feature-enhanced image features; the cell localization result and the cell classification result constitute a cell detection result; Figure 2 A flowchart of model training is shown.
[0052] In this step, the shared encoder outputs image features at multiple scales. The input of the random feature enhancement module is the intermediate layer features output by the shared encoder, and the output is the image features after feature enhancement that are consistent with the length, width, and number of channels of the input features. The random feature enhancement module retains the domain-invariant information of the input intermediate features by using random convolution, while perturbing the domain-specific information. The image features after feature enhancement are sent to the first decoder and the second decoder together with the original image features to complete cell detection.
[0053] Here, the first decoder includes multiple 3x3 convolutional layers and pooling layers, and the output of this module is used for cell localization; the second decoder includes multiple 3x3 convolutional layers, pooling layers and Sigmoid layers, which are used to predict the probability values of cells belonging to positive tumor cells, negative tumor cells and other cells. The output of the second decoder is used for cell classification.
[0054] Step S3: Train the deep learning model for cell detection based on the model-built dataset to obtain a trained deep learning model for cell detection. Here, an end-to-end approach is used for training, employing a domain-invariant information consistency learning strategy to learn and capture the content information of similar cells in different domains.
[0055] In the above embodiment, a random feature enhancement module is used to increase the breadth of the source domain data in the feature space, and the domain-invariant information consistency learning strategy prompts the cell detection model to learn more generalized feature representations, thereby improving the generalization performance of the pathological image cell detection model; the random feature enhancement module is used, which can be plug-and-play. Compared with existing methods, it does not require sophisticated design, has low computational cost, and is easy to implement; the random feature enhancement module used only works in the model training phase and is not called in the inference phase, and does not increase the computational complexity of the cell detection model inference, which is beneficial to clinical deployment.
[0056] In one embodiment, in step S2, the random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple enhanced image features, including:
[0057] Multiple random feature enhancement operations are performed on the original image features; here, specifically three random feature enhancement operations can be performed to obtain three feature-enhanced image features.
[0058] Each random feature enhancement operation can be performed in the following ways:
[0059] First, the original image features are convolved with a randomly determined convolution kernel to obtain the perturbed features. The following formula can be used:
[0060] f rc =f*Θ
[0061] Among them, f rc is the perturbed feature, f is the original image feature, Θ is the randomly determined convolution kernel. Here, an initial convolution size pool P = [1, 3, ...] can be set, and the convolution size k is randomly selected from the convolution size pool as the convolution size of the randomly determined convolution kernel.
[0062] Then, the perturbed features are fused with the original image features using a randomly determined mixing factor to obtain the enhanced image features, which can be obtained using the following formula:
[0063]
[0064] in, is the image feature after feature enhancement, β is a randomly determined mixing factor, β~U(0,l), where U is uniformly distributed, l is a hyperparameter of the weak feature mixer, indicating the strength of feature mixing, f rc is the feature after perturbation, and f is the original image feature.
[0065] In one embodiment, in step S3, the cell detection deep learning model is trained based on the model construction dataset, and the loss function used is:
[0066]
[0067] Among them, L consisy is the loss function, n is the sample number, N is the number of samples in the model construction data set, To calculate the cross entropy loss between the cell detection results and the cell labeling results, j is the label of the image feature after feature enhancement, J is the number of image features after feature enhancement, J can be 3, is the cell detection result corresponding to the image feature after the j-th feature enhancement of the n-th sample, u n is the cell labeling result of the nth sample, μ is the balance factor, which can be 0.1. is the cell detection result corresponding to the original image feature of the nth sample.
[0068] In this embodiment, the original image features and the image features after multiple feature enhancements are sequentially sent to the first decoder and the second decoder to obtain the corresponding cell detection results respectively; in the model training process, a domain-invariant information consistency loss L is added. consist , so that the trained model has better generalization performance.
[0069] In addition, the trained model can be tested using a test set, and the Hungarian algorithm can be used to match the model-predicted cells with the labeled cells. The precision, recall, and F1 index of the prediction results can be calculated, and the cell detection results can be visualized.
[0070] Based on the same inventive concept as the method for constructing a deep learning model for multi-center pathology image cell detection, this embodiment also provides a corresponding device for constructing a deep learning model for multi-center pathology image cell detection. Figure 3 The following is a block diagram of a multi-center pathology image cell detection deep learning model construction device according to an embodiment of the present application, including:
[0071] A data set acquisition module 31 is used to acquire a model construction data set, where the model construction data set includes multiple immunohistochemical pathology images from multiple centers;
[0072] The model construction module 32 is used to construct a cell detection deep learning model and a random feature enhancement module; the cell detection deep learning model includes a shared encoder, a first decoder, and a second decoder; the shared encoder is used to extract original image features; the random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple feature-enhanced image features; the first decoder is used to determine a cell localization result based on the original image features and the multiple feature-enhanced image features; the second decoder is used to determine a cell classification result based on the original image features and the multiple feature-enhanced image features; the cell localization result and the cell classification result constitute a cell detection result;
[0073] The model training module 33 is used to train the cell detection deep learning model based on the model construction data set to obtain a trained cell detection deep learning model.
[0074] In this embodiment, a random feature enhancement module is used to increase the breadth of the source domain data in the feature space, and the domain-invariant information consistency learning strategy prompts the cell detection model to learn more generalized feature representations, thereby improving the generalization performance of the pathological image cell detection model; the random feature enhancement module is used, which can be plug-and-play. Compared with existing methods, it does not require sophisticated design, has low computational cost, and is easy to implement; the random feature enhancement module used only works in the model training phase and is not called in the inference phase, and does not increase the computational complexity of the cell detection model inference, which is beneficial to clinical deployment.
[0075] The above descriptions are merely examples of various embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A method for constructing a deep learning model for multi-center pathological image cell detection, characterized in that: include: Acquiring a model construction dataset, wherein the model construction dataset includes multiple immunohistochemical pathology images from multiple centers; Build a deep learning model for cell detection and a random feature enhancement module; The cell detection deep learning model includes a shared encoder, a first decoder, and a second decoder; the shared encoder is used to extract original image features; the random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple feature-enhanced image features; the first decoder is used to determine cell positioning results based on the original image features and the multiple feature-enhanced image features; the second decoder is used to determine cell classification results based on the original image features and the multiple feature-enhanced image features; The cell positioning result and the cell classification result constitute a cell detection result; Training the cell detection deep learning model based on the model construction data set to obtain a trained cell detection deep learning model; The random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple feature-enhanced image features, including: Perform multiple random feature enhancement operations on the original image features; each random feature enhancement operation adopts the following method: Performing a convolution operation on the original image features using a randomly determined convolution kernel to obtain disturbed features; Fusing the disturbed features with the original image features using a randomly determined mixing factor to obtain feature-enhanced image features; The perturbed features are fused with the original image features using a randomly determined mixing factor to obtain feature-enhanced image features, using the following formula: in, is the image feature after feature enhancement, is a randomly determined mixing factor, is the characteristic after perturbation, is the original image feature.
2. The method according to claim 1, wherein in, The cell detection deep learning model is trained based on the dataset constructed by the model, and the loss function used is: in, is the loss function, n is the sample number, The number of samples in the dataset for model construction, To calculate the cross entropy loss between the cell detection results and the cell labeling results, j is the label of the image feature after feature enhancement, J is the number of image features after feature enhancement, is the cell detection result corresponding to the image feature after the j-th feature enhancement of the n-th sample, The cell labeling result for the nth sample, is the balance factor, is the cell detection result corresponding to the original image feature of the nth sample.
3. The method according to claim 1, wherein The first decoder includes multiple convolutional layers and pooling layers; the second decoder includes multiple convolutional layers, pooling layers and Sigmoid layers.
4. A multi-center pathology image cell detection deep learning model construction device, characterized in that: include: A data set acquisition module is used to acquire a model construction data set, wherein the model construction data set includes multiple immunohistochemical pathology images from multiple centers; A model construction module is used to construct a cell detection deep learning model and a random feature enhancement module; the cell detection deep learning model includes a shared encoder, a first decoder, and a second decoder; the shared encoder is used to extract original image features; the random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple feature-enhanced image features; the first decoder is used to determine cell positioning results based on the original image features and the multiple feature-enhanced image features; the second decoder is used to determine cell classification results based on the original image features and the multiple feature-enhanced image features; The cell positioning result and the cell classification result constitute a cell detection result; A model training module, configured to train the cell detection deep learning model based on the model construction data set to obtain a trained cell detection deep learning model; The random feature enhancement module is used to perform random feature enhancement on the original image features to obtain multiple feature-enhanced image features, including: Perform multiple random feature enhancement operations on the original image features; each random feature enhancement operation adopts the following method: Performing a convolution operation on the original image features using a randomly determined convolution kernel to obtain disturbed features; Fusing the disturbed features with the original image features using a randomly determined mixing factor to obtain feature-enhanced image features; The perturbed features are fused with the original image features using a randomly determined mixing factor to obtain feature-enhanced image features, using the following formula: in, is the image feature after feature enhancement, is a randomly determined mixing factor, is the characteristic after perturbation, is the original image feature.
5. The device according to claim 4, characterized in that in, The cell detection deep learning model is trained based on the dataset constructed by the model, and the loss function used is: in, is the loss function, n is the sample number, The number of samples in the dataset for model construction, To calculate the cross entropy loss between the cell detection results and the cell labeling results, j is the label of the image feature after feature enhancement, J is the number of image features after feature enhancement, is the cell detection result corresponding to the image feature after the j-th feature enhancement of the n-th sample, The cell labeling result for the nth sample, is the balance factor, is the cell detection result corresponding to the original image feature of the nth sample.
6. The device according to claim 4, characterized in that The first decoder includes multiple convolutional layers and pooling layers; the second decoder includes multiple convolutional layers, pooling layers and Sigmoid layers.