Immunohistochemical molecular expression prediction method, device, equipment and storage medium

Abstract

The invention discloses an immunohistochemical molecular expression prediction method and device, equipment and a storage medium, and relates to the technical field of medical image processing, and the method comprises the steps: obtaining a plurality of sub-images of lung tissue; each sub-image comprises any one of tumor tissues or non-tumor tissues; classifying the sub-images to obtain sub-images classified as tumor tissues as tumor sub-images; constructing an expression prediction model of the immunohistochemical molecules according to the tumor sub-images and the expression tags of the immunohistochemical molecules; and inputting each tumor sub-image into the expression prediction model to obtain an expression prediction result of the immunohistochemical molecules. According to the method, the expression result of the immunohistochemical molecule can be predicted according to the expression prediction model, cascade amplification of prediction errors by multiple models can be reduced, the sample size for training the expression prediction model is reduced, the prediction stability of the expression prediction model is improved, the expression prediction accuracy of the immunohistochemical molecule is improved, and the prediction accuracy of the immunohistochemical molecule is improved. And meanwhile, the calculation efficiency is also greatly improved.

Description

Technical Field

[0001] This application relates to the technical field of medical image processing, and in particular, to a method, device, equipment, and storage medium for predicting immunohistochemical molecular expression. Background Art

[0002] Currently, in clinical practice, pathological sections stained with hematoxylin and eosin (H&E) are routinely used to observe tissue morphology. However, for some cases with complex tissue morphology and low differentiation, it is difficult to determine the specific tissue morphology only by observing H&E-stained sections. At this time, immunohistochemistry (IHC) staining is usually required to detect the expression of specific molecular markers, and then to judge the tissue morphology.

[0003] Immunohistochemistry is a molecular detection technique based on the specific binding of antibodies and antigens. Its experimental process includes steps such as fixation, sectioning, antigen repair, antibody incubation, color reaction, and microscopic observation of tissue samples. IHC can qualitatively or quantitatively detect the expression status of specific proteins, but the existing immunohistochemical molecular detection techniques are still not accurate enough. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a method, device, equipment, and storage medium for predicting immunohistochemical molecular expression, so as to improve the accuracy of detecting whether immunohistochemical molecules are expressed.

[0005] To achieve the above purpose, on the one hand, an embodiment of this application proposes a method for predicting immunohistochemical molecular expression, and the method includes the following steps:

[0006] Obtain a plurality of sub-images of lung tissue; wherein, each of the sub-images includes either tumor tissue or non-tumor tissue;

[0007] Classify each of the sub-images, and obtain the sub-images classified as tumor tissue as tumor sub-images;

[0008] Construct an expression prediction model of the immunohistochemical molecule according to each of the tumor sub-images and the expression labels of the immunohistochemical molecule;

[0009] Input each of the tumor sub-images into the expression prediction model to obtain the expression prediction result of the immunohistochemical molecule.

[0010] In some embodiments, the obtaining a plurality of sub-images of lung tissue includes the following steps:

[0011] Obtain a whole-slide image of lung tissue and a tumor tissue annotation file;

[0012] Segment the whole-slide image into a plurality of sub-images of the same size according to the tumor tissue annotation file.

[0013] In some embodiments, the segmenting the whole-slide image into a plurality of sub-images of the same size according to the tumor tissue annotation file includes the following steps:

[0014] Generate a tumor mask according to the tumor tissue annotation file and the whole-slide image;

[0015] Segment the whole-slide image into non-tumor sub-images not covered by the tumor mask and tumor sub-images covered by the tumor mask; wherein, the sizes of the non-tumor sub-images and the tumor sub-images are the same.

[0016] In some embodiments, the classifying each of the sub-images to obtain the sub-images classified as tumor tissue as tumor sub-images includes the following steps:

[0017] Construct a binary classification model based on a convolutional neural network according to each of the sub-images and the corresponding classification labels of the sub-images;

[0018] Use the binary classification model to classify each of the sub-images to obtain the tumor sub-images.

[0019] In some embodiments, the constructing an expression prediction model of the immunohistochemical molecule according to each of the tumor sub-images and the expression labels of the immunohistochemical molecule includes the following steps:

[0020] Construct the expression prediction model based on a convolutional neural network according to each of the tumor sub-images and the expression labels of different immunohistochemical molecules.

[0021] In some embodiments, the inputting each of the tumor sub-images into the expression prediction model to obtain the expression prediction result of the immunohistochemical molecule includes the following steps:

[0022] Input each of the tumor sub-images into the expression prediction model to obtain an expression prediction probability vector of the immunohistochemical molecule by the expression prediction model; wherein, each component in the expression prediction probability vector corresponds to a positive expression probability of one of the immunohistochemical molecules;

[0023] Determine whether each of the immunohistochemical molecules is positively expressed according to the expression prediction probability vector as the expression prediction result.

[0024] In some embodiments, the determining whether each of the immunohistochemical molecules is positively expressed according to the expression prediction probability vector as the expression prediction result includes the following steps:

[0025] Convert the positive expression probabilities in the expression prediction probability vector that are greater than or equal to the clinical empirical threshold and the grid search optimal threshold into positive expression labels;

[0026] Convert the positive expression probabilities in the expression prediction probability vector that are less than the clinical empirical threshold and the grid search optimal threshold into negative expression labels; wherein, the positive expression labels and the negative expression labels are used as the expression prediction results.

[0027] To achieve the above object, on the other hand, an immunohistochemical molecular expression prediction device is proposed in an embodiment of the present application. The device includes:

[0028] An image acquisition unit, configured to acquire a plurality of sub-images of lung tissue; wherein, each of the sub-images includes any one of tumor tissue or non-tumor tissue;

[0029] An image classification unit, configured to classify each of the sub-images, and obtain the sub-images classified as tumor tissue as tumor sub-images;

[0030] A model construction unit, configured to construct an expression prediction model of the immunohistochemical molecule according to each of the tumor sub-images and the expression labels of the immunohistochemical molecule;

[0031] An expression prediction unit, configured to input each of the tumor sub-images into the expression prediction model to obtain the expression prediction result of the immunohistochemical molecule.

[0032] To achieve the above object, on the other hand, an electronic device is proposed in an embodiment of the present application. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above method is implemented.

[0033] To achieve the above object, on the other hand, a computer-readable storage medium is proposed in an embodiment of the present application. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0034] The embodiments of the present application at least include the following beneficial effects:

[0035] This application can obtain multiple sub-images of lung tissue; among them, each sub-image includes either tumor tissue or non-tumor tissue; classify each sub-image, and obtain the sub-images classified as tumor tissue as tumor sub-images; construct an expression prediction model of immunohistochemical molecules based on each tumor sub-image and the expression labels of immunohistochemical molecules; input each tumor sub-image into the expression prediction model to obtain the expression prediction result of immunohistochemical molecules. Compared with the existing multi-model prediction scheme and complex cross-domain image generation scheme, this application can predict the expression result of immunohistochemical molecules according to the expression prediction model, which can reduce the amplification of prediction errors by multiple models in cascade, reduce the sample size for training the expression prediction model, and improve the prediction stability of the expression prediction model, thereby improving the accuracy of the expression prediction of immunohistochemical molecules, and at the same time greatly improving the calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0037] Figure 1 It is a schematic flowchart of a method for predicting the expression of immunohistochemical molecules provided by an embodiment of this application;

[0038] Figure 2 It is an example flowchart for predicting the expression of IHC molecules based on H&E images provided by an embodiment of this application;

[0039] Figure 3 It is an example structural diagram of a convolutional neural network provided by an embodiment of this application;

[0040] Figure 4 It is a schematic structural diagram of a device for predicting the expression of immunohistochemical molecules provided by an embodiment of this application;

[0041] Figure 5 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] In order to make the objectives, technical solutions, and advantages of this application more clearly understood, the following further details this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit this application. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this application. They are merely examples of devices and methods that are consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0043] It can be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if", "when" as used herein may be interpreted as "when...", "while...", or "in response to determining".

[0044] The terms "at least one", "multiple", "each", "any one", etc. used in this application, at least one includes one, two, or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any one refers to any one of the multiple.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0046] Before elaborating on the embodiments of this application in detail, first, some terms and related technologies that may be involved in the embodiments of this application are described. The terms and related technologies involved in the embodiments of this application are applicable to the following explanations:

[0047] 1. Non-small cell lung cancer (NSCLC): Non-small cell lung cancer includes squamous cell carcinoma (SCC), adenocarcinoma, and large cell carcinoma. Compared with small cell carcinoma, its cancer cells grow and divide more slowly, and the spread and metastasis are relatively late. Non-small cell lung cancer accounts for about 80% of all lung cancers. Approximately 75% of patients are found to be in the middle and late stages when diagnosed, and the 5-year survival rate is very low.

[0048] 2. Hematoxylin and Eosin (H&E) staining method: One of the commonly used staining methods in paraffin section technology. The hematoxylin staining solution is alkaline and mainly stains the chromatin in the nucleus and the nucleic acid in the cytoplasm purple-blue; eosin is an acidic dye and mainly stains the components in the cytoplasm and extracellular matrix red. The HE staining method is the most basic and widely used technical method in histology, embryology, and pathology teaching and research.

[0049] 3. Whole Slide Image (WSI): Digital slides are obtained by using a fully automated microscope scanning system in combination with virtual slide software to scan and seamlessly stitch traditional glass slides to generate a whole digital slide (also known as a virtual slide) of the entire field of view (Whole Slide Image, abbreviated as WSI). In the practice of pathology medical treatment, teaching, and research, digital slides have all the functions of traditional slides and have the advantages of being unrestricted by space and time. Digital slides are not a static picture. They contain all the lesion information on the glass slide. On a computer, just like under a microscope, different magnifications can be observed (4x, 10x, 20x, 40x, 100x, etc.), and within a certain range (1X - 100X), seamless continuous magnification browsing of the slide can be achieved; and by uniformly classifying, archiving, and storing these massive digital slides in a server, a digital slide library is formed.

[0050] 4. Immunohistochemistry (IHC): Immunohistochemistry applies the basic principle of immunology - the antigen-antibody reaction, that is, the principle of specific binding of antigen and antibody. By chemical reaction, the chromogenic agent (fluorescein, enzyme, metal ion, isotope) of the labeled antibody is made to develop color to determine the antigen (polypeptide and protein) in tissue cells, and its localization, qualitative, and relative quantitative research is carried out, which is called immunohistochemistry technology or immunocytochemistry technology.

[0051] 5. Convolutional Neural Network (CNN): A type of feedforward neural network that contains convolutional calculations and has a deep structure, and is one of the representative algorithms of deep learning. Convolutional neural networks have the ability of feature learning and can perform translation-invariant classification on input information according to their hierarchical structure.

[0052] Currently, in clinical practice, pathological sections stained with hematoxylin and eosin (H&E) are routinely used to observe the morphological features of lung tissues. However, for some cases with complex tissue morphology and low differentiation, it is difficult to determine the specific morphology of lung tissues only by observing H&E-stained sections. At this time, immunohistochemistry (IHC) staining is usually needed to detect the expression of specific molecular markers, and then to judge the tissue morphology.

[0053] Immunohistochemistry is a molecular detection technique based on the specific binding of antibodies and antigens. Its experimental process includes steps such as tissue sample fixation, sectioning, antigen retrieval, antibody incubation, color reaction, and microscopic observation. IHC can qualitatively or quantitatively detect the expression status of specific proteins, but the existing immunohistochemical molecular detection techniques are still not accurate enough.

[0054] The IHC experiment has the following main limitations:

[0055] 1) Long experimental cycle: It usually takes several hours to one day from sample processing to result observation.

[0056] 2) High cost: The IHC experiment relies on expensive antibody reagents and professional equipment.

[0057] 3) Difficulty in multi-marker detection: The IHC experiment needs to detect multiple lung cancer molecular markers, so it requires relatively more tissue section samples. As the number of types of molecular markers increases, the available tissue samples from patients may be difficult to meet the IHC detection requirements. This contradiction is particularly prominent in small non-surgical biopsy samples. In addition, detecting multiple markers simultaneously in limited tissue samples faces technical and spatial limitations, and requires repeated section processing and staining.

[0058] 4) Strong subjectivity in evaluation: The IHC results are inherently subjective because they rely on the visual evaluation of pathologists on the localization of markers in tumors (such as nucleus, cytoplasm or membrane), staining intensity, and the percentage of positive cells.

[0059] Some technical solutions related to this application:

[0060] 1. Style transfer method:

[0061] First, the H&E image is stylized into an IHC image. The basic idea is that each image can be regarded as a combination of content and style. By calculating the style loss and content loss, the H&E image is iteratively updated to make its staining style similar to the IHC image and its content similar to the original H&E image. Then, on the pseudo IHC image generated after migration, the prediction of IHC molecular markers is performed. The main drawback is that style transfer relies on low-level statistics and often fails to capture semantic structures, and the generated pseudo IHC image is prone to losing tissue structure features.

[0062] 2. Cross-domain image generation method:

[0063] It generally refers to the generation algorithm based on the generative adversarial network (GAN). The GAN network consists of an encoder (Generator, G) and a decoder (Decoder, D). The image conversion network is used as G, and the image encoder is used as D. Among them, G is specifically used to generate pseudo images, and D is used to extract Content and Style information as loss to train G in turn. The IHC image corresponding to the H&E image is generated, and then the expression of IHC molecular markers is predicted. The disadvantage of this method is that the training process of the GAN network is relatively complex and prone to collapse problems.

[0064] To solve the bottleneck problems of long IHC experiment cycle, large sample demand, and high cost, and to make up for the deficiencies of existing computational alternatives in terms of image fitting degree, computational efficiency, and model training, this application proposes a solution for directly predicting the expression of immunohistochemical molecules from conventional H&E-stained pathological sections. This application combines deep learning and multi-instance learning techniques to accurately predict the expression status of immunohistochemical molecular markers by analyzing the rich morphological and structural information in H&E sections without the need for additional IHC experiments. This application has the following advantages: simplified process, reduced experimental cost, improved efficiency, and extended scope.

[0065] The embodiments of the present application provide a method, apparatus, device and storage medium for predicting the expression of immunohistochemical molecules. The technical solution of the present application includes: obtaining a plurality of sub-images of lung tissue; wherein each sub-image includes either tumor tissue or non-tumor tissue; classifying each sub-image to obtain the sub-images classified as tumor tissue as tumor sub-images; constructing an expression prediction model of immunohistochemical molecules according to each tumor sub-image and the expression label of the immunohistochemical molecule; and inputting each tumor sub-image into the expression prediction model to obtain the expression prediction result of the immunohistochemical molecule. Compared with the existing multi-model prediction scheme and complex cross-domain image generation scheme, the present application can predict the expression result of immunohistochemical molecules according to the expression prediction model, which can reduce the amplification of prediction errors by multiple models in cascade, reduce the sample size for training the expression prediction model, and improve the prediction stability of the expression prediction model, thereby improving the accuracy of predicting the expression of immunohistochemical molecules and greatly improving the calculation efficiency at the same time.

[0066] The embodiments of the present application provide a method, apparatus, device and storage medium for predicting the expression of immunohistochemical molecules, which relates to the technical field of medical image processing. The method, apparatus, device and storage medium for predicting the expression of immunohistochemical molecules provided by the embodiments of the present application can be applied to a terminal, or to a server, or can also be software running on a terminal or a server. In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto; the server side can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application for implementing the knowledge extraction method, etc., but is not limited to the above forms.

[0067] This application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are executed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0068] Referring to Figure 1 , an embodiment of this application provides an immunohistochemical molecular expression prediction method, which may include but is not limited to S100 to S130, specifically as follows:

[0069] S100: Obtain multiple sub-images of lung tissue; wherein, each of the sub-images includes either tumor tissue or non-tumor tissue.

[0070] Further, S100 may include S101 to S102:

[0071] S101: Obtain a whole-slide image of lung tissue and a tumor tissue annotation file;

[0072] S102: Segment the whole-slide image into multiple sub-images of the same size according to the tumor tissue annotation file.

[0073] More specifically, S102 may include S1021 to S1022:

[0074] S1021: Generate a tumor mask according to the tumor tissue annotation file and the whole-slide image;

[0075] S1022: Segment the whole-slide image into non-tumor sub-images not covered by the tumor mask and tumor sub-images covered by the tumor mask; wherein, each of the non-tumor sub-images and each of the tumor sub-images are of the same size.

[0076] S110: Classify each of the sub-images to obtain the sub-images classified as tumor tissue as tumor sub-images.

[0077] Further, S110 may include S111 to S112:

[0078] S111: Based on a convolutional neural network, construct a binary classification model according to each of the sub-images and the classification labels corresponding to the sub-images.

[0079] S112: Use the binary classification model to classify each of the sub-images to obtain the tumor sub-images.

[0080] S120: Construct an expression prediction model for the immunohistochemical molecule according to each of the tumor sub-images and the expression labels of the immunohistochemical molecule.

[0081] Further, S120 may include S121:

[0082] S121: Based on a convolutional neural network, construct the expression prediction model according to each of the tumor sub-images and the expression labels of different immunohistochemical molecules.

[0083] S130: Input each of the tumor sub-images into the expression prediction model to obtain the expression prediction result of the immunohistochemical molecule.

[0084] Further, S130 may include S131 - S132:

[0085] S131: Input each of the tumor sub-images into the expression prediction model to obtain an expression prediction probability vector of the immunohistochemical molecule by the expression prediction model; wherein, each component in the expression prediction probability vector corresponds to a positive expression probability of one of the immunohistochemical molecules.

[0086] S132: Determine whether each of the immunohistochemical molecules is positively expressed according to the expression prediction probability vector as the expression prediction result.

[0087] More specifically, S132 may include S1321 - S1322:

[0088] S1321: Convert the positive expression probabilities in the expression prediction probability vector that are greater than or equal to the clinical empirical threshold and the grid search optimal threshold into positive expression labels.

[0089] S1322: Convert the positive expression probabilities in the expression prediction probability vector that are less than the clinical empirical threshold and the grid search optimal threshold into negative expression labels; wherein, the positive expression labels and the negative expression labels are used as the expression prediction result.

[0090] Next, the solution of the embodiment of the present application will be introduced and described in detail in combination with specific application examples.

[0091] Exemplarily, the overall process of this embodiment is as Figure 2 shown.

[0092] The steps of this embodiment may include image reading, preprocessing, binary classification model construction, multi-label classification model construction, and IHC molecular positive expression prediction. The specific description is as follows:

[0093] 1) Read the whole slide image (WSI) stained with H&E through a computer program;

[0094] 2) Preprocess the WSI and cut it into several sub-images (Patches);

[0095] 3) Construct a binary classification model for normal tissue and tumor tissue based on deep learning methods;

[0096] 4) For the samples predicted as "tumor" by the binary classifier, construct a multi-label classifier by combining the IHC molecular expression labels. This step is significantly different from the existing methods: among them, the existing style transfer methods use statistical methods to first convert the H&E stained image into an IHC stained image, and then use the IHC molecular expression labels to construct a tumor region detection model and a molecular expression prediction model. Therefore, at least three types of models need to be constructed, and the prediction errors are cascaded and amplified; the existing cross-domain image generation algorithms use generative adversarial methods such as GAN networks and contrast learning methods. According to the paired H&E stained images and IHC stained images, generate "pseudo IHC stained" images, and then use the "pseudo IHC stained" images and IHC molecular expression labels to construct a tumor region detection model and a molecular expression prediction model. Therefore, in terms of data, H&E stained images, IHC molecular expression labels, and IHC stained images are required, and 3 types of models are constructed. The training of the GAN network is relatively difficult and the results are unstable; while the solution of this step only requires H&E stained images and IHC molecular expression labels, only 2 models need to be constructed, using a convolutional neural network structure that is relatively easy to train, and can synchronously predict any type and any combination of IHC molecules.

[0097] 5) IHC molecular positive expression prediction: The multi-label model outputs the positive prediction probability values (floating-point numbers between 0 and 1) of multiple IHC molecules, and judges the positive expression according to the threshold.

[0098] Next, more specific implementation manners will be described as follows:

[0099] First step, view the WSI image and its tumor region delineation information on the digital pathology image browsing software (such as ASAP software) on the computer side, read the WSI image (TIFF or SVS format) and the tumor region annotation file (XML format) through the Python toolkit OpenSlide, and generate a mask file for the tumor region.

[0100] Step 2: Preprocessing of WSI images. According to the WSI images and their corresponding Mask files, normal tissues (not covered by the Mask) and tumor regions (covered by the Mask) are extracted respectively, and all are uniformly cut into Patches of size 224×224 as the input of the convolutional neural network.

[0101] Step 3: Construct a binary classification model. Based on a convolutional neural network (such as the ResNet-50 model), a binary classification model Model1 is constructed using normal tissue Patches, tumor tissue Patches, and a label file (normal tissue: 0, tumor tissue: 1). The network structure of Model1 is as Figure 3 shown. The Patch is classified as normal or tumor. Both Ρ_normal and Ρ_tumor are decimals between 0 and 1 (can be equal to 0 or 1), and their sum is equal to 1.

[0102] Step 4: For the Patches predicted as tumors by the binary classification model Model1, an expression prediction model Model2 of IHC molecules is jointly constructed with the IHC molecular expression labels (0 or 1). Model2 is a multi-label classification model based on ResNet-50. Except for the final output, the rest of the network structure is the same as the Figure 3 network structure shown.

[0103] Specifically, Figure 3 is a schematic diagram of the network structure of Model1. Among them, ZeroPad means padding with 0; Conv means convolutional layer; BatchNorm means batch normalization layer; ReLU means ReLU activation function layer; MaxPool means max pooling layer; ConvBlock means convolutional block; IdBlock means IdentityBlock with the same input and output dimensions; ConvBlock means convolutional layer with different input and output dimensions; AvgPool means average pooling layer; Flatten means flattening vector layer; FC means Fullyconnected layer, that is, fully connected layer. The output of the model is a two-dimensional vector. P_normal represents the probability that the Patch is predicted as normal tissue, while Ρ 肿瘤 represents the probability that the Patch is predicted as tumor tissue.

[0104] Each Patch can be positively expressed with multiple molecular markers simultaneously, and its label form is as follows:

[0105] 〔〔110001000〕;

[0106] 〔010100010〕;

[0107] 〔000100100〕〕;

[0108] Among them, each row represents a Patch sample, each column represents an IHC molecular marker, 0 represents negative, and 1 represents positive.

[0109] The output of Model2 is a probability vector, and its components correspond to the probability values of positive expression of an IHC molecule. The probability values of positive expression of each molecule are between 0 and 1, and the sum of the positive expression probabilities of each molecule may be greater than 1. For example: [0.723, 0.432, 0.23, 0.455, 0.379, 0.002, 0.081, 0.062, 0.617].

[0110] In the fifth step, prediction of positive expression of IHC molecules. According to the clinical empirical threshold and the optimal threshold of grid search, the positive prediction probability of IHC molecules is converted into a positive expression label: those greater than or equal to the threshold are predicted as positive expression, otherwise predicted as negative expression, thereby determining the IHC molecular expression in lung tissue.

[0111] The beneficial effects of this embodiment include:

[0112] First of all, this embodiment can predict immunohistochemical molecular markers based on digital pathological sections of conventional H&E staining. Compared with other existing methods, this embodiment does not require IHC staining sections, thus fundamentally reducing the demand for tissue samples, saving the time and reagent costs of immunohistochemistry, and providing a feasible solution for predicting the expression of multiple IHC molecules in small biopsy samples. Secondly, compared with complex cross-domain image generation algorithms, this embodiment only needs to construct two homogeneous models - the two models only differ in the output layer, and the rest of the structures are exactly the same, greatly improving the computational efficiency and model stability. In addition, the expression prediction model of this embodiment can be easily extended to the prediction tasks of any type of IHC molecular marker and tumor molecular markers such as tumor drivers (such as PD-L1), and has a wide range of applications.

[0113] Refer to Figure 4 This application embodiment also provides an immunohistochemical molecular expression prediction device, which can implement the above-mentioned immunohistochemical molecular expression prediction method. The device includes:

[0114] An image acquisition unit, configured to acquire a plurality of sub-images of lung tissue; wherein, each of the sub-images includes either tumor tissue or non-tumor tissue.

[0115] An image classification unit, configured to classify each of the sub-images to obtain the sub-images classified as tumor tissue as tumor sub-images.

[0116] A model construction unit, configured to construct the immunohistochemical molecular expression prediction model according to each of the tumor sub-images and the expression labels of immunohistochemical molecules.

[0117] An expression prediction unit for inputting each of the tumor sub-images into the expression prediction model to obtain an expression prediction result of the immunohistochemical molecule.

[0118] It can be understood that the content in the above method embodiments is applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0119] An embodiment of the present application also provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above method for predicting the expression of immunohistochemical molecules. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0120] It can be understood that the content in the above method embodiments is applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0121] Please refer to Figure 5 , Figure 5 which shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0122] A processor 501, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0123] A memory 502, which can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 502, and the processor 501 is used to call and execute a method for predicting the expression of immunohistochemical molecules in the embodiments of the present application;

[0124] An input / output interface 503 for implementing information input and output;

[0125] A communication interface 504, which is used to implement the communication interaction between this device and other devices. It can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0126] A bus 505, which transmits information between various components of the device (such as a processor 501, a memory 502, an input / output interface 503, and a communication interface 504);

[0127] Among them, the processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are communicatively connected to each other inside the device through the bus 505.

[0128] The embodiment of the present application also provides a computer-readable storage medium. This computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned method for predicting immunohistochemical molecular expression.

[0129] It can be understood that the content in the above method embodiment is applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those of the above method embodiment.

[0130] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely provided relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0131] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation to the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0132] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine some steps, or different steps.

[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations.

[0135] As used in the specification of this application and the above drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0136] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0137] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0138] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store programs.

[0141] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of rights of the embodiments of the present application.

Claims

1. A method for predicting immunohistochemical molecule expression, characterized in that: The method comprises the following steps: Acquire multiple sub-images of lung tissue; wherein each of the sub-images includes any one of tumor tissue or non-tumor tissue; Classifying each of the sub-images to obtain the sub-image classified as tumor tissue as a tumor sub-image; Constructing an expression prediction model of the immunohistochemical molecule according to each of the tumor sub-images and the expression labels of the immunohistochemical molecule; Each of the tumor sub-images is input into the expression prediction model to obtain the expression prediction results of the immunohistochemical molecules.

2. The method for predicting immunohistochemical molecule expression according to claim 1, characterized in that: The step of acquiring multiple sub-images of lung tissue comprises the following steps: Obtain full-field slice images of lung tissue and tumor tissue annotation files; The full-view slice image is divided into a plurality of sub-images of the same size according to the tumor tissue annotation file.

3. The method for predicting immunohistochemical molecule expression according to claim 2, characterized in that: The step of dividing the full-view slice image into a plurality of sub-images of the same size according to the tumor tissue annotation file comprises the following steps: generating a tumor mask according to the tumor tissue annotation file and the full-field slice image; The full-field slice image is divided into non-tumor sub-images not covered by the tumor mask and the tumor sub-images covered by the tumor mask; wherein the sizes of the non-tumor sub-images and the tumor sub-images are the same.

4. The method for predicting immunohistochemical molecule expression according to claim 1, characterized in that: The step of classifying each of the sub-images to obtain the sub-image classified as tumor tissue as a tumor sub-image comprises the following steps: Constructing a binary classification model based on a convolutional neural network according to each of the sub-images and the classification labels corresponding to each of the sub-images; The binary classification model is used to classify each of the sub-images to obtain the tumor sub-image.

5. The method for predicting immunohistochemical molecule expression according to claim 1, characterized in that: The step of constructing the expression prediction model of the immunohistochemical molecule according to each of the tumor sub-images and the expression labels of the immunohistochemical molecule comprises the following steps: The expression prediction model is constructed based on a convolutional neural network according to each of the tumor sub-images and the expression labels of different immunohistochemical molecules.

6. The method for predicting immunohistochemical molecule expression according to claim 1, characterized in that: The step of inputting each of the tumor sub-images into the expression prediction model to obtain the expression prediction results of the immunohistochemical molecules comprises the following steps: Inputting each of the tumor sub-images into the expression prediction model to obtain an expression prediction probability vector of the expression prediction model regarding the immunohistochemical molecule; wherein each component in the expression prediction probability vector corresponds to a positive expression probability of the immunohistochemical molecule; Whether various immunohistochemical molecules are positively expressed is determined according to the expression prediction probability vector as the expression prediction result.

7. The method for predicting immunohistochemical molecule expression according to claim 6, characterized in that: The step of determining whether various immunohistochemical molecules are positively expressed as the expression prediction result according to the expression prediction probability vector comprises the following steps: Converting the positive expression probability in the expression prediction probability vector that is greater than or equal to the clinical empirical threshold and the grid search optimal threshold into a positive expression label; The positive expression probability in the expression prediction probability vector that is less than the clinical empirical threshold and the grid search optimal threshold is converted into a negative expression label; wherein the positive expression label and the negative expression label serve as the expression prediction result.

8. An immunohistochemical molecule expression prediction device, characterized in that: The device comprises: An image acquisition unit, configured to acquire a plurality of sub-images of lung tissue; wherein each of the sub-images includes any one of tumor tissue or non-tumor tissue; An image classification unit, used for classifying each of the sub-images to obtain the sub-image classified as tumor tissue as a tumor sub-image; A model building unit, used for building an expression prediction model of the immunohistochemical molecule according to each of the tumor sub-images and the expression labels of the immunohistochemical molecule; The expression prediction unit is used to input each of the tumor sub-images into the expression prediction model to obtain the expression prediction results of the immunohistochemical molecules.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images