Breast cancer HER2 immunohistochemical digital slice automatic interpretation method, device and equipment and storage medium

Abstract

The invention provides a breast cancer HER2 immunohistochemical digital slice automatic interpretation method, device and equipment and a storage medium. Relates to the technical field of digital pathological image analysis. Acquiring a full-slice image, automatically identifying an external contrast area and a tissue area through a multi-task head real-time instance segmentation model, and segmenting an ROI (Region of Interest) area of tumor cell aggregation; extracting image blocks by adopting a fixed sliding window based on the ROI region, and constructing a double-branch multi-task feature extraction model containing a backbone network: realizing HER2 expression pattern classification by an image block classification head, and outputting a continuous protein expression intensity quantized value by a dyeing intensity numerical head through regression analysis; and establishing a slice-level interpretation model, integrating the classification results and the intensity quantized values of all the image blocks, and outputting an automatic HER2 interpretation result. According to the invention, end-to-end analysis from cell-level characteristic quantification to slice-level diagnosis is realized, and an objective and quantifiable intelligent interpretation scheme is provided for HER2 immunohistochemical evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital pathology image analysis, and particularly relates to a breast cancer HER2 immunohistochemistry digital slice automatic interpretation method and device, equipment and a storage medium. BACKGROUND

[0002] Breast cancer is one of the common malignant tumors in women. Overexpression of HER2 protein or amplification of HER2 gene is an important prognostic indicator and basis for targeted therapy of breast cancer. Immunohistochemistry (IHC) is a commonly used method for detecting HER2 protein expression levels. Pathologists observe HER2 stained slices under a microscope and interpret them according to the proportion of positively stained tumor cells and staining intensity, which are divided into four grades: 0, 1+, 2+, and 3+.

[0003] The traditional manual interpretation method has the following problems:

[0004] Strong subjectivity: Different pathologists and the same pathologist at different times may have different interpretations of the same slice, affecting the consistency and accuracy of the interpretation.

[0005] Low efficiency: The interpretation process requires a lot of time and effort, especially when dealing with a large number of samples.

[0006] Lack of accurate quantification: The human eye cannot accurately quantify staining intensity and positive cell proportion, especially in cases where the boundaries are ambiguous. SUMMARY

[0007] The present application provides a breast cancer HER2 immunohistochemistry digital slice automatic interpretation method, device, equipment and storage medium to solve the problems that existing artificial intelligence methods face challenges in computational efficiency and model robustness when dealing with large-scale, high-resolution digital pathology slices, and are not perfect in accurately simulating the comprehensive interpretation logic of pathologists based on diagnostic guidelines.

[0008] In a first aspect, the present application provides a breast cancer HER2 immunohistochemistry digital slice automatic interpretation method, comprising:

[0009] Obtaining a breast cancer HER2 immunohistochemistry digital whole slice image WSI;

[0010] Based on the real-time instance segmentation structure of the multi-task head, automatically identifying and locating the external control area and the tissue area in the WSI, and performing region of interest segmentation on the located tissue area to obtain a tumor cell aggregation area with HER2 expression as a ROI area;

[0011] Based on the ROI area, a fixed sliding window is used to extract image blocks to obtain a plurality of image blocks;

[0012] construct a multi-task feature extraction model; wherein the multi-task feature extraction model comprises a backbone network, an image block classification head and a staining intensity numerical head; input the image block into the backbone network, extract image features after processing by the backbone network, and input the image features into the image block classification head and the staining intensity numerical head respectively, the image block classification head classifies the image block according to the HER2 expression pattern in response to the input image features, and obtains the HER2 expression pattern classification result of the image block, and the staining intensity numerical head performs numerical regression analysis of the staining intensity in response to the input image features, and outputs a continuous numerical value to quantify the expression intensity of the HER2 protein;

[0013] establish a slice-level HER2 interpretation model, which outputs a slice-level HER2 interpretation result in response to the input HER2 expression pattern classification result and the expression intensity of the HER2 protein.

[0014] In a possible design, the backbone network comprises at least three convolutional layers connected in sequence, and the output features of the last three convolutional layers of the backbone network are input into an attention connection network to obtain image features.

[0015] In a possible design, the HER2 expression pattern classification result of the image block comprises one of negative, weak positive, moderate positive, strong positive, specific staining area of non-tumor cells and non-tumor cell area.

[0016] In a possible design, the multi-task feature extraction model is trained based on a first loss function, and the slice-level HER2 interpretation model is realized by the trained multi-task feature extraction model in response to the input HER2 expression pattern classification result and the expression intensity of the HER2 protein; wherein the first loss function comprises a classification loss and a regression loss, the classification loss is used to measure the prediction accuracy of the multi-task feature extraction model for the HER2 expression pattern category of the image block, and the regression loss is used to measure the prediction accuracy of the multi-task feature extraction model for the staining intensity value of the image block.

[0017] In a possible design, the first loss function is represented as:

[0018]

[0019]

[0020]

[0021] In the formula, L patch is the total loss, is the classification loss, is the regression loss, and a and b are the weights of the classification loss and the regression loss, respectively, y i is the true class label of the i-th image patch, is the probability of the i-th image patch belonging to each class predicted by the model, and N is the number of image patches, v i is the weakly supervised staining intensity numerical label of the i-th image patch, is the staining intensity value of the i-th image patch predicted by the model.

[0022] In one possible design, the weakly supervised staining intensity numerical label of the image patch is obtained by:

[0023] The R channel information and the B channel information of the image patch are extracted;

[0024] The R channel information and the B channel information are clustered to obtain K clusters;

[0025] Each cluster is associated with a preset staining intensity level, and a numerical label representing the overall staining intensity of the cluster is generated as the weakly supervised staining intensity numerical label of the image patch included in the cluster.

[0026] In one possible design, the slice-level HER2 interpretation model is trained based on a second loss function, and the output slice-level HER2 interpretation result of the trained slice-level HER2 interpretation model; wherein the second loss function is represented as:

[0027]

[0028] In the formula, L slice represents the second loss, M is the number of slices, z j is the true HER2 status label of slice j, is the probability of each state of slice j predicted by the slice-level HER2 interpretation model.

[0029] In a second aspect, the present application provides a breast cancer HER2 immunohistochemistry digital slice automatic interpretation device, comprising:

[0030] An image acquisition module configured to acquire a breast cancer HER2 immunohistochemistry digital whole slice image WSI;

[0031] A positioning and identification module configured to automatically identify and position the external control area and the tissue area in the WSI based on a real-time instance segmentation structure of a multi-task head, and to segment the tissue area positioned into a region of interest to obtain a tumor cell aggregation area with HER2 expression as a ROI area;

[0032] The sliding window extraction module is configured to extract image blocks by using a fixed sliding window based on the ROI region to obtain a plurality of image blocks.

[0033] The multi-task parallel module is configured to construct a multi-task feature extraction model; wherein the multi-task feature extraction model comprises a backbone network, an image block classification head and a staining intensity numerical head; the image block is input into the backbone network, and image features are extracted after being processed by the backbone network; the image features are input into the image block classification head and the staining intensity numerical head respectively; the image block classification head classifies the image block according to the HER2 expression pattern in response to the input image features, and obtains the HER2 expression pattern classification result of the image block; the staining intensity numerical head performs numerical regression analysis on the staining intensity in response to the input image features, and outputs a continuous numerical value to quantify the expression intensity of the HER2 protein.

[0034] The slice interpretation module is configured to establish a slice-level HER2 interpretation model, and the slice-level HER2 interpretation model outputs a slice-level HER2 interpretation result in response to the input HER2 expression pattern classification result and the expression intensity of the HER2 protein.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the breast cancer HER2 immunohistochemical digital slice automatic interpretation method as described in the first aspect and various possible designs of the first aspect.

[0036] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and when a processor executes the computer execution instructions, the breast cancer HER2 immunohistochemical digital slice automatic interpretation method as described in the first aspect and various possible designs of the first aspect is implemented.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the breast cancer HER2 immunohistochemical digital slice automatic interpretation method as described in the first aspect and various possible designs of the first aspect.

[0038] 1) The present application can improve the interpretation accuracy and consistency: through the fine extraction and objective quantification of image features by the deep learning model, the subjective difference of artificial interpretation is significantly reduced, and the accuracy and repeatability of HER2 state evaluation are improved.

[0039] 2) The application can improve interpretation efficiency: the automated processing process significantly shortens the time for pathologists to review and analyze slides, especially when dealing with a large number of samples, which can significantly improve work efficiency.

[0040] 3) The application can realize fine quantitative analysis: not only provides qualitative classification results, but also provides quantitative numerical values through staining intensity regression, providing more rich diagnostic information for clinical.

[0041] 4) The application can simulate expert diagnosis logic: the slice-level interpretation model incorporates the interpretation rules of the clinical diagnosis guidelines, making the final output result closer to the comprehensive diagnostic thinking of pathologists.

[0042] 5) The application can assist in interpreting complex cases: for complex cases with uneven HER2 expression or in a critical state, it can provide objective assistance to help pathologists make more accurate judgments. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0044] Figure 1 The flow of the breast cancer HER2 immunohistochemical digital slice automatic interpretation method provided for the embodiments of the application Figure 1 ;

[0045] Figure 2 The flow of the breast cancer HER2 immunohistochemical digital slice automatic interpretation method provided for the embodiments of the application Figure 2 ;

[0046] Figure 3 The data labeling flowchart provided for the embodiments of the application

[0047] Figure 4 The structure diagram of the multi-task feature extraction model provided for the embodiments of the application

[0048] Figure 5 An exemplary HER2 classification interpretation result of a digital slice provided for the embodiments of the application

[0049] Figure 6 The structural diagram of the breast cancer HER2 immunohistochemical digital slice automatic interpretation device provided for the embodiments of the application.

[0050] Through the above drawings, the specific embodiments of the application have been shown, and will be described in more detail in the following. These drawings and written descriptions are not intended to limit the scope of the concept of the application in any way, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description of exemplary embodiments is not representative of all possible embodiments consistent with the present application. Instead, it is merely intended to describe some illustrative embodiments consistent with some aspects of the present application as detailed in the appended claims.

[0052] In the technical solutions of the present application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data, user data or medical image data comply with relevant laws and regulations and do not violate public order and good customs.

[0053] It should be noted that in the embodiments of the present application, some existing industry solutions such as software, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0054] It should be noted that in the embodiments of the present application, some existing industry solutions such as software, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0055] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0056] Manual interpretation of HER2 immunohistochemistry results is subjective, and the interpretation results of different pathologists and the same doctor at different time points may differ, affecting the consistency of diagnosis. Traditional image processing methods are difficult to accurately process uneven staining, complex background, etc. Existing artificial intelligence methods face challenges in computing efficiency and model robustness when processing large-scale, high-resolution digital pathology slides, and are not perfect in accurately simulating the comprehensive interpretation logic of pathologists based on diagnostic guidelines. The present invention aims to solve the above technical problems and provide an automated interpretation scheme with high precision, high efficiency and consistent with clinical diagnostic logic.

[0057] The present application provides a breast cancer HER2 immunohistochemistry digital slide automatic interpretation method. Please refer to Figure 1 , Figure 1The method provided by the embodiment of the present application is a breast cancer HER2 immunohistochemical digital slice automatic interpretation method Figure 1 The breast cancer HER2 immunohistochemical digital slice automatic interpretation method can be implemented based on an electronic terminal. According to an input WSI image, a real-time instance segmentation model is used to distinguish a control area and a tissue area, and a Patch (image block) is segmented in the area. After preprocessing, the Patch is sent to a Patch feature extraction model (i.e., a multi-task feature extraction model) and a slice-level HER2 interpretation model, respectively. The Patch feature extraction model outputs a Patch staining intensity value and a Patch classification category, respectively. The slice-level HER2 interpretation model generates a slice-level HER2 interpretation result according to the preprocessed Patch and the Patch staining intensity value and the Patch classification category output by the Patch feature extraction model.

[0058] Specifically, refer to Figure 2 , Figure 2 The method provided by the embodiment of the present application is a breast cancer HER2 immunohistochemical digital slice automatic interpretation method Figure 2 The breast cancer HER2 immunohistochemical digital slice automatic interpretation method includes the following steps S10 to S50.

[0059] S10, acquire a breast cancer HER2 immunohistochemical digital whole slice image WSI.

[0060] Refer to Figure 3 , Figure 3 The data labeling flowchart provided by the embodiment of the present application. Through the data labeling flowchart shown in Figure 3 , the breast cancer HER2 immunohistochemical digital whole slice image WSI obtained is used as the input of the subsequent steps or as the training process of the model constructed in the subsequent steps. For example, according to the WSI image, a minimum magnification thumbnail is obtained, the tissue and control area position and size are labeled on the minimum magnification thumbnail, and the Patch (image block) is cut in the tissue area. The Patch staining value intensity is labeled, and the different staining categories of the Patch are labeled.

[0061] In some embodiments, the data set labeling stage flowchart is implemented by the following steps S101-S103 to obtain the breast cancer HER2 immunohistochemical digital whole slice image WSI.

[0062] S101: data set collection.

[0063] Collect breast cancer HER2 immunohistochemical whole slice images from multiple medical institutions, covering different staining batches and scanner devices, and classify them according to the four-grade diagnosis standard (0, 1+, 2+, 3+) specified in the 2024 HER2-positive breast cancer clinical diagnosis and treatment guidelines.

[0064] S102: Slice thumbnail annotation.

[0065] Read the thumbnail of the full slice image at the minimum magnification, and make the following annotations.

[0066] Control area: Use a rectangular bounding box to annotate its precise location and range, and record the overall staining intensity grade synchronously.

[0067] Tissue area: Annotate the bounding box of tumor tissue and non-tumor tissue, and record the staining intensity grade.

[0068] S103: Patch-level annotation.

[0069] Cut the tissue area images of all slices into 1024x1024 pixel image blocks at 20x magnification. Complete manual annotation processing, with 6 categories of annotation classes: 0 (negative), 1+ (weakly positive), 2+ (moderately positive), 3+ (strongly positive), specific staining area of non-tumor cells, and non-tumor cell area.

[0070] S20, real-time instance segmentation structure based on multi-task head, automatically identifies and locates the external control area and tissue area in the WSI, and performs region of interest segmentation on the located tissue area to obtain the tumor cell aggregation area with HER2 expression as the ROI area.

[0071] In some embodiments, a deep learning-based target detection model can be constructed, which uses a real-time instance segmentation structure based on a multi-task head to automatically identify and locate the external control area and effective tissue area in the WSI. This step ensures that the subsequent analysis focuses on areas with diagnostic value.

[0072] Further precise segmentation of the region of interest is performed within the detected tissue area. In some embodiments, image segmentation algorithms can be used to distinguish tumor cell aggregation areas with HER2 expression, excluding irrelevant areas such as blanks, impurities, and backgrounds, to provide high-quality input for subsequent patch analysis. The real-time instance segmentation algorithm uses an improved ConvNextv2tiny model as the backbone network, and an improved Mask-RCNN structure as the task head.

[0073] S30, based on the ROI area, fixed sliding window is used to extract image blocks to obtain a number of image blocks.

[0074] In some embodiments, based on the ROI region obtained by segmentation, a fixed size sliding window strategy can be used to extract image patches. The sliding window size is set to 1024x1024 pixels to ensure that each patch contains sufficient cell and tissue information while taking into account computational efficiency. The system slides with a certain overlap rate to ensure comprehensive coverage of the tissue area.

[0075] S40, constructing a multi-task feature extraction model; wherein the multi-task feature extraction model comprises a backbone network, an image patch classification head, and a staining intensity numerical head; inputting the image patch into the backbone network, extracting image features after processing by the backbone network, inputting the image features into the image patch classification head and the staining intensity numerical head, respectively, the image patch classification head classifies the image patch according to the HER2 expression pattern in response to the input image features, obtaining the HER2 expression pattern classification result of the image patch, and the staining intensity numerical head performs numerical regression analysis of the staining intensity in response to the input image features, and outputs a continuous numerical value to quantify the expression intensity of the HER2 protein.

[0076] Please refer to Figure 4 , Figure 4 for the structure diagram of the multi-task feature extraction model. The backbone network is a shared feature extraction module for the image patch classification head and the staining intensity numerical head. In some embodiments, the features output by the last 3 layers of the backbone network can be output to different task heads after passing through an attention connection network according to the needs of different task heads.

[0077] The image patch classification head is used to implement the HER2 expression pattern classification of the Patch. In some embodiments, each image patch extracted by the backbone network is input into an improved multi-task head ConvNext v2 Base model to perform fine classification of the HER2 expression pattern. This model can accurately classify each Patch into one of the following six predefined categories: 0 (negative), 1+ (weakly positive), 2+ (moderately positive), 3+ (strongly positive), specific staining area of non-tumor cells, and non-tumor cell area. This classification result provides a basis for subsequent quantitative analysis and diagnosis.

[0078] The staining intensity numerical head is used to implement the numerical regression of the staining intensity of the Patch. While performing HER2 expression pattern classification, different task heads of the same multi-task feature extraction model perform numerical regression analysis of the staining intensity of each Patch. This regression task is trained using weakly supervised learning. By analyzing the staining features of a large number of Patches, a continuous numerical value is output to accurately quantify the expression intensity of the HER2 protein. This numerical value provides a more objective basis for subsequent fine interpretation.

[0079] In some embodiments, the multi-task feature extraction model is trained based on the first loss function, and the slice-level HER2 interpretation model is implemented based on the trained multi-task feature extraction model to output a classification result of the input HER2 expression pattern and an expression intensity of the HER2 protein.

[0080] The multi-task feature extraction model adopts multi-task learning, and the first loss function used in the training of the multi-task feature extraction model is a weighted combination of a classification task loss and a regression task loss. The classification task loss The cross-entropy loss is used to measure the prediction accuracy of the model for the Patch HER2 expression pattern category:

[0081]

[0082] In the formula, y i is the true category label of the i-th image block, is the probability of the i-th image block belonging to each category predicted by the model, and N is the number of image blocks.

[0083] The regression task adopts weak supervision learning, and the generation process of the weak supervision label vi is as follows: for each Patch, the system extracts the R channel and B channel information of the image data. Since the DAB developing agent in the HER2 immunohistochemical staining presents a brownish yellow color, it has specific response characteristics in the R and B channels, and the use of information of the two channels helps to distinguish the depth of staining. The system performs K-means clustering on the R channel and B channel pixel values of the Patch, and clusters the pixel points into K clusters (for example, K=3 or K=4, depending on the staining complexity and the required intensity level to be distinguished). By analyzing the pixel features (for example, the average pixel value of the cluster center) of each cluster, and associating these clusters with predefined staining intensity levels (such as 0, 1+, 2+, 3+), the system generates a numerical label v i for the Patch representing the overall staining intensity of the Patch. This label is used as a weak supervision signal to train the regression model.

[0084] The regression task loss The mean squared error loss (MSE) is used to measure the prediction accuracy of the model for the numerical value of the Patch staining intensity:

[0085]

[0086] In the formula, v i is the weak supervision staining intensity numerical label of the i-th image block, is the numerical value of the staining intensity of the i-th image block predicted by the model.

[0087] The total loss is expressed as the weighted sum of classification loss and regression loss, so the first loss function is expressed as:

[0088]

[0089] Where, L patch is the total loss, is the classification loss, is the regression loss, α and β are the weights of classification loss and regression loss respectively.

[0090] S50. Establish a slice-level HER2 interpretation model. The slice-level HER2 interpretation model responds to the input HER2 expression pattern classification result and the expression intensity of the HER2 protein, and outputs a slice-level HER2 interpretation result.

[0091] The slice-level HER2 interpretation model is used to implement slice-level interpretation based on diagnostic guidelines. In some embodiments, the slice-level HER2 interpretation model can be a sequence encoding model based on an attention mechanism, which summarizes the HER2 expression pattern classification results and staining intensity regression values ​​of all patches. These patch-level information are input into a sequence encoding model based on an attention mechanism. The model simulates the logic of a pathologist's comprehensive evaluation based on the 2024-HER2 diagnostic guidelines, comprehensively considering the proportion, spatial distribution and staining intensity information of different categories of patches, and finally outputs the interpretation result of the HER2 classification of the entire digital slice. The interpretation result of the HER2 classification of an exemplary digital slice is as follows: Figure 5 shown.

[0092] In some embodiments, the slice-level HER2 interpretation model is trained using a second loss function, which can be a cross-entropy loss, to measure the model's prediction accuracy for the entire slice HER2 grade. The second loss function is expressed as:

[0093]

[0094] Where, L slice represents the second loss, M is the number of slices, z j is the true HER2 status label of slice j, is the probability of each state described in slice j predicted by the slice-level HER2 interpretation model.

[0095] Figure 6 This is a schematic diagram of the structure of the automatic interpretation device for HER2 immunohistochemistry digital slices of breast cancer provided in the embodiment of the present application. This embodiment of the present application also provides an automatic interpretation device for HER2 immunohistochemistry digital slices of breast cancer. Figure 6 As shown, the breast cancer HER2 immunohistochemistry digital slice automatic interpretation device includes:

[0096] The image acquisition module 601 is configured to acquire a breast cancer HER2 immunohistochemistry digital whole-slice image WSI;

[0097] A positioning identification module 602 is configured as a real-time instance segmentation structure based on a multi-task head to automatically identify and locate the external control area and tissue area in the WSI, and perform region of interest segmentation on the located tissue area to obtain a tumor cell cluster area with HER2 expression as the ROI area;

[0098] The sliding window extraction module 603 is configured to extract image blocks based on the ROI region using a fixed sliding window to obtain a plurality of image blocks;

[0099] The multi-task parallel module 604 is configured to construct a multi-task feature extraction model; wherein the multi-task feature extraction model includes a backbone network, an image block classification head, and a staining intensity numerical head; the image block is input into the backbone network, and after being processed by the backbone network, image features are extracted, and the image features are respectively input into the image block classification head and the staining intensity numerical head. The image block classification head classifies the HER2 expression pattern of the image block in response to the input image features to obtain a HER2 expression pattern classification result of the image block. The staining intensity numerical head performs a numerical regression analysis of the staining intensity in response to the input image features, and outputs a continuous numerical value to quantify the expression intensity of the HER2 protein;

[0100] The slice interpretation module 605 is configured to establish a slice-level HER2 interpretation model, which outputs a slice-level HER2 interpretation result in response to the input HER2 expression pattern classification result and the expression intensity of the HER2 protein.

[0101] In some embodiments, the backbone network includes at least three sequentially connected convolutional layers, and the output features of the last three convolutional layers of the backbone network are passed through an attention connection network to obtain image features.

[0102] In some embodiments, the HER2 expression pattern classification result of the image block includes one of negative, weakly positive, moderately positive, strongly positive, a non-tumor cell specific staining area, and a non-tumor cell area.

[0103] In some embodiments, the multi-task parallel module is further configured to train the multi-task feature extraction model based on a first loss function, so that the trained multi-task feature extraction model realizes the output of the slice-level HER2 interpretation model in response to the input HER2 expression pattern classification results and the expression intensity of the HER2 protein; wherein the first loss function includes classification loss and regression loss, the classification loss is used to measure the prediction accuracy of the multi-task feature extraction model for the image block HER2 expression pattern category, and the regression loss is used to measure the prediction accuracy of the multi-task feature extraction model for the image block staining intensity value.

[0104] In some embodiments, the first loss function is expressed as:

[0105]

[0106]

[0107]

[0108] Where, L patch is the total loss, is the classification loss, is the regression loss, α and β are the weights of classification loss and regression loss respectively, y i is the true category label of the i-th image patch, is the probability that the model predicts that the i-th image block belongs to each category, N is the number of image blocks, and v i is the weakly supervised staining intensity numerical label of the i-th image patch, is the staining intensity value of the i-th image block predicted by the model.

[0109] In some embodiments, the multi-task parallel module is further configured to obtain the weakly supervised staining intensity numerical label of the image block by:

[0110] Extract the R channel information and B channel information of the image block;

[0111] Clustering the R channel information and the B channel information to obtain K clusters;

[0112] Each cluster is associated with a preset staining intensity level, and a numerical label representing the overall staining intensity of the cluster is generated as a weakly supervised staining intensity numerical label for the image blocks contained in the cluster.

[0113] In some embodiments, the slice interpretation module is further configured to train the slice-level HER2 interpretation model based on a second loss function, to obtain an output slice-level HER2 interpretation result of the trained slice-level HER2 interpretation model; wherein the second loss function is represented as:

[0114]

[0115] wherein, L slice represents the second loss, M is the number of slices, z j is the true HER2 status label of slice j, is the probability of each state of slice j predicted by the slice-level HER2 interpretation model.

[0116] The breast cancer HER2 immunohistochemical digital slice automatic interpretation device provided by the embodiments of the present application can be used to execute the technical solutions of the breast cancer HER2 immunohistochemical digital slice automatic interpretation method in the above embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0117] The embodiments of the present application provide an electronic device. The electronic device can include a processor, a memory, wherein the processor and the memory can communicate; for example, the processor and the memory communicate through a communication bus.

[0118] The processor executes computer execution instructions stored in the memory, so that the processor executes the schemes in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0119] The communication bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries and read-only libraries). The memory can include random access memory (RAM), and can also include non-volatile memory.

[0120] The electronic device provided by the embodiments of the present application can be the terminal device of the above embodiments.

[0121] The embodiments of the present application further provide a computer readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the technical solutions of the breast cancer HER2 immunohistochemical digital slice automatic interpretation method of the above embodiments.

[0122] The embodiments of the present application further provide a computer program product, which includes a computer program stored in a computer readable storage medium. At least one processor can read the computer program from the computer readable storage medium, and when the at least one processor executes the computer program, the technical solutions of the breast cancer HER2 immunohistochemical digital slice automatic interpretation method of the above embodiments can be implemented.

[0123] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and the division of modules can be different, for example, the division of modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0124] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to implement the embodiments of the present application.

[0125] In addition, each functional module in the embodiments of the present application can be integrated in one processing unit, or each module can be physically present alone, or two or more modules can be integrated in one unit. The unit formed by the above modules can be realized in the form of hardware, or in the form of hardware plus software function unit.

[0126] The integrated modules realized in the form of software function modules can be stored in a computer readable storage medium. The software function modules stored in the storage medium include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the embodiments of the present application.

[0127] It should be appreciated that the above-mentioned processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0128] The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0129] The bus can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0130] The above-mentioned storage medium can be realized by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0131] An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a host device.

[0132] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various storage media that can store program codes, such as ROM, RAM, magnetic disk or optical disk.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for automatically interpreting digital sections of breast cancer HER2 immunohistochemistry, characterized in that: include: Acquire digital whole-slice images (WSI) of HER2 immunohistochemistry of breast cancer; Based on the real-time instance segmentation structure of the multi-task head, the external control area and tissue area in the WSI are automatically identified and located, and the located tissue area is segmented into regions of interest, and the tumor cell aggregation area with HER2 expression is obtained as the ROI area; Based on the ROI area, a fixed sliding window is used to extract image blocks to obtain a plurality of image blocks; Constructing a multi-task feature extraction model; wherein the multi-task feature extraction model includes a backbone network, an image block classification head, and a staining intensity numerical head; inputting the image block into the backbone network, extracting image features after processing by the backbone network, and inputting the image features into the image block classification head and the staining intensity numerical head, respectively; the image block classification head classifies the HER2 expression pattern of the image block in response to the input image features to obtain a HER2 expression pattern classification result of the image block; the staining intensity numerical head performs numerical regression analysis of the staining intensity in response to the input image features, and outputs a continuous numerical value to quantify the expression intensity of the HER2 protein; A slice-level HER2 interpretation model is established, wherein the slice-level HER2 interpretation model outputs a slice-level HER2 interpretation result in response to the input HER2 expression pattern classification result and the expression intensity of the HER2 protein.

2. The method according to claim 1, characterized in that The backbone network includes at least three sequentially connected convolutional layers, and the output features of the last three convolutional layers of the backbone network are passed through an attention connection network to obtain image features.

3. The method according to claim 1, characterized in that The HER2 expression pattern classification result of the image block includes one of negative, weakly positive, moderately positive, strongly positive, a non-tumor cell specific staining area, and a non-tumor cell area.

4. The method according to claim 1, wherein The multi-task feature extraction model is trained based on a first loss function, and the trained multi-task feature extraction model is used to implement the output of the slice-level HER2 interpretation model in response to the input HER2 expression pattern classification result and the expression intensity of the HER2 protein; wherein the first loss function includes classification loss and regression loss, the classification loss is used to measure the prediction accuracy of the multi-task feature extraction model for the HER2 expression pattern category of the image block, and the regression loss is used to measure the prediction accuracy of the multi-task feature extraction model for the image block staining intensity value.

5. The method according to claim 4, characterized in that The first loss function is expressed as: Where, L patch is the total loss, is the classification loss, is the regression loss, α and β are the weights of classification loss and regression loss respectively, y i is the true category label of the i-th image patch, is the probability that the model predicts that the i-th image block belongs to each category, N is the number of image blocks, and v i is the weakly supervised staining intensity numerical label of the i-th image patch, is the staining intensity value of the i-th image block predicted by the model.

6. The method according to claim 5, characterized in that The weakly supervised staining intensity numerical label of the image patch is obtained as follows: Extract the R channel information and B channel information of the image block; Clustering the R channel information and the B channel information to obtain K clusters; Each cluster is associated with a preset staining intensity level, and a numerical label representing the overall staining intensity of the cluster is generated as a weakly supervised staining intensity numerical label for the image blocks contained in the cluster.

7. The method according to claim 1, characterized in that The slice-level HER2 interpretation model is trained based on a second loss function, and the trained slice-level HER2 interpretation model outputs a slice-level HER2 interpretation result; wherein the second loss function is expressed as: Where, L slice represents the second loss, M is the number of slices, z j is the true HER2 status label of slice j, is the probability of each state described in slice j predicted by the slice-level HER2 interpretation model.

8. An automatic interpretation device for digital slices of breast cancer HER2 immunohistochemistry, characterized by: include: An image acquisition module is configured to acquire a breast cancer HER2 immunohistochemistry digital whole-slice image WSI; a positioning recognition module, configured as a real-time instance segmentation structure based on a multi-task head, automatically identifying and locating the external control area and tissue area in the WSI, and performing region of interest segmentation on the located tissue area to obtain a tumor cell aggregation area with HER2 expression as a ROI area; A sliding window extraction module is configured to extract image blocks based on the ROI area using a fixed sliding window to obtain a plurality of image blocks; A multi-task parallel module is configured to construct a multi-task feature extraction model; wherein the multi-task feature extraction model includes a backbone network, an image block classification head, and a staining intensity numerical head; the image block is input into the backbone network, and after being processed by the backbone network, image features are extracted, and the image features are respectively input into the image block classification head and the staining intensity numerical head; the image block classification head classifies the HER2 expression pattern of the image block in response to the input image features to obtain a HER2 expression pattern classification result of the image block; the staining intensity numerical head performs numerical regression analysis of the staining intensity in response to the input image features, and outputs a continuous numerical value to quantify the expression intensity of the HER2 protein; The slice interpretation module is configured to establish a slice-level HER2 interpretation model, which outputs a slice-level HER2 interpretation result in response to the input HER2 expression pattern classification result and the expression intensity of the HER2 protein.

9. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images