An automatic scoring system for immunohistochemical images of P53 protein in colorectal cancer

Abstract

The present invention discloses an automatic scoring system for immunohistochemical images of colorectal cancer P53 protein. The system comprises a staining channel separation module, a cell nucleus region extraction module, an effective tissue region extraction module, and an immunohistochemical scoring module. The system separates the colorectal cancer P53 immunohistochemical staining image into hematoxylin, DAB, and residual channels; extracts the cell nucleus region staining image from the colorectal cancer P53 immunohistochemical staining image; obtains a tissue region mask; extracts the first eigenvector of the original colorectal cancer P53 immunohistochemical staining image; uses an attention mechanism to learn the cell nucleus region staining intensity and positive staining percentage based on the cell nucleus region staining image and the tissue region mask to obtain a second eigenvector; and concatenates the first and second eigenvectors and outputs them through a fully connected layer to obtain the immunohistochemical scoring result. The present invention enables automatic inspection of immunohistochemical images.

Description

Technical Field

[0001] The present invention relates to the fields of digital pathological images and computers, and in particular to an automatic scoring system for colorectal cancer P53 immunohistochemistry images integrating auxiliary features. Background Art

[0002] In pathological examinations, approximately 5% to 10% of cases are difficult to make a definitive morphological diagnosis based solely on H&E staining, especially for early-stage, poorly differentiated or undifferentiated tumors. The addition of immunohistochemistry can significantly improve diagnostic accuracy. The score of immunohistochemical images indicates the expression level of a specific gene protein. Pathologists assess the images based on staining intensity and percentage of positive staining, a laborious and time-consuming process. Furthermore, due to visual perceptual biases, scoring results may vary between pathologists. Therefore, an automated scoring system is needed that leverages the diagnostic experience of professional pathologists to efficiently and accurately generate immunohistochemical scoring results. Summary of the Invention

[0003] In order to overcome the deficiencies of the prior art, the present invention provides an automatic scoring system for P53 immunohistochemistry images integrating auxiliary features.

[0004] The technical solution adopted by the present invention to solve the technical problem is as follows: an embodiment of the present invention provides an automatic scoring system for P53 immunohistochemistry images of colorectal cancer, the system comprising:

[0005] The staining channel separation module is used to separate the colorectal cancer P53 immunohistochemical staining image into hematoxylin, DAB and residual channels through the color deconvolution algorithm;

[0006] The cell nucleus region extraction module is used to transform the colorectal cancer P53 immunohistochemical staining image into HRD space, use the maximum inter-class variance method to segment the cell nucleus region mask, and obtain the cell nucleus region staining map in the colorectal cancer P53 immunohistochemical staining image;

[0007] The effective tissue region extraction module obtains the DAB channel staining image output by the staining channel separation module, detects the tissue region using the LC saliency detection algorithm, and then uses the adaptive binarization method to obtain a binary image of the tissue region. The tissue region mask is obtained by connecting adjacent regions in the binary image using a closing operation.

[0008] The immunohistochemistry scoring module is used to extract the first eigenvector of the original colorectal cancer P53 immunohistochemical staining image; based on the cell nuclear region staining image output by the cell nuclear region extraction module and the tissue region mask output by the effective tissue region extraction module, the attention mechanism is used to learn the cell nuclear region staining intensity and positive staining percentage to obtain the second eigenvector; the first eigenvector and the second eigenvector are spliced ​​and output through the fully connected layer to obtain the immunohistochemistry scoring result.

[0009] Furthermore, the cell nucleus region segmentation module transforms the colorectal cancer P53 immunohistochemical staining image into optical density space and then into HRD space through log operation, obtains the grayscale image of the hematoxylin channel in the HRD space, and uses the maximum inter-class variance method to segment the cell nucleus region mask, thereby obtaining the cell nucleus region staining image in the colorectal cancer P53 immunohistochemical staining image.

[0010] Furthermore, the process of transforming from the optical density space to the HRD space includes: transforming the optical density values ​​of the RGB channels of the optical density space using the dye, and the formula is as follows:

[0011] x hrd =M -1 x rgb

[0012] Where x hrd is the pixel point in the P53 immunohistochemical staining image of colorectal cancer in HRD space, x rgb It is a pixel point in the P53 immunohistochemical staining image of colorectal cancer in the optical density space. Each row in the matrix M represents the optical density value of a certain dye in the RGB channel.

[0013] Furthermore, the effective tissue region extraction module first enhances the dark area of ​​the DAB channel staining image output by the staining channel separation module through gamma transformation, uses the LC significance detection algorithm to detect the effective tissue area of ​​the dark-enhanced image, and then uses the adaptive binarization method to obtain a binarized image of the tissue region, and uses a closing operation to connect adjacent areas in the binarized image to obtain a relatively complete tissue region mask; sets a connected domain size threshold, uses connected domain detection to obtain the size of each connected domain in the tissue region mask, removes small areas caused by false detection and / or missed detection areas inside the tissue, and further optimizes the tissue region mask.

[0014] Furthermore, the process of detecting effective tissue areas using the LC saliency detection algorithm includes: calculating the Euclidean distance from each pixel to other pixels as the global contrast, obtaining a contrast matrix with the same size as the original image, taking the contrast mean of all pixels as the threshold k, setting the grayscale value of pixels above the threshold k to 255, and setting the grayscale value of pixels below the threshold k to 0, and obtaining a binary mask of the tissue area.

[0015] Furthermore, the immunohistochemistry scoring module is specifically:

[0016] The immunohistochemistry scoring module includes two branch network units. The first branch network unit is used to extract the first eigenvector of the original colorectal cancer P53 immunohistochemical staining image. The second branch network unit takes the cell nucleus region staining image output by the cell nucleus region extraction module and the tissue region mask output by the effective tissue region extraction module as input, and learns the cell nucleus region staining intensity and positive staining percentage based on the attention mechanism to obtain a second eigenvector; the first eigenvector and the second eigenvector are spliced ​​and input into two fully connected layers with the activation function being Softmax. The fully connected layer outputs the immunohistochemistry scoring result; the immunohistochemistry scoring result includes four categories: strongly positive, positive, weakly positive, and negative.

[0017] Furthermore, the first branch network unit is specifically:

[0018] The first branch network unit uses a densely connected convolutional network as a skeleton network, including a first convolutional layer, a first dense convolutional block, a first transition layer, a first FCANet multi-spectral channel attention unit, a second dense convolutional block, a second transition layer, a second FCANet multi-spectral channel attention unit, a third dense convolutional block, a third transition layer, a third FCANet multi-spectral channel attention unit, and a fourth dense convolutional block connected in sequence.

[0019] Furthermore, the second branch network unit is specifically:

[0020] The second branch network unit specifically includes: an input layer and a fully connected layer connected in sequence.

[0021] Furthermore, the process of obtaining the second eigenvector also includes:

[0022] Vectorize the cell nucleus region staining image output by the input cell nucleus region extraction module and the tissue region mask output by the effective tissue region extraction module;

[0023] Specifically:

[0024] (1) Grayscale value division: The grayscale value range of each channel of RGB image is 0~255, and the grayscale value of each channel is divided into 26 levels every 10 grayscale values ​​starting from 0. Assume that the grayscale value of a pixel point in the R, G, and B channels of RGB image is expressed as [r i ,g i ,b i ], after grayscale division, you can use [R i ,G i ,B i ] is replaced by Ri =Floor(r i / 10), G i =Floor(g i / 10), B i =Floor(b i / 10);

[0025] (2) Constructing the feature vector: In order to map a point in the three-dimensional coordinate system to one dimension of a one-dimensional vector, the following mapping relationship is constructed: j = (R i *26+G i )*26+B i , j represents the gray value R i , G i 、B i The position after the point is mapped to a one-dimensional vector;

[0026] (3) Counting the number of pixels: Through steps (1) and (2), a mapping relationship from a point in the RGB space to a position in a one-dimensional vector has been established. All points in the image are mapped to one dimension so that the value of the corresponding position of the vector is equal to the number of pixels mapped to that position; and the feature vector is obtained.

[0027] Compared with the existing technology, the beneficial effects of the present invention are as follows: the present invention establishes an automated colorectal cancer P53 image scoring system, combines it with clinical immunohistochemistry scoring standards, and uses deep learning network learning to obtain the staining intensity and positive staining percentage characteristics of the cell nuclear region and calculate the immunohistochemistry score results. P53 is usually only expressed in the cell nucleus, and staining in the cytoplasm is considered a false positive when reading the film. In order to learn the staining intensity characteristics of the cell nuclear region, the cell nuclear region in the original image is segmented; in order to learn the positive staining percentage characteristics, the tissue region mask is extracted as a basis for calculating the positive staining percentage; the cell nuclear region image and tissue region mask are vectorized, and a color representation vector is constructed and input into the second branch network; the above-mentioned second branch network is spliced ​​with the first branch network through a parallel network structure, which enhances the model's attention to the characteristics of cell nuclear staining intensity and positive staining percentage, and improves the consistency of the scoring results with the scores of professional pathologists. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of the automatic scoring system for P53 immunohistochemistry images integrating auxiliary features provided by the present invention;

[0029] Figure 2 Schematic diagram of the process of cell nucleus segmentation and effective tissue area extraction in an embodiment of the present invention;

[0030] Figure 3 Schematic diagram of the process of separating the DAB staining channel and the hematoxylin staining channel in an embodiment of the present invention;

[0031] Figure 4 is the image of the cell nucleus region obtained by segmentation;

[0032] Figure 5 This is the original P53 immunohistochemical staining image;

[0033] Figure 6 This is the image after DAB channel enhancement;

[0034] Figure 7 Schematic diagram of tissue regions extracted for LC significance detection;

[0035] Figure 8 is the image after region connection and elimination;

[0036] Figure 9 is a schematic diagram of the first branch network structure;

[0037] Figure 10 Schematic diagram of the overall structure of the parallel network. DETAILED DESCRIPTION

[0038] To facilitate technical personnel to understand and implement the present invention, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings and operating examples.

[0039] The present invention discloses an automatic scoring system for P53 immunohistochemistry images integrating auxiliary features, such as Figure 1 The system includes a staining channel separation module, a cell nucleus region extraction module, an effective tissue region extraction module and an immunohistochemistry scoring module.

[0040] The staining channel separation module is used to separate the colorectal cancer P53 immunohistochemical staining image into hematoxylin, DAB and residual channels through a color deconvolution algorithm, such as Figure 2 shown.

[0041] The cell nucleus region extraction module is used to transform the colorectal cancer P53 immunohistochemical staining image into the HRD space, use the maximum inter-class variance method to segment the cell nucleus region mask, and then obtain the cell nucleus region staining image in the colorectal cancer P53 immunohistochemical staining image.

[0042] Specifically, the cell nucleus region segmentation module transforms the colorectal cancer P53 immunohistochemical staining image into the optical density space first by log operation, and then transforms it into the HRD space, obtains the grayscale image of the hematoxylin channel in the HRD space, and uses the characteristics of hematoxylin staining cell nuclei to segment the cell nucleus region mask using the maximum inter-class variance method, thereby obtaining the cell nucleus region staining image in the colorectal cancer P53 immunohistochemical staining image ( Figure 4 ).

[0043] The process of transforming from the optical density space to the HRD space includes: using the optical density value of the dye in the RGB channel of the optical density space to transform, the formula is as follows:

[0044] x hrd =M -1 x rgb (a)

[0045]

[0046] Where x hrd is the pixel point in the P53 immunohistochemical staining image of colorectal cancer in HRD space, x rgb is a pixel point in the P53 immunohistochemical staining image of colorectal cancer in the optical density space. Each row in the matrix M represents the optical density value of a certain dye in the RGB channel. In this example, the value of the matrix M is shown in formula (b).

[0047] The effective tissue region extraction module obtains the staining image of the DAB channel output by the staining channel separation module, detects the tissue region using the LC significance detection algorithm, then uses the adaptive binarization method to obtain a binary image of the tissue region, and uses a closing operation to connect adjacent regions in the binary image to obtain a relatively complete tissue region mask.

[0048] Specifically, the effective tissue region extraction module first performs dark enhancement on the DAB channel staining image output by the staining channel separation module. In this example, the dark region is enhanced by gamma transformation, and the gamma value is set to 0.5 ( Figure 6 ). The LC saliency detection algorithm is used to detect the effective tissue area in the image after dark enhancement, and then the adaptive binarization method is used to obtain the binarized image of the tissue area. The adjacent areas in the binarized image are connected using a closed operation to obtain a relatively complete tissue area mask, where the convolution kernel size of the closed operation is 9. The tissue area mask is further optimized, and the size of each connected domain in the tissue area mask is obtained using connected domain detection. In this example, the connected domain size threshold is set to 3000. The connected domain size threshold can be adjusted according to the image size to remove small areas caused by false detection and / or missed detection areas inside the tissue ( Figure 8 ).

[0049] Furthermore, the process of detecting effective tissue regions using the LC saliency detection algorithm includes: calculating the Euclidean distance from each pixel to other pixels as the global contrast, obtaining a contrast matrix with the same size as the original image, calculating the mean contrast of all pixels as the threshold k, setting the grayscale value of pixels above the threshold k to 255, and setting the grayscale value of pixels below the threshold k to 0, and obtaining a binary mask of the tissue region ( Figure 7 ).

[0050] The immunohistochemistry scoring module includes two branch network units, such as Figure 10 As shown, the first branch network unit is used to extract the first eigenvector of the original colorectal cancer P53 immunohistochemical staining image. The second branch network unit takes the cell nucleus region staining image output by the cell nucleus region extraction module and the tissue region mask output by the effective tissue region extraction module as input, and learns the cell nucleus region staining intensity and positive staining percentage based on the attention mechanism to obtain the second eigenvector. The first and second eigenvectors are concatenated and input into two fully connected layers with a softmax activation function. The fully connected layers output the immunohistochemical score results, which include four categories: strongly positive, positive, weakly positive, and negative.

[0051] Furthermore, the process of obtaining the second eigenvector through the second branch network unit also includes: vectorizing the cell nucleus region staining image output by the input cell nucleus region extraction module and the tissue region mask output by the effective tissue region extraction module: 1) grayscale value division. The grayscale value range of each channel of the RGB image is 0 to 255. In order to compress information and avoid too high a dimension after vectorization, the grayscale value of each channel is divided into 26 levels starting from 0 and every 10 grayscale values ​​are one level. Assume that the grayscale value of a pixel point in the R, G, and B channels of the RGB image is expressed as [r i ,g i ,b i ], after grayscale division, you can use [R i ,G i ,B i ] is replaced by R i =Floor(r i / 10), G i =Floor(g i / 10), B i =Floor(b i / 10). 2) Construct the feature vector. In order to map a point in the three-dimensional coordinate system to one dimension of the one-dimensional vector, the following mapping relationship is constructed: j = (R i *26+G i )*26+B i , j represents the gray value R i , G i 、B i The position of the point after being mapped to a one-dimensional vector. Because [R i ,G i ,B i] ranges from 0 to 25, so the length of the constructed feature vector is 26 cubed, or 17576. 3) Count the number of pixels. Through steps 1) and 2), a mapping relationship has been established from a point in RGB space to a position in a one-dimensional vector. All points in the image are mapped to one dimension, so that the value at the corresponding position in the vector equals the number of pixels mapped to that position. After these steps, a feature vector of length 17576 is obtained. The value at each position in the vector represents the number of points in a certain RGB range of the image, so this vector can be used to describe the pixel value distribution of the entire image.

[0052] The structure of the first branch network unit is: using a densely connected convolutional network (DenseNet) as the skeleton network, and setting the channel compression rate to 0.5 in this example, as shown in Figure 9 As shown, the first branch network unit includes a first convolutional layer, a first dense convolutional block, a first transition layer, a first FCANet multi-spectral channel attention unit, a second dense convolutional block, a second transition layer, a second FCANet multi-spectral channel attention unit, a third dense convolutional block, a third transition layer, a third FCANet multi-spectral channel attention unit, and a fourth dense convolutional block connected in sequence; wherein, in this example, the first dense convolutional block consists of 6 basic convolutional units, the second dense convolutional block consists of 12 basic convolutional units, the third dense convolutional block consists of 24 basic convolutional units, and the fourth dense convolutional block consists of 16 basic convolutional units, and each basic convolutional unit includes a convolutional layer with a convolution kernel size of 1×1 and a convolution layer with a convolution kernel size of 3×3.

[0053] Furthermore, the first branch network unit extracts the first feature vector of the original colorectal cancer P53 immunohistochemical staining image by inputting the original colorectal cancer P53 immunohistochemical staining image into the first convolutional layer, then into the first dense convolutional block, inputting the feature map extracted by the previous convolutional layer into the FCANet multi-spectral channel attention unit to learn the combination weights between channels, and finally inputting it into the transition layer for downsampling. This process is repeated in this way to obtain the first feature vector.

[0054] The second branch network unit consists of an input layer with 35,152 neurons and a fully connected layer with 1,024 neurons. The color representation vectors of the tissue region mask and the cell nucleus region staining map are quantized and input into the second branch network unit. The fully connected layer extracts features and performs dimensionality reduction to generate the second feature vector.

[0055] This paper establishes an automated colorectal cancer P53 image scoring system, integrated with clinical immunohistochemistry scoring criteria. Using a deep learning network, the system learns staining intensity and positive staining percentage features, and calculates the immunohistochemistry score. P53 is typically expressed only in the cell nucleus; staining in the cytoplasm is considered a false positive when reading images. To learn the nuclear region staining intensity features, the nuclear region in the original image is segmented. To learn the positive staining percentage features, a tissue region mask is extracted as a basis for calculating the positive staining percentage. The nuclear region and tissue region masks are synthesized into a multi-channel image and input into an auxiliary feature extraction network. This auxiliary feature extraction network is then spliced ​​with the original image feature extraction network through a parallel network structure, enhancing the model's focus on nuclear staining intensity and positive staining percentage features, and improving the consistency of the scoring results with those of professional pathologists.

[0056] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.

[0057] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. An automatic scoring system for colorectal cancer P53 immunohistochemistry images, characterized in that: The system comprises: The staining channel separation module is used to separate the colorectal cancer P53 immunohistochemical staining image into hematoxylin, DAB and residual channels through the color deconvolution algorithm; The cell nucleus region extraction module is used to transform the colorectal cancer P53 immunohistochemical staining image into HRD space, use the maximum inter-class variance method to segment the cell nucleus region mask, and obtain the cell nucleus region staining map in the colorectal cancer P53 immunohistochemical staining image; The effective tissue region extraction module obtains the DAB channel staining image output by the staining channel separation module, detects the tissue region using the LC saliency detection algorithm, and then uses the adaptive binarization method to obtain a binary image of the tissue region. The tissue region mask is obtained by connecting adjacent regions in the binary image using a closing operation. The immunohistochemistry scoring module is used to extract the first eigenvector of the original colorectal cancer P53 immunohistochemical staining image. Based on the cell nuclear region staining image output by the cell nuclear region extraction module and the tissue region mask output by the effective tissue region extraction module, the attention mechanism is used to learn the cell nuclear region staining intensity and positive staining percentage to obtain the second eigenvector. The first and second eigenvectors are concatenated and output through the fully connected layer to obtain the immunohistochemical scoring result. The immunohistochemistry scoring module is specifically: The immunohistochemistry scoring module includes two branch network units, the first branch network unit is used to extract the first eigenvector of the original colorectal cancer P53 immunohistochemical staining image, and the second branch network unit takes the cell nucleus region staining image output by the cell nucleus region extraction module and the tissue region mask output by the effective tissue region extraction module as input, and learns the cell nucleus region staining intensity and positive staining percentage based on the attention mechanism to obtain a second eigenvector; the first eigenvector and the second eigenvector are spliced ​​and input into two fully connected layers with the activation function being Softmax, and the fully connected layers output an immunohistochemical scoring result; the immunohistochemical scoring result includes four categories: strongly positive, positive, weakly positive, and negative; Among them, the first branch network unit is specifically: the first branch network unit uses a densely connected convolutional network as a skeleton network, including a first convolutional layer, a first dense convolutional block, a first transition layer, a first FCANet multi-spectral channel attention unit, a second dense convolutional block, a second transition layer, a second FCANet multi-spectral channel attention unit, a third dense convolutional block, a third transition layer, a third FCANet multi-spectral channel attention unit, and a fourth dense convolutional block connected in sequence; Among them, the second branch network unit is specifically: the second branch network unit specifically includes: an input layer and a fully connected layer connected in sequence.

2. The automatic scoring system for colorectal cancer P53 immunohistochemistry images according to claim 1, characterized in that: The cell nucleus region segmentation module transforms the colorectal cancer P53 immunohistochemical staining image into optical density space and then into HRD space through logarithmic operation, obtains the grayscale image of the hematoxylin channel in the HRD space, and uses the maximum inter-class variance method to segment the cell nucleus region mask, thereby obtaining the cell nucleus region staining image in the colorectal cancer P53 immunohistochemical staining image.

3. The automatic scoring system for colorectal cancer P53 immunohistochemistry images according to claim 2, characterized in that: The process of transforming from optical density space to HRD space includes: using the optical density value of the dye in the RGB channel of the optical density space to transform, the formula is as follows: ； Where, is the pixel point in the P53 immunohistochemical staining image of colorectal cancer in hrd space, It is a pixel point in the P53 immunohistochemical staining image of colorectal cancer in the optical density space. Each row in the matrix M represents the optical density value of a certain dye in the RGB channel.

4. The automatic scoring system for colorectal cancer P53 immunohistochemistry images according to claim 1, characterized in that: The effective tissue region extraction module first enhances the dark area of ​​the DAB channel staining image output by the staining channel separation module through gamma transformation, detects the effective tissue area of ​​the dark-enhanced image using the LC saliency detection algorithm, then uses the adaptive binarization method to obtain a binary image of the tissue area, and uses a closing operation to connect adjacent areas in the binary image to obtain a relatively complete tissue area mask; A connected domain size threshold is set, and connected domain detection is used to obtain the size of each connected domain in the tissue region mask. Small regions caused by false detection and / or missed detection regions within the tissue are removed, and the tissue region mask is further optimized.

5. The automatic scoring system for colorectal cancer P53 immunohistochemistry images according to claim 4, characterized in that: The process of detecting effective tissue areas using the LC saliency detection algorithm includes: calculating the Euclidean distance from each pixel to other pixels as the global contrast, obtaining a contrast matrix with the same size as the original image, taking the contrast mean of all pixels as the threshold k, setting the grayscale value of pixels above the threshold k to 255, and setting the grayscale value of pixels below the threshold k to 0, and obtaining a binary mask of the tissue area.

6. The automatic scoring system for colorectal cancer P53 immunohistochemistry images according to claim 1, characterized in that: The process of obtaining the second eigenvector also includes: Vectorize the cell nucleus region staining image output by the input cell nucleus region extraction module and the tissue region mask output by the effective tissue region extraction module; Specifically: (1) Grayscale value division: The grayscale value range of each channel of RGB image is 0~255. Starting from 0, every 10 grayscale values ​​are one level, and the grayscale value of each channel is divided into 26 levels. Assume that the grayscale value of a pixel point in the R, G, and B channels of the RGB image is expressed as , after grayscale division, can be used Instead, , , ; (2) Constructing eigenvectors: In order to map a point in a three-dimensional coordinate system to one dimension of a one-dimensional vector, the following mapping relationship is constructed: , j represents the gray value 、 、 The position after the point is mapped to a one-dimensional vector; (3) Counting the number of pixels: Through steps (1) and (2), a mapping relationship from a point in the RGB space to a position in a one-dimensional vector has been established. All points in the image are mapped to one dimension so that the value of the corresponding position of the vector is equal to the number of pixels mapped to that position; the feature vector is obtained.

Images