Immunohistochemistry slide grading method, apparatus, device, and storage medium

Abstract

The application discloses an immunohistochemical slice grading method, device, equipment and storage medium. The application determines an infiltration area in a slice image of an immunohistochemical slice, divides the infiltration area into a plurality of image blocks, inputs the image blocks into a pre-trained feature extraction model to obtain a feature map output by the feature extraction model, inputs the feature map into a pre-trained cell detection model to determine tumor cells of various intensity levels in the corresponding image blocks through the cell detection model, wherein the cell detection model jointly performs multi-task learning with a cell counting model during a training process, and according to the tumor cells of various intensity levels in the plurality of image blocks, the number proportions of the tumor cells of various intensity levels in the infiltration area are counted; and according to the number proportions of the tumor cells of various intensity levels, the grade of the immunohistochemical slice is evaluated, so that automatic grading of the immunohistochemical slice is realized, and dependence on a dyeing threshold value set by artificial experience and dependence on morphological operation are eliminated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an immunohistochemical slice grading method, device, equipment and storage medium. BACKGROUND

[0002] Human epidermal growth factor receptor 2 (HER2) status evaluation is a key link of precise diagnosis and treatment of breast cancer, and the immunohistochemical (IHC) score result directly determines whether the patient is suitable for targeted therapy. According to the ASCO / CAP international guidelines, the HER2 status needs to be graded by the integrity, intensity and distribution range of membrane staining. The traditional manual grading of HER2 by pathologists has significant limitations: on the one hand, due to the difference in experience of doctors, subjective judgment can easily lead to inconsistency of grading results; on the other hand, in the face of high-throughput detection needs, the low efficiency of manual grading has become a bottleneck in clinical diagnosis. Therefore, it is of great significance to develop an efficient and standardized automatic grading system to improve the consistency and efficiency of diagnosis.

[0003] Currently, based on traditional image processing technology, automatic grading of HER2 slices can be realized. Specifically, the pathological slice RGB image is converted to HSV color space, the H (hue), S (saturation), and V (lightness) channel extreme values of the deep staining, medium staining, and light staining regions are counted, and the HSV threshold range of membrane staining features is dynamically set; then the membrane contour is extracted based on morphological operation; finally, according to the grading indicators such as membrane integrity ratio and staining intensity distribution, the HER2 slice is divided into grades (0 / 1+ / 2+ / 3+) combined with fixed threshold. However, such methods have the following technical defects: morphological operation relies on pre-set structure elements (such as circular, rectangular kernel) for image processing, which has limited fitting ability for irregular membrane contours, reduces segmentation accuracy, and affects the accuracy of grading; relying on manual experience to set global or dynamic threshold, sensitive to interference factors such as uneven staining (such as local over-staining or decolorization), tissue folding or slice artifacts during slicing process, easy to cause threshold range deviating from the true staining characteristics, and then cause over-segmentation or under-segmentation, affecting the accuracy of grading; fixed morphological parameters and threshold strategy are difficult to adapt to the diversity of different staining batches, scanning devices or pathological subtypes, and manual adjustment of algorithm parameters is required to cope with heterogeneous samples in actual application, affecting the universality of automatic grading. SUMMARY

[0004] The application provides an immunohistochemical slice grading method, device, equipment and storage medium, which uses a cell detection model to detect tumor cells of various intensity levels in the image blocks corresponding to the infiltration area, so as to count the proportion of the number of tumor cells of various intensity levels in the infiltration area, grade the immunohistochemical slice according to the proportion of the number of tumor cells of various intensity levels, realize automatic grading of the immunohistochemical slice, get rid of the dependence on the dyeing threshold set by artificial experience and the dependence on morphological operation, adapt to various application scenarios, and improve the accuracy and universality of automatic grading of the immunohistochemical slice.

[0005] In a first aspect, the application provides an immunohistochemical slice grading method, comprising:

[0006] determining an infiltration area in a slice image of an immunohistochemical slice, and dividing the infiltration area into a plurality of image blocks;

[0007] inputting the image blocks into a pre-trained feature extraction model to obtain a feature map output by the feature extraction model;

[0008] inputting the feature map into a pre-trained cell detection model to determine tumor cells of various intensity levels in the corresponding image blocks by using the cell detection model; wherein the cell detection model jointly performs multi-task learning with a cell counting model during the training process;

[0009] counting the proportion of the number of tumor cells of various intensity levels in the infiltration area according to the tumor cells of various intensity levels in the plurality of image blocks;

[0010] grading the immunohistochemical slice according to the proportion of the number of tumor cells of various intensity levels.

[0011] In a second aspect, the application provides an immunohistochemical slice grading device, comprising:

[0012] An image block division module is configured to determine an infiltration area in a slice image of an immunohistochemical slice, and divide the infiltration area into a plurality of image blocks;

[0013] A feature map determination module is configured to input the image blocks into a pre-trained feature extraction model to obtain a feature map output by the feature extraction model;

[0014] A cell detection module is configured to input the feature map into a pre-trained cell detection model to determine tumor cells of various intensity levels in the corresponding image blocks by using the cell detection model; wherein the cell detection model jointly performs multi-task learning with a cell counting model during the training process;

[0015] The percentage statistics module is configured to count the percentage of tumor cells of various intensity levels in the infiltrated region based on the tumor cells of various intensity levels in the multiple image blocks.

[0016] The slide grading module is configured to assess the grade of the immunohistochemical slides based on the percentage of tumor cells at each of the various intensity levels.

[0017] Thirdly, this application provides an immunohistochemical slide grading device, comprising:

[0018] One or more processors; a storage device storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the immunohistochemical slide grading method as described in the first aspect.

[0019] Fourthly, this application provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the immunohistochemical slide grading method as described in the first aspect.

[0020] In this application, the infiltration region is determined in the slice image of an immunohistochemical slide and divided into multiple image blocks. These image blocks are then input into a pre-trained feature extraction model to obtain feature maps output by the model. The feature maps are then input into a pre-trained cell detection model, which identifies tumor cells of various intensities within the corresponding image blocks. During training, the cell detection model is combined with a cell counting model for multi-task learning. Based on the tumor cells of various intensities in the multiple image blocks, the proportion of tumor cells of each intensity in the infiltration region is statistically analyzed. The grade of the immunohistochemical slide is then evaluated based on the proportion of tumor cells of each intensity. Through these techniques, the infiltration region is segmented from the slice image and divided into multiple image blocks for tumor cell detection, avoiding the need to divide the entire slice image into blocks. This effectively reduces the number of image blocks, saves computational resources, and improves tumor cell detection efficiency. The cell detection model and cell counting model are trained using a multi-task learning approach. Through feature sharing and joint optimization, the cell detection model acquires global statistical information and feature space constraints from the counting task, thereby improving the detection accuracy, robustness, and generalization ability of the cell detection model. Based on the detection results of the cell detection model, the proportion of tumor cells of various intensity levels is determined, and the grade of immunohistochemical sections is quantitatively evaluated. This enables automatic grading of immunohistochemical sections while eliminating dependence on staining thresholds set by human experience and morphological manipulation. It is adaptable to various application scenarios and improves the accuracy and universality of automatic grading of immunohistochemical sections. Attached Figure Description

[0021] Figure 1 This is a flowchart of an immunohistochemical slide grading method provided in an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating the determination of the infiltrated area of ​​a sliced ​​image, provided in an embodiment of this application.

[0023] Figure 3 This is a flowchart of the training region segmentation model provided in the embodiments of this application;

[0024] Figure 4 This is a flowchart of the feature map extraction of image blocks provided in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram of an image block provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the tumor cell mask provided in the embodiments of this application;

[0027] Figure 7 This is a flowchart of the training feature extraction model and multi-task learning model provided in the embodiments of this application;

[0028] Figure 8 This is a model architecture diagram of the feature extraction model and multi-task learning model provided in the embodiments of this application;

[0029] Figure 9 This is a schematic diagram of the structure of an immunohistochemical slide grading device provided in an embodiment of this application;

[0030] Figure 10 This is a schematic diagram of the structure of an immunohistochemical slide grading device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0032] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0033] In some implementations, HER2 slides can be automatically graded using traditional image processing techniques. First, the RGB image of the pathological slide is converted to the HSV color space. Then, by statistically analyzing the extreme values ​​of the HSV channels in the dark, medium, and light stained regions, the HSV threshold range for membrane staining features is dynamically set. Next, the membrane contour is extracted based on morphological operations. Finally, the HER2 slides are graded according to grading indicators such as membrane integrity ratio and staining intensity distribution, combined with a fixed threshold. However, morphological operations rely on preset structural elements for image processing, which have limited ability to fit irregular membrane contours, reducing segmentation accuracy and affecting grading accuracy. Global or dynamic thresholds rely on manual experience and are easily affected by factors such as uneven staining, tissue folding, or slide artifacts during slide preparation, causing the threshold range to deviate from the true staining features, leading to over-segmentation or under-segmentation, and reducing the accuracy of slide grading. Fixed morphological parameters and threshold strategies are difficult to adapt to the diversity of different staining batches, scanning equipment, or pathological subtypes, requiring manual adjustment of algorithm parameters to cope with heterogeneous samples in practical applications, affecting the universality of automated grading.

[0034] To address the aforementioned issues, this embodiment provides an immunohistochemical slide grading method. This method utilizes a cell detection model to detect tumor cells of various intensity levels within image blocks corresponding to the infiltrated region. It then statistically analyzes the proportion of tumor cells of different intensity levels within the infiltrated region and grades the immunohistochemical slides based on this proportion. This achieves automatic grading of immunohistochemical slides while eliminating reliance on manually set staining thresholds and morphological manipulations. It adapts to various application scenarios, improving the accuracy and universality of automatic grading of immunohistochemical slides.

[0035] The immunohistochemical slide grading method provided in this embodiment can be executed by an immunohistochemical slide grading device. This device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. For example, the immunohistochemical slide grading device can be a powerful intelligent terminal such as a computer or server. The server can be a standalone server or a server cluster consisting of multiple servers.

[0036] The immunohistochemical slide grading device is equipped with at least one type of operating system. The device can install at least one application program based on this operating system. This application program can be a built-in application of the operating system or an application downloaded from a third-party device or server. In this embodiment, the immunohistochemical slide grading device has at least one application program installed that can execute the immunohistochemical slide grading method.

[0037] For ease of understanding, this embodiment uses a computer as the main device for performing the immunohistochemical slide grading method.

[0038] Figure 1 A flowchart of an immunohistochemical slide grading method provided in an embodiment of this application is given. (Reference) Figure 1 The immunohistochemical section grading method specifically includes:

[0039] S110. Identify the infiltrated region in the slice image of the immunohistochemical section and divide the infiltrated region into multiple image blocks.

[0040] Immunohistochemical sections, also known as immunohistochemical sections, are sections created by the specific binding of labeled antibodies to antigens in tissue samples to reveal the antigens. The process involves preparing tissue sections from the tissue sample, using fluorescein- or enzyme-labeled antibodies to specifically bind to antigens in the tissue sections, and a chemical reaction causing a color change at the binding site. The location of the color change indicates the antigen location, resulting in an immunohistochemical tissue section marked with the antigen location. Immunohistochemical sections include estrogen receptor (ER) sections, progesterone receptor (PR) sections, and human epidermal growth factor receptor 2 (HER2) sections. ER sections are tissue sections labeled with estrogen receptors, PR sections with progesterone receptors, and HER2 sections with human epidermal growth factor receptor 2. In the medical field, detecting the expression status of progesterone receptors, progesterone receptors, and human epidermal growth factor receptor 2 in breast cancer tissue sections can help determine the molecular subtype of breast cancer, providing a basis for targeted therapy and endocrine therapy.

[0041] The slice image of an immunohistochemical slide is a digital image obtained by scanning the immunohistochemical slide using a whole-slice scanner. Slice images can have resolutions ranging from 50,000 to 200,000 pixels. Directly detecting the infiltration region in the slice image is computationally intensive. Therefore, the slice image can be divided into multiple sub-images. The infiltration region in each sub-image is detected, and the infiltration regions in the sub-images are then stitched together to obtain the infiltration region in the slice image. For example, a sliding window of a fixed size (typically 1024*1024) is used to crop the slice image. The sliding window moves according to a preset step size, and each movement crops the area of ​​the slice image within the sliding window as a sub-image. After the sliding window has traversed the entire slice image, all sub-images are obtained. A pre-trained region segmentation model performs semantic segmentation on the sub-images to obtain infiltration region masks, in-situ region masks, and background region masks. Based on the position of each sub-image in the slice image, the corresponding infiltration region mask is mapped onto the slice image to obtain the infiltration region mask in the slice image. The infiltration region in the slice image is then segmented using the infiltration region mask in the slice image.

[0042] Optionally, detecting infiltration regions using sub-images of a sliced ​​image requires semantic segmentation of multiple sub-images, resulting in a large computational load and necessitating substantial computing resources. To improve the detection efficiency of infiltration regions, they can be detected directly within the scaled-down sliced ​​image, effectively reducing the number of semantic segmentation operations and lowering the computational cost. Specifically, Figure 2 This is a flowchart illustrating the determination of the infiltration region of a sliced ​​image, provided in an embodiment of this application. For example... Figure 2 As shown, the step of determining the infiltrated area of ​​the slice image specifically includes S1101-S1103:

[0043] S1101. Downsample the slice image of the immunohistochemical section into a slice thumbnail.

[0044] For example, the slice image is downsampled by 32 times, that is, reduced by 32 times, to obtain a slice thumbnail.

[0045] S1102. Input the slice thumbnail into the pre-trained region segmentation model to obtain the infiltrated region mask output by the region segmentation model.

[0046] For example, the Unet network is used as the region segmentation model. The Unet network includes an encoder and a decoder, with the encoder employing the MobileNet V3-large network. The slice thumbnail is input to the encoder, which progressively compresses the spatial size of the slice thumbnail and increases its depth through multiple convolutional blocks, ultimately outputting a feature map with a small spatial size but high depth. This feature map is then input to the decoder, which progressively recovers the spatial size of the feature map through upsampling. The upsampled feature map is then concatenated with feature maps of the same size from the encoder to combine the low-level features extracted by the encoder with the high-level features extracted by the decoder, preserving both detailed and global information and improving segmentation accuracy. Finally, the decoder outputs a semantic segmentation mask with the same size as the slice thumbnail. Each pixel in the semantic segmentation mask corresponds to semantic information of a region category, such as an infiltrated region, an in-situ region, or a background region. Pixels in the semantic segmentation mask that are semantically classified as infiltrated regions form the infiltrated region mask.

[0047] In this embodiment, Figure 3 This is a flowchart of the training region segmentation model provided in an embodiment of this application. For example... Figure 3 As shown, the specific steps for training the region segmentation model include S210-S240:

[0048] S210. Obtain the slice sample image and the corresponding infiltration region annotation mask, and downsample the slice sample image and the infiltration region annotation mask to obtain the slice sample thumbnail and the mask thumbnail.

[0049] The slice sample image is used to train the region segmentation model, and the infiltrated region labeling mask is the label information of the slice sample image. Each pixel in the infiltrated region labeling mask is labeled with the corresponding region category. After downsampling the slice sample image and the corresponding infiltrated region labeling mask by 32 times, slice sample thumbnails and mask thumbnails are obtained respectively.

[0050] S220. Input the slice sample thumbnail into the region segmentation model to obtain the first predicted probability of each pixel in the slice sample thumbnail output by the region segmentation model belonging to various regions.

[0051] For example, a slice sample thumbnail is input into a region segmentation model. The region segmentation model performs semantic segmentation on the slice sample thumbnail and outputs a corresponding semantic segmentation mask. Each pixel in the semantic segmentation mask is marked with the first predicted probability that the pixel belongs to the infiltrated region, the in-situ region, or the background region, as predicted by the region segmentation model.

[0052] S230. Determine the dice loss value and focus loss value based on the first predicted probability corresponding to each pixel and the first true value corresponding to each pixel in the mask thumbnail.

[0053] For example, the first true value is used to characterize the true region category of a pixel. When the region category of a pixel in the mask thumbnail is a wetted region, the first true value of the pixel belonging to the wetted region is 1, and the first true value of the pixel belonging to the in-situ region and the background region is 0; when the region category of a pixel in the mask thumbnail is an in-situ region, the first true value of the pixel belonging to the in-situ region is 1, and the first true value of the pixel belonging to the wetted region and the background region is 0; when the region category of a pixel in the mask thumbnail is a background region, the first true value of the pixel belonging to the background region is 1, and the first true value of the pixel belonging to the wetted region and the background region is 0.

[0054] Based on the first predicted probability and the first true value corresponding to each pixel, the dice loss value can be calculated using the calculation expression of the dice loss function. The calculation expression of the dice loss function is as follows:

[0055]

[0056] Where C represents the total number of region categories. In this embodiment, the region is divided into in-situ region, infiltration region, and background region, so C=3. For pixels Category The first true value, For pixels Category The first predicted probability, It is a very small constant used to prevent the denominator from being zero.

[0057] Based on the first predicted probability and the first true value corresponding to each pixel, the focus loss value can be calculated using the expression for the focus loss function. The expression for the focus loss function is as follows:

[0058]

[0059] Where N is the total number of pixels. For category The balance factor This is the modulation index.

[0060] S240. The first total loss value is obtained by weighted summation of the dice loss value and the focus loss value. The model parameters of the region segmentation model are then optimized in reverse using the first total loss value.

[0061] For example, after determining the dice loss value and the focus loss value, the dice loss value and the focus loss value are multiplied by their corresponding weighting coefficients and then added together to obtain the first total loss value. The expression for calculating the first total loss value is:

[0062]

[0063] Where 0.8 is the weighting coefficient for the dice loss value. The weighting coefficients for the focus loss value.

[0064] Next, the model parameters of the decoder and encoder in the region segmentation model are optimized in reverse based on the first total loss value, thus completing one training iteration of the region segmentation model. When the number of training iterations of the region segmentation model reaches the upper limit, or when the model parameters of the region segmentation model meet the convergence condition, the region segmentation model is confirmed to have completed training, and the trained region segmentation model is applied to the real-time slice detection process.

[0065] S1103. Map the immersion region mask onto the slice image to obtain the immersion region in the slice image.

[0066] For example, based on the image ratio between the immersion region mask and the slice image, the immersion region mask is enlarged proportionally, and then the enlarged immersion region mask is mapped onto the slice image to segment the immersion region in the slice image.

[0067] This embodiment downsamples the slice image into a slice thumbnail and then uses a region segmentation model to detect the infiltration region mask in the slice thumbnail in one step. The infiltration region of the slice image is determined by the infiltration region mask. The detection of the infiltration region in the slice image is achieved by semantic segmentation only once, which effectively improves the detection efficiency of the infiltration region.

[0068] Furthermore, when the slice image size is large, it needs to be significantly downsized to obtain a size suitable for region segmentation models. However, large-scale downsampling can distort the content of the slice image, affecting the segmentation accuracy of the infiltrated region. To address this, when the slice image size exceeds a preset threshold, a sliding window of a first preset size can be used to extract multiple sub-images from the slice image. Each sub-image is then downsampled by a preset factor to obtain a thumbnail. A pre-trained region segmentation model performs semantic segmentation on the thumbnail sub-images to obtain an infiltrated region mask. The infiltrated region in the slice image is then determined based on the infiltrated region mask of the thumbnail sub-image. In essence, when the slice image size is large, it can be divided into a small number of sub-images, and the thumbnails of these sub-images can be used for infiltrated region detection, ensuring both accuracy and efficiency in region detection.

[0069] After identifying the infiltrated region in the sliced ​​image, a 1024*1024 sliding window can be used to crop the infiltrated region to obtain multiple image blocks. Pixels in the image blocks that do not belong to the infiltrated region are filled with 0 or 255.

[0070] In addition, when the infiltration region is detected based on a thumbnail of the slice image, the accuracy of the infiltration region may decrease. To ensure the accuracy of subsequent tumor cell detection, a 1024*1024 sliding window can be used to slice the slice image to obtain multiple sub-images. The sub-image containing the infiltration region is used as an image block. The image block will include the infiltration region and the corresponding surrounding region. Tumor cells can then be detected in the infiltration region and the corresponding surrounding region, thereby reducing the impact of the infiltration region's accuracy on tumor cell detection.

[0071] S120. Input the image patch into the pre-trained feature extraction model to obtain the feature map output by the feature extraction model.

[0072] For example, the feature extraction model can be a convolutional neural network, which extracts feature maps from image patches. The feature extraction model is trained together with the cell detection model.

[0073] Optionally, CSRNet (Crowd Scene Recognition Network) can be used as the feature extraction model. CSRNet includes a front-end network (Encoder) and a back-end network (backend), which extract feature maps of image patches. Figure 4 This is a flowchart illustrating the feature map extraction of image blocks provided in an embodiment of this application. For example... Figure 4 As shown, the steps for extracting the feature map of the image patch specifically include S1201-S1202:

[0074] S1201. Input the image block into the front-end network, and downsample the image block through multiple 3*3 convolutional layers of the front-end network to obtain the downsampled image block.

[0075] For example, the front-end network includes five 3*3 convolutional layers, which downsample the 1024*1024 image block by 8 times to obtain a 128*128 image block.

[0076] S1202. Input the downsampled image patch into the backend network, and convolve the downsampled image patch through multiple dilated convolutional layers of the backend network to obtain the feature map.

[0077] For example, a 128*128 image patch is input into the backend network, which includes 5 dilated convolutional layers. The 128*128 image patch is further enlarged by the 5 dilated convolutional layers, and a 128*128 feature map is output.

[0078] It should be noted that the feature extraction model is trained together with the multi-task learning model for cell detection and cell counting. CSRNet is used as the front-end feature extraction model of the multi-task learning model. The front-end network of CSRNet can effectively capture the detailed information of tumor cells in image patches, providing high-quality feature representations for detection and counting tasks. The back-end network of CSRNet significantly expands the receptive field to capture a wider range of contextual information without increasing the number of parameters and computational cost through dilated convolution. This helps to handle cell-dense scenes, reduce detection and counting errors caused by cell overlap or occlusion, and effectively improve the detection and counting accuracy of the multi-task learning model.

[0079] S130. Input the feature map into the pre-trained cell detection model, and use the cell detection model to determine the tumor cells of various intensity levels in the corresponding image block; wherein, the cell detection model performs multi-task learning in conjunction with the cell counting model during the training process.

[0080] For example, during training, the cell counting model and the cell detection model form a multi-task learning model. The model parameters of the multi-task learning model and the feature extraction model are jointly optimized based on the counting and detection results output by the counting task head and the detection task head of the multi-task learning model. This enables the cell detection model to obtain global statistical information and feature space constraints from the counting task, thereby improving the detection accuracy, robustness and generalization ability of the cell detection model.

[0081] In practical applications, the detection task head from the multi-task learning model is retained as the cell detection model. The feature map output from the feature extraction model is input into the cell detection model. The cell detection model upsamples the 128*128 feature map to output a tumor cell mask for each intensity level. The size of the tumor cell mask is 1024*1024. The pixel value of each pixel in the tumor cell mask represents whether the pixel is a tumor cell of the corresponding intensity level. For example, when the pixel value of pixel X in the tumor cell mask of intensity level 1+ is 255, it indicates that the pixel is a tumor cell of intensity level 1+; when the pixel value of pixel X is 0, it indicates that the pixel is not a tumor cell of intensity level 1+. For example, Figure 5 This is a schematic diagram of an image block provided in an embodiment of this application. Figure 6 This is a schematic diagram of the tumor cell mask provided in an embodiment of this application. Figure 5 and Figure 6 As shown, Figure 5 After the image patch shown is input into the feature extraction model, the feature extraction model inputs the corresponding extracted feature map into the cell detection model. The cell detection model outputs a tumor cell mask with four channels, where each channel represents an intensity level, thus obtaining... Figure 6Four tumor cell masks are shown. Category 0 represents tumor cells with an intensity level of 0 (negative), Category 1 represents tumor cells with an intensity level of 1+ (weakly positive), Category 2 represents tumor cells with an intensity level of 2+ (moderately positive), and Category 3+ represents tumor cells with an intensity level of 3+ (strongly positive). (Reference) Figure 6 In the tumor cell mask corresponding to category 0, a white pixel corresponds to a negative tumor cell; in the tumor cell mask corresponding to category 1, a white pixel corresponds to a weakly positive tumor cell; and in the tumor cell masks corresponding to categories 2 and 3, there are no white pixels, meaning that there are no moderately positive or strongly positive tumor cells in this image patch.

[0082] It needs to be explained that, Figure 6 The four intensity levels of tumor cell masks shown are for HER2 sections, as tumor cells in HER2 sections are classified into four intensity levels: 0, 1+, 2+, and 3+. Other types of sections may be classified into two intensity levels, such as 0 and 1, or other numbers of intensity levels. The number of intensity levels depends on the type of immunohistochemical section and is not specifically limited here.

[0083] In one embodiment, Figure 7 This is a flowchart of the training feature extraction model and multi-task learning model provided in the embodiments of this application. Figure 7 As shown, the specific steps for training the feature extraction model and the multi-task learning model include S310-S340:

[0084] S310. Obtain the sample image block and the corresponding tumor cell annotation masks of various intensity levels and the number of tumor cell annotations. Input the sample image block into the feature extraction model to obtain the sample feature map output by the feature extraction model.

[0085] For example, a sliding window is used to capture images of the infiltrated region in a sliced ​​sample image to obtain a sample image patch. Then, a sliding window is used to capture images of tumor cell annotation masks of various intensity levels within the infiltrated region to obtain the tumor cell annotation mask corresponding to the sample image patch. Finally, the number of tumor cells at the corresponding intensity level in the tumor cell annotation mask is counted to obtain the number of tumor cell annotations in the sample image patch. Here, each pixel in the tumor cell annotation mask is labeled with its corresponding tumor cell category.

[0086] It should be noted that the tumor cell annotation mask can be generated simply by doctors marking the corresponding tumor cells in the slide sample image. This annotation cost is lower than that of cell segmentation annotation and is simpler and faster to operate. Furthermore, the multiple intensity levels of the tumor cell annotation mask allow the cell detection model to directly output tumor cells of different intensity levels. Compared to the cell nuclear segmentation model, which still requires additional cell nuclear analysis to obtain cell categories, the cell detection model's operating logic is also simpler, reducing computational complexity and improving the efficiency of tumor cell detection.

[0087] Figure 8 This is a model architecture diagram of the feature extraction model and multi-task learning model provided in the embodiments of this application. For example... Figure 8 As shown, the sample image patch is input into the front-end network of the feature extraction model. The front-end network downsamples the 1024*1024 sample image patch into a 128*128 sample image patch through a convolutional layer and then inputs it into the back-end network. The back-end network expands the receptive field of the 128*128 sample image patch through a dilated convolutional layer and outputs a 128*128 sample feature map.

[0088] S320. Input the sample feature map into the cell detection model and the cell counting model to obtain the second predicted probability of each pixel point output by the cell detection model as a tumor cell of various intensity levels, and to obtain the predicted number of tumor cells of various intensity levels output by the cell counting model.

[0089] For example, the sample feature map is input into a cell detection model and a cell counting model. The cell detection model restores the size of the sample feature map to 1024*1024 and classifies tumor cells of various intensity levels, finally outputting tumor cell prediction masks for each intensity level. The tumor cell prediction mask labels a second prediction probability for each pixel, which represents the probability that the pixel is a tumor cell of the corresponding intensity level. The cell counting model classifies tumor cells of various intensity levels in the sample feature map, counts the number of tumor cells of each intensity level based on the classification results, and outputs the predicted number of tumor cells of each intensity level.

[0090] refer to Figure 8 The cell detection model acts as a decoder, composed of multiple transposed convolutions. These transposed convolutions upsample the sample feature map back to a 1024*1024 size, outputting tumor cell prediction masks of various intensity levels. The cell counting model consists of global pooling layers and fully connected layers (FC). The global pooling layer compresses the spatial dimension of the sample feature map into a single value while preserving channel dimension information to obtain a feature vector. This feature vector is then input into the fully connected layer, which maps the feature vector to output the predicted number of tumor cells at various intensity levels.

[0091] S330. Determine the cell detection loss value based on the second predicted probability corresponding to each pixel and the second true value of each pixel in the tumor cell annotation mask, and determine the cell counting loss value based on the predicted number of tumor cells of various intensity levels and the number of tumor cells annotated.

[0092] The second true value indicates whether a pixel is a tumor cell of the corresponding intensity level in the tumor cell annotation mask; if so, the second true value is 1, otherwise it is 0. The cell detection loss value between the second predicted probability and the second true value can be determined using a variance loss function. The expression for the variance loss function is as follows:

[0093]

[0094] Where H and W are the height and width of the sample image patch, and x and y are the x and y coordinates of the pixel. The number of intensity levels, Let (x, y) be the second predicted probability of pixel (x, y) being intensity level n. Let pixel (x, y) be the second true value for intensity level n. It can be understood that the variance loss function can be used to calculate the sum of the variances of the second predicted probability and the second true value for each pixel at various intensity levels, and this sum of variances can be used as the cell detection loss value.

[0095] The cell counting loss value between the predicted number of tumor cells and the labeled number of tumor cells is determined using an absolute error loss function. The expression for the absolute error loss function is as follows:

[0096]

[0097] in, Predict the number of tumor cells for intensity level n. Let be the number of tumor cells labeled at intensity level n. This can be understood as the sum of the absolute values ​​of the differences between the predicted number of tumor cells at each intensity level and the number of labeled tumor cells, used as the cell detection loss value.

[0098] S340. Determine the second total loss value based on the cell detection loss value and the cell counting loss value, and use the second total loss value to back-optimize the model parameters of the cell detection model, the cell counting model, and the feature extraction model.

[0099] For example, the cell detection model and the cell counting model serve as task heads in a multi-task learning model. The loss values ​​of the two task heads can be combined to calculate a second total loss value. This second total loss value is then used to inversely optimize the model parameters of the multi-task learning model and the feature extraction model. Specifically, the sum of the cell detection loss value and the cell counting loss value can be used as the second total loss value. This second total loss value is then used to inversely optimize the model parameters of the cell detection model, the cell counting model, and the feature extraction model, thus completing one training iteration of the cell detection model, the cell counting model, and the feature extraction model. When the training iterations reach their maximum limit, or when the model parameters of the cell detection model, the cell counting model, and the feature extraction model converge, the training of the cell detection model, the cell counting model, and the feature extraction model is complete. The trained cell detection model and the feature extraction model can then be deployed on a computer to detect the number of tumor cells of various intensity levels during real-time slice detection.

[0100] This embodiment uses a feature extraction model as a shared backbone network for both the detection and counting tasks in a multi-task learning model, thereby extracting common cell morphology features. Feature sharing enables the feature extraction model to learn expressions that are effective for both tasks, avoiding overfitting of the cell detection model to local noise. The counting task is used as an auxiliary task, providing additional supervision signals, constraining the shared feature space, and reducing the sensitivity of the detection task to noise in the training data. The counting task requires the model to not miss any cells, which can improve the detection task's ability to detect dense, overlapping, or low-contrast regions. Furthermore, the shared features enhance the detection task's ability to distinguish noise and false positives, effectively improving the detection accuracy.

[0101] S140. Based on the tumor cells of various intensity levels in multiple image patches, calculate the percentage of tumor cells of various intensity levels in the infiltrated area.

[0102] For example, based on the white pixels in the mask of tumor cells of each intensity level in the image patch, the number of tumor cells of each intensity level in the image patch is counted. The number of tumor cells of each intensity level in multiple image patches is accumulated to obtain the total number of tumor cells of each intensity level in the infiltrated region. The total number of tumor cells of each intensity level is then accumulated again to obtain the total number of tumor cells in the infiltrated region. The ratio of the total number of tumor cells of each intensity level to the total number of tumor cells in the infiltrated region is taken as the proportion of the number of tumor cells of each intensity level in the infiltrated region.

[0103] S150. Assess the grade of immunohistochemical sections based on the proportion of tumor cells of various intensity levels.

[0104] For example, when the immunohistochemical section is a HER2 section, tumor cells are classified into four intensity levels: negative, weakly positive, moderately positive, and strongly positive. Similarly, HER2 sections are also classified into four intensity levels: negative, weakly positive, moderately positive, and strongly positive. The intensity level of the HER2 section can be assessed based on the proportion of tumor cells in these four intensity levels.

[0105] For example, the percentage of strongly positive tumor cells is compared with a preset percentage threshold. If the percentage of strongly positive tumor cells is greater than the preset percentage threshold, the HER2 section is classified as strongly positive. Otherwise, the cumulative percentage of moderately positive and strongly positive tumor cells is compared with the preset percentage threshold. If the cumulative percentage of the first percentage is greater than the preset percentage threshold, the HER2 section is classified as moderately positive. Otherwise, the cumulative percentage of the second percentage of weakly positive, moderately positive, and strongly positive tumor cells is compared with the preset percentage threshold. If the cumulative percentage of the second percentage is greater than the preset percentage threshold, the HER2 section is classified as weakly positive; otherwise, the HER2 section is classified as negative. The preset percentage threshold is the minimum percentage of tumor cells higher than or belonging to the intensity level of the HER2 section, typically ranging from 10% to 30%. For example, when the HER2 section is strongly positive, the percentage of strongly positive tumor cells can be as high as 10% to 30%; when the HER2 section is moderately positive, the total percentage of strongly positive and moderately positive tumor cells can be as high as 10% to 30%. In other words, when the percentage of strongly positive tumor cells is greater than a preset percentage threshold, it indicates that a significant number of tumor cells within the infiltrated area have converted to strongly positive, thus determining the HER2 section as strongly positive (3+). If the percentage of strongly positive tumor cells is less than or equal to the preset percentage threshold, it indicates that not many tumor cells within the infiltrated area have converted to strongly positive. In this case, the percentage of moderately positive tumor cells can be added to the percentage of strongly positive tumor cells to obtain a first cumulative percentage value. The first cumulative percentage value is compared with the preset percentage threshold to confirm whether there are enough strongly positive and moderately positive tumor cells. When the first cumulative percentage value exceeds the preset percentage threshold, it indicates that a significant number of tumor cells within the infiltrated area have converted to moderately and strongly positive, thus determining the HER2 section as moderately positive (2+). If the first cumulative percentage value is less than or equal to the preset percentage threshold, it indicates that not many tumor cells within the infiltrated area have converted to strongly or moderately positive. In this case, the percentages of weakly positive tumor cells, moderately positive tumor cells, and strongly positive tumor cells can be added to obtain a second cumulative percentage value. The cumulative value of the second percentage is compared with a preset percentage threshold to confirm whether there are enough weakly positive tumor cells. When the cumulative value of the second percentage exceeds the preset percentage threshold, it indicates that a large number of tumor cells in the infiltrated area have converted to weakly positive, moderately positive, or strongly positive, thus determining the HER2 section as weakly positive (1+). If the cumulative value of the second percentage is less than or equal to the preset percentage threshold, it indicates that not many tumor cells in the infiltrated area have converted to strongly positive, moderately positive, or weakly positive, and the HER2 section can be determined as negative (0).

[0106] This embodiment calculates the proportion of tumor cells of various intensity levels within the infiltration area, and quantitatively assesses the intensity level of immunohistochemical sections based on the proportion of cells and a preset proportion threshold. This achieves automatic grading of immunohistochemical sections while eliminating reliance on manually set staining thresholds and morphological manipulations, adapting to various application scenarios and improving the accuracy and universality of automatic grading of immunohistochemical sections.

[0107] In summary, the immunohistochemical slide grading method provided in this application identifies the infiltrative region in the slide image and divides it into multiple image blocks. The image blocks are then input into a pre-trained feature extraction model to obtain feature maps output by the model. These feature maps are then input into a pre-trained cell detection model to identify tumor cells of various intensity levels within the corresponding image blocks. During training, the cell detection model is combined with a cell counting model for multi-task learning. Based on the tumor cells of various intensity levels in the multiple image blocks, the proportion of tumor cells of each intensity level in the infiltrative region is statistically analyzed. Finally, the grade of the immunohistochemical slide is evaluated based on the proportion of tumor cells of each intensity level. Through these techniques, the infiltrative region is segmented from the slide image and divided into multiple image blocks for tumor cell detection. This avoids dividing the entire slide image into image blocks, effectively reducing the number of image blocks, saving computational resources, and improving tumor cell detection efficiency. The cell detection and cell counting models are trained using a multi-task learning approach. Through feature sharing and joint optimization, multi-task learning enables the cell detection model to acquire global statistical information and feature space constraints from the counting task, thereby improving the detection accuracy, robustness, and generalization ability of the cell detection model. Based on the detection results of the cell detection model, the proportion of tumor cells at various intensity levels is determined, and the grade of immunohistochemical sections is quantitatively evaluated. This achieves automatic grading of immunohistochemical sections while eliminating dependence on manually set staining thresholds and morphological manipulations, adapting to various application scenarios and improving the accuracy and universality of automatic grading of immunohistochemical sections.

[0108] Based on the above embodiments, Figure 9 This is a schematic diagram of an immunohistochemical slide grading device provided in an embodiment of this application. (Reference) Figure 9 The immunohistochemical slide grading device provided in this embodiment specifically includes: an image block division module 41, a feature map determination module 42, a cell detection module 43, a proportion statistics module 44, and a slide grading module 45.

[0109] The image block division module 41 is configured to determine the infiltration region in the slice image of the immunohistochemical slide and divide the infiltration region into multiple image blocks.

[0110] The feature map determination module 42 is configured to input image patches into a pre-trained feature extraction model to obtain feature maps output by the feature extraction model.

[0111] The cell detection module 43 is configured to input the feature map into a pre-trained cell detection model, and use the cell detection model to identify tumor cells of various intensity levels in the corresponding image patch; wherein, the cell detection model performs multi-task learning in conjunction with the cell counting model during the training process;

[0112] The percentage statistics module 44 is configured to count the percentage of tumor cells of various intensity levels in the infiltrated region based on the tumor cells of various intensity levels in multiple image blocks.

[0113] The slide grading module 45 is configured to assess the grading of immunohistochemical slides based on the percentage of tumor cells at various intensity levels.

[0114] Based on the above embodiments, the image block segmentation module 41 includes: a thumbnail acquisition unit, configured to downsample the slice image of the immunohistochemical slice into a slice thumbnail; an infiltrate region segmentation unit, configured to input the slice thumbnail into a pre-trained region segmentation model to obtain an infiltrate region mask output by the region segmentation model; and an infiltrate region determination unit, configured to map the infiltrate region mask onto the slice image to obtain the infiltrate region in the slice image.

[0115] Based on the above embodiments, the immunohistochemical slide grading device includes a first model training module, which includes: a first sample acquisition unit configured to acquire a slide sample image and a corresponding infiltration region annotation mask, and downsample the slide sample image and the infiltration region annotation mask to obtain a slide sample thumbnail and a mask thumbnail; a first prediction unit configured to input the slide sample thumbnail into a region segmentation model to obtain a first predicted probability that each pixel in the slide sample thumbnail output by the region segmentation model belongs to various regions; a first loss value determination unit configured to determine a dice loss value and a focus loss value based on the first predicted probability corresponding to each pixel and the first true value corresponding to each pixel in the mask thumbnail; and a first model optimization unit configured to perform a weighted summation of the dice loss value and the focus loss value to obtain a first total loss value, and back-optimize the model parameters of the region segmentation model using the first total loss value.

[0116] Based on the above embodiments, the feature extraction model includes a front-end network and a back-end network; correspondingly, the feature map determination module 42 includes: a downsampling unit, configured to input an image block into the front-end network, and downsample the image block through multiple 3*3 convolutional layers of the front-end network to obtain a downsampled image block; and a feature map determination unit, configured to input the downsampled image block into the back-end network, and convolve the downsampled image block through multiple dilated convolutional layers of the back-end network to obtain a feature map.

[0117] Based on the above embodiments, the immunohistochemical slide grading device includes a second model training module, which includes: a second sample acquisition unit configured to acquire sample image blocks and corresponding tumor cell annotation masks of various intensity levels and the number of tumor cell annotations, input the sample image blocks into a feature extraction model, and obtain a sample feature map output by the feature extraction model; a second prediction unit configured to input the sample feature map into a cell detection model and a cell counting model, obtain a second prediction probability that each pixel point output by the cell detection model is a tumor cell of various intensity levels, and obtain a predicted number of tumor cells of various intensity levels output by the cell counting model; a second loss value calculation unit configured to determine a cell detection loss value based on the second prediction probability corresponding to each pixel point and the second true value of each pixel point in the tumor cell annotation mask, and determine a cell counting loss value based on the predicted number of tumor cells of various intensity levels and the number of tumor cell annotations; and a second model optimization unit configured to determine a second total loss value based on the cell detection loss value and the cell counting loss value, and back-optimize the model parameters of the cell detection model, the cell counting model, and the feature extraction model through the second total loss value.

[0118] Based on the above embodiments, the cell detection loss value is determined based on the variance loss function, and the cell counting loss value is determined based on the absolute error loss function.

[0119] Based on the above embodiments, the immunohistochemical sections are HER2 sections, with various intensity levels including negative, weakly positive, moderately positive, and strongly positive. Correspondingly, the section grading module 45 includes: a comparison unit configured to compare the percentage of strongly positive tumor cells with a preset percentage threshold; a first grading unit configured to determine the HER2 section grade as strongly positive if the percentage of strongly positive tumor cells is greater than the preset percentage threshold, otherwise comparing the cumulative first percentage of moderately positive and strongly positive tumor cells with the preset percentage threshold; a second grading unit configured to determine the HER2 section grade as moderately positive if the cumulative first percentage is greater than the preset percentage threshold, otherwise comparing the cumulative second percentage of weakly positive, moderately positive, and strongly positive tumor cells with the preset percentage threshold; and a third grading unit configured to determine the HER2 section grade as weakly positive if the cumulative second percentage is greater than the preset percentage threshold, otherwise determining the HER2 section grade as negative.

[0120] The immunohistochemical slide grading device provided in this application, as described above, determines the infiltrative region in the slide image of the immunohistochemical slide and divides the infiltrative region into multiple image blocks; inputs the image blocks into a pre-trained feature extraction model to obtain feature maps output by the feature extraction model; inputs the feature maps into a pre-trained cell detection model to determine tumor cells of various intensity levels in the corresponding image blocks; wherein, the cell detection model performs multi-task learning in conjunction with a cell counting model during training; based on the tumor cells of various intensity levels in the multiple image blocks, the proportion of tumor cells of various intensity levels in the infiltrative region is statistically analyzed; and the grade of the immunohistochemical slide is evaluated based on the proportion of tumor cells of various intensity levels. Through the above technical means, the infiltrative region is segmented from the slide image and divided into multiple image blocks for detecting tumor cells, avoiding the need to divide the entire slide image into image blocks, effectively reducing the number of image blocks, saving computational resources, and improving the efficiency of tumor cell detection. The cell detection and cell counting models are trained using a multi-task learning approach. Through feature sharing and joint optimization, multi-task learning enables the cell detection model to acquire global statistical information and feature space constraints from the counting task, thereby improving the detection accuracy, robustness, and generalization ability of the cell detection model. Based on the detection results of the cell detection model, the proportion of tumor cells at various intensity levels is determined, and the grade of immunohistochemical sections is quantitatively evaluated. This achieves automatic grading of immunohistochemical sections while eliminating dependence on manually set staining thresholds and morphological manipulations, adapting to various application scenarios and improving the accuracy and universality of automatic grading of immunohistochemical sections.

[0121] The immunohistochemical slide grading device provided in this application embodiment can be used to perform the immunohistochemical slide grading method provided in the above embodiment, and has the corresponding functions and beneficial effects.

[0122] Figure 10 This is a schematic diagram of the structure of an immunohistochemical slide grading device provided in an embodiment of this application, with reference to... Figure 10 The immunohistochemical slide grading device includes a processor 51, a memory 52, a communication device 53, an input device 54, and an output device 55. The number of processors 51 and the number of memories 52 in the immunohistochemical slide grading device can be one or more. The processor 51, memory 52, communication device 53, input device 54, and output device 55 of the immunohistochemical slide grading device can be connected via a bus or other means.

[0123] The memory 52, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the immunohistochemical slide grading method in any embodiment of this application (e.g., image block segmentation module 41, feature map determination module 42, cell detection module 43, proportion statistics module 44, and slide grading module 45). The memory 52 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 52 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0124] The communication device 53 is used for data transmission.

[0125] The processor 51 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 52, thereby realizing the above-mentioned immunohistochemical slide grading method.

[0126] Input device 54 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 55 may include display devices such as a display screen.

[0127] The immunohistochemical slide grading equipment provided above can be used to perform the immunohistochemical slide grading method provided in the above embodiments, and has the corresponding functions and beneficial effects.

[0128] This application embodiment also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an immunohistochemical slide grading method. The immunohistochemical slide grading method includes: determining an infiltrative region in an image of an immunohistochemical slide; dividing the infiltrative region into multiple image blocks; inputting the image blocks into a pre-trained feature extraction model to obtain feature maps output by the feature extraction model; inputting the feature maps into a pre-trained cell detection model to determine tumor cells of various intensity levels in the corresponding image blocks; wherein the cell detection model performs multi-task learning in conjunction with a cell counting model during training; statistically analyzing the proportion of tumor cells of various intensity levels in the infiltrative region based on the proportion of tumor cells of various intensity levels in the multiple image blocks; and evaluating the grade of the immunohistochemical slide based on the proportion of tumor cells of various intensity levels.

[0129] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0130] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the immunohistochemical slide grading method described above, but can also perform related operations in the immunohistochemical slide grading method provided in any embodiment of this application.

[0131] The immunohistochemical slide grading device, immunohistochemical slide grading system, storage medium, and immunohistochemical slide grading equipment provided in the above embodiments can execute the immunohistochemical slide grading method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the immunohistochemical slide grading method provided in any embodiment of this application.

[0132] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application. The scope of this application is determined by the scope of the claims.

Claims

1. A method for grading immunohistochemical sections, characterized in that, include: In the slice images of immunohistochemical sections, the infiltrative regions are identified and the infiltrative regions are divided into multiple image blocks; The image patch is input into a pre-trained feature extraction model to obtain the feature map output by the feature extraction model; The feature map is input into a pre-trained cell detection model, which identifies tumor cells of various intensity levels in the corresponding image patch. During training, the cell detection model is combined with a cell counting model for multi-task learning. The training process of the feature extraction model and the cell detection model includes: acquiring sample image patches and corresponding tumor cell annotation masks of various intensity levels and the number of tumor cell annotations; inputting the sample image patches into the feature extraction model to obtain the sample feature map output by the feature extraction model; and inputting the sample feature map into the cell detection model and the cell counting model to obtain the output of the cell detection model. Each pixel represents a second predicted probability of a tumor cell at various intensity levels, and the predicted number of tumor cells at various intensity levels is obtained from the cell counting model output. A cell detection loss value is determined based on the second predicted probability corresponding to each pixel and the second true value of each pixel in the tumor cell annotation mask. A cell counting loss value is determined based on the predicted number of tumor cells at various intensity levels and the number of tumor cells labeled. A second total loss value is determined based on the cell detection loss value and the cell counting loss value. The model parameters of the cell detection model, the cell counting model, and the feature extraction model are then optimized using the second total loss value. Based on the tumor cells of various intensity levels in the multiple image blocks, the percentage of tumor cells of various intensity levels in the infiltrative region is calculated. The grade of the immunohistochemical sections is assessed based on the percentage of tumor cells of each intensity level.

2. The immunohistochemical section grading method according to claim 1, characterized in that, The step of identifying the infiltrative region in the slice image of the immunohistochemical section includes: The slice images of the immunohistochemical sections were downsampled into slice thumbnails; The slice thumbnail is input into a pre-trained region segmentation model to obtain the infiltrated region mask output by the region segmentation model; The immersion region mask is mapped onto the slice image to obtain the immersion region in the slice image.

3. The immunohistochemical section grading method according to claim 2, characterized in that, The training process of the region segmentation model includes: Obtain a slice sample image and the corresponding infiltration region annotation mask, and downsample the slice sample image and the infiltration region annotation mask to obtain a slice sample thumbnail and a mask thumbnail; The slice sample thumbnail is input into the region segmentation model to obtain the first predicted probability of each pixel in the slice sample thumbnail output by the region segmentation model belonging to various regions. Based on the first predicted probability corresponding to each pixel and the first true value corresponding to each pixel in the mask thumbnail, determine the dice loss value and the focus loss value. The first total loss value is obtained by weighted summation of the dice loss value and the focus loss value, and the model parameters of the region segmentation model are optimized in reverse using the first total loss value.

4. The immunohistochemical section grading method according to claim 1, characterized in that, The feature extraction model includes a front-end network and a back-end network; correspondingly, the step of inputting the image patch into the pre-trained feature extraction model to obtain the feature map output by the feature extraction model includes: The image patch is input into the front-end network, and the image patch is downsampled through multiple 3*3 convolutional layers of the front-end network to obtain the downsampled image patch; The downsampled image patch is input into the backend network, and the downsampled image patch is convolved by multiple dilated convolutional layers of the backend network to obtain a feature map.

5. The immunohistochemical section grading method according to claim 1, characterized in that, The cell detection loss value is determined based on the variance loss function, and the cell counting loss value is determined based on the absolute error loss function.

6. The immunohistochemical section grading method according to claim 1, characterized in that, The immunohistochemical sections are HER2 sections, and the various intensity grades include negative, weakly positive, moderately positive, and strongly positive; correspondingly, the assessment of the immunohistochemical section grade based on the proportion of tumor cells at each intensity grade includes: Compare the percentage of strongly positive tumor cells with a preset percentage threshold. If the proportion of strongly positive tumor cells is greater than a preset proportion threshold, the HER2 slice is determined to be strongly positive; otherwise, the sum of the first proportions of moderately positive tumor cells and strongly positive tumor cells is compared with the preset proportion threshold. If the cumulative percentage of the first quantity is greater than the preset percentage threshold, the HER2 slice is determined to be of medium positive grade; otherwise, the cumulative percentage of the second quantity of weak positive tumor cells, the medium positive tumor cells, and the strong positive tumor cells is compared with the preset percentage threshold. If the cumulative value of the second quantity percentage is greater than the preset percentage threshold, the HER2 slice is determined to be weakly positive; otherwise, the HER2 slice is determined to be negative.

7. An immunohistochemical slide grading device, characterized in that, include: The image block segmentation module is configured to identify infiltration regions in the slice images of immunohistochemical sections and divide the infiltration regions into multiple image blocks; The feature map determination module is configured to input the image patch into a pre-trained feature extraction model to obtain a feature map output by the feature extraction model. The cell detection module is configured to input the feature map into a pre-trained cell detection model, and use the cell detection model to determine tumor cells of various intensity levels in the corresponding image patch; wherein, the cell detection model performs multi-task learning in conjunction with a cell counting model during the training process; The percentage statistics module is configured to count the percentage of tumor cells of various intensity levels in the infiltrated region based on the tumor cells of various intensity levels in the multiple image blocks. The slide grading module is configured to assess the grade of the immunohistochemical slides based on the percentage of tumor cells of each intensity level. The immunohistochemical slide grading device also includes a second model training module, configured to acquire sample image patches and corresponding tumor cell annotation masks of various intensity levels and the number of tumor cell annotations; input the sample image patches into the feature extraction model to obtain a sample feature map output by the feature extraction model; input the sample feature map into the cell detection model and the cell counting model to obtain a second predicted probability that each pixel point output by the cell detection model represents a tumor cell of various intensity levels, and to obtain a predicted number of tumor cells of various intensity levels output by the cell counting model; determine a cell detection loss value based on the second predicted probability corresponding to each pixel point and the second true value of each pixel point in the tumor cell annotation mask; determine a cell counting loss value based on the predicted number of tumor cells of various intensity levels and the number of tumor cell annotations; determine a second total loss value based on the cell detection loss value and the cell counting loss value; and back-optimize the model parameters of the cell detection model, the cell counting model, and the feature extraction model using the second total loss value.

8. An immunohistochemical slide grading device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the immunohistochemical slide grading method as described in any one of claims 1-6.

9. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the immunohistochemical slide grading method as described in any one of claims 1-6.

Images