A tumor region segmentation method and device for pathological slice images

Abstract

The present application provides a method and device for tumor region segmentation of pathological slice images, which belongs to the field of image recognition technology. The method obtains several first images with pre-annotated tumor regions, and inputs each first image of the first resolution into a tumor region segmentation model for training until the model verification value of the tumor region segmentation model is less than the first preset threshold, and updates the tumor region segmentation model. Receive a second image of the tumor region to be annotated. Input the second image of the first resolution into the tumor region segmentation model to determine whether there is a tumor region to be determined in each first tumor segmentation map output by the tumor region segmentation model. If so, determine the image to be spliced ​​at the corresponding position of the tumor region to be determined in the second image of the second resolution. Generate a second tumor segmentation map based on the image to be spliced ​​and each first tumor segmentation map, and use the second tumor segmentation map as the final tumor segmentation map of the second image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for segmenting a tumor region in a pathological slice image. Background Art

[0002] Breast cancer is one of the most common cancer-related deaths in women. Pathologists’ diagnosis, risk factor assessment, and tissue morphology analysis are all inseparable from the tumor area. By analyzing the tumor area, the type and grade of the tumor can be determined, and the patient's diagnosis and treatment strategy and prognosis can be established. In traditional pathological diagnosis, pathologists usually use naked eye observation to capture this category information, which has a large workload and a long detection cycle, and the results obtained by different experts are also different.

[0003] At present, the tumor region segmentation method based on deep learning has advantages in throughput and generalization compared with the method based on manual feature identification of tumor regions. In addition, the automatic detection of tumor regions is helpful for the construction of tumor analysis models, which helps pathologists diagnose the disease and make in-depth assessments of prognosis.

[0004] Pathological slice images contain a huge amount of information, and the structural complexity of each region of pathological tissue also varies. The existing deep learning tumor region segmentation method has low segmentation efficiency and performance when segmenting pathological images with a huge amount of information.

[0005] When the inventors used the deep learning tumor region segmentation method, they found that the segmentation of high-scale images had better segmentation results, but the speed was slow, and the low-scale images had a faster speed, but the segmentation results were not as good as those of high-scale images. Summary of the invention

[0006] The embodiments of the present application provide a method and device for segmenting a tumor region in a pathological slice image, which are used to solve the current technical problems of poor segmentation effect and low segmentation efficiency of the tumor region in a pathological slice image.

[0007] On the one hand, an embodiment of the present application provides a tumor region segmentation method for a pathological slice image, the method comprising:

[0008] Acquire several first images with tumor regions pre-annotated. Input each first image of the first resolution into the tumor region segmentation model for training until the model verification value of the tumor region segmentation model is less than the first preset threshold, and update the tumor region segmentation model. Receive a second image of the tumor region to be annotated. The pathological tissue type of the second image is the same as the pathological tissue type of the first image. Input the second image of the first resolution into the tumor region segmentation model to determine whether there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model. In the case of determining that there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model, determine the image to be spliced ​​at the corresponding position of the pending tumor region in the second image of the second resolution. The second resolution is greater than the first resolution. Generate a second tumor segmentation map based on the image to be spliced ​​and each first tumor segmentation map, and use the second tumor segmentation map as the final tumor segmentation map of the second image. The second tumor segmentation map includes the annotated tumor region, and the annotated tumor region is annotated by the tumor region segmentation model.

[0009] In one implementation of the present application, each first image of the first resolution is input into the first segmentation network in the tumor region segmentation model to determine a plurality of first segmentation sub-graphs. The first segmentation network is used to segment the first image of the first resolution, and the first segmentation sub-graphs include the predicted tumor region and the pre-marked tumor region marked by the first segmentation network. The credibility of each first segmentation sub-graph is determined by the discriminant network of the tumor region segmentation model. The credibility characterizes the credibility of each predicted tumor region and each pre-marked tumor region in the first segmentation sub-graph. When the credibility is less than the second preset threshold, the network parameters of the first segmentation network and the discriminant network are updated respectively to train the tumor region segmentation model.

[0010] In one implementation of the present application, in each first tumor segmentation map output by the tumor region segmentation model, if there is no pending tumor region, the second image of the first resolution is used as the final tumor segmentation map. After determining the image to be spliced ​​at the corresponding position of the pending tumor region in the second image of the second resolution, the method also includes: determining whether there is a pending tumor region in the image to be spliced. In the case of determining whether there is a pending tumor region in the image to be spliced, determine the image to be spliced ​​at the corresponding position of the pending tumor region in the second image of the third resolution to determine whether there is a pending tumor region in the image to be spliced ​​corresponding to the third resolution, until there is no pending tumor region in the image to be spliced ​​corresponding to the Nth resolution or the resolution of the image to be spliced ​​reaches a preset value. Wherein, the third resolution is greater than the second resolution, and N is a natural number greater than 1.

[0011] In one implementation of the present application, a corresponding second segmentation network is determined according to the resolution of the image to be spliced. The second segmentation network is a pre-trained neural network model. The image to be spliced ​​is segmented by the second segmentation network, and it is determined whether there is a tumor region to be determined in the segmented image to be spliced. When it is determined that there is no tumor region to be determined in the segmented image to be spliced, the pixel points of the image to be spliced ​​are subjected to image downsampling processing, so as to generate the second tumor segmentation map when the image to be spliced ​​after the downsampling processing and the first tumor segmentation map meet the splicing rule; wherein the splicing rule is that the image pixel points of the image to be spliced ​​match the image pixel points of the first tumor segmentation map.

[0012] In one implementation of the present application, several first sample images with tumor areas pre-labeled are obtained. Each first sample image is input into a tumor labeling model to train the tumor labeling model. The tumor labeling model is a neural network model. Several second sample images are input into the trained tumor labeling model to label the tumor area of ​​each second sample image and obtain a tumor label for each second sample image. Each tumor label output by the tumor labeling model and the corresponding second sample image are sent to a display device to obtain a second sample image after adjusting the tumor label based on the operation of the medical staff. The second sample image is stored in a preset database as the first image.

[0013] In one implementation of the present application, the credibility of the undetermined tumor region is less than a second preset threshold.

[0014] In one implementation of the present application, the first loss function of the first segmentation network and the second loss function of the discriminant network are gradient descent processed by the AdamW optimizer to update the network parameters of the first segmentation network and the discriminant network respectively to train the tumor area segmentation model.

[0015] In an implementation of the present application, the pathological slice image includes a full-field digital slice WSI.

[0016] In one implementation of the present application, when there is a tumor area to be determined in the image to be spliced ​​and the corresponding resolution of the image to be spliced ​​reaches a preset value, an expert annotation prompt information is generated. The expert annotation prompt information includes: sound, text, and image. The expert annotation prompt information is sent to the corresponding medical terminal.

[0017] On the other hand, the embodiment of the present application further provides a tumor region segmentation device for pathological slice images, characterized in that the device includes:

[0018] At least one processor; and a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0019] Acquire several first images with tumor regions pre-annotated. Input each first image of the first resolution into the tumor region segmentation model for training until the model verification value of the tumor region segmentation model is less than the first preset threshold, and update the tumor region segmentation model. Receive a second image of the tumor region to be annotated. The pathological tissue type of the second image is the same as the pathological tissue type of the first image. Input the second image of the first resolution into the tumor region segmentation model to determine whether there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model. In the case of determining that there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model, determine the image to be spliced ​​at the corresponding position of the pending tumor region in the second image of the second resolution. The second resolution is greater than the first resolution. Generate a second tumor segmentation map based on the image to be spliced ​​and each first tumor segmentation map, and use the second tumor segmentation map as the final tumor segmentation map of the second image. The second tumor segmentation map includes the annotated tumor region, and the annotated tumor region is annotated by the tumor region segmentation model.

[0020] Through the above scheme, the present application trains the tumor region segmentation model through the pre-labeled first image, and then uses the trained tumor region segmentation model to segment the tumor region in the second image, which can improve the speed of tumor region segmentation, ensure the tumor region segmentation effect, and improve the efficiency of tumor region segmentation. At the same time, it can also improve the user experience when performing tumor region segmentation in pathological slice images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 A schematic diagram of a flow chart of a tumor region segmentation method for a pathological slice image in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of a pathological slice image in a tumor region segmentation method for a pathological slice image in an embodiment of the present application;

[0024] Figure 3 Another schematic diagram of a flow chart of a tumor region segmentation method for pathological slice images in an embodiment of the present application;

[0025] Figure 4 A structural schematic diagram of a tumor region segmentation model in a tumor region segmentation method for pathological slice images in an embodiment of the present application;

[0026] Figure 5 Another structural schematic diagram of a tumor region segmentation model in a tumor region segmentation method for pathological slice images in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of a structure of a segmentation network in a tumor region segmentation model in a tumor region segmentation method for pathological slice images in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of an attention module of a segmentation network in a tumor region segmentation model in a tumor region segmentation method for pathological slice images in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of a structure of a discriminant network in a tumor region segmentation model in a tumor region segmentation method for pathological slice images in an embodiment of the present application;

[0030] Fig. 9 Schematic diagram of the structure of a tumor region segmentation device for pathological slice images in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0032] Cancer is a serious disease, causing one in six deaths on average. Breast cancer is one of the most common cancer-related deaths in women of all ages. Early diagnosis and treatment can significantly prevent the progression of the disease and reduce its incidence, so women are advised to perform self-examination by palpation and regular examinations by ultrasound or mammography; if abnormalities are found, a breast tissue biopsy is performed. Through biopsy, the collected tissue samples are stained with hematoxylin and eosin (H&E), which can distinguish cell nuclei from parenchyma, and observed through an optical microscope. These samples can be scanned into 10,000-gigapixel-sized images, called whole slide images (WSI), for digital processing by scanners. During the evaluation process, pathologists look for signs of cancer on microscopic sections of tissue by analyzing the histological properties of the tissue. This process can distinguish malignant tissue from non-malignant (benign) tissue, which presents normal structural changes in the breast parenchyma that are not associated with malignant progression. These malignant lesions can be further classified as carcinoma in situ, where the cancer cells are confined to the breast lobular system, and invasive if the cancer cells spread outside the duct. Benign and malignant (in situ or invasive) correspond to different treatment options. Benign lesions can usually be treated clinically without surgery, but malignant tumors almost always require surgery after chemotherapy. The treatment plan is closely related to the patient's life and health, so accurate diagnosis is essential. In the analysis of breast cancer WSI, a large amount of data needs to be visualized, and the evaluation of the tissue structure of the lesion area on the image is very time-consuming. In this case, the computer-aided diagnosis (CAD) system can provide pathologists with complementary and objective evaluations.

[0033] The pathologist's approach to diagnosis and risk factor stratification begins with the identification of tumor regions. These regions are typically analyzed by pathologists using conventional stained tissues via light microscopy. The entire tumor region consists of cancer cells and their supporting tissue matrix. Histomorphometric analysis of these sections has been performed for decades and can generally reliably identify tumor type and extent. However, due to the lack of effective annotation tools and the vulnerability of annotations to intra- and inter-observer variability, pathologists are unable to effectively and reproducibly identify or quantify these histological landmarks. In the past few years, various quantitative digital pathology image analyses using machine learning, especially deep learning, have shown the ability to effectively identify histological primitives from digitized H&E-stained tissue sections. The current segmentation neural network structures based on deep learning mainly include: fully convolutional neural networks, encoder-decoder networks, neural networks with attention mechanisms, and generative adversarial neural networks. The direction of improvement is mainly in the structure, organization, and loss function of the model. Pathological images are stored in a pyramidal manner, with each layer being an image of a different scale.

[0034] In practical applications, the inventors found that the segmentation of high-scale images has better segmentation results, but the speed is slow, and the low-scale images have faster speed, but the segmentation results are not as good as those of high-scale images. Pathological images have a huge amount of information, some regions have rich and complex structures, and some regions have simple structures.

[0035] Based on this, an embodiment of the present application provides a method and device for tumor region segmentation of pathological slice images, which is used to solve the technical problems of poor tumor region segmentation effect and low segmentation efficiency in the prior art for pathological slice images.

[0036] The following describes in detail various embodiments of the present application in conjunction with the accompanying drawings.

[0037] The present application embodiment provides a tumor region segmentation method for pathological slice images, such as Figure 1 As shown, the method may include steps S101-S106:

[0038] S101: The server obtains a plurality of first images with tumor regions pre-labeled.

[0039] In an embodiment of the present application, the server may obtain a plurality of first images of a pre-marked tumor region from a pre-set database, or crawl a plurality of first images of a pre-marked tumor region from a website through crawler software. The pathological slice image includes a whole-view digital slice (WSI) image, and the pathological slice image may also include other types of H&E-stained pathological images or other pathological images, which are not specifically limited in the present application.

[0040] Before acquiring a plurality of first images of a pre-marked tumor region, the method includes:

[0041] First, the server obtains a number of first sample images with tumor areas pre-labeled.

[0042] In the embodiment of the present application, the first sample image may be an image in which a pathologist marks a tumor area on a WSI or other type of H&E stained pathological image, such as Figure 2 As shown in the figure, the pathologist can also annotate the tumor type and grade in the tumor region, and use the tumor region, tumor type, and tumor grade as tumor labels.

[0043] Secondly, the server inputs each first sample image into a tumor annotation model to train the tumor annotation model. The tumor annotation model is a neural network model.

[0044] In the embodiment of the present application, the tumor annotation model can be any neural network model for segmentation, or other segmentation models, which is not specifically limited in the present application. The tumor annotation model uses the first sample image as a training sample, extracts the tumor region features of the tumor region annotated in the first sample image, and then trains the tumor annotation model. When using the trained tumor annotation model, the tumor annotation model can annotate the tumor region of the newly input image without annotated tumor region, thereby obtaining a similar Figure 2 of the image.

[0045] Next, the server inputs a number of second sample images into the trained tumor labeling model to label the tumor region of each second sample image, thereby obtaining a tumor label of each second sample image.

[0046] The second sample image is an image in which the tumor region has not been annotated by a pathologist. The tumor annotation model can annotate the tumor region in the second sample image and generate a tumor label for the tumor region.

[0047] Next, the server sends each tumor label output by the tumor labeling model and the corresponding second sample image to the display device, so as to obtain the second sample image after adjusting the tumor label based on the operation of the medical staff.

[0048] In the embodiment of the present application, after the tumor annotation model outputs the second sample image and the corresponding tumor label, the server can send it to a display device, such as a pathologist's computer, mobile phone, etc. The pathologist can modify the tumor area with incorrect annotations, adjust the tumor label with incorrect annotations, and modify the tumor area with incorrect annotations according to the tumor area and tumor label annotated by the tumor annotation model, thereby obtaining the second sample image after adjusting the tumor label.

[0049] Finally, the server stores the second sample image as the first image in a preset database.

[0050] In the embodiment of the present application, the second sample image may include the first sample image, or may not include the first sample image.

[0051] Through the above solution, the time for acquiring the first image can be saved, the time spent by the pathologist on annotation can be saved, and sample data for training the tumor region segmentation model can be efficiently obtained.

[0052] It should be noted that the server, as the executor of the tumor region segmentation method for pathological slice images, is only an example, and the executor is not limited to the server, and this application does not make any specific limitation on this.

[0053] S102, the server inputs each first image of the first resolution into the tumor region segmentation model for training until the model verification value of the tumor region segmentation model is less than a first preset threshold, and updates the tumor region segmentation model.

[0054] In the embodiment of the present application, the model validation value is obtained according to the overall loss function of the tumor region segmentation model, and the formula is as follows:

[0055] L=L S +λL D

[0056] Among them, L is the overall loss function value, L S is the first loss function of the first segmentation network, L D is the second loss function of the discriminant network, and λ is an empirical constant, which can be set to 0.5. In the embodiment of the present application, during the training of the tumor region segmentation model, the difference between the values ​​of the overall loss function is used as the model verification value, and it is determined whether the model verification value is less than the first preset threshold value. The first preset threshold value can be used to determine whether the overall loss function value converges, thereby determining the training status of the tumor region segmentation model to update the tumor region segmentation model.

[0057] The first preset threshold can be set according to actual use, and this application does not make any specific limitation on this.

[0058] In the embodiment of the present application, the server inputs each first image of the first resolution into the tumor region segmentation model for training, such as Figure 3 As shown, the specific steps include:

[0059] S301, the server inputs each first image of the first resolution into a first segmentation network in a tumor region segmentation model to determine a plurality of first segmentation sub-graphs.

[0060] The first segmentation network is used to segment the first image of the first resolution, and the first segmentation sub-image includes the predicted tumor area marked by the first segmentation network and the pre-marked tumor area.

[0061] In an embodiment of the present application, the server may determine, among a plurality of first images, a plurality of first images of a first resolution, for example, a first resolution of 512×512 pixels or 256×256 pixels. After the first image of the first resolution is input into the first segmentation network, the first segmentation network may segment the first image. The segmentation may be performed according to a preset rule to divide the first image into four equal parts, that is, to divide a first image of 512×512 pixels into four non-overlapping sub-images of corresponding sizes of 256×256 pixels. Figure 4 As shown, the first segmentation network divides the first image into four first segmentation sub-images.

[0062] When the first segmentation network segments the first image, the first segmentation network can be trained according to the pre-annotated tumor region, and the predicted tumor region in the first image can be determined according to the tumor region features obtained by the first segmentation network training, and the predicted tumor region can be annotated. The first segmentation network is used for segmenting an image of a first resolution, and the first resolution can correspond to a low resolution in a pathological slice image.

[0063] S302: The server determines the credibility of each first segmentation sub-image through the discriminant network of the tumor region segmentation model.

[0064] The credibility represents the credibility of each predicted tumor region and each pre-labeled tumor region in the first segmentation sub-image.

[0065] In the embodiment of the present application, the server can input a plurality of first segmentation sub-images and the first image obtained by segmentation of the first segmentation network into the discriminant network, such as Figure 4 As shown. The discriminant network determines the credibility of each first segmentation sub-image, and the credibility is, for example, 0, 1, or a value between the interval [0, 1]. In the embodiment of the present application, when the discriminant network is trained, when the first segmentation sub-image is a predicted tumor area, the credibility is 0, and when the first segmentation sub-image is a pre-marked tumor area, the credibility is 1.

[0066] S303: When the credibility is less than a second preset threshold, the server updates the network parameters of the first segmentation network and the discrimination network respectively to train the tumor region segmentation model.

[0067] In the embodiment of the present application, for example, the second preset threshold value can be set to a value such as 0.5 or 0.6. Figure 4As shown, three first segmentation sub-images are greater than the second preset threshold, and one first segmentation sub-image is less than the first preset threshold. Since the first image has a first segmentation sub-image with a low degree of credibility, the server can train and update the tumor region segmentation model by updating the network parameters of the first segmentation network and the discriminant network.

[0068] The network parameters of the first segmentation network and the discriminant network are updated respectively to train the tumor region segmentation model, specifically including:

[0069] The server uses the AdamW optimizer to perform gradient descent processing on the first loss function of the first segmentation network and the second loss function of the discriminant network, respectively, to update the network parameters of the first segmentation network and the discriminant network, so as to train and update the tumor area segmentation model.

[0070] Among them, the specific formula of the first loss function is:

[0071] L s =l(S(x),y)+λl(D(x,S(x)),1)

[0072] Among them, L S is the first loss function value, S(x) is the output value of the segmentation network, y is the tumor annotation area, the tumor annotation area is the tumor label image generated by the pathologist, l is the cross entropy loss function, and D(x,S(x)) is the output value when the first image and several first segmentation sub-images are used as the discriminant network input.

[0073] The specific formula of the second loss function is:

[0074] L D =l(D(x,y),1)+l(D(x,S(x)),0)

[0075] Among them, L D is the value of the second loss function, and D(x,y) is the output value of the discriminant network combining the first image and the tumor label.

[0076] Through the above scheme, the training speed of the tumor region segmentation model can be improved while ensuring the output accuracy of the tumor region segmentation model.

[0077] S103: The server receives a second image of the tumor area to be labeled.

[0078] The pathological tissue type of the second image is the same as the pathological tissue type of the first image.

[0079] In the application embodiment, the server can receive the second image of the tumor area to be annotated sent by the external terminal device through the network or Bluetooth, and the external terminal device includes but is not limited to a mobile phone and a tablet computer. The pathological tissue type of the second image is the same as the pathological tissue type of the first image. For example, the first image and the second image are both WSI images of pathological tissue of breast cancer or other types of H&E stained pathological images.

[0080] S104: The server inputs the second image of the first resolution into a tumor region segmentation model to determine whether there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model.

[0081] In an embodiment of the present application, the server can determine the image features in the second image of the first resolution through the tumor region segmentation model, and segment the second image to obtain a first tumor segmentation map. The first tumor segmentation map includes the predicted tumor region of the trained first segmentation network. Then, the first tumor segmentation map is input into the trained discriminant network to identify the first tumor segmentation map whose credibility is less than the second preset threshold, and the tumor region in the map is used as the tumor region to be determined.

[0082] In the embodiment of the present application, the credibility of the undetermined tumor area is less than the second preset threshold.

[0083] The discriminant network is trained by pre-labeled tumor regions and predicted tumor regions of the first segmentation sub-graphs and corresponding first images, so that images that do not conform to the pre-labeled tumor regions in each first tumor segmentation graph can be identified. Through the above scheme, the accuracy of segmenting and identifying tumor regions can be guaranteed.

[0084] S105, when determining that there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model, the server determines an image to be spliced ​​at a position corresponding to the pending tumor region in the second image of the second resolution.

[0085] The second resolution is greater than the first resolution.

[0086] When determining that there is a pending tumor region in each of the first tumor segmentation maps output by the tumor region segmentation model, the server may determine the first tumor segmentation map A corresponding to the pending tumor region. The server may determine the position of the first tumor segmentation map A in the second image of the second resolution according to the position of the first tumor segmentation map A corresponding to the first tumor segmentation map A in the second image. The server uses the image in the second image of the second resolution at the same position as the first tumor segmentation map A as the image to be spliced.

[0087] In one embodiment of the present application, in each first tumor segmentation map output by the tumor region segmentation model, when there is no undetermined tumor region, the second image with the first resolution is used as the final tumor segmentation map.

[0088] After the server determines the image to be spliced ​​at the position corresponding to the tumor region to be determined in the second image with the second resolution, the server may further perform the following method:

[0089] First, the server determines whether there is a tumor area to be determined in the image to be stitched.

[0090] In the embodiments of the present application, Figure 5 As shown, the image to be spliced ​​can be input into the second segmentation network, and the server determines whether there is a tumor region to be determined in the image to be spliced ​​through the second segmentation network. The second segmentation network is a neural network model with a processing resolution higher than the first resolution. The difference between the second segmentation network and the first segmentation network is that the image resolutions processed are different.

[0091] Then, when it is determined that there is a pending tumor area in the image to be spliced, the server determines the image to be spliced ​​at the corresponding position of the pending tumor area in the second image of the third resolution to determine whether there is a pending tumor area in the image to be spliced ​​corresponding to the third resolution, until there is no pending tumor area in the image to be spliced ​​corresponding to the Nth resolution or the corresponding resolution of the image to be spliced ​​reaches a preset value.

[0092] The third resolution is greater than the second resolution, and N is a natural number greater than 1.

[0093] In an embodiment of the present application, for example, the resolution of several pathological slice images has 5 levels, that is, N is 5, the higher the level, the higher the resolution, when the first resolution is level 1, then the second resolution is level 2, if the image to be spliced ​​corresponding to the second resolution has a pending tumor area, then the server can determine the image to be spliced ​​corresponding to the third resolution (level 3), if there is still a pending tumor area, then determine in turn whether the corresponding images to be spliced ​​at levels 4 and 5 have a pending tumor area.

[0094] In the example, the highest resolution of the pathological slice image is level five. When there is still an undetermined tumor area at level five, the server can use the image to be spliced ​​corresponding to the highest resolution as the image to be spliced ​​for generating the second tumor segmentation image.

[0095] In another embodiment of the present application, when there is still a tumor area to be determined in the image to be stitched corresponding to the highest resolution, the server may also execute the following method:

[0096] First, the server generates expert annotation prompt information when there is a pending tumor area in the image to be stitched and the corresponding resolution of the image to be stitched reaches a preset value.

[0097] Expert annotation prompt information includes: sound, text, and image.

[0098] For example, the server generates a sound or text message: "The tumor area cannot be determined", or displays an image such as a red background.

[0099] Then, the server sends the expert annotation prompt information to the corresponding medical terminal.

[0100] In the embodiment of the present application, the medical terminal can be a pathologist's mobile phone, computer, tablet computer or other device, and the present application does not make any specific limitation on this.

[0101] S106: The server generates a second tumor segmentation map according to the image to be stitched and each first tumor segmentation map, and uses the second tumor segmentation map as the final tumor segmentation map of the second image.

[0102] The second tumor segmentation map includes a labeled tumor region, and the labeled tumor region is labeled by the tumor region segmentation model.

[0103] In the embodiment of the present application, generating a second tumor segmentation map according to the image to be stitched and the first tumor segmentation map specifically includes:

[0104] First, the server determines the corresponding second segmentation network according to the resolution of the image to be stitched.

[0105] The second segmentation network is a pre-trained neural network model. This step is the same as the second segmentation network in S105 above. Figure 5 shown.

[0106] Then, the server segments the image to be spliced ​​through a second segmentation network, and determines whether there is a pending tumor area in the segmented image to be spliced.

[0107] The above step S105 has described the solution of whether there is a pending tumor area in the image to be stitched, which will not be described in detail here.

[0108] Finally, when the server determines that there is no undetermined tumor area in the segmented image to be stitched, it downsamples the pixels of the image to be stitched, so as to generate a second tumor segmentation map when the downsampled image to be stitched and the first tumor segmentation maps meet the stitching rules.

[0109] The stitching rule is that the image pixel points of the image to be stitched match the image pixel points of the first tumor segmentation image.

[0110] In the embodiment of the present application, the stitching rules can add matching conditions according to the needs of the actual stitching effect, for example, the pixel points of the image to be stitched are downsampled to 256×256 pixel values, and the image to be stitched is a ten-fold magnified image of the pathological slice image, and each first tumor segmentation image is a 256×256 pixel value and a ten-fold magnified image of the pathological slice image. At this point, it is achieved that the image to be stitched and each first tumor segmentation map meet the stitching rules. After the server completes the stitching of the image to be stitched and each first tumor segmentation map, it obtains the second tumor segmentation map, and the server uses the second tumor segmentation map as the final tumor segmentation map.

[0111] Taking the above embodiment of the present application as an example, at a magnification of 10 times of the pathological slice image, non-overlapping image blocks are sequentially cut according to the size of 512×512 pixels. The image block (second image) is sent to the segmentation network to obtain each first tumor segmentation map at a magnification of 10 times. Each first tumor segmentation map is cut into 4 256×256 pixel images at half the width and height of the image block (second image), corresponding to 4 256×256 pixel sub-image block maps, which are input into the discriminant network. The discriminant network makes a "true or false" judgment on these 4 images. In the embodiment of the present application, if it is true (the discriminant network outputs 1), it is considered that this sub-image block map is credible. If it is false (the discriminant network outputs 0), it is considered that the prediction of this sub-image block is unreliable and requires segmentation of a higher resolution model. When a higher resolution model is required for segmentation, the same area of ​​the 256×256 pixel sub-image block at 10 times magnification is restored to 20 times magnification, that is, a 512×512 pixel image block, and the 512×512 pixel image block is input into the segmentation network at 20 times magnification to obtain the corresponding 512×512 pixel image to be spliced ​​at 20 times magnification, and then the image to be spliced ​​is downsampled to 10 times magnification, that is, a 256×256 pixel prediction map at 20 times magnification at 10 times magnification, which replaces the original unreliable 10 times magnification first tumor segmentation map, and is spliced ​​with other reliable first tumor segmentation maps to form a complete second tumor segmentation map. Repeat the same operation for each image block (second image), and finally splice into a complete breast cancer tumor area prediction map of the WSI image or other types of H&E stained pathological images.

[0112] The present application uses several first images with tumor regions pre-labeled to train a tumor region segmentation model, and through the tumor region segmentation model, determines the location of the tumor region in the second image of the tumor region to be labeled, and labels the tumor region. Moreover, the tumor region segmentation model of the present application is performed at a low resolution, which can speed up the tumor region segmentation speed; and when the tumor region cannot be accurately determined (a tumor region to be determined appears), the tumor region is segmented using a high-resolution image, thereby ensuring the tumor region segmentation effect.

[0113] To sum up, the present application can ensure the tumor area segmentation speed and tumor area segmentation effect of pathological slice images through the above scheme, thereby improving the user experience when performing tumor area segmentation in pathological slice images and improving the work efficiency of medical staff.

[0114] In the embodiment of the present application, the segmentation network structure of the tumor region segmentation model is as follows: Figure 6 As shown in the figure, it is mainly divided into two parts: encoder and decoder. In the encoder, it is mainly composed of a combination of convolution groups, attention modules and pooling layers. In the decoder, it is mainly composed of a combination of convolution groups, transposed convolution layers and output convolution layers. A convolution group contains a convolutional layer (Convolutional Layer) with a kernel size of 3, padding of 1, and a stride of 1, a batch normalization layer (BatchNormalization Layer), and a rectified linear unit activation layer (Rectified Linear Unit ActivationLayer). The kernel size of a transposed convolution layer (Transposed Convolution Layer) is 2, the padding is 0, and the stride is 2. The kernel size of an output convolution layer is 1, the padding is 0, and the stride is 1. The structure of the attention module is as follows Figure 7 As shown in the figure, it first performs channel-level average pooling on the output features of the previous layer to obtain the mean on each channel, and then passes through two fully connected layers to reduce and then enlarge the dimension of the average pooling result to obtain the attention weight on each channel. Finally, the attention weight is multiplied by the output features of the previous layer to obtain the output of this attention module.

[0115] In the embodiment of the present application, the model structure of the discriminant network of the tumor region segmentation model is as follows: Figure 8As shown in the figure, it is mainly composed of a convolutional layer and a leaky rectified linear unit activation layer (Leaky Rectified Linear Unit ActivationLayer). In particular, the kernel size of the convolutional layer is 4, the padding is 2, and the stride is 2. Except for the last convolutional layer, the other convolutional layers double the number of channels of the previous layer output. All convolutional layers halve the spatial dimension of the previous layer output. Finally, through a fully connected layer, the output of the last convolutional layer is mapped into a scalar, and through a Sigmoid activation layer, a probability of 0 or 1 is obtained.

[0116] Fig. 9 A tumor region segmentation device for pathological slice images provided in an embodiment of the present application includes:

[0117] At least one processor; and a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0118] Acquire several first images with tumor regions pre-annotated. Input each first image of the first resolution into the tumor region segmentation model for training until the model verification value of the tumor region segmentation model is less than the first preset threshold, and update the tumor region segmentation model. Receive a second image of the tumor region to be annotated. The pathological tissue type of the second image is the same as the pathological tissue type of the first image. Input the second image of the first resolution into the tumor region segmentation model to determine whether there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model. In the case of determining that there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model, determine the image to be spliced ​​at the corresponding position of the pending tumor region in the second image of the second resolution. The second resolution is greater than the first resolution. Generate a second tumor segmentation map based on the image to be spliced ​​and each first tumor segmentation map, and use the second tumor segmentation map as the final tumor segmentation map of the second image. The second tumor segmentation map includes the annotated tumor region, and the annotated tumor region is annotated by the tumor region segmentation model.

[0119] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0120] The device and method provided in the embodiments of the present application correspond one to one, and therefore, the device also has similar beneficial technical effects as its corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device will not be repeated here.

[0121] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0122] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A tumor region segmentation method for pathological slice images, characterized in that: The method comprises: Acquire a plurality of first images with tumor regions pre-marked; Inputting each of the first images at the first resolution into a tumor region segmentation model for training until a model verification value of the tumor region segmentation model is less than a first preset threshold, and updating the tumor region segmentation model; Receiving a second image of the tumor area to be annotated; wherein the pathological tissue type of the second image is the same as the pathological tissue type of the first image; Inputting the second image of the first resolution into the tumor region segmentation model, and determining whether there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model; If so, determining the image to be stitched at the position corresponding to the tumor region to be determined in the second image with a second resolution; wherein the second resolution is greater than the first resolution; A second tumor segmentation map is generated according to the image to be spliced ​​and each of the first tumor segmentation maps, and the second tumor segmentation map is used as the final tumor segmentation map of the second image; wherein the second tumor segmentation map includes a labeled tumor area, and the labeled tumor area is labeled by the tumor area segmentation model.

2. The method according to claim 1, characterized in that: Inputting each of the first images at the first resolution into a tumor region segmentation model for training, specifically comprising: Input each of the first images of the first resolution into a first segmentation network in the tumor region segmentation model to determine a plurality of first segmentation sub-graphs; wherein the first segmentation network is used to segment the first image of the first resolution, and the first segmentation sub-graphs include the predicted tumor region annotated by the first segmentation network and the pre-annotated tumor region; Determining the credibility of each of the first segmentation sub-images through the discriminant network of the tumor region segmentation model; the credibility represents the credibility of each of the predicted tumor regions and each of the pre-marked tumor regions in the first segmentation sub-image; When the credibility is less than a second preset threshold, the network parameters of the first segmentation network and the discriminant network are updated respectively to train the tumor region segmentation model.

3. The method according to claim 1, characterized in that: The method further comprises: In each of the first tumor segmentation maps output by the tumor region segmentation model, when the undetermined tumor region does not exist, using the second image of the first resolution as the final tumor segmentation map; After determining the image to be spliced ​​at a position corresponding to the tumor region to be determined in the second image of the second resolution, the method further includes: Determining whether the to-be-determined tumor region exists in the to-be-stitched image; If it exists, determine the image to be spliced ​​at the position corresponding to the tumor area to be determined in the second image of the third resolution, so as to determine whether the tumor area to be determined exists in the image to be spliced ​​corresponding to the third resolution, until the tumor area to be determined does not exist in the image to be spliced ​​corresponding to the Nth resolution or the resolution corresponding to the image to be spliced ​​reaches a preset value; wherein, the third resolution is greater than the second resolution, and N is a natural number greater than 1.

4. The method according to claim 1, characterized in that: Generating a second tumor segmentation map according to the image to be stitched and the first tumor segmentation map specifically includes: Determine a corresponding second segmentation network according to the resolution of the image to be stitched; wherein the second segmentation network is a pre-trained neural network model; Segmenting the image to be spliced ​​by the second segmentation network, and determining whether the segmented image to be spliced ​​contains the tumor region to be determined; If it does not exist, image downsampling processing is performed on the pixel points of the image to be stitched, so as to generate the second tumor segmentation map when the image to be stitched after the downsampling processing and the first tumor segmentation map meet the stitching rule; wherein the stitching rule is that the image pixel points of the image to be stitched match the image pixel points of the first tumor segmentation map.

5. The method according to claim 1, characterized in that: Before acquiring a plurality of first images with pre-labeled tumor regions, the method further includes: Acquire a plurality of first sample images with tumor areas pre-marked; Inputting each of the first sample images into a tumor labeling model to train the tumor labeling model; the tumor labeling model is a neural network model; Inputting a plurality of second sample images into the trained tumor labeling model to label the tumor region of each second sample image to obtain a tumor label of each second sample image; Sending each of the tumor labels output by the tumor labeling model and the corresponding second sample image to a display device, so as to obtain the second sample image after adjusting the tumor label based on the operation of the medical staff; The second sample image is used as the first image and stored in a preset database.

6. The method according to claim 2, characterized in that: The credibility of the undetermined tumor region is less than the second preset threshold.

7. The method according to claim 2, characterized in that: Updating the network parameters of the first segmentation network and the discriminant network respectively to train the tumor region segmentation model specifically includes: Through the AdamW optimizer, the first loss function of the first segmentation network and the second loss function of the discriminant network are respectively subjected to gradient descent processing to update the network parameters of the first segmentation network and the discriminant network respectively to train the tumor region segmentation model.

8. The method according to claim 1, characterized in that: The pathological slice image includes a full-view digital slice WSI.

9. The method according to claim 3, characterized in that: The method further comprises: When the to-be-determined tumor region exists in the to-be-spliced ​​images and the corresponding resolution of the to-be-spliced ​​images reaches a preset value, generating expert annotation prompt information; the expert annotation prompt information includes: sound, text, and image; The expert marking prompt information is sent to the corresponding medical terminal.

10. A tumor region segmentation device for pathological slice images, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Acquire a plurality of first images with tumor regions pre-marked; Inputting each of the first images at the first resolution into a tumor region segmentation model for training until a model verification value of the tumor region segmentation model is less than a first preset threshold, and updating the tumor region segmentation model; Receiving a second image of the tumor area to be annotated; wherein the pathological tissue type of the second image is the same as the pathological tissue type of the first image; Inputting the second image of the first resolution into the tumor region segmentation model, and determining whether there is a pending tumor region in each first tumor segmentation map output by the tumor region segmentation model; If so, determining the image to be stitched at the position corresponding to the tumor region to be determined in the second image with a second resolution; wherein the second resolution is greater than the first resolution; A second tumor segmentation map is generated according to the image to be spliced ​​and each of the first tumor segmentation maps, and the second tumor segmentation map is used as the final tumor segmentation map of the second image; wherein the second tumor segmentation map includes the marked tumor area.

Images