Cancer cell segmentation method, device and computer equipment based on digital pathology images

Abstract

The present invention relates to the field of medical image segmentation, and in particular to a method, apparatus, and computer equipment for cancer cell segmentation based on digital pathology images. The method extracts feature information and frame prompt information from the digital pathology images to be segmented, performs encoding and decoding processing, and performs cancer cell segmentation based on the obtained decoded feature map, thereby improving the accuracy and efficiency of cancer cell segmentation.

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and in particular to a method, device, computer equipment and storage medium for cancer cell segmentation based on digital pathology images. Background Art

[0002] With the development of artificial intelligence, the technology in the field of image segmentation has made great progress. Cancer cell segmentation is one of the important tasks in pathological image analysis. However, due to the huge data format of digital pathology images (WSI) and the complex and diverse features of each image, current technical solutions for cancer cell segmentation in WSI often require a large amount of computing resources, and the extraction of detailed features is still not comprehensive, resulting in limited cancer cell segmentation results and difficulty in performing accurate cancer cell segmentation tasks. Summary of the Invention

[0003] Based on this, the purpose of the present invention is to provide a method, device, equipment and storage medium for cancer cell segmentation based on digital pathology images. By extracting feature information and frame prompt information from the digital pathology images to be segmented, encoding and decoding are performed, and cancer cell segmentation is performed based on the obtained decoded feature map, thereby improving the accuracy and efficiency of cancer cell segmentation.

[0004] In a first aspect, an embodiment of the present application provides a method for cancer cell segmentation based on digital pathology images, comprising the following steps:

[0005] Obtaining a digital pathological image to be segmented and a cancer cell segmentation model, wherein the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module;

[0006] Dividing the digital pathology image to be segmented into a plurality of raster images, inputting the plurality of raster images into the feature extraction module for information feature extraction and frame prompt mask map construction, and obtaining an information feature map and a frame prompt mask map for each of the raster images;

[0007] Scaling and pooling the frame hint mask map of each of the raster images to obtain frame hint pooling feature maps of several scales corresponding to each of the raster images;

[0008] Inputting the information feature map of the same raster image and the frame prompt pooling feature maps of several scales into the encoding module to obtain several encoding feature maps of each raster image;

[0009] Inputting the plurality of encoded feature maps of the same raster image and the frame prompt pooling feature maps of a plurality of scales into the decoding module to obtain a plurality of decoded feature maps of each raster image;

[0010] Inputting the plurality of decoded feature maps of each raster image into the detection module for probability prediction to obtain cancer cell probability prediction data of each raster image;

[0011] The digital pathology image to be segmented is segmented according to the cancer cell probability prediction data of each of the grid images to obtain a cancer cell segmentation result of the digital pathology image to be segmented.

[0012] In a second aspect, an embodiment of the present application provides a cancer cell segmentation device based on digital pathology images, comprising:

[0013] A data acquisition module, configured to obtain a digital pathology image to be segmented and a cancer cell segmentation model, wherein the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module;

[0014] A feature construction module is used to divide the digital pathology image to be segmented into a plurality of raster images, input the plurality of raster images into the feature extraction module to perform information feature extraction and frame prompt mask map construction, and obtain an information feature map and a frame prompt mask map for each of the raster images;

[0015] A feature pooling module, configured to scale and pool the frame hint mask images of each of the raster images to obtain frame hint pooling feature maps of several scales corresponding to each of the raster images;

[0016] A feature encoding module, configured to input the information feature map of the same raster image and the frame prompt pooling feature maps of several scales into the encoding module to obtain several encoded feature maps of each raster image;

[0017] A feature decoding module, configured to input the plurality of encoded feature maps of the same raster image and the frame prompt pooling feature maps of a plurality of scales into the decoding module to obtain a plurality of decoded feature maps of each raster image;

[0018] A probability prediction module, configured to input the plurality of decoded feature maps of each raster image into the detection module for probability prediction, thereby obtaining cancer cell probability prediction data of each raster image;

[0019] The cancer cell segmentation module is used to perform segmentation processing on the digital pathology image to be segmented according to the cancer cell probability prediction data of each of the grid images, so as to obtain a cancer cell segmentation result of the digital pathology image to be segmented.

[0020] In a third aspect, an embodiment of the present application provides a computer device comprising: a processor, a memory, and a computer program stored on the memory and executable on the processor; when the computer program is executed by the processor, the steps of the method for cancer cell segmentation based on digital pathology images as described in the first aspect are implemented.

[0021] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for cancer cell segmentation based on digital pathology images as described in the first aspect.

[0022] In an embodiment of the present application, a method, apparatus, computer device, and storage medium for cancer cell segmentation based on digital pathology images are provided. By extracting feature information and frame prompt information from the digital pathology image to be segmented, encoding and decoding are performed, and cancer cell segmentation is performed based on the obtained decoded feature map, thereby improving the accuracy and efficiency of cancer cell segmentation.

[0023] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic flow chart of a method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application;

[0025] Figure 2 A schematic diagram of the process of step S2 in a method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application;

[0026] Figure 3 A schematic diagram of the process of step S4 in a method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application;

[0027] Figure 4 A schematic diagram of the flow of S41 in a method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application;

[0028] Figure 5 This is a flow chart of S412 in a method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application;

[0029] Figure 6 A schematic diagram of the flow of S5 in the method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application;

[0030] Figure 7 A schematic diagram of the flow of S51 in a method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application;

[0031] Figure 8 A schematic diagram of the structure of a cancer cell segmentation device based on digital pathology images provided in one embodiment of the present application;

[0032] Figure 9 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0034] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0035] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to a determination."

[0036] The data sending end may be a computer device or a mobile terminal device, which is used to establish a network connection with the data receiving end, and can encode the data information sent to the data receiving end and parse the data information sent from the data receiving end.

[0037] The data receiving end may be a computer device or a mobile terminal device, which is used to establish a network connection with the data sending end, and can encode the data information sent to the data sending end and parse the data information sent from the data sending end.

[0038] See also Figure 1 , Figure 1A flowchart of a method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application is provided. The method includes the following steps:

[0039] S1: Obtain the digital pathology image to be segmented and the cancer cell segmentation model.

[0040] The execution subject of the cancer cell segmentation method based on digital pathology images is a segmentation device of the cancer cell segmentation method based on digital pathology images (hereinafter referred to as the segmentation device). In an optional embodiment, the segmentation device can be a computer device, a server, or a server cluster composed of multiple computer devices.

[0041] In this embodiment, the segmentation device may obtain a digital pathology image to be segmented input by a user, or may select a digital pathology image to be segmented from a preset database.

[0042] The segmentation device obtains a preset cancer cell segmentation model, wherein the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module and a detection module.

[0043] S2: Divide the digital pathology image to be segmented into a plurality of raster images, input the plurality of raster images into the feature extraction module for information feature extraction and frame prompt mask map construction, and obtain the information feature map and frame prompt mask map of each raster image.

[0044] In this embodiment, the segmentation device divides the digital pathology image to be segmented into several raster images, inputs the several raster images into the feature extraction module for information feature extraction and frame prompt mask map construction, and obtains the information feature map and frame prompt mask map of each raster image.

[0045] See also Figure 2 , Figure 2 The flowchart of S2 in the method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application includes steps S21 to S22, which are specifically as follows:

[0046] S21: performing edge detection on each of the raster images to obtain a Laplacian graph of each of the raster images; performing clustering operation and median filtering processing on each of the raster images to obtain a median filtered graph of each of the raster images; cross-joining the Laplacian graph and the median filtered graph of the same raster image to obtain an information feature graph of each of the raster images.

[0047] Edge detection captures edge details by quantifying the difference between each pixel and its neighbors. In this embodiment, the segmentation device uses a Laplacian operator to perform edge detection on the raster image, obtaining several Laplacian graphs of the raster image to extract color features from the digital pathology image to be segmented, thereby reducing the interference effects of color and brightness factors.

[0048] Traditional image clustering algorithms such as K-means can roughly extract information. In this embodiment, the segmentation device adopts the K-means clustering algorithm to perform clustering operations and median filtering on the raster image, and divides the pixels of the raster image into two categories, namely the target area and the non-target area. The median filter is used to eliminate isolated noise points, and finally obtains an image that can preliminarily distinguish the foreground and background, enhance the texture features of the lesion area, and obtain several median filtered images of the raster image.

[0049] The segmentation device cross-splices the Laplacian graph and the median filter graph of the same raster image to obtain information feature graphs of several raster images, so as to fuse color features and texture features, fully extract feature information of the digital pathology image to be segmented, and improve the accuracy of cancer cell segmentation in the digital pathology image to be segmented.

[0050] S22: Obtaining initial detection frames of several of the raster images, solving the minimum circumscribed rectangle of the initial detection frames, and obtaining frame prompt information of several of the raster images; constructing frame prompt mask maps based on the several raster images and the corresponding frame prompt information, and obtaining frame prompt mask maps of several of the raster images.

[0051] Because the location of cancer cells in WSI slices, i.e., raster images, is always changing, they may appear in various parts of the raster image, and some slices may even contain all cancer cells or none of them. Even manually screening the range of cancer cells is a tedious task.

[0052] In order to simulate manual labeling and enable the network to perceive the approximate area where the cancer cells are located, in this embodiment, the segmentation device adopts the generalized Hough transform method to obtain the initial detection frame of several of the raster images, solves the minimum circumscribed rectangle of the initial detection frame, obtains the frame prompt information of several of the raster images, constructs the frame prompt mask map based on the several raster images and the corresponding frame prompt information, and obtains the frame prompt mask map of the several raster images.

[0053] S3: scaling and pooling the frame hint mask map of each of the raster images to obtain frame hint pooling feature maps of several scales corresponding to each of the raster images.

[0054] In this embodiment, the segmentation device scales and pools the box prompt mask images of each of the raster images to obtain box prompt pooled feature maps of several scales corresponding to each of the raster images to highlight the cancer cell area while suppressing background information, which helps the network focus more on the target area and improves the network's performance in detecting and locating the target area.

[0055] S4: Inputting the information feature map of the same raster image and the frame prompt pooling feature maps of several scales into the encoding module to obtain several encoding feature maps of each raster image.

[0056] In this embodiment, the segmentation device inputs the information feature map of the same raster image and the frame prompt pooling feature maps of several scales into the encoding module to obtain several encoding feature maps of each raster image. During the encoding process, the frame prompt pooling feature maps of the corresponding scales are introduced for interactive information processing, thereby improving the accuracy of feature encoding.

[0057] The encoding module includes several stacked encoding units; see Figure 3 , Figure 3 The flowchart of S4 in the method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application includes steps S41 to S43, which are specifically as follows:

[0058] S41: performing encoding processing on the information feature map of the raster image as input data of the first encoding unit of the encoding module, and obtaining a plurality of encoding feature maps of the raster image output by the first encoding unit.

[0059] In this embodiment, the segmentation device uses the information feature map of the raster image as input data of the first encoding unit of the encoding module to perform encoding processing, and obtains a plurality of encoding feature maps of the raster image output by the first encoding unit.

[0060] The encoding unit includes a small visual field encoding subunit, a large visual field encoding subunit and an attention extraction subunit; see Figure 4 , Figure 4 The flowchart of S41 in the method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application includes steps S411 to S412, which are specifically as follows:

[0061] S411: Input the information feature map into the small field of view encoding subunit, and sequentially process it through the first convolution layer, the first normalization layer, the first activation function layer and the maximum pooling layer to obtain several small field of view encoding feature maps of the raster image; input the information feature map into the large field of view encoding subunit, and sequentially process it through the second convolution layer, the second normalization layer, the second activation function layer and the average pooling layer to obtain several large field of view encoding feature maps of the raster image.

[0062] In this embodiment, the segmentation device inputs the information feature map into the small field of view encoding subunit, and sequentially processes it through the first convolution layer, the first normalization layer, the first activation function layer, and the maximum pooling layer to obtain a plurality of small field of view encoding feature maps of the raster image, wherein the first convolution layer uses a convolution layer with a 3*3 convolution kernel, and the small kernel convolution is used to capture fine local features, while the maximum pooling layer can retain the key information of the vector while downsampling. The small field of view encoding feature map is:

[0063]

[0064] Where, is the small field of view encoding feature map, is the information feature map, is the convolution function of the 3*3 convolution kernel, is the normalization function, is the ReLU activation function, is the maximum pooling function.

[0065] The segmentation device inputs the information feature map into the large field of view encoding subunit, and sequentially processes it through the second convolution layer, the second normalization layer, the second activation function layer, and the average pooling layer to obtain a plurality of large field of view encoding feature maps of the raster image. The second convolution layer uses a convolution layer with a 7*7 convolution kernel. Using a larger convolution kernel can enable the network to better understand the global structure. The normalization layer accelerates the training process and improves stability through standardization. The ReLU activation function effectively increases the nonlinear expression ability of the network. The average pooling layer network can smoothly retain the important features in the input data instead of just focusing on the most prominent features. The large field of view encoding feature map is:

[0066]

[0067] Where, Encode feature maps for large field of view, is the convolution function of the 7*7 convolution kernel, is the average pooling function.

[0068] S412: Cross-joining and segmenting the small-field coding feature map and the large-field coding feature map of the same raster image along the channel dimension to obtain several first segmentation feature maps and second segmentation feature maps of the raster image; inputting the first segmentation feature map into the attention extraction subunit for attention extraction to obtain several first attention feature maps of the raster image; and performing feature fusion on the first attention feature map and the second segmentation feature map of the same raster image to obtain several coding feature maps of the raster image.

[0069] In this embodiment, the segmentation device cross-splices and segments the small-field coding feature map and the large-field coding feature map of the same raster image along the channel dimension to obtain a plurality of first segmentation feature maps and second segmentation feature maps of the raster image to effectively extract features of different receptive fields.

[0070] The segmentation device inputs the first segmentation feature map into the attention extraction subunit for attention extraction to obtain several first attention feature maps of the raster images; the segmentation device performs feature fusion on the first attention feature map and the second segmentation feature map of the same raster image to obtain several encoding feature maps of the raster images. Specifically, the segmentation device uses a matrix product method to perform feature fusion on the first attention feature map and the second segmentation feature map of the same raster image to obtain several encoding feature maps of the raster images.

[0071] The attention extraction subunit includes a global attention extraction block and a local attention extraction block; the global attention extraction block includes a global average pooling layer and a first multi-head self-attention network; the local attention extraction block includes a spatial pyramid pooling network and a second multi-head self-attention network; please refer to Figure 5 , Figure 5 The flowchart of S412 in the method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application includes steps S4121 to S4123, which are specifically as follows:

[0072] S4121: Inputting the first segmentation feature map into the global average pooling layer for pooling processing to obtain a global average pooling feature map; inputting the global average pooling feature map into the first multi-head self-attention network for attention feature extraction to obtain a first sub-attention feature map;

[0073] In this embodiment, the segmentation device inputs the first segmentation feature map into the global attention extraction block, performs pooling processing based on the first segmentation feature map and the global average pooling layer to obtain a global average pooling feature map, and uses the global average pooling layer to compress the first segmentation feature map into a global feature vector, effectively extracting the most significant global context features while eliminating spatial dimension differences.

[0074] The segmentation device extracts attention features based on the global average pooling feature map and the first multi-head self-attention network to obtain a first sub-attention feature map.

[0075] Specifically, the first multi-head self-attention network adopts a multi-head attention mechanism, which is an extension of the self-attention mechanism. The self-attention mechanism generates output by dynamically calculating the weighted average to capture the dependencies within the sequence. The core calculation formula of self-attention is:

[0076]

[0077] Where, Q 、 K 、 V Represent query, key, and value matrices respectively, is the dimension of the key vector, used to scale the dot product to stabilize Function, that is, the normalized exponential function.

[0078] The multi-head attention mechanism expands upon the self-attention mechanism by introducing multiple attention heads to process input information in parallel, distributing attention across features of different dimensions and capturing more comprehensive and rich features and context. The multi-head attention mechanism performs multiple sets of self-attention operations in parallel on compressed features. Through multiple sets of independent linear transformations, a single global feature is decoupled into multidimensional feature subspaces, enabling the model to simultaneously focus on global information interactions across different semantic dimensions.

[0079] S4122: Input the first segmentation feature map into the spatial pyramid pooling network for multi-scale pooling feature extraction, sharpening operation and splicing processing to obtain a local pooling feature map; input the local pooling feature map into the second multi-head self-attention network for attention feature extraction to obtain a second sub-attention feature map.

[0080] In this embodiment, the segmentation device inputs the first segmentation feature map into the local attention extraction block, performs multi-scale pooling feature extraction, sharpening operation and splicing processing based on the first segmentation feature map and the spatial pyramid pooling network to obtain a local pooling feature map.

[0081] Specifically, if the size of the first segmentation feature map is h×w×c, the scale set is {M1, M2, …, Mn}, based on each scale Mi, the step size is set to [h / M i] and [w / M i]. For the first segmentation feature maps of several grid images, the step sizes [h / M i] and [w / M i] are divided into A grid is formed, sub-regions; the maximum pooling operation is used for each sub-region to realize multi-scale pooling feature extraction, and the The pooling results of several scales of the same raster image are sharpened and spliced ​​to obtain the local pooling feature map.

[0082] The segmentation device extracts attention features based on the local pooling feature map and the second multi-head self-attention network to obtain a second sub-attention feature map, and realizes semantic association capture across spatial regions through multi-dimensional feature interaction modeling. For specific embodiments, please refer to step S4121 and will not be repeated here.

[0083] S4123: Reconstruct the spatial dimension of the first sub-attention feature map and the second sub-attention feature map to obtain a spatial dimension reconstructed feature map as the first attention feature map.

[0084] In this embodiment, the segmentation device reconstructs the spatial dimension of the first sub-attention feature map and the second sub-attention feature map to obtain a spatial dimension reconstructed feature map as the first attention feature map.

[0085] Specifically, the segmentation device splits the first sub-attention feature map into feature sub-maps of several scales, and through bilinear interpolation, uniformly upsamples the feature sub-maps of several scales to the size of the second sub-attention feature map, and adds the upsampled feature sub-maps and the second sub-attention feature map element by element to obtain a spatial dimension reconstructed feature map, realizing the "deconstruction-interaction-reconstruction" process of cross-scale feature fusion, which not only retains the multi-scale advantages of the spatial pyramid pooling network, but also enhances the semantic consistency of spatial features through the multi-head self-attention mechanism, thereby improving the accuracy of cancer cell segmentation.

[0086] S42: performing splicing processing on the coded feature map output by the first coding unit and the frame prompt pooling feature map of the first scale of the same raster image to obtain a coded spliced ​​map of the first scale of each raster image; performing encoding processing on the coded spliced ​​map of the first scale as input data of the next coding unit to obtain a coded feature map of each raster image output by the next coding unit.

[0087] In this embodiment, the segmentation device splices the encoding feature map output by the first encoding unit with the frame prompt pooling feature map of the first scale of the same raster image to obtain the encoding splicing map of the first scale of each raster image; and encodes the encoding splicing map of the first scale as input data of the next encoding unit to obtain the encoding feature map of each raster image output by the next encoding unit.

[0088] S43: performing splicing processing on the coding feature map of each of the raster images output by the next coding unit and the next scale box prompt pooling feature map of the same raster image to obtain the coding splicing map of the next scale of each of the raster images; repeating the coding processing according to the coding splicing map of the next scale of each of the raster images until the coding feature map of each of the raster images output by the last coding unit is obtained.

[0089] In this embodiment, the segmentation device splices the encoding feature map of each of the raster images output by the next encoding unit with the next scale box prompt pooling feature map of the same raster image to obtain the encoding splicing map of the next scale of each of the raster images; and repeatedly performs encoding processing according to the encoding splicing map of the next scale of each of the raster images until the encoding feature map of each of the raster images output by the last encoding unit is obtained.

[0090] S5: Inputting the plurality of encoded feature maps of the same raster image and the frame prompt pooling feature maps of a plurality of scales into the decoding module to obtain a plurality of decoded feature maps of each raster image.

[0091] The decoding module is used to gradually restore the spatial resolution of the image and further fuse and extract features. In this embodiment, the segmentation device inputs several encoded feature maps of the same raster image and several scales of the frame hint pooling feature maps into the decoding module to obtain several decoded feature maps for each raster image. During the decoding process, the frame hint pooling feature maps of the corresponding scales are introduced for interactive information processing, thereby improving the accuracy of feature decoding.

[0092] The decoding module includes several stacked decoding units; see Figure 6 , Figure 6 The flowchart of S5 in the method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application includes steps S51 to S53, which are specifically as follows:

[0093] S51: using the encoded feature map of the raster image as input data of the first decoding unit of the decoding module for decoding processing, and obtaining a plurality of decoded feature maps of the raster image output by the first decoding unit.

[0094] In this embodiment, the segmentation device uses the encoded feature map of the raster image as input data of the first decoding unit of the decoding module for decoding processing, and obtains a plurality of decoded feature maps of the raster image output by the first decoding unit.

[0095] The decoding unit includes a deconvolution unit, a small field of view decoding subunit, a large field of view decoding subunit and an attention extraction subunit; see Figure 7 , Figure 7 The flowchart of S51 in the method for cancer cell segmentation based on digital pathology images provided in one embodiment of the present application includes steps S511 to S513, which are specifically as follows:

[0096] S511: Input the encoded feature map into the deconvolution unit for deconvolution processing to obtain a plurality of deconvolution feature maps of the raster images.

[0097] In this embodiment, the segmentation device inputs the encoded feature map into the deconvolution unit for deconvolution processing to obtain several deconvolution feature maps of the raster image, wherein the deconvolution unit uses a deconvolution layer with a 2*2 convolution kernel to restore the encoded feature map to the original input size.

[0098] S512: Input the deconvolution feature map into the small field of view decoding subunit, and process it sequentially through the third convolution layer, the third normalization layer, and the third activation function layer to obtain several small field of view decoding feature maps of the raster image; input the deconvolution feature map into the large field of view decoding subunit, and process it sequentially through the fourth convolution layer, the fourth normalization layer, and the fourth activation function layer to obtain several large field of view decoding feature maps of the raster image.

[0099] In this embodiment, the segmentation device inputs the deconvolution feature map into the small field of view decoding subunit, and processes it in sequence through the third convolution layer, the third normalization layer and the third activation function layer to obtain several small field of view decoding feature maps of the raster image, wherein the convolution kernel of the third convolution layer is the same as that of the first convolution layer.

[0100] The segmentation device inputs the deconvolution feature map into the large field of view decoding subunit, and processes it in sequence through the fourth convolution layer, the fourth normalization layer, and the fourth activation function layer to obtain a plurality of large field of view decoding feature maps of the raster image, wherein the convolution kernel of the fourth convolution layer is the same as that of the second convolution layer.

[0101] S513: Cross-joining and segmenting the small-field decoding feature map and the large-field decoding feature map of the same raster image along the channel dimension to obtain a plurality of third segmentation feature maps and a fourth segmentation feature map of the raster image, inputting the third segmentation feature map into the attention extraction subunit for attention extraction to obtain a plurality of second attention feature maps of the raster image, and performing feature fusion on the second attention feature map and the fourth segmentation feature map of the same raster image to obtain a plurality of decoding feature maps of the raster image.

[0102] In this embodiment, the segmentation device cross-joins and segments the small-field decoding feature map and the large-field decoding feature map of the same raster image along the channel dimension to obtain a plurality of third segmentation feature maps and a fourth segmentation feature map of the raster image. The third segmentation feature map is input into the attention extraction subunit for attention extraction to obtain a plurality of second attention feature maps of the raster image. The second attention feature map and the fourth segmentation feature map of the same raster image are feature-fused to obtain a plurality of decoding feature maps of the raster image. For a specific embodiment, reference may be made to step S412, which will not be described in detail here.

[0103] S52: splicing the last scale frame prompt pooling feature map of the same raster image with the decoded feature map of the raster image output by the first decoding unit to obtain a last scale decoded splicing map, and using the last scale decoded splicing map as input data of the next decoding unit for decoding, to obtain several decoded feature maps of the raster image output by the next decoding unit.

[0104] In this embodiment, the segmentation device splices the frame prompt pooling feature map of the last scale of the same raster image with the decoded feature map of the raster image output by the first decoding unit to obtain a decoded spliced ​​map of the last scale, and uses the decoded spliced ​​map of the last scale as input data of the next decoding unit for decoding processing to obtain several decoded feature maps of the raster image output by the next decoding unit.

[0105] S53: Splicing the box prompt pooling feature map of the previous scale of the same raster image with the decoded feature map of the raster image output by the next decoding unit to obtain a decoded spliced ​​map of the previous scale, and repeating the decoding process until obtaining the decoded feature maps of several raster images output by the last decoding unit.

[0106] In this embodiment, the segmentation device splices the box prompt pooling feature map of the previous scale of the same raster image with the decoded feature map of the raster image output by the next decoding unit to obtain a decoded spliced ​​map of the previous scale, and repeats the decoding process until several decoded feature maps of the raster image output by the last decoding unit are obtained.

[0107] S6: Inputting the plurality of decoded feature maps of each raster image into the detection module for probability prediction to obtain cancer cell probability prediction data of each raster image.

[0108] In this embodiment, the segmentation device inputs the decoded feature maps of several of the raster images into the detection module and performs probability prediction through Sigmoid activation function processing to obtain cancer cell probability prediction data of several of the raster images, wherein the cancer cell probability prediction data includes cancer cell probability prediction vectors of several dimensions of several pixels of the raster image.

[0109] S7: performing segmentation processing on the digital pathology image to be segmented according to the cancer cell probability prediction data of each of the grid images to obtain a cancer cell segmentation result of the digital pathology image to be segmented.

[0110] In this embodiment, the segmentation device extracts the dimension of the largest cancer cell probability prediction vector as the recognition result of the pixel based on the cancer cell probability prediction data of the plurality of raster images, and performs segmentation processing on the digital pathology image to be segmented to obtain a cancer cell segmentation result of the digital pathology image to be segmented.

[0111] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of a cancer cell segmentation device based on digital pathology images provided in one embodiment of the present application. The device can implement all or part of the cancer cell segmentation device based on digital pathology images through software, hardware, or a combination of both. The device 8 includes:

[0112] A data acquisition module 81 is used to obtain a digital pathology image to be segmented and a cancer cell segmentation model, wherein the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module;

[0113] A feature construction module 82 is configured to divide the digital pathology image to be segmented into a plurality of raster images, input the plurality of raster images into the feature extraction module for information feature extraction and frame prompt mask map construction, and obtain an information feature map and a frame prompt mask map for each of the raster images;

[0114] A feature pooling module 83 is configured to scale and pool the frame hint mask images of each of the raster images to obtain frame hint pooling feature maps of multiple scales corresponding to each of the raster images;

[0115] A feature encoding module 84 is configured to input the information feature map of the same raster image and the frame hint pooling feature maps of several scales into the encoding module to obtain several encoded feature maps of each raster image;

[0116] A feature decoding module 85 is configured to input the plurality of encoded feature maps of the same raster image and the frame hint pooling feature maps of a plurality of scales into the decoding module to obtain a plurality of decoded feature maps of each raster image;

[0117] A probability prediction module 86 is configured to input the decoded feature maps of each raster image into the detection module for probability prediction, thereby obtaining cancer cell probability prediction data for each raster image;

[0118] The cancer cell segmentation module 87 is configured to segment the digital pathology image to be segmented according to the cancer cell probability prediction data of each of the grid images, and obtain a cancer cell segmentation result of the digital pathology image to be segmented.

[0119] In an embodiment of the present application, a digital pathology image to be segmented and a cancer cell segmentation model are obtained through a data acquisition module, wherein the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module and a detection module; the digital pathology image to be segmented is divided into a plurality of raster images through a feature construction module, and the plurality of raster images are input into the feature extraction module for information feature extraction and frame prompt mask map construction to obtain information feature maps and frame prompt mask maps of each raster image; the frame prompt mask map of each raster image is scaled and pooled through a feature pooling module to obtain frame prompt pooling feature maps of a plurality of scales corresponding to each raster image; the information feature maps of the same raster image are pooled through a feature encoding module. The information feature map and the frame prompt pooling feature maps at several scales are input into the encoding module to obtain several encoded feature maps for each raster image; the feature decoding module inputs several encoded feature maps and the frame prompt pooling feature maps at several scales of the same raster image into the decoding module to obtain several decoded feature maps for each raster image; the probability prediction module inputs several decoded feature maps of each raster image into the detection module for probability prediction to obtain cancer cell probability prediction data for each raster image; the cancer cell segmentation module performs segmentation processing on the digital pathology image to be segmented based on the cancer cell probability prediction data of each raster image to obtain cancer cell segmentation results for the digital pathology image to be segmented. By extracting feature information and frame prompt information from the digital pathology image to be segmented, performing encoding and decoding processing, and performing cancer cell segmentation based on the obtained decoded feature maps, the accuracy and efficiency of cancer cell segmentation are improved.

[0120] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. The computer device 9 includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91; the computer device may store multiple instructions, which are suitable for being loaded and executed by the processor 91. Figures 1 to 7 The method steps of the embodiment shown, the specific execution process can be found in Figures 1 to 7 The detailed description of the illustrated embodiment will not be repeated here.

[0121] The processor 91 may include one or more processing cores. The processor 91 utilizes various interfaces and circuits to connect to various components within the server. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 92 and accessing data within the memory 92, the processor 91 performs various functions and processes data in the digital pathology image-based cancer cell segmentation device 8. Optionally, the processor 91 may be implemented in the form of at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 91 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the touchscreen display; and the modem handles wireless communications. It is understood that the modem may also be implemented as a separate chip, rather than integrated into the processor 91.

[0122] The memory 92 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 92 includes non-transitory computer-readable storage medium. The memory 92 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 92 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch control instructions), instructions for implementing the aforementioned method embodiments, and the data storage area may store data related to the aforementioned method embodiments. The memory 92 may also optionally be at least one storage device located remotely from the aforementioned processor 91.

[0123] The embodiment of the present application also provides a storage medium, which can store multiple instructions, which are suitable for the processor to load and execute the above Figures 1 to 7 The method steps of the embodiment shown, the specific execution process can be found in Figures 1 to 7 The detailed description of the illustrated embodiment will not be repeated here.

[0124] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0125] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0126] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraint algorithm of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0127] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0129] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0130] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form.

[0131] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications of the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims of the present invention and equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A cancer cell segmentation method based on digital pathology images, characterized in that: The following steps are involved: Obtaining a digital pathology image to be segmented and a cancer cell segmentation model, wherein the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module; the encoding module includes a plurality of stacked encoding units; the encoding unit includes a small field of view encoding subunit, a large field of view encoding subunit, and an attention extraction subunit; Dividing the digital pathology image to be segmented into a plurality of raster images, inputting the plurality of raster images into the feature extraction module for information feature extraction and frame prompt mask map construction, and obtaining an information feature map and a frame prompt mask map for each of the raster images; Scaling and pooling the frame hint mask map of each of the raster images to obtain frame hint pooling feature maps of several scales corresponding to each of the raster images; Inputting the information feature map of the same raster image and the frame hint pooling feature maps of several scales into the encoding module to obtain several encoding feature maps of each raster image, including using the information feature map of the raster image as input data of a first encoding unit of the encoding module for encoding processing, and obtaining several encoding feature maps of the raster image output by the first encoding unit; The method of using the information feature map of the raster image as input data of the first encoding unit of the encoding module for encoding processing to obtain a plurality of encoding feature maps of the raster image output by the first encoding unit comprises the following steps: Inputting the information feature map into the small field of view encoding subunit, sequentially processing it through a first convolution layer, a first normalization layer, a first activation function layer, and a maximum pooling layer to obtain a plurality of small field of view encoding feature maps of the raster image; inputting the information feature map into the large field of view encoding subunit, sequentially processing it through a second convolution layer, a second normalization layer, a second activation function layer, and an average pooling layer to obtain a plurality of large field of view encoding feature maps of the raster image; Cross-joining and segmenting the small-view coding feature map and the large-view coding feature map of the same raster image along the channel dimension to obtain a plurality of first segmentation feature maps and second segmentation feature maps of the raster image; inputting the first segmentation feature map into the attention extraction subunit for attention extraction to obtain a plurality of first attention feature maps of the raster image; and performing feature fusion on the first attention feature map and the second segmentation feature map of the same raster image to obtain a plurality of coding feature maps of the raster image; Inputting the plurality of encoded feature maps of the same raster image and the frame prompt pooling feature maps of a plurality of scales into the decoding module to obtain a plurality of decoded feature maps of each raster image; Inputting the plurality of decoded feature maps of each raster image into the detection module for probability prediction to obtain cancer cell probability prediction data of each raster image; The digital pathology image to be segmented is segmented according to the cancer cell probability prediction data of each of the grid images to obtain a cancer cell segmentation result of the digital pathology image to be segmented.

2. The method for cancer cell segmentation based on digital pathology images according to claim 1, characterized in that: The method of dividing the digital pathology image to be segmented into a plurality of raster images, inputting the plurality of raster images into the feature extraction module for information feature extraction and frame prompt mask map construction, and obtaining the information feature map and frame prompt mask map of each raster image comprises the following steps: Performing edge detection on each of the raster images to obtain a Laplacian graph of each of the raster images; performing clustering and median filtering on each of the raster images to obtain a median filtered graph of each of the raster images; cross-joining the Laplacian graph and the median filtered graph of the same raster image to obtain an information feature graph of each of the raster images; Initial detection frames of several of the raster images are obtained, and a minimum circumscribed rectangle is solved for the initial detection frames to obtain frame prompt information of several of the raster images; and frame prompt mask maps are constructed based on the several raster images and the corresponding frame prompt information to obtain frame prompt mask maps of several of the raster images.

3. The method for cancer cell segmentation based on digital pathology images according to claim 2, characterized in that: The information feature map of the same raster image and the frame prompt pooling feature maps of several scales are input into the encoding module to obtain several encoding feature maps of each raster image, including the steps of: Using the information feature map of the raster image as input data of the first encoding unit of the encoding module for encoding processing, and obtaining a plurality of encoding feature maps of the raster image output by the first encoding unit; The coded feature map output by the first coding unit is spliced ​​with the frame hint pooling feature map of the first scale of the same raster image to obtain a coded spliced ​​map of the first scale of each raster image; the coded spliced ​​map of the first scale is used as input data of the next coding unit for encoding to obtain a coded feature map of each raster image output by the next coding unit; The encoding feature map of each raster image output by the next encoding unit is spliced ​​with the next scale box prompt pooling feature map of the same raster image to obtain the encoding splicing map of the next scale of each raster image; the encoding process is repeatedly performed according to the encoding splicing map of the next scale of each raster image until the encoding feature map of each raster image output by the last encoding unit is obtained.

4. The method for cancer cell segmentation based on digital pathology images according to claim 3, wherein: The attention extraction subunit includes a global attention extraction block and a local attention extraction block; the global attention extraction block includes a global average pooling layer and a first multi-head self-attention network; the local attention extraction block includes a spatial pyramid pooling network and a second multi-head self-attention network; The step of inputting the first segmentation feature map into the attention extraction subunit for attention extraction to obtain a plurality of first attention feature maps of the raster images comprises the following steps: Inputting the first segmentation feature map into the global average pooling layer for pooling processing to obtain a global average pooling feature map; inputting the global average pooling feature map into the first multi-head self-attention network for attention feature extraction to obtain a first sub-attention feature map; Inputting the first segmentation feature map into the spatial pyramid pooling network for multi-scale pooling feature extraction, sharpening operation and splicing processing to obtain a local pooling feature map; inputting the local pooling feature map into the second multi-head self-attention network for attention feature extraction to obtain a second sub-attention feature map; The first sub-attention feature map and the second sub-attention feature map are spatially reconstructed to obtain a spatial dimension reconstructed feature map as the first attention feature map.

5. The method for cancer cell segmentation based on digital pathology images according to claim 4, characterized in that: The decoding module includes a plurality of stacked decoding units; The step of inputting the plurality of encoded feature maps of the same raster image and the frame prompt pooling feature maps of the plurality of scales into the decoding module to obtain the plurality of decoded feature maps of the respective raster images comprises the following steps: Using the encoded feature map of the raster image as input data of the first decoding unit of the decoding module for decoding processing, and obtaining a plurality of decoded feature maps of the raster image output by the first decoding unit; splicing the last scale frame hint pooling feature map of the same raster image with the decoded feature map of the raster image output by the first decoding unit to obtain a decoded spliced ​​map of the last scale, and performing decoding on the decoded spliced ​​map of the last scale as input data of the next decoding unit to obtain a plurality of decoded feature maps of the raster image output by the next decoding unit; The frame prompt pooling feature map of the previous scale of the same raster image is spliced ​​with the decoded feature map of the raster image output by the next decoding unit to obtain a decoded spliced ​​map of the previous scale, and the decoding process is repeated until several decoded feature maps of the raster images output by the last decoding unit are obtained.

6. The method for cancer cell segmentation based on digital pathology images according to claim 5, characterized in that: The decoding unit includes a deconvolution unit, a small field of view decoding subunit, a large field of view decoding subunit and an attention extraction subunit; The method comprises the steps of: using the encoded feature map of the raster image as input data of the first decoding unit of the decoding module for decoding processing, and obtaining a plurality of decoded feature maps of the raster image output by the first decoding unit. Inputting the encoded feature map into the deconvolution unit for deconvolution processing to obtain a plurality of deconvolution feature maps of the raster image; Inputting the deconvolution feature map into the small field of view decoding subunit, and sequentially processing it through a third convolution layer, a third normalization layer, and a third activation function layer to obtain a plurality of small field of view decoding feature maps of the raster image; Inputting the deconvolution feature map into the large field of view decoding subunit, and sequentially processing it through a fourth convolution layer, a fourth normalization layer, and a fourth activation function layer to obtain a plurality of large field of view decoding feature maps of the raster image; The small-field decoding feature map and the large-field decoding feature map of the same raster image are cross-joined and segmented along the channel dimension to obtain a plurality of third segmentation feature maps and a fourth segmentation feature map of the raster image. The third segmentation feature map is input into the attention extraction subunit for attention extraction to obtain a plurality of second attention feature maps of the raster image. The second attention feature map and the fourth segmentation feature map of the same raster image are feature-fused to obtain a plurality of decoding feature maps of the raster image.

7. A cancer cell segmentation device based on digital pathology images, characterized in that: include: A data acquisition module, configured to obtain a digital pathology image to be segmented and a cancer cell segmentation model, wherein the cancer cell segmentation model comprises a feature extraction module, an encoding module, a decoding module, and a detection module; the encoding module comprises a plurality of stacked encoding units; the encoding units comprise a small field of view encoding subunit, a large field of view encoding subunit, and an attention extraction subunit; A feature construction module is used to divide the digital pathology image to be segmented into a plurality of raster images, input the plurality of raster images into the feature extraction module to perform information feature extraction and frame prompt mask map construction, and obtain an information feature map and a frame prompt mask map for each of the raster images; A feature pooling module, configured to scale and pool the frame hint mask images of each of the raster images to obtain frame hint pooling feature maps of several scales corresponding to each of the raster images; a feature encoding module, configured to input the information feature map of the same raster image and the frame hint pooling feature maps of several scales into the encoding module to obtain several encoded feature maps of each raster image, including using the information feature map of the raster image as input data of a first encoding unit of the encoding module for encoding processing, and obtaining several encoded feature maps of the raster image output by the first encoding unit; The method of using the information feature map of the raster image as input data of the first encoding unit of the encoding module for encoding processing to obtain a plurality of encoding feature maps of the raster image output by the first encoding unit comprises the following steps: Inputting the information feature map into the small field of view encoding subunit, sequentially processing it through a first convolution layer, a first normalization layer, a first activation function layer, and a maximum pooling layer to obtain a plurality of small field of view encoding feature maps of the raster image; inputting the information feature map into the large field of view encoding subunit, sequentially processing it through a second convolution layer, a second normalization layer, a second activation function layer, and an average pooling layer to obtain a plurality of large field of view encoding feature maps of the raster image; Cross-joining and segmenting the small-view coding feature map and the large-view coding feature map of the same raster image along the channel dimension to obtain a plurality of first segmentation feature maps and second segmentation feature maps of the raster image; inputting the first segmentation feature map into the attention extraction subunit for attention extraction to obtain a plurality of first attention feature maps of the raster image; and performing feature fusion on the first attention feature map and the second segmentation feature map of the same raster image to obtain a plurality of coding feature maps of the raster image; A feature decoding module, configured to input the plurality of encoded feature maps of the same raster image and the frame prompt pooling feature maps of a plurality of scales into the decoding module to obtain a plurality of decoded feature maps of each raster image; A probability prediction module, configured to input the plurality of decoded feature maps of each raster image into the detection module for probability prediction, thereby obtaining cancer cell probability prediction data of each raster image; The cancer cell segmentation module is used to perform segmentation processing on the digital pathology image to be segmented according to the cancer cell probability prediction data of each of the grid images, so as to obtain a cancer cell segmentation result of the digital pathology image to be segmented.

8. A computer device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the method for cancer cell segmentation based on digital pathology images as claimed in any one of claims 1 to 6 are implemented.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for cancer cell segmentation based on digital pathology images according to any one of claims 1 to 6 are implemented.

Images