Cancer cell segmentation method and device based on digital pathological image, and computer equipment

Abstract

The invention relates to the field of medical image segmentation, in particular to a cancer cell segmentation method and device based on a digital pathological image and computer equipment, and aims to extract feature information and box prompt information in the digital pathological image to be segmented, perform coding processing and decoding processing, and perform cancer cell segmentation based on the obtained decoded feature map. And the accuracy and efficiency of cancer cell segmentation are improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and particularly to a method, device, computer device, and storage medium for cancer cell segmentation based on digital pathology images. Background Art

[0002] With the development of artificial intelligence, the technology in the field of image segmentation has made great progress. Cancer cell segmentation is one of the important tasks in pathological image analysis. However, due to the huge data format of digital pathology images (WSIs) and the complex and diverse features of each image, the current technical solutions often require a large amount of computing resources for cancer cell segmentation of WSIs, and the extraction of detailed features is still not comprehensive enough, resulting in limited cancer cell segmentation effects and difficult to perform accurate cancer cell segmentation tasks. Summary of the Invention

[0003] Based on this, the purpose of the present invention is to provide a method, device, equipment, and storage medium for cancer cell segmentation based on digital pathology images. By extracting the feature information and box hint information in the digital pathology image to be segmented, performing encoding processing and decoding processing, and performing cancer cell segmentation based on the obtained decoded feature map, the accuracy and efficiency of cancer cell segmentation are improved.

[0004] In the first aspect, an embodiment of the present application provides a method for cancer cell segmentation based on digital pathology images, including the following steps: Obtain a digital pathology image to be segmented and a cancer cell segmentation model, where the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module; Divide the digital pathology image to be segmented into several grid images, and input the several grid images into the feature extraction module for information feature extraction and box hint mask map construction to obtain the information feature map and box hint mask map of each grid image; Perform scale scaling and pooling processing on the box hint mask maps of each grid image to obtain several scale box hint pooled feature maps corresponding to each grid image; Input the information feature map and several scale box hint pooled feature maps of the same grid image into the encoding module to obtain several encoded feature maps of each grid image; Input several encoded feature maps and several scale box hint pooled feature maps of the same grid image into the decoding module to obtain several decoded feature maps of each grid image; Input several decoded feature maps of each grid image into the detection module for probability prediction to obtain cancer cell probability prediction data of each grid image; According to the cancer cell probability prediction data of each of the grid images, the digital pathological image to be segmented is segmented to obtain the cancer cell segmentation result of the digital pathological image to be segmented.

[0005] In a second aspect, an embodiment of the present application provides a cancer cell segmentation device based on a digital pathological image, including: A data acquisition module, configured to acquire a digital pathological image to be segmented and a cancer cell segmentation model, where the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module; A feature construction module, configured to divide the digital pathological image to be segmented into a plurality of grid images, input the plurality of grid images into the feature extraction module for information feature extraction and frame hint mask map construction, and obtain the information feature map and the frame hint mask map of each of the grid images; A feature pooling module, configured to perform scale scaling and pooling processing on the frame hint mask maps of each of the grid images to obtain a plurality of scale frame hint pooled feature maps corresponding to each of the grid images; A feature encoding module, configured to input the information feature map of the same grid image and the plurality of scale frame hint pooled feature maps into the encoding module to obtain a plurality of encoded feature maps of each grid image; A feature decoding module, configured to input the plurality of encoded feature maps of the same grid image and the plurality of scale frame hint pooled feature maps into the decoding module to obtain a plurality of decoded feature maps of each grid image; A probability prediction module, configured to input the plurality of decoded feature maps of each grid image into the detection module for probability prediction to obtain the cancer cell probability prediction data of each of the grid images; A cancer cell segmentation module, configured to segment the digital pathological image to be segmented according to the cancer cell probability prediction data of each of the grid images to obtain the cancer cell segmentation result of the digital pathological image to be segmented.

[0006] In a third aspect, an embodiment of the present application provides a computer device, including: a processor, a memory, and a computer program stored on the memory and executable on the processor; when the computer program is executed by the processor, the steps of the cancer cell segmentation method based on a digital pathological image as described in the first aspect are implemented.

[0007] In a fourth aspect, an embodiment of the present application provides a storage medium, where the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the cancer cell segmentation method based on a digital pathological image as described in the first aspect are implemented.

[0008] In an embodiment of the present application, a method, apparatus, computer device, and storage medium for cancer cell segmentation based on digital pathology images are provided. By extracting the feature information and box hint information in the digital pathology image to be segmented, performing encoding processing and decoding processing, and performing cancer cell segmentation based on the obtained decoded feature map, the accuracy and efficiency of cancer cell segmentation are improved.

[0009] For better understanding and implementation, the present invention will be described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic flowchart of a method for cancer cell segmentation based on digital pathology images provided by an embodiment of the present application; Figure 2 A schematic flowchart of S2 in the method for cancer cell segmentation based on digital pathology images provided by an embodiment of the present application; Figure 3 A schematic flowchart of S4 in the method for cancer cell segmentation based on digital pathology images provided by an embodiment of the present application; Figure 4 A schematic flowchart of S41 in the method for cancer cell segmentation based on digital pathology images provided by an embodiment of the present application; Figure 5 A schematic flowchart of S412 in the method for cancer cell segmentation based on digital pathology images provided by an embodiment of the present application; Figure 6 A schematic flowchart of S5 in the method for cancer cell segmentation based on digital pathology images provided by an embodiment of the present application; Figure 7 A schematic flowchart of S51 in the method for cancer cell segmentation based on digital pathology images provided by an embodiment of the present application; Figure 8 A schematic structural diagram of a device for cancer cell segmentation based on digital pathology images provided by an embodiment of the present application; Figure 9 A schematic structural diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0011] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0012] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit the application. The singular forms "a", "the", and "said" used in this application and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0013] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0014] The data sending end can be a computer device or a mobile terminal device, used to establish a network connection with the data receiving end, capable of encoding the data information sent to the data receiving end, and parsing the data information sent from the data receiving end.

[0015] The data receiving end can be a computer device or a mobile terminal device, used to establish a network connection with the data sending end, capable of encoding the data information sent to the data sending end, and parsing the data information sent from the data sending end.

[0016] Please refer to Figure 1 , Figure 1 which is a schematic flow diagram of a method for segmenting cancer cells based on digital pathology images provided for an embodiment of this application. The method includes the following steps: S1: Obtain the digital pathology image to be segmented and the cancer cell segmentation model.

[0017] The execution subject of the method for segmenting cancer cells based on digital pathology images is the segmentation device of the method for segmenting cancer cells based on digital pathology images (hereinafter referred to as the segmentation device). In an optional embodiment, the segmentation device can be a computer device, which can be a server, or a server cluster formed by combining multiple computer devices.

[0018] In this embodiment, the segmentation device can obtain the digital pathology image to be segmented input by the user, or select the digital pathology image to be segmented from a preset database.

[0019] The segmentation device obtains a preset cancer cell segmentation model, where the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module.

[0020] S2: Divide the digital pathology image to be segmented into a plurality of grid images, and input the plurality of grid images into the feature extraction module for information feature extraction and box prompt mask map construction, so as to obtain the information feature map and box prompt mask map of each grid image.

[0021] In this embodiment, the segmentation device divides the digital pathology image to be segmented into a plurality of grid images, and inputs the plurality of grid images into the feature extraction module for information feature extraction and box prompt mask map construction, so as to obtain the information feature map and box prompt mask map of each grid image.

[0022] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of S2 in the cancer cell segmentation method based on digital pathology images provided by an embodiment of the present application, including steps S21 to S22, specifically as follows: S21: Perform edge detection on each grid image to obtain the Laplacian graph of each grid image; perform clustering operation and median filtering processing on each grid image to obtain the median filtered graph of each grid image; cross splice the Laplacian graph and the median filtered graph of the same grid image to obtain the information feature map of each grid image.

[0023] Edge detection reflects edge details by quantifying the difference between each pixel and its adjacent pixels. In this embodiment, the segmentation device uses a Laplacian operator to perform edge detection on the grid image to obtain the Laplacian graphs of a plurality of grid images, so as to extract the color features of the digital pathology image to be segmented, and reduce the interference effects of color and brightness factors.

[0024] Traditional image clustering algorithms such as K-means can roughly extract information. In this embodiment, the segmentation device uses the K-means clustering algorithm to perform clustering operation and median filtering processing on the grid image, classifies the pixels of the grid image into two categories, namely the target area and the non-target area, uses median filtering to eliminate isolated noise points, and finally obtains a graph that can initially distinguish the foreground and the background, strengthens the texture features of the lesion area, and obtains the median filtered graphs of a plurality of grid images.

[0025] The segmentation device cross - splices the Laplacian graph and the median - filtered graph of the same raster image to obtain information feature graphs of a number of the raster images, so as to fuse color features and texture features, fully extract the feature information of the digital pathology image to be segmented, and improve the accuracy of cancer cell segmentation of the digital pathology image to be segmented.

[0026] S22: Obtain initial detection frames of a number of the raster images, solve the minimum bounding rectangles for the initial detection frames to obtain box hint information of a number of the raster images; construct box hint mask graphs according to a number of the raster images and the corresponding box hint information to obtain box hint mask graphs of a number of the raster images.

[0027] Since in the WSI slices, that is, raster images, the positions of cancer cells are always changing, and may appear in various parts of the raster images. Even in some slices, there are all cancer cells or none of them; and even manually screening the range of cancer cells is a cumbersome task.

[0028] In order to simulate manual annotation and at the same time enable the network to perceive the approximate area where cancer cells are located, in this embodiment, the segmentation device uses the generalized Hough transform method to obtain initial detection frames of a number of the raster images, solve the minimum bounding rectangles for the initial detection frames to obtain box hint information of a number of the raster images, and construct box hint mask graphs according to a number of the raster images and the corresponding box hint information to obtain box hint mask graphs of a number of the raster images.

[0029] S3: Scale and pool the box hint mask graphs of each raster image to obtain box hint pooled feature graphs of several scales corresponding to each raster image.

[0030] In this embodiment, the segmentation device scales and pools the box hint mask graphs of each raster image to obtain box hint pooled feature graphs of several scales corresponding to each raster image, so as to highlight the cancer cell area and at the same time suppress background information, which helps the network to focus more on the target area and improve the performance of the network in detecting and locating the target area.

[0031] S4: Input the information feature graph of the same raster image and the box hint pooled feature graphs of several scales into the encoding module to obtain several encoded feature graphs of each raster image.

[0032] In this embodiment, the segmentation device inputs the information feature map of the same grid image and the box hint pooling feature maps of several scales into the encoding module, and obtains several encoded feature maps of each grid image. During the encoding process, the box hint pooling feature maps of the corresponding scales are introduced for interactive information processing, improving the accuracy of feature encoding.

[0033] The encoding module includes several stacked encoding units; refer to Figure 3 , Figure 3 which is a schematic flowchart of S4 in the cancer cell segmentation method based on digital pathology images provided by an embodiment of the present application, including steps S41 to S43, specifically as follows: S41: Use the information feature map of the grid image as the input data of the first encoding unit of the encoding module for encoding processing, and obtain several encoded feature maps of the grid image output by the first encoding unit.

[0034] In this embodiment, the segmentation device uses the information feature map of the grid image as the input data of the first encoding unit of the encoding module for encoding processing, and obtains several encoded feature maps of the grid image output by the first encoding unit.

[0035] The encoding unit includes a small-field encoding subunit, a large-field encoding subunit, and an attention extraction subunit; refer to Figure 4 , Figure 4 which is a schematic flowchart of S41 in the cancer cell segmentation method based on digital pathology images provided by an embodiment of the present application, including steps S411 to S412, specifically as follows: S411: Input the information feature map into the small-field encoding subunit, and successively perform processing through the first convolutional layer, the first normalization layer, the first activation function layer, and the max pooling layer to obtain several small-field encoded feature maps of the grid image; input the information feature map into the large-field encoding subunit, and successively perform processing through the second convolutional layer, the second normalization layer, the second activation function layer, and the average pooling layer to obtain several large-field encoded feature maps of the grid image.

[0036] In this embodiment, the segmentation device inputs the information feature map into the small-field encoding subunit, and successively performs processing through the first convolutional layer, the first normalization layer, the first activation function layer, and the max pooling layer to obtain several small-field encoded feature maps of the grid image. Among them, the first convolutional layer is a convolutional layer with a 3*3 convolutional kernel. The small kernel convolution is used to capture fine local features, and the max pooling layer can retain the key information of the vector while downsampling. The small-field encoded feature map is:

[0037] In the formula, is the small field-of-view encoded feature map, is the information feature map, is the convolution function with a 3*3 convolution kernel, is the normalization function, is the ReLU activation function, is the max pooling function.

[0038] The segmentation device inputs the information feature map into the large field-of-view encoding subunit, and successively processes it through the second convolutional layer, the second normalization layer, the second activation function layer, and the average pooling layer to obtain the large field-of-view encoded feature maps of several grid images. Among them, the second convolutional layer is a convolutional layer with a 7*7 convolution kernel. Using a larger convolution kernel enables the network to better understand the global structure. The normalization layer accelerates the training process and improves stability through standardization. The ReLU activation function effectively increases the non-linear expression ability of the network. The average pooling layer network can smoothly retain the important features in the input data instead of only focusing on the most prominent features. The large field-of-view encoded feature map is:

[0039] In the formula, is the large field-of-view encoded feature map, is the convolution function with a 7*7 convolution kernel, is the average pooling function.

[0040] S412: Cross-join and segment the small field-of-view encoded feature map and the large field-of-view encoded feature map of the same grid image along the channel dimension to obtain the first segmentation feature maps and the second segmentation feature maps of several grid images; input the first segmentation feature maps into the attention extraction subunit for attention extraction to obtain the first attention feature maps of several grid images, and fuse the first attention feature maps and the second segmentation feature maps of the same grid image to obtain the encoded feature maps of several grid images.

[0041] In this embodiment, the segmentation device cross-joins and segments the small field-of-view encoded feature map and the large field-of-view encoded feature map of the same grid image along the channel dimension to obtain the first segmentation feature maps and the second segmentation feature maps of several grid images, so as to effectively extract the features of different receptive fields.

[0042] The segmentation device inputs the first segmentation feature map into the attention extraction sub-unit for attention extraction to obtain the first attention feature maps of several grid images; the segmentation device performs feature fusion on the first attention feature maps and the second segmentation feature maps of the same grid image to obtain the encoded feature maps of several grid images. Specifically, the segmentation device uses the method of matrix multiplication to perform feature fusion on the first attention feature maps and the second segmentation feature maps of the same grid image to obtain the encoded feature maps of several grid images.

[0043] The attention extraction sub-unit includes a global attention extraction block and a local attention extraction block; the global attention extraction block includes a global average pooling layer and a first multi-head self-attention network; the local attention extraction block includes a spatial pyramid pooling network and a second multi-head self-attention network; please refer to Figure 5 , Figure 5 which is a schematic flowchart of S412 in the cancer cell segmentation method based on digital pathology images provided by an embodiment of the present application, including steps S4121 to S4123, specifically as follows: S4121: Input the first segmentation feature map into the global average pooling layer for pooling processing to obtain a global average pooling feature map; input the global average pooling feature map into the first multi-head self-attention network for attention feature extraction to obtain a first sub-attention feature map; In this embodiment, the segmentation device inputs the first segmentation feature map into the global attention extraction block, performs pooling processing according to the first segmentation feature map and the global average pooling layer to obtain a global average pooling feature map, and uses the global average pooling layer to compress the first segmentation feature map into a global feature vector, effectively extracting the most significant global context features and eliminating the spatial dimension differences at the same time.

[0044] The segmentation device performs attention feature extraction according to the global average pooling feature map and the first multi-head self-attention network to obtain a first sub-attention feature map.

[0045] Specifically, the first multi-head self-attention network adopts a multi-head attention mechanism, which is an extension of the self-attention mechanism. The self-attention mechanism generates an output by dynamically calculating a weighted average value to capture the dependencies within the sequence. The core calculation formula of self-attention is:

[0046] In the formula, Q , K , V respectively represent the query, key, and value matrices, is the dimension of the key vector, used to scale the dot product to stabilize The function, namely the normalized exponential function.

[0047] The multi-head attention mechanism is an extension based on the self-attention mechanism. By introducing multiple attention heads to process the input information in parallel, it realizes the attention allocation for features in different dimensions, and further captures more comprehensive and rich features and context information. The multi-head attention mechanism performs multiple groups of self-attention operations in parallel on the compressed features, decouples the single global feature into multi-dimensional feature subspaces through multiple groups of independent linear transformations, enabling the model to simultaneously focus on the global information interaction in different semantic dimensions.

[0048] S4122: Input the first segmentation feature map into the spatial pyramid pooling network for multi-scale pooling feature extraction, sharpening operation, and splicing process to obtain a local pooling feature map; input the local pooling feature map into the second multi-head self-attention network for attention feature extraction to obtain a second sub-attention feature map.

[0049] In this embodiment, the segmentation device inputs the first segmentation feature map into the local attention extraction block, and performs multi-scale pooling feature extraction, sharpening operation, and splicing process according to the first segmentation feature map and the spatial pyramid pooling network to obtain a local pooling feature map.

[0050] Specifically, if the size of the first segmentation feature map is h×w×c and the scale set is {M1, M2, …, Mn}, based on each scale Mi, set the stride as [h / Mi] and [w / Mi]. For the first segmentation feature map of several grid images, divide them into grids with the strides [h / Mi] and [w / Mi], that is, form sub-regions; perform max pooling operation on each sub-region to achieve multi-scale pooling feature extraction, and obtain pooling results. Perform sharpening operation and splicing process on the pooling results of several scales of the same grid image to obtain the local pooling feature map.

[0051] The segmentation device performs attention feature extraction according to the local pooling feature map and the second multi-head self-attention network to obtain a second sub-attention feature map, and realizes the capture of semantic associations across spatial regions through multi-dimensional feature interaction modeling. For specific embodiments, reference can be made to step S4121, which will not be elaborated here.

[0052] S4123: Perform spatial dimension reconstruction on the first sub-attention feature map and the second sub-attention feature map to obtain a spatial dimension reconstruction feature map as the first attention feature map.

[0053] In this embodiment, the segmentation device performs spatial dimension reconstruction on the first sub-attention feature map and the second sub-attention feature map to obtain a spatially dimensionally reconstructed feature map as the first attention feature map.

[0054] Specifically, the segmentation device splits the first sub-attention feature map into feature sub-maps of several scales, and through bilinear interpolation, uniformly upsamples the feature sub-maps of several scales to the size of the second sub-attention feature map. The upsampled feature sub-maps of several scales and the second sub-attention feature map are added element-wise to obtain a spatially dimensionally reconstructed feature map, realizing the "deconstruction-interaction-reconstruction" process of cross-scale feature fusion. This not only retains the multi-scale advantages of the spatial pyramid pooling network but also enhances the semantic consistency of spatial features through the multi-head self-attention mechanism, thereby improving the accuracy of cancer cell segmentation.

[0055] S42: Concatenate the encoded feature map output by the first encoding unit with the box hint pooling feature map of the first scale of the same raster image to obtain the encoded concatenated map of the first scale of each raster image; use the encoded concatenated map of the first scale as the input data for the next encoding unit for encoding processing to obtain the encoded feature maps of each raster image output by the next encoding unit.

[0056] In this embodiment, the segmentation device concatenates the encoded feature map output by the first encoding unit with the box hint pooling feature map of the first scale of the same raster image to obtain the encoded concatenated map of the first scale of each raster image; uses the encoded concatenated map of the first scale as the input data for the next encoding unit for encoding processing to obtain the encoded feature maps of each raster image output by the next encoding unit.

[0057] S43: Concatenate the encoded feature maps of each raster image output by the next encoding unit with the box hint pooling feature map of the next scale of the same raster image to obtain the encoded concatenated map of the next scale of each raster image; repeat the encoding process according to the encoded concatenated map of the next scale of each raster image until the encoded feature maps of each raster image output by the last encoding unit are obtained.

[0058] In this embodiment, the segmentation device concatenates the encoded feature maps of each raster image output by the next encoding unit with the box hint pooling feature map of the next scale of the same raster image to obtain the encoded concatenated map of the next scale of each raster image; repeats the encoding process according to the encoded concatenated map of the next scale of each raster image until the encoded feature maps of each raster image output by the last encoding unit are obtained.

[0059] S5: Input several of the encoded feature maps of the same raster image and the box hint pooling feature maps of several scales into the decoding module to obtain several decoded feature maps of each raster image.

[0060] The decoding module is used to gradually restore the spatial resolution of the image and further fuse and extract features. In this embodiment, the segmentation device inputs several of the encoded feature maps of the same raster image and the box hint pooling feature maps of several scales into the decoding module to obtain several decoded feature maps of each raster image, and introduces the box hint pooling feature maps of the corresponding scale for interactive information processing during the decoding process, improving the accuracy of feature decoding.

[0061] The decoding module includes several stacked decoding units; please refer to Figure 6 , Figure 6 is a schematic flowchart of S5 in the method for segmenting cancer cells based on digital pathology images provided by an embodiment of the present application, including steps S51 - S53, specifically as follows: S51: Use the encoded feature map of the raster image as the input data of the first decoding unit of the decoding module for decoding processing to obtain several decoded feature maps of the raster image output by the first decoding unit.

[0062] In this embodiment, the segmentation device uses the encoded feature map of the raster image as the input data of the first decoding unit of the decoding module for decoding processing to obtain several decoded feature maps of the raster image output by the first decoding unit.

[0063] The decoding unit includes a transposed convolution unit, a small - field - of - view decoding subunit, a large - field - of - view decoding subunit, and an attention extraction subunit; please refer to Figure 7 , Figure 7 is a schematic flowchart of S51 in the method for segmenting cancer cells based on digital pathology images provided by an embodiment of the present application, including steps S511 - S513, specifically as follows: S511: Input the encoded feature map into the transposed convolution unit for transposed convolution processing to obtain several transposed convolution feature maps of the raster image.

[0064] In this embodiment, the segmentation device inputs the encoded feature map into the transposed convolution unit for transposed convolution processing to obtain several transposed convolution feature maps of the raster image, where the transposed convolution unit uses a transposed convolution layer with a 2*2 convolution kernel and is used to restore the encoded feature map to the original input size.

[0065] S512: Input the deconvolution feature map into the small-field-of-view decoding subunit, and successively process it through the third convolutional layer, the third normalization layer, and the third activation function layer to obtain the small-field-of-view decoding feature maps of several grid images; input the deconvolution feature map into the large-field-of-view decoding subunit, and successively process it through the fourth convolutional layer, the fourth normalization layer, and the fourth activation function layer to obtain the large-field-of-view decoding feature maps of several grid images.

[0066] In this embodiment, the segmentation device inputs the deconvolution feature map into the small-field-of-view decoding subunit, and successively processes it through the third convolutional layer, the third normalization layer, and the third activation function layer to obtain the small-field-of-view decoding feature maps of several grid images, where the convolutional kernel of the third convolutional layer is the same as that of the first convolutional layer.

[0067] The segmentation device inputs the deconvolution feature map into the large-field-of-view decoding subunit, and successively processes it through the fourth convolutional layer, the fourth normalization layer, and the fourth activation function layer to obtain the large-field-of-view decoding feature maps of several grid images, where the convolutional kernel of the fourth convolutional layer is the same as that of the second convolutional layer.

[0068] S513: Cross-join and segment the small-field-of-view decoding feature map and the large-field-of-view decoding feature map of the same grid image along the channel dimension to obtain the third segmentation feature maps and the fourth segmentation feature maps of several grid images, input the third segmentation feature maps into the attention extraction subunit for attention extraction to obtain the second attention feature maps of several grid images, and perform feature fusion on the second attention feature map and the fourth segmentation feature map of the same grid image to obtain the decoding feature maps of several grid images.

[0069] In this embodiment, the segmentation device cross-joins and segments the small-field-of-view decoding feature map and the large-field-of-view decoding feature map of the same grid image along the channel dimension to obtain the third segmentation feature maps and the fourth segmentation feature maps of several grid images, inputs the third segmentation feature maps into the attention extraction subunit for attention extraction to obtain the second attention feature maps of several grid images, and performs feature fusion on the second attention feature map and the fourth segmentation feature map of the same grid image to obtain the decoding feature maps of several grid images. For specific embodiments, reference can be made to step S412, which will not be elaborated here.

[0070] S52: Concatenate the box hint pooling feature map of the last scale of the same raster image with the decoded feature map of the raster image output by the first decoding unit to obtain the decoded concatenated map of the last scale, and use the decoded concatenated map of the last scale as the input data of the next decoding unit for decoding processing to obtain a plurality of decoded feature maps of the raster image output by the next decoding unit.

[0071] In this embodiment, the segmentation device concatenates the box hint pooling feature map of the last scale of the same raster image with the decoded feature map of the raster image output by the first decoding unit to obtain the decoded concatenated map of the last scale, and uses the decoded concatenated map of the last scale as the input data of the next decoding unit for decoding processing to obtain a plurality of decoded feature maps of the raster image output by the next decoding unit.

[0072] S53: Concatenate the box hint pooling feature map of the previous scale of the same raster image with the decoded feature map of the raster image output by the next decoding unit to obtain the decoded concatenated map of the previous scale, and repeatedly perform decoding processing until a plurality of decoded feature maps of the raster image output by the last decoding unit are obtained.

[0073] In this embodiment, the segmentation device concatenates the box hint pooling feature map of the previous scale of the same raster image with the decoded feature map of the raster image output by the next decoding unit to obtain the decoded concatenated map of the previous scale, and repeatedly performs decoding processing until a plurality of decoded feature maps of the raster image output by the last decoding unit are obtained.

[0074] S6: Input the plurality of decoded feature maps of each raster image into the detection module for probability prediction to obtain the cancer cell probability prediction data of each raster image.

[0075] In this embodiment, the segmentation device inputs the decoded feature maps of the plurality of raster images into the detection module for probability prediction through the Sigmoid activation function processing to obtain the cancer cell probability prediction data of the plurality of raster images, where the cancer cell probability prediction data includes the cancer cell probability prediction vectors of a plurality of dimensions of a plurality of pixels of the raster image.

[0076] S7: According to the cancer cell probability prediction data of each raster image, perform segmentation processing on the digital pathology image to be segmented to obtain the cancer cell segmentation result of the digital pathology image to be segmented.

[0077] In this embodiment, the segmentation device predicts the cancer cell probability prediction vectors of several dimensions of the same pixel according to the cancer cell probability prediction data of several grid images, and extracts the dimension of the maximum cancer cell probability prediction vector as the recognition result of the pixel, where the recognition result is used to indicate that the pixel is a diseased pixel, that is, a cancer cell pixel, and performs segmentation processing on the digital pathology image to be segmented to obtain the cancer cell segmentation result of the digital pathology image to be segmented.

[0078] Please refer to Figure 8 , Figure 8 FIG. 7 is a schematic structural diagram of a cancer cell segmentation device based on a digital pathology image provided by an embodiment of the present application. The device can implement all or part of the cancer cell segmentation device based on a digital pathology image through software, hardware, or a combination of both. The device 8 includes: A data acquisition module 81, configured to acquire a digital pathology image to be segmented and a cancer cell segmentation model, where the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module; A feature construction module 82, configured to divide the digital pathology image to be segmented into several grid images, input the several grid images into the feature extraction module for information feature extraction and frame hint mask map construction, and obtain the information feature map and frame hint mask map of each grid image; A feature pooling module 83, configured to perform scale scaling and pooling processing on the frame hint mask maps of each grid image to obtain several scale frame hint pooled feature maps corresponding to each grid image; A feature encoding module 84, configured to input the information feature map of the same grid image and the several scale frame hint pooled feature maps into the encoding module to obtain several encoded feature maps of each grid image; A feature decoding module 85, configured to input the several encoded feature maps of the same grid image and the several scale frame hint pooled feature maps into the decoding module to obtain several decoded feature maps of each grid image; A probability prediction module 86, configured to input the several decoded feature maps of each grid image into the detection module for probability prediction to obtain the cancer cell probability prediction data of each grid image; A cancer cell segmentation module 87, configured to perform segmentation processing on the digital pathology image to be segmented according to the cancer cell probability prediction data of each grid image to obtain the cancer cell segmentation result of the digital pathology image to be segmented.

[0079] In an embodiment of the present application, through a data acquisition module, a digital pathological image to be segmented and a cancer cell segmentation model are acquired. The cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module. Through a feature construction module, the digital pathological image to be segmented is divided into a plurality of grid images, and the plurality of grid images are input into the feature extraction module for information feature extraction and frame hint mask map construction, so as to obtain an information feature map and a frame hint mask map of each grid image. Through a feature pooling module, the frame hint mask maps of each grid image are subjected to scale scaling and pooling processing to obtain a plurality of scale frame hint pooled feature maps corresponding to each grid image. Through a feature encoding module, the information feature map and the plurality of scale frame hint pooled feature maps of the same grid image are input into the encoding module to obtain a plurality of encoded feature maps of each grid image. Through a feature decoding module, the plurality of encoded feature maps and the plurality of scale frame hint pooled feature maps of the same grid image are input into the decoding module to obtain a plurality of decoded feature maps of each grid image. Through a probability prediction module, the plurality of decoded feature maps of each grid image are input into the detection module for probability prediction to obtain cancer cell probability prediction data of each grid image. Through a cancer cell segmentation module, according to the cancer cell probability prediction data of each grid image, the digital pathological image to be segmented is segmented to obtain a cancer cell segmentation result of the digital pathological image to be segmented. By extracting the feature information and frame hint information in the digital pathological image to be segmented, performing encoding processing and decoding processing, and performing cancer cell segmentation based on the obtained decoded feature map, the accuracy and efficiency of cancer cell segmentation are improved.

[0080] Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of a computer device provided by an embodiment of the present application. The computer device 9 includes: a processor 91, a memory 92, and a computer program 93 stored on the memory 92 and executable on the processor 91. The computer device can store multiple instructions, and the instructions are suitable for being loaded and executed by the processor 91 to perform the method steps of the above Figures 1 to 7 shown embodiment. The specific execution process can refer to the specific description of the Figures 1 to 7 shown embodiment and will not be elaborated here.

[0081] Among them, the processor 91 may include one or more processing cores. The processor 91 is connected to various parts within the server using various interfaces and lines. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 92, and by invoking the data stored in the memory 92, it performs various functions of the cancer cell segmentation device 8 based on digital pathology images and processes data. Optionally, the processor 91 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 91 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the processor 91 and may be implemented separately by a single chip.

[0082] Among them, the memory 92 may include a random access memory (RAM) and may also include a read-only memory. Optionally, the memory 92 includes a non-transitory computer-readable storage medium. The memory 92 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 92 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above method embodiments, etc.; the data storage area may store the data involved in the above method embodiments. Optionally, the memory 92 may also be at least one storage device located far from the aforementioned processor 91.

[0083] The embodiments of the present application also provide a storage medium. The storage medium may store multiple instructions, and the instructions are suitable for being loaded and executed by a processor to perform the Figures 1 to 7 method steps of the embodiments shown above. The specific execution process can be referred to the Figures 1 to 7 specific description of the embodiments shown above, and details will not be elaborated here.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments and will not be elaborated herein.

[0085] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0086] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0087] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, in each embodiment of the present invention, each functional unit may be integrated in a processing unit, may exist separately physically for each unit, or two or more units may be integrated in one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0090] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, all or part of the processes in the above-described embodiment methods of the present invention may also be completed by a computer program instructing related hardware. The computer program may be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments may be implemented. Among them, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, an executable file, or some intermediate form, etc.

[0091] The present invention is not limited to the above-described embodiments. If various modifications or deformations of the present invention do not depart from the spirit and scope of the present invention, and if these modifications and deformations fall within the scope of the claims of the present invention and equivalent technical scope, then the present invention also intends to include these modifications and deformations.

Claims

1. A method for segmenting cancer cells based on digital pathological images, characterized in that, It includes the following steps: Obtain the digital pathological image to be segmented and the cancer cell segmentation model, where the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module; Divide the digital pathological image to be segmented into a plurality of grid images, and input the plurality of grid images into the feature extraction module for information feature extraction and box prompt mask map construction to obtain the information feature maps and box prompt mask maps of each grid image; Perform scale scaling and pooling processing on the box prompt mask maps of each grid image to obtain a plurality of scale box prompt pooled feature maps corresponding to each grid image; Input the information feature map of the same grid image and the box prompt pooled feature maps of a plurality of scales into the encoding module to obtain a plurality of encoded feature maps of each grid image; Input the plurality of encoded feature maps of the same grid image and the box prompt pooled feature maps of a plurality of scales into the decoding module to obtain a plurality of decoded feature maps of each grid image; Input the plurality of decoded feature maps of each grid image into the detection module for probability prediction to obtain the cancer cell probability prediction data of each grid image; According to the cancer cell probability prediction data of each grid image, perform segmentation processing on the digital pathological image to be segmented to obtain the cancer cell segmentation result of the digital pathological image to be segmented.

2. The method for segmenting cancer cells based on digital pathological images according to claim 1, wherein The step of dividing the digital pathological image to be segmented into a plurality of grid images, inputting the plurality of grid images into the feature extraction module for information feature extraction and box prompt mask map construction to obtain the information feature maps and box prompt mask maps of each grid image includes the steps: Perform edge detection on each grid image to obtain the Laplacian map of each grid image; perform clustering operation and median filtering processing on each grid image to obtain the median filtered map of each grid image; perform cross splicing on the Laplacian map and the median filtered map of the same grid image to obtain the information feature map of each grid image; Obtain the initial detection boxes of the plurality of grid images, solve the minimum bounding rectangles of the initial detection boxes to obtain the box prompt information of the plurality of grid images; construct box prompt mask maps according to the plurality of grid images and the corresponding box prompt information to obtain the box prompt mask maps of the plurality of grid images.

3. The method for segmenting cancer cells based on digital pathological images according to claim 2, wherein: The encoding module includes a plurality of stacked encoding units; The step of inputting the information feature map of the same grid image and the box prompt pooled feature maps of a plurality of scales into the encoding module to obtain a plurality of encoded feature maps of each grid image includes the steps: Use the information feature map of the grid image as the input data of the first encoding unit of the encoding module for encoding processing to obtain the encoded feature maps of the plurality of grid images output by the first encoding unit; The encoded feature map output by the first encoding unit is concatenated with the box hint pooling feature map of the first scale of the same raster image to obtain the encoded concatenated map of the first scale of each raster image; the encoded concatenated map of the first scale is used as the input data of the next encoding unit for encoding processing to obtain the encoded feature maps of each raster image output by the next encoding unit; The encoded feature maps of each raster image output by the next encoding unit are concatenated with the box hint pooling feature map of the next scale of the same raster image to obtain the encoded concatenated map of the next scale of each raster image; the encoding processing is repeatedly executed according to the encoded concatenated maps of the next scale of each raster image until the encoded feature maps of each raster image output by the last encoding unit are obtained.

4. The method for segmenting cancer cells based on digital pathological images according to claim 3, wherein: The encoding unit includes a small receptive field encoding subunit, a large receptive field encoding subunit, and an attention extraction subunit; Encoding the information feature map of the raster image as the input data of the first encoding unit of the encoding module to obtain the encoded feature maps of several raster images output by the first encoding unit, including the steps of: Inputting the information feature map into the small receptive field encoding subunit, and sequentially processing it through a first convolutional layer, a first normalization layer, a first activation function layer, and a max pooling layer to obtain the small receptive field encoded feature maps of several raster images; Inputting the information feature map into the large receptive field encoding subunit, and sequentially processing it through a second convolutional layer, a second normalization layer, a second activation function layer, and an average pooling layer to obtain the large receptive field encoded feature maps of several raster images; Cross-concatenating and segmenting the small receptive field encoded feature map and the large receptive field encoded feature map of the same raster image along the channel dimension to obtain the first segmented feature map and the second segmented feature map of several raster images; inputting the first segmented feature map into the attention extraction subunit for attention extraction to obtain the first attention feature maps of several raster images, and fusing the first attention feature map and the second segmented feature map of the same raster image to obtain the encoded feature maps of several raster images.

5. The method for segmenting cancer cells based on digital pathological images according to claim 4, wherein: The attention extraction subunit includes a global attention extraction block and a local attention extraction block; the global attention extraction block includes a global average pooling layer and a first multi-head self-attention network; the local attention extraction block includes a spatial pyramid pooling network and a second multi-head self-attention network; Inputting the first segmented feature map into the attention extraction subunit for attention extraction to obtain the first attention feature maps of several raster images, including the steps of: Inputting the first segmented feature map into the global average pooling layer for pooling processing to obtain a global average pooling feature map; inputting the global average pooling feature map into the first multi-head self-attention network for attention feature extraction to obtain a first sub-attention feature map; Input the first segmentation feature map into the spatial pyramid pooling network for multi-scale pooling feature extraction, sharpening operation and splicing process to obtain a local pooling feature map; input the local pooling feature map into the second multi-head self-attention network for attention feature extraction to obtain a second sub-attention feature map; Perform spatial dimension reconstruction on the first sub-attention feature map and the second sub-attention feature map to obtain a spatial dimension reconstruction feature map as the first attention feature map.

6. The method for segmenting cancer cells based on digital pathological images according to claim 5, wherein: The decoding module includes a plurality of stacked decoding units; Inputting the encoding feature maps of several of the same raster images and the box hint pooling feature maps of several scales into the decoding module to obtain the decoding feature maps of several raster images respectively, including the steps of: Taking the encoding feature map of the raster image as the input data of the first decoding unit of the decoding module for decoding processing to obtain the decoding feature maps of several raster images output by the first decoding unit; Splice the box hint pooling feature map of the last scale of the same raster image with the decoding feature map of the raster image output by the first decoding unit to obtain a decoding splicing map of the last scale, and take the decoding splicing map of the last scale as the input data of the next decoding unit for decoding processing to obtain the decoding feature maps of several raster images output by the next decoding unit; Splice the box hint pooling feature map of the previous scale of the same raster image with the decoding feature map of the raster image output by the next decoding unit to obtain a decoding splicing map of the previous scale, and repeat the decoding process until the decoding feature maps of several raster images output by the last decoding unit are obtained.

7. The method for segmenting cancer cells based on digital pathological images according to claim 6, wherein: The decoding unit includes a transposed convolution unit, a small field-of-view decoding sub-unit, a large field-of-view decoding sub-unit and an attention extraction sub-unit; Taking the encoding feature map of the raster image as the input data of the first decoding unit of the decoding module for decoding processing to obtain the decoding feature maps of several raster images output by the first decoding unit, including the steps of: Input the encoding feature map into the transposed convolution unit for transposed convolution processing to obtain the transposed convolution feature maps of several raster images; Input the transposed convolution feature map into the small field-of-view decoding sub-unit, and process it through the third convolutional layer, the third normalization layer and the third activation function layer in sequence to obtain the small field-of-view decoding feature maps of several raster images; Input the transposed convolution feature map into the large field-of-view decoding sub-unit, and process it through the fourth convolutional layer, the fourth normalization layer and the fourth activation function layer in sequence to obtain the large field-of-view decoding feature maps of several raster images; Cross - splice and segment the small - field decoded feature maps and large - field decoded feature maps of the same raster image along the channel dimension to obtain the third segmented feature maps and fourth segmented feature maps of several raster images. Input the third segmented feature maps into the attention extraction sub - unit for attention extraction to obtain the second attention feature maps of several raster images. Feature - fuse the second attention feature maps and the fourth segmented feature maps of the same raster image to obtain the decoded feature maps of several raster images.

8. A cancer cell segmentation device based on digital pathological images, characterized in that, Comprising: A data acquisition module, configured to acquire a digital pathology image to be segmented and a cancer cell segmentation model, wherein the cancer cell segmentation model includes a feature extraction module, an encoding module, a decoding module, and a detection module; A feature construction module, configured to divide the digital pathology image to be segmented into several raster images, and input the several raster images into the feature extraction module for information feature extraction and frame prompt mask map construction to obtain the information feature maps and frame prompt mask maps of each raster image; A feature pooling module, configured to perform scale scaling and pooling processing on the frame prompt mask maps of each raster image to obtain several scale - level frame prompt pooled feature maps corresponding to each raster image; A feature encoding module, configured to input the information feature map and several scale - level frame prompt pooled feature maps of the same raster image into the encoding module to obtain several encoded feature maps of each raster image; A feature decoding module, configured to input the several encoded feature maps and several scale - level frame prompt pooled feature maps of the same raster image into the decoding module to obtain several decoded feature maps of each raster image; A probability prediction module, configured to input the several decoded feature maps of each raster image into the detection module for probability prediction to obtain the cancer cell probability prediction data of each raster image; A cancer cell segmentation module, configured to perform segmentation processing on the digital pathology image to be segmented according to the cancer cell probability prediction data of each raster image to obtain the cancer cell segmentation result of the digital pathology image to be segmented.

9. A computer device, characterized in that, Comprising: A processor, a memory, and a computer program stored on the memory and executable on the processor; when the computer program is executed by the processor, it implements the steps of the cancer cell segmentation method based on digital pathology images according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the cancer cell segmentation method based on digital pathology images according to any one of claims 1 to 7.

Images