A cell nucleus segmentation method based on context feature fusion and related equipment

Abstract

The present invention discloses a cell nucleus segmentation method based on context feature fusion and related equipment. The method comprises: obtaining a medical cell image, inputting the medical cell image into a convolutional neural network for stacking to obtain multiple feature maps, and inputting each feature map into a feature pyramid. The context enhancement module in the feature pyramid performs feature fusion on the feature map to obtain output features; inputting the output features into a region extraction network to obtain a candidate box, and the region of interest alignment layer obtains the region of interest features based on the candidate box; sending the region of interest features to a feature adaptive selection module to obtain a combined feature representation at the feature level; and calculating the category classification, candidate box coordinates, and edge-aware mask of the region of interest based on the combined feature representation, thereby achieving cell nucleus segmentation. The present invention processes medical cell images through the context enhancement module and the feature adaptive selection module, significantly improving the segmentation accuracy of cell nuclei in pathological sections.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a cell nucleus segmentation method, system, terminal and computer-readable storage medium based on context feature fusion. Background Art

[0002] Medical image segmentation is an essential component of medical image processing and analysis and a key step in computer-assisted diagnosis (CAD). Its purpose is to more clearly define anatomical or pathological structural changes in medical images. It plays a crucial role in CAD and smart healthcare, significantly improving diagnostic efficiency and accuracy. Common medical image segmentation tasks include liver and liver tumor segmentation, brain and brain tumor segmentation, cell segmentation, lung segmentation, and lung nodule segmentation. Nucleus segmentation is primarily used in cancer diagnosis. Nucleus segmentation is a critical step in tissue and cell image analysis. In digital microscopic tissue images, nucleus segmentation can extract high-quality features for nuclear morphology detection and other computational pathology analyses. Inaccurate segmentation results can significantly impact subsequent analysis steps. Accurate nucleus segmentation in tissue images from diverse patients, organs, and disease states can also contribute significantly to the development of clinical and medical research software. Accurate segmentation can yield detailed morphological and appearance features, such as the nucleus-to-cytoplasm ratio, average nuclear size, and polygonal shape. Based on the results of cell nucleus segmentation, different types of cell nuclei can be identified, and information about gland shape can be obtained. This information can not only be used to assess cancer grade, but also to predict treatment effects and help develop anti-cancer drugs.

[0003] However, medical cell images often suffer from blurred cell edges and multiple cells stacked on top of each other. Existing techniques segment the entire image pixel by pixel, then perform clustering to identify individual cell instances. This approach often fails to achieve satisfactory segmentation performance when multiple cells are stacked and occlude each other. However, recent studies have shown that the feature pyramid network architecture still has some issues. First, it simply stacks different feature layers to expand the receptive field without encouraging information propagation. Multi-level feature maps are merged solely through a top-down, pixel-by-pixel addition process, ignoring the semantic gap between feature maps at different levels. Furthermore, during the merging process, lower-level features can gain stronger semantic features by fusing higher-level features, while higher-level features suffer information loss when the number of channels is increased through 1×1 convolution. Therefore, preventing information loss in the highest-level feature maps and suboptimal feature fusion strategies within the feature pyramid network architecture to improve the accuracy of cell nucleus segmentation in pathology slides is a pressing issue in the current landscape.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a cell nucleus segmentation method and related equipment based on contextual feature fusion, aiming to solve the problem in the prior art that the feature pyramid network structure causes information loss in the highest level feature mapping and suboptimal feature fusion strategy, resulting in low accuracy of cell nucleus segmentation in pathological sections.

[0006] To achieve the above object, the present invention provides a cell nucleus segmentation method based on context feature fusion, the cell nucleus segmentation method based on context feature fusion comprising the following steps:

[0007] Acquire a medical cell image, input the medical cell image into a convolutional neural network for stacking to obtain multiple feature maps, and input each of the feature maps into a feature pyramid. A context enhancement module in the feature pyramid performs feature fusion on each of the feature maps to obtain output features;

[0008] The output features are input into a region extraction network, the region extraction network obtains a candidate box according to the output features, and the region of interest alignment layer obtains a region of interest feature according to the candidate box;

[0009] Sending the region of interest features to a feature adaptive selection module, wherein the feature adaptive selection module obtains a combined feature representation of all feature levels in the region of interest based on the region of interest features;

[0010] Based on the combined feature representation, the category classification of the region of interest, the coordinates of the candidate box and the edge-aware mask are calculated respectively, and the cell nucleus of the pathological image is segmented based on the category classification, the coordinates and the edge-aware mask.

[0011] Optionally, the cell nucleus segmentation method based on contextual feature fusion, wherein the medical cell image is acquired, the medical cell image is input into a convolutional neural network for stacking to obtain multiple feature maps, each of the feature maps is input into a feature pyramid, and the context enhancement module in the feature pyramid performs feature fusion on each of the feature maps to obtain output features, specifically includes:

[0012] Acquiring the medical cell image, inputting the medical cell image into the convolutional neural network for stacking, changing the resolution of the medical cell image according to downsampling, and obtaining feature maps of different stages according to the different resolutions;

[0013] Obtain multiple dilated convolutions with different dilation rates, combine the dilated convolutions with different dilation rates in a dense connection manner to obtain a feature pyramid, and input the feature map into the feature pyramid;

[0014] In the feature pyramid, the coarse-grained information of the feature map is enhanced according to the context enhancement module, and the enhanced coarse-grained information is pixel-wise added and fused with the original coarse-grained information to obtain the output feature.

[0015] Optionally, the cell nucleus segmentation method based on context feature fusion, wherein the step of inputting the feature map into the feature pyramid further includes:

[0016] The output signal obtained by performing a dilated convolution operation on the input signal of the feature map is:

[0017]

[0018] Where y[i] is the output signal obtained after the dilated convolution operation, i is the position subscript of a pixel in the feature map, d is the dilation rate, w[k] is the kth parameter of the convolution kernel, and when d = 1, the dilated convolution is a normal convolution.

[0019] The output of the dense connection is: l =H l ([x1,x2,…,x l ]);

[0020] Among them, l is the number of layers, x l is the output of the lth layer, H l is the nonlinear transformation of the lth layer.

[0021] Optionally, in the cell nucleus segmentation method based on context feature fusion, the calculation formula of the dilated convolution is:

[0022] Among them, y i is the output feature map of the i-th layer in the previous stage, H is the height of the input feature map, d i is the expansion rate of the i-th layer, K is the size of the hole convolution layer, [y i ,y i-1 ,…,y1] is the feature map generated by concatenating all previous feature layers, and y1 represents the output feature map of the first layer in the previous stage.

[0023] Optionally, the cell nucleus segmentation method based on contextual feature fusion, wherein the step of obtaining multiple dilated convolutions with different dilation rates, combining the dilated convolutions with different dilation rates in a dense connection manner to obtain a dense pyramid, and inputting the feature map into the dense pyramid, further comprises:

[0024] Processing the global average pooling layer to obtain a global feature vector, upsampling the global feature vector to obtain a target global feature vector, and concatenating the target global feature vector with the output of the dilated convolutional layer to obtain a feature map;

[0025] The feature map is input into a 1×1 convolutional layer, and the coarse-grained and fine-grained features of the feature map are fused.

[0026] Optionally, the cell nucleus segmentation method based on context feature fusion, wherein the context-based enhancement module enhances the coarse-grained information of the feature map, and performs pixel addition and fusion of the enhanced coarse-grained information and the original coarse-grained information to obtain output features, further comprises:

[0027] Get the input feature map F, according to the convolution layer W q and convolutional layer W k Convert the feature maps of the input feature map to the latent feature space to obtain feature maps Q and K';

[0028] The feature map Q and the feature map K' are reorganized in terms of feature dimensions to obtain a target feature map, the target feature map is processed to obtain a relationship matrix, and the relationship matrix is normalized to obtain an attention matrix R.

[0029] Optionally, in the cell nucleus segmentation method based on context feature fusion, the calculation formula of the feature map Q is:

[0030] Among them, Q is the convolution layer W q The feature map obtained later, T is the matrix transpose operation;

[0031] The calculation formula of the characteristic map K' is:

[0032] Among them, K' is the convolution layer W k The feature map obtained later;

[0033] The calculation formula of the attention matrix R is: R = Q T K'.

[0034] Optionally, the cell nucleus segmentation method based on context feature fusion, wherein the normalization processing of the relationship matrix is performed to obtain the attention matrix R, further comprising:

[0035] Get the convolutional layer W v , based on the convolutional layer W v Processing the input feature map to obtain a new feature map V, and multiplying the attention matrix R by the feature map V pixel by pixel to obtain an attention-enhanced feature map E;

[0036] The expression of the characteristic graph V is:

[0037] Among them, F is the input feature map;

[0038] The expression of the characteristic graph E is: i =R⊙V i ;

[0039] Among them, ⊙ is the element-by-element multiplication operation, E i is the i-th feature map.

[0040] Optionally, the cell nucleus segmentation method based on context feature fusion, wherein the output features are input into a region extraction network, the region extraction network obtains a candidate box according to the output features, and the region of interest alignment layer obtains a region of interest feature according to the candidate box, specifically includes:

[0041] Performing feature mapping on each feature layer of the feature pyramid according to the region of interest alignment layer to obtain a regional feature pyramid of the region of interest, inputting the regional feature pyramid into a feature fusion module, and splicing the regional feature maps of the regional feature pyramid according to channels based on the feature fusion module to obtain an output feature map;

[0042] Obtaining a channel of the output feature map, encoding the spatial features of the channel according to global average pooling to obtain a global feature, compressing the dimension of the global feature, and expanding the compressed dimension to obtain an attention weight coefficient of the channel;

[0043] The attention weight coefficient is split into four weight vectors, the weight vectors are multiplied by the features of the regional feature pyramid layer by layer and pixel by pixel to obtain a weighted regional feature pyramid, and the features in the weighted regional feature pyramid are added pixel by pixel to obtain the region of interest features.

[0044] Optionally, in the cell nucleus segmentation method based on context feature fusion, the calculation formula of the output feature map is:

[0045]

[0046] Among them, q[i] is the output feature map, p[i] is the input feature map, ⊙ is the element-by-element multiplication operation, i is the position subscript of a pixel in the feature map, and W is the width of the input feature map;

[0047] Among them, the The calculation process is:

[0048] u = Conv(X);

[0049]

[0050] Among them, X is the input feature, Conv is the convolution layer operation, u is the output after the convolution operation, c is a channel of the feature map, i and j are the position subscripts of a pixel in the feature map, H is the height of the input feature map, z c for The output after the operation.

[0051] In addition, to achieve the above-mentioned purpose, the present invention further provides a cell nucleus segmentation system based on context feature fusion, wherein the cell nucleus segmentation system based on context feature fusion includes:

[0052] a data input module, configured to acquire medical cell images, input the medical cell images into a convolutional neural network for stacking to obtain a plurality of feature maps, and input each of the feature maps into a feature pyramid;

[0053] A context enhancement module, configured to perform feature fusion on each of the feature maps to obtain output features;

[0054] A candidate box acquisition module is used to input the output features into a region extraction network, and the region extraction network obtains a candidate box according to the output features;

[0055] The candidate frame processing module is used for the ROI alignment layer to obtain ROI features based on the candidate frame and send the ROI features to the feature adaptive selection module;

[0056] A feature adaptive selection module, configured to obtain a combined feature representation of all feature levels in the region of interest based on the features of the region of interest;

[0057] A cell nucleus segmentation module is used to calculate the category classification of the region of interest, the coordinates of the candidate box and the edge-aware mask based on the combined feature representation, and to segment the cell nuclei of the pathological image based on the category classification, the coordinates and the edge-aware mask.

[0058] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a cell nucleus segmentation program based on context feature fusion stored on the memory and runnable on the processor, and when the cell nucleus segmentation program based on context feature fusion is executed by the processor, the steps of the cell nucleus segmentation method based on context feature fusion as described above are implemented.

[0059] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a cell nucleus segmentation program based on context feature fusion, and when the cell nucleus segmentation program based on context feature fusion is executed by a processor, the steps of the cell nucleus segmentation method based on context feature fusion as described above are implemented.

[0060] In the present invention, a medical cell image is obtained, and the medical cell image is input into a convolutional neural network for stacking to obtain multiple feature maps, and each feature map is input into a feature pyramid. The context enhancement module in the feature pyramid performs feature fusion on each feature map to obtain output features; the output features are input into a region extraction network, and the region extraction network obtains a candidate box based on the output features. The region of interest alignment layer obtains region of interest features based on the candidate box; the region of interest features are sent to a feature adaptive selection module, and the feature adaptive selection module obtains a combined feature representation of all feature levels in the region of interest based on the region of interest features; based on the combined feature representation, the category classification of the region of interest, the coordinates of the candidate box and the edge-aware mask are respectively calculated, and the cell nucleus of the pathological image is segmented based on the category classification, the coordinates and the edge-aware mask. The present invention focuses on mining a wide range of contextual information in a context enhancement module based on an attention mechanism, capturing important semantics to reduce information loss in the highest pyramid-level feature map. The features are then sent to a feature adaptive selection module to adaptively obtain a combined feature representation at all feature levels of each region of interest. Finally, the classification of the region of interest category, the coordinates of the candidate box, and the edge-aware mask are calculated based on these feature representations, thereby realizing the segmentation of medical pathology slice images. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flow chart of a preferred embodiment of the cell nucleus segmentation method based on context feature fusion in the present invention;

[0062] Figure 2 Schematic diagram of ordinary convolution in a preferred embodiment of the cell nucleus segmentation method based on context feature fusion in the present invention;

[0063] Figure 3Schematic diagram of dilated convolution in a preferred embodiment of the cell nucleus segmentation method based on context feature fusion in the present invention;

[0064] Figure 4 is a schematic diagram of a context information enhancement module of the cell nucleus segmentation method based on context feature fusion in the present invention;

[0065] Figure 5 is a schematic diagram of an adaptive feature fusion module of the cell nucleus segmentation method based on context feature fusion in the present invention;

[0066] Figure 6 1 is a schematic diagram of the overall process of a preferred embodiment of the cell nucleus segmentation method based on context feature fusion in the present invention;

[0067] Figure 7 Schematic diagram of the principle of a preferred embodiment of a cell nucleus segmentation system based on context feature fusion in the present invention;

[0068] Figure 8 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0069] In order to make the objectives, technical solutions and advantages of the present invention more clear and explicit, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0070] The cell nucleus segmentation method based on context feature fusion described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the cell nucleus segmentation method based on context feature fusion includes the following steps:

[0071] Step S10: Acquire a medical cell image, input the medical cell image into a convolutional neural network for stacking to obtain multiple feature maps, and input each feature map into a feature pyramid. The context enhancement module in the feature pyramid performs feature fusion on each feature map to obtain output features.

[0072] The step S10 includes:

[0073] Step S11: acquiring the medical cell image, inputting the medical cell image into the convolutional neural network for stacking, changing the resolution of the medical cell image according to downsampling, and obtaining feature maps of different stages according to the different resolutions;

[0074] Step S12: obtaining multiple dilated convolutions with different dilation rates, combining the dilated convolutions with different dilation rates in a dense connection manner to obtain a feature pyramid, and inputting the feature map into the feature pyramid;

[0075] Step S13: In the feature pyramid, the coarse-grained information of the feature map is enhanced according to the context enhancement module, and the enhanced coarse-grained information is pixel-wise added and fused with the original coarse-grained information to obtain the output feature.

[0076] Specifically, the medical cell image is obtained and input into a convolutional neural network for stacking. During the stacking process, the convolutional neural network uses downsampling to change the resolution. The convolutional neural network is divided into different stages according to the different resolutions, and the feature maps obtained in different stages are input into the feature pyramid for feature fusion. During the feature fusion process of the feature pyramid, the features of the feature map will be propagated in a top-down manner, and the bottom features can be improved by the stronger semantic information from the high-level features. However, the features of the highest pyramid level lose semantic information due to the reduction in the number of channels caused by the use of 1×1 convolution for dimensionality reduction, and there is no other higher-level information to supplement the information. Therefore, the present invention proposes a context enhancement module to supplement the information loss of the highest-level feature map.

[0077] In order to solve the information loss problem faced by the highest-level feature map in the feature pyramid structure, the context enhancement module is used to obtain the context information integrated on the receptive fields of different sizes. Since the resolution of the highest-level feature map itself is already very low, most of the information will be lost if downsampling and then upsampling are performed. Therefore, in order to obtain context information of different receptive fields while maintaining the resolution of the feature map (receptive field: the size of the area where the pixels on the feature map output by each layer of the convolutional neural network are mapped back to the input image, that is, the size of a point on the feature map relative to the original image, and also the area of the input image that the convolutional neural network feature can see), the present invention adopts hollow convolution layers with different expansion rates as basic modules. Figure 2 In the middle is ordinary convolution, Figure 3 The middle one is the dilated convolution, Figure 2 and Figure 3 This shows the difference between ordinary convolution and dilated convolution. Figure 3 The figure shows a dilated convolution with a dilation rate of 2. It is also a 3×3 convolution. The receptive field of the dilated convolution is expanded from 3×3 to 5×5. In the one-dimensional case, applying a dilated convolution operation to the input x can be expressed as: Where y[i] is the output signal after the dilated convolution operation, i is the position index of a pixel in the feature map, d is the dilation rate, and w[k] is the kth parameter of the convolution kernel. When d=1, the dilated convolution is the same as the normal convolution operation. The dilated convolution does not fill some blank pixels between pixels, but skips some pixels on the existing pixels, or inserts d-1 zero weights in the middle of the convolution kernel parameters while keeping the input unchanged. Figure 3In fact, a 3×3 convolution with d=2 is equivalent to a 5×5 convolution, where only 9 positions have non-zero weights, and the rest are all 0. Figure 2 and Figure 3 The two layers are stacked to obtain a 7×7 receptive field; therefore, a large expansion rate means a large receptive field. The actual size R' of the dilated convolution is: R' = K + (K-1) × (d-1), where K is the size of the dilated convolution. Stacking two convolutional layers can obtain a larger receptive field. For example, if there are two convolutional layers with sizes K1 and K2 respectively, then the receptive field K0 obtained by stacking the two convolutional layers is K0 = K1 + K2-1. However, there are also certain problems with dilated convolution layers. When the receptive field is large, only a few pixels are sampled and calculated for the dilated convolution with a large expansion rate. Although a larger receptive field is obtained in this process, a large amount of information is lost in the calculation process. At the same time, only using the information of dilated convolution with a large expansion rate may only be effective for the segmentation of some large objects, but will have no benefit for small objects.

[0078] Furthermore, how to properly handle the relationship between objects of different sizes is also the key to designing a dilated convolutional network. In this invention, the dilated convolutions of different expansion rates are combined in a dense connection manner. The expansion rates are {3, 12, 24} respectively, so that the subsequent neurons can obtain larger and larger receptive fields while sharing information through skip-layer connections. The dilated convolutions with large and small expansion rates depend on each other to form a dense pyramid, while also avoiding the convolution degradation caused by excessive expansion rate. In a densely connected module of the network, the output feature map of each layer is the result of merging the output feature maps of the previous layers on the channel, which is equivalent to a fusion of the features of the high and low layers. The calculation method of the densely connected output layer is: l =H l ([x1,x2,…,x l ]); where l is the number of layers, x l is the output of the lth layer, H l is a nonlinear transformation for the lth layer; through the calculation formula of dense connection and void convolution, it can be deduced that in a module with a series of densely connected void convolution layers, the calculation formula for each layer of void convolution is: Among them, y i is the output feature map of the i-th layer in the previous stage, H is the height of the input feature map, d i is the expansion rate of the i-th layer, K is the size of the hole convolution layer, [y i ,y i-1,…,y1] is the feature map generated by concatenating all previous feature layers, and y1 represents the output feature map of the first layer in the previous stage. To preserve the coarse-grained information of the initial input, a global feature vector is obtained through a global average pooling layer. This global feature vector is upsampled and concatenated with the output of the dilated convolutional layer to obtain a feature map. This feature map is then input into a convolutional layer to fuse coarse-grained and fine-grained features. This method fuses contextual information of varying sizes while maintaining the resolution of the feature map.

[0079] Furthermore, in order to capture the semantic dependencies between regions, the present invention also introduces a context attention module based on the attention mechanism, such as Figure 4 As shown in the figure, the context attention module adaptively pays more attention to the relationship between related partitions. The specific model is established as follows Figure 4 As shown in the dotted part, by using two convolutional layers W q and W k Input feature map The feature map is transformed into a latent feature space, where Represents the number domain, C represents the number of channels of the input feature map F, H represents the height of the input feature map F, and W represents the width of the input feature map F. The converted feature map is calculated by equations (1) and (2):

[0080]

[0081]

[0082] Among them, Q is the convolution layer W q The feature map obtained after the convolution layer W k The feature map obtained later, T represents the matrix transpose operation; the dimensions of the feature map Q and feature map K' are converted to Where N = H × W. Then, in order to calculate the correlation between sub-regions, the attention matrix is obtained by normalizing a relationship matrix and then processing it through a Sigmoid activation function and an average pooling layer. The calculation formula of the attention matrix R is: R = Q T K'; While calculating the attention matrix R through the feature map, the feature map F is operated through the convolution layer to obtain another new feature map V. The expression of the feature map V is: in Finally, the attention matrix R is multiplied pixel by pixel by the feature map V to obtain the attention-enhanced feature map E. The expression of the feature map E is: E i =R⊙Vi ; Among them, ⊙ is the element-by-element multiplication operation, E i is the i-th feature map; to preserve the coarse-grained information of the initial input, this algorithm also fuses the context-enhanced information with the original information by pixel-by-pixel addition. Because the input and output dimensions of the context-enhanced module are exactly the same, it can be flexibly embedded in various types of convolutional networks.

[0083] Step S20: input the output features into a region extraction network, the region extraction network obtains a candidate box according to the output features, and the region of interest alignment layer obtains the region of interest features according to the candidate box.

[0084] The step S20 includes:

[0085] Step S21: feature mapping each feature layer of the feature pyramid according to the region of interest alignment layer to obtain a regional feature pyramid of the region of interest, inputting the regional feature pyramid into a feature fusion module, and splicing the regional feature maps of the regional feature pyramid according to channels based on the feature fusion module to obtain an output feature map;

[0086] Step S22: obtaining a channel of the output feature map, encoding the spatial features of the channel according to global average pooling to obtain a global feature, compressing the dimension of the global feature, and expanding the compressed dimension to obtain an attention weight coefficient of the channel;

[0087] Step S23: split the attention weight coefficient to obtain four weight vectors, multiply the weight vectors by the features of the region feature pyramid layer by layer and pixel by pixel to obtain a weighted region feature pyramid, and add the features in the weighted region feature pyramid pixel by pixel to obtain the region of interest features.

[0088] Specifically, each candidate box is mapped to a different feature level in the feature pyramid, and then the ROI alignment layer is used to map ROIs of different sizes to the feature map of a certain feature level to obtain the regional features corresponding to the ROI candidate box; each ROI regional feature is obtained by mapping the ROI coordinates to the feature map coordinates at a specific feature level, and this specific feature level is heuristically selected based on the size of the ROI. Typically, smaller ROIs are assigned to lower levels, while larger ROIs are assigned to higher levels. Therefore, there are two problems. First, in order to more efficiently batch calculate, candidate ROIs of different scales need to be scaled to a uniform scale through ROI pooling or ROI alignment, and the size of all cropped regions is adjusted to 14×14 (upsampled to 28×28 using deconvolution). This limits the output resolution of instance segmentation, and large instances require higher resolution to preserve boundary details. Second, heuristically selecting the mapping feature level based on the size of the ROI is not necessarily the optimal solution, and may miss useful information about the target object in other feature levels.

[0089] To address the second problem mentioned above, the performance of instance segmentation is improved by fusing the region of interest features of all feature levels and selecting the region of interest feature points with the maximum response generated by the fully connected layer to generate the final region feature map for subsequent candidate box adjustment. However, the additional fully connected layer significantly increases the number of parameters. In addition, the maximum value operation only selects the feature point with the highest response, while ignoring the features with lower responses in other levels, which may hinder the full utilization of features at different levels. After obtaining the output features {P2, P3, P4, P5} of the feature pyramid network, these features are fused and adjusted, and then the adjusted {P'2, P'3, P'4, P'5} are subjected to region of interest feature mapping through region of interest Align to obtain regional features. In the present invention, an adaptive feature fusion module based on the attention mechanism is proposed. This adaptive feature fusion module can solve this problem by learning to generate more discriminative region of interest features from the feature pyramid network.

[0090] Among them, the structure of the adaptive feature fusion module is as follows: Figure 5As shown in FIG, first, each feature layer (for example, P2, P3, P4, P5) in the feature pyramid network is feature mapped through the region of interest alignment layer to obtain a regional feature pyramid belonging to the region of interest, that is, a feature with 4 dimensions of C×7×7, where C is the number of channels, usually set to 256; then the regional feature pyramid obtained by mapping feature maps at different levels is input into the adaptive feature fusion module, and the attention mechanism is used to adaptively select useful features; in the adaptive feature fusion module, first, the regional feature maps containing semantic information at different levels in the regional feature pyramid are spliced according to the channels to obtain a 4C×7×7 feature map; secondly, the feature map is globally averaged. The pooling encodes the entire spatial feature on a channel into a global feature with a dimension of 4C×1×1. Then, to reduce the complexity of the model, a 1×1 convolution is used to reduce parameters and compress the weight dimension to C / 4×1×1. Then, a 1×1 convolution is used to expand the dimension back to 4C×1×1 to obtain the attention weight coefficient of each channel. The attention weight coefficients of each channel are combined into a total weight coefficient, and this total weight coefficient is divided into 4 C×1×1 vectors. Finally, an activation layer is used to normalize the weight vector, and the weight vector is multiplied layer by layer and pixel by pixel with the features in the original regional feature pyramid to obtain the regional feature pyramid weighted by the attention mechanism. For the input feature map, the calculation process of the output feature map is as follows: Where q[i] is the output feature map, p[i] is the input feature map, ⊙ is the element-by-element multiplication operation, i is the position subscript of a pixel in the feature map, and W is the width of the input feature map; The calculation process is:

[0091] u = Conv(X);

[0092]

[0093] Among them, X is the input feature, Conv is the convolution layer operation, u is the output after the convolution operation, c is a channel of the feature map, i and j are the position subscripts of a pixel in the feature map, H is the height of the input feature map, z c for The output obtained after the operation will be divided into several parts corresponding to the weights of different feature levels. After obtaining the converted regional feature pyramid, these features are summed pixel by pixel to obtain the final region of interest feature.

[0094] Step S30: Send the region of interest features to a feature adaptive selection module, and the feature adaptive selection module obtains a combined feature representation of all feature levels in the region of interest based on the region of interest features.

[0095] Specifically, after the final region of interest feature is obtained, the region of interest feature is sent to a feature adaptive selection module, and the feature adaptive selection module obtains a combined feature representation of all feature levels in the region of interest according to the region of interest feature.

[0096] Step S40 : calculating the category classification of the region of interest, the coordinates of the candidate box, and the edge-aware mask based on the combined feature representation, and segmenting the cell nuclei of the pathological image based on the category classification, the coordinates, and the edge-aware mask.

[0097] Specifically, based on the combined feature representation, the category classification of the region of interest and the coordinates of the candidate box are calculated respectively. In order to better segment multiple cells with overlapping parts, the present invention also proposes an instance mask prediction module supervised by edge features. This instance mask prediction module obtains an edge-aware mask by combining boundary extreme points as additional supervision, and segments the cell nuclei of the pathological image based on the category classification, the coordinates and the edge-aware mask to obtain better segmentation results.

[0098] Further, if Figure 6 As shown, the overall process of the preferred embodiment of the cell nucleus segmentation method based on context feature fusion in the present invention is specifically as follows: obtaining a medical cell image, inputting the medical cell image into a convolutional neural network for stacking to obtain multiple feature maps, and inputting each feature map into a feature pyramid. The context enhancement module in the feature pyramid performs feature fusion on each feature map to obtain output features; in the context enhancement module, the context enhancement module based on the attention mechanism focuses on mining a wide range of context information and capturing important semantics to reduce the information loss of the highest pyramid level feature map; secondly, the output features are input into the region extraction network, and the region extraction network obtains a candidate box based on the output features. The region of interest alignment layer obtains the region of interest features based on the candidate box; then the features are sent to the feature adaptive selection module to adaptively obtain a combined feature representation at all feature levels of each region of interest, and finally, the classification of the region of interest category, the coordinates of the candidate box and the edge-aware mask are calculated based on the combined feature representation, and the cell nucleus of the pathological image is segmented based on the category classification, the coordinates and the edge-aware mask to obtain a better segmentation result.

[0099] Further, if Figure 7 As shown, based on the above-mentioned cell nucleus segmentation method based on context feature fusion, the present invention also provides a cell nucleus segmentation system based on context feature fusion, wherein the cell nucleus segmentation system based on context feature fusion includes:

[0100] A data input module 51 is used to obtain a medical cell image, input the medical cell image into a convolutional neural network for stacking to obtain multiple feature maps, and input each feature map into a feature pyramid;

[0101] A context enhancement module 52 is configured to perform feature fusion on each of the feature maps to obtain output features;

[0102] A candidate box acquisition module 53 is configured to input the output features into a region extraction network, and the region extraction network obtains a candidate box based on the output features;

[0103] The candidate frame processing module 54 is used for the ROI alignment layer to obtain ROI features based on the candidate frame and send the ROI features to the feature adaptive selection module;

[0104] A feature adaptive selection module 55 is configured to obtain a combined feature representation of all feature levels in the region of interest based on the features of the region of interest;

[0105] The cell nucleus segmentation module 56 is configured to calculate the category classification of the region of interest, the coordinates of the candidate box, and the edge-aware mask based on the combined feature representation, and segment the cell nuclei of the pathological image based on the category classification, the coordinates, and the edge-aware mask.

[0106] Further, if Figure 8 As shown, based on the above-mentioned cell nucleus segmentation method and system based on context feature fusion, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 8 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0107] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code of the installation terminal, etc. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a cell nucleus segmentation program 40 based on context feature fusion is stored on the memory 20, and the cell nucleus segmentation program 40 based on context feature fusion can be executed by the processor 10, thereby realizing the cell nucleus segmentation method based on context feature fusion in the present application.

[0108] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the cell nucleus segmentation method based on context feature fusion.

[0109] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0110] In one embodiment, when the processor 10 executes the cell nucleus segmentation program 40 based on context feature fusion in the memory 20, the following steps are implemented:

[0111] Acquire a medical cell image, input the medical cell image into a convolutional neural network for stacking to obtain multiple feature maps, and input each of the feature maps into a feature pyramid. A context enhancement module in the feature pyramid performs feature fusion on each of the feature maps to obtain output features;

[0112] The output features are input into a region extraction network, the region extraction network obtains a candidate box according to the output features, and the region of interest alignment layer obtains a region of interest feature according to the candidate box;

[0113] Sending the region of interest features to a feature adaptive selection module, wherein the feature adaptive selection module obtains a combined feature representation of all feature levels in the region of interest based on the region of interest features;

[0114] Based on the combined feature representation, the category classification of the region of interest, the coordinates of the candidate box and the edge-aware mask are calculated respectively, and the cell nucleus of the pathological image is segmented based on the category classification, the coordinates and the edge-aware mask.

[0115] The step of acquiring a medical cell image, inputting the medical cell image into a convolutional neural network for stacking to obtain a plurality of feature maps, and inputting each of the feature maps into a feature pyramid specifically includes:

[0116] Acquiring the medical cell image, inputting the medical cell image into the convolutional neural network for stacking, changing the resolution of the medical cell image according to downsampling, and obtaining feature maps of different stages according to the different resolutions;

[0117] Obtain multiple dilated convolutions with different dilation rates, combine the dilated convolutions with different dilation rates in a dense connection manner to obtain a feature pyramid, and input the feature map into the feature pyramid;

[0118] In the feature pyramid, the coarse-grained information of the feature map is enhanced according to the context enhancement module, and the enhanced coarse-grained information is pixel-wise added and fused with the original coarse-grained information to obtain the output feature.

[0119] The step of inputting the feature map into the feature pyramid further includes:

[0120] The output signal obtained by performing a dilated convolution operation on the input signal of the feature map is:

[0121] Where y[i] is the output signal obtained after the dilated convolution operation, i is the position subscript of a pixel in the feature map, d is the dilation rate, w[k] is the kth parameter of the convolution kernel, and when d = 1, the dilated convolution is a normal convolution.

[0122] The output of the dense connection is: l =H l ([x1,x2,…,x l ]);

[0123] Among them, l is the number of layers, x l is the output of the lth layer, H l is the nonlinear transformation of the lth layer.

[0124] The calculation formula of the dilated convolution is:

[0125] Among them, y i is the output feature map of the i-th layer in the previous stage, H is the height of the input feature map, d i is the expansion rate of the i-th layer, K is the size of the hole convolution layer, [y i ,y i-1 ,…,y1] is the feature map generated by concatenating all previous feature layers, and y1 represents the output feature map of the first layer in the previous stage.

[0126] The method further comprises: obtaining a plurality of dilated convolutions with different dilation rates, combining the dilated convolutions with different dilation rates in a dense connection manner to obtain a dense pyramid, and inputting the feature map into the dense pyramid. The method further comprises:

[0127] Processing the global average pooling layer to obtain a global feature vector, upsampling the global feature vector to obtain a target global feature vector, and concatenating the target global feature vector with the output of the dilated convolutional layer to obtain a feature map;

[0128] The feature map is input into a 1×1 convolutional layer, and the coarse-grained and fine-grained features of the feature map are fused.

[0129] The context-based enhancement module enhances the coarse-grained information of the feature map, and performs pixel addition and fusion of the enhanced coarse-grained information and the original coarse-grained information to obtain an output feature, and then further includes:

[0130] Get the input feature map F, according to the convolution layer W q and convolutional layer W k Convert the feature maps of the input feature map to the latent feature space to obtain feature maps Q and K';

[0131] The feature map Q and the feature map K' are reorganized in terms of feature dimensions to obtain a target feature map, the target feature map is processed to obtain a relationship matrix, and the relationship matrix is normalized to obtain an attention matrix R.

[0132] Among them, the calculation formula of the characteristic map Q is:

[0133] Among them, Q is the convolution layer W q The feature map obtained later, T is the matrix transpose operation;

[0134] The calculation formula of the characteristic map K' is:

[0135] Among them, K' is the convolution layer Wk The feature map obtained later;

[0136] The calculation formula of the attention matrix R is: R = Q T K'.

[0137] The normalization process of the relationship matrix is performed to obtain an attention matrix R, and then the following steps are further included:

[0138] Get the convolutional layer W v , based on the convolutional layer W v Processing the input feature map to obtain a new feature map V, and multiplying the attention matrix R by the feature map V pixel by pixel to obtain an attention-enhanced feature map E;

[0139] The expression of the characteristic graph V is:

[0140] Among them, F is the input feature map;

[0141] The expression of the characteristic graph E is: i =R⊙V i ;

[0142] Among them, ⊙ is the element-by-element multiplication operation, E i is the i-th feature map.

[0143] The output features are input into a region extraction network, the region extraction network obtains a candidate box according to the output features, and the region of interest alignment layer obtains the region of interest features according to the candidate box, specifically including:

[0144] Performing feature mapping on each feature layer of the feature pyramid according to the region of interest alignment layer to obtain a regional feature pyramid of the region of interest, inputting the regional feature pyramid into a feature fusion module, and splicing the regional feature maps of the regional feature pyramid according to channels based on the feature fusion module to obtain an output feature map;

[0145] Obtaining a channel of the output feature map, encoding the spatial features of the channel according to global average pooling to obtain a global feature, compressing the dimension of the global feature, and expanding the compressed dimension to obtain an attention weight coefficient of the channel;

[0146] The attention weight coefficient is split into four weight vectors, the weight vectors are multiplied by the features of the regional feature pyramid layer by layer and pixel by pixel to obtain a weighted regional feature pyramid, and the features in the weighted regional feature pyramid are added pixel by pixel to obtain the region of interest features.

[0147] The calculation formula of the output feature map is:

[0148] Among them, q[i] is the output feature map, p[i] is the input feature map, ⊙ is the element-by-element multiplication operation, i is the position subscript of a pixel in the feature map, and W is the width of the input feature map;

[0149] Among them, the The calculation process is:

[0150] u = Conv(X);

[0151]

[0152] Among them, X is the input feature, Conv is the convolution layer operation, u is the output after the convolution operation, c is a channel of the feature map, i and j are the position subscripts of a pixel in the feature map, H is the height of the input feature map, z c for The output after the operation.

[0153] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a cell nucleus segmentation program based on context feature fusion, and when the cell nucleus segmentation program based on context feature fusion is executed by a processor, the steps of the cell nucleus segmentation method based on context feature fusion as described above are implemented.

[0154] In summary, the present invention focuses on mining a wide range of contextual information in the context enhancement module based on the attention mechanism, capturing important semantics to reduce the information loss of the highest pyramid level feature map; then sends the features to the feature adaptive selection module to adaptively obtain a combined feature representation at all feature levels of each region of interest, and finally calculates the classification of the region of interest category, the coordinates of the candidate box and the edge-aware mask based on these feature representations, thereby realizing the segmentation of medical pathology slice images.

[0155] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0156] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0157] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A cell nucleus segmentation method based on context feature fusion, characterized in that: The cell nucleus segmentation method based on context feature fusion includes: Acquire a medical cell image, input the medical cell image into a convolutional neural network for stacking to obtain multiple feature maps, and input each of the feature maps into a feature pyramid. A context enhancement module in the feature pyramid performs feature fusion on each of the feature maps to obtain output features; The method of acquiring a medical cell image, inputting the medical cell image into a convolutional neural network for stacking to obtain a plurality of feature maps, and inputting each of the feature maps into a feature pyramid, wherein a context enhancement module in the feature pyramid performs feature fusion on each of the feature maps to obtain output features, specifically includes: Acquiring the medical cell image, inputting the medical cell image into the convolutional neural network for stacking, changing the resolution of the medical cell image according to downsampling, and obtaining feature maps of different stages according to the different resolutions; Obtain multiple dilated convolutions with different dilation rates, combine the dilated convolutions with different dilation rates in a dense connection manner to obtain a feature pyramid, and input the feature map into the feature pyramid; The method further includes obtaining a plurality of dilated convolutions with different dilation rates, combining the dilated convolutions with different dilation rates in a dense connection manner to obtain a feature pyramid, and inputting the feature map into the feature pyramid. The method further includes: Processing the global average pooling layer to obtain a global feature vector, upsampling the global feature vector to obtain a target global feature vector, and concatenating the target global feature vector with the output of the dilated convolutional layer to obtain a feature map; Input the feature map into a and fuse the coarse-grained and fine-grained features of the feature map; In the feature pyramid, the coarse-grained information of the feature map is enhanced according to the context enhancement module, and the enhanced coarse-grained information is pixel-wise fused with the original coarse-grained information to obtain the output feature; The output features are input into a region extraction network, the region extraction network obtains a candidate box according to the output features, and the region of interest alignment layer obtains a region of interest feature according to the candidate box; Sending the region of interest features to a feature adaptive selection module, wherein the feature adaptive selection module obtains a combined feature representation of all feature levels in the region of interest based on the region of interest features; Based on the combined feature representation, the category classification of the region of interest, the coordinates of the candidate box and the edge-aware mask are calculated respectively, and the cell nucleus of the pathological image is segmented based on the category classification, the coordinates and the edge-aware mask.

2. The cell nucleus segmentation method based on context feature fusion according to claim 1, characterized in that: The step of inputting the feature map into the feature pyramid further includes: The output signal obtained by performing a dilated convolution operation on the input signal of the feature map is: ； in, is the output signal obtained after the dilated convolution operation, i is the subscript of the position of a pixel in the feature map, is the input signal of the feature map, d is the expansion rate, is the convolution kernel k Parameters, where when d=1, the dilated convolution is a normal convolution; The output mode of the dense connection is: ; in, is the number of layers, For the The output of the layer, For the Layer nonlinear transformation.

3. The cell nucleus segmentation method based on context feature fusion according to claim 1, characterized in that: The calculation formula of the dilated convolution is: ; in, is the output feature map of the i-th layer in the previous stage, H is the height of the input feature map, For the i The expansion rate of the layer, K is the size of the hole convolution layer, The feature map generated by cascading all previous feature layers, Represents the output feature map of the first layer in the previous stage.

4. The cell nucleus segmentation method based on context feature fusion according to claim 1, characterized in that: The coarse-grained information of the feature map is enhanced according to the context enhancement module, and the enhanced coarse-grained information is pixel-wise added and fused with the original coarse-grained information to obtain the output feature, and then the following steps are further included: Get the input feature map F, according to the convolution layer and convolutional layers Convert the feature maps of the input feature map to the latent feature space to obtain feature maps Q and K'; The feature map Q and the feature map K' are reorganized in terms of feature dimensions to obtain a target feature map, the target feature map is processed to obtain a relationship matrix, and the relationship matrix is normalized to obtain an attention matrix R.

5. The cell nucleus segmentation method based on context feature fusion according to claim 4, characterized in that: The calculation formula of the feature map Q is: ; Among them, Q is the convolution layer The feature map obtained later, T is the matrix transpose operation; The calculation formula of the characteristic map K' is: ; Among them, K' is the convolution layer The feature map obtained later; The calculation formula of the attention matrix R is: '.

6. The cell nucleus segmentation method based on context feature fusion according to claim 4, characterized in that: The normalization process of the relationship matrix is performed to obtain an attention matrix R, and then the following steps are further included: Get the convolutional layer , based on the convolutional layer Processing the input feature map to obtain a new feature map V, and multiplying the attention matrix R by the feature map V pixel by pixel to obtain an attention-enhanced feature map E; The expression of the characteristic graph V is: ; Among them, F is the input feature map; The expression of the characteristic graph E is: ; in, For the i feature maps, For the i A new feature map.

7. The cell nucleus segmentation method based on context feature fusion according to claim 1, characterized in that: The output features are input into a region extraction network, the region extraction network obtains a candidate box according to the output features, and the region of interest alignment layer obtains a region of interest feature according to the candidate box, specifically including: Performing feature mapping on each feature layer of the feature pyramid according to the region of interest alignment layer to obtain a regional feature pyramid of the region of interest, inputting the regional feature pyramid into a feature fusion module, and splicing the regional feature maps of the regional feature pyramid according to channels based on the feature fusion module to obtain an output feature map; Obtaining a channel of the output feature map, encoding the spatial features of the channel according to global average pooling to obtain a global feature, compressing the dimension of the global feature, and expanding the compressed dimension to obtain an attention weight coefficient of the channel; The attention weight coefficient is split into four weight vectors, the weight vectors are multiplied by the features of the regional feature pyramid layer by layer and pixel by pixel to obtain a weighted regional feature pyramid, and the features in the weighted regional feature pyramid are added pixel by pixel to obtain the region of interest features.

8. The cell nucleus segmentation method based on context feature fusion according to claim 7, characterized in that: The calculation formula of the output feature map is: ； in, is the output feature map, is the input feature map, , i Subscript the position of a pixel in the feature map; Among them, the The calculation process is: ； ； in, X is the input feature, is the convolution layer operation, is the output after the convolution operation, c is a channel of the feature map, i and j They are all the position subscripts of a pixel in the feature map. H is the height of the input feature map, W is the width of the input feature map, for The output after the operation is, For channel c middle i Pixels and j Output after pixel convolution operation.

9. A cell nucleus segmentation system based on context feature fusion, characterized in that: The cell nucleus segmentation system based on context feature fusion is applied to the cell nucleus segmentation method based on context feature fusion according to any one of claims 1 to 8, and the cell nucleus segmentation system based on context feature fusion comprises: A data input module is used to obtain medical cell images, input the medical cell images into a convolutional neural network for stacking to obtain multiple feature maps, and input each of the feature maps into a feature pyramid; A context enhancement module, configured to perform feature fusion on each of the feature maps to obtain output features; A candidate box acquisition module is used to input the output features into a region extraction network, and the region extraction network obtains a candidate box according to the output features; The candidate frame processing module is used for the ROI alignment layer to obtain ROI features based on the candidate frame and send the ROI features to the feature adaptive selection module; A feature adaptive selection module, configured to obtain a combined feature representation of all feature levels in the region of interest based on the features of the region of interest; A cell nucleus segmentation module is used to calculate the category classification of the region of interest, the coordinates of the candidate box and the edge-aware mask based on the combined feature representation, and to segment the cell nuclei of the pathological image based on the category classification, the coordinates and the edge-aware mask.

10. A terminal, characterized in that: The terminal includes a memory, a processor, and a program stored in the memory and executable on the processor. When the program is executed by the processor, the steps of the cell nucleus segmentation method based on context feature fusion as described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the cell nucleus segmentation method based on context feature fusion according to any one of claims 1 to 8.

Images