A multi-scale spatial feature enhancement method and device

Abstract

The present invention provides a multi-scale spatial feature enhancement method and device, comprising: processing a pathology image to be segmented to obtain a target pathology image; inputting the target pathology image into an image feature extraction network to extract features of different scales from the target pathology image, thereby obtaining feature images at different scale levels; inputting the feature images at different scale levels into a self-attention mechanism for spatial feature enhancement processing to obtain a final feature map; and inputting the final feature map into a baseline model based on a two-stage detector to generate a segmentation result. By inputting the feature images at different scale levels into a self-attention mechanism for spatial feature enhancement processing, the present invention extracts a feature map with richer information, enabling better identification and location of targets in the image, avoiding reliance solely on pixels in the last feature map, and improving segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of medical pathology image segmentation, and in particular to a multi-scale spatial feature enhancement method and device. Background Art

[0002] Medical pathology image segmentation is a key step in medical image analysis and recognition, and cell nucleus segmentation is an important component of pathology image segmentation. In recent years, many cell nucleus segmentation algorithms have achieved excellent performance by utilizing multi-level features with positional and semantic information.

[0003] With the development of deep learning, image segmentation methods based on deep learning have achieved significant improvements in accuracy compared to traditional image segmentation methods and have garnered widespread attention in recent years. Modern visual recognition systems are closely tied to the environment. Due to the hierarchical structure of convolutional neural networks, contextual information is often encoded over increasingly larger receptive fields through pooling, convolution strides, or dilated convolutions. Therefore, the predictions output by the last feature map layer are essentially based on rich contextual information. Even for smaller objects, such as a cup, which only has a few "feature pixels," the cup can be recognized using this auxiliary information due to the perception of a larger context, such as a table. At the same time, scale is also important. The recognition of a cup should be assigned more feature pixels than just those in the last feature map. Relying solely on the pixels in the last feature map will overlook many small objects.

[0004] That is, existing cell instance segmentation methods only rely on the pixels on the last feature map, resulting in low segmentation accuracy.

[0005] Therefore, the existing technology has defects and needs to be improved and developed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a multi-scale spatial feature enhancement method and device in response to the above-mentioned defects of the prior art, aiming to solve the problem that the cell instance segmentation method in the prior art only relies on the pixels on the last feature map, resulting in low segmentation accuracy.

[0007] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0008] A multi-scale spatial feature enhancement method, comprising:

[0009] Processing the pathological image to be segmented to obtain the target pathological image;

[0010] Inputting the target pathological image into an image feature extraction network, extracting features of different scales in the target pathological image, and obtaining feature images of different scale levels;

[0011] The feature images at different scale levels are input into the self-attention mechanism to perform spatial feature enhancement processing to obtain the final feature map;

[0012] The final feature map is input into a baseline model based on a two-stage detector to generate segmentation results.

[0013] In one implementation, processing the pathological image to be segmented to obtain a target pathological image includes:

[0014] receiving a pathological image to be segmented, and scaling the side length of the pathological image to be segmented to a preset range to obtain a scaled image;

[0015] The zoomed image is randomly cropped to generate a target pathological image with a preset resolution.

[0016] In one implementation, the target pathological image is input into an image feature extraction network, and features of different scales in the target pathological image are extracted to obtain feature images of different scale levels, including:

[0017] Inputting the target pathological image into an image feature extraction network, wherein the image feature extraction network uses a 50-layer network pre-trained on an image dataset as a basic feature extractor;

[0018] The target pathological image outputs feature images of different scale levels after passing through each stage of the network.

[0019] In one implementation, the resolutions of the feature images at different scale levels are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the target pathology image, and the dimensions are 256, 512, 1024, and 2048, respectively.

[0020] In one implementation, the pre-training formula of the image feature extraction network is expressed as:

[0021] I=f resnet (V);

[0022] Wherein, I is the extracted feature image I={i1,i2,…,i n}, n represents the number of images in the image dataset, f resnet (.) is the image feature extraction network ResNet, and V is the image in the image dataset.

[0023] In one implementation, the feature images at different scale levels are input into a self-attention mechanism to perform spatial feature enhancement processing to obtain a final feature map, including:

[0024] Construct a feature pyramid structure based on feature images at different scale levels;

[0025] Obtaining a high-level feature map and a low-level feature map in the feature pyramid structure;

[0026] Converting and mapping the low-level feature map into a first feature map and a second feature map, and mapping the high-level feature map into a third feature map;

[0027] The coarse-grained information on the low-level feature map is merged into the high-level feature map through the first feature map, the second feature map, and the third feature map to obtain a final feature map.

[0028] In one implementation, merging the coarse-grained information on the low-level feature map into the high-level feature map through the first feature map, the second feature map, and the third feature map to obtain a final feature map includes:

[0029] Input the second feature map into a global average pooling layer to obtain a weight;

[0030] Multiplying the weight and the third feature map pixel by pixel to obtain weighted information;

[0031] Pass the first feature map through convolution layers with different strides to obtain a low-scale feature map;

[0032] The weighted information and the low-scale feature map are added pixel by pixel and processed through a self-attention mechanism to obtain a final feature map.

[0033] In one implementation, the weighted information and the low-scale feature map are added pixel by pixel and processed through a self-attention mechanism to obtain a final feature map, including:

[0034] Adding the weighted information and the low-scale feature map pixel by pixel to obtain a superimposed feature map, and passing the superimposed feature map through a 1×1 convolution layer to obtain a target feature map;

[0035] The pixels in the target feature map are mapped to a feature space to obtain a final feature map that learns the dependency relationship between pixels.

[0036] In one implementation, the final feature map is input into a baseline model based on a two-stage detector to generate a segmentation result, including:

[0037] Inputting the final feature map and the region candidate box belonging to the foreground into the region of interest alignment layer, mapping the region candidate box onto the final feature map to obtain a region feature map;

[0038] Scaling the regional feature maps of different sizes to a preset uniform size to obtain the target region feature map; inputting the target region feature map into two full-volume layers with a dimension of 1024, and performing category prediction of the region of interest through an activation layer, and performing bounding box regression at the same time to obtain the position coordinates of the detection box;

[0039] The target area feature map is input into a feature pyramid network with 4 convolutional layers. According to the position coordinates of the detection box, the target area feature map is upsampled twice, and the instance mask is predicted to obtain a segmentation result.

[0040] In one implementation, the final feature map and the region candidate box belonging to the foreground are input into the region of interest alignment layer, and the region candidate box is mapped onto the final feature map to obtain the region feature map, including:

[0041] Input the final feature map and the candidate box of the region belonging to the foreground into the region of interest alignment layer;

[0042] Identifying image coordinates on the final feature map, and converting the image coordinates into feature map coordinates through bilinear interpolation;

[0043] Converting the feature map coordinates into feature coordinates of the region of interest through bilinear interpolation;

[0044] The region candidate frame is mapped onto the final feature map according to the feature coordinates of the region of interest to obtain a region feature map.

[0045] The present invention also provides a multi-scale spatial feature enhancement device, comprising:

[0046] A preprocessing module is used to process the pathological image to be segmented to obtain a target pathological image;

[0047] a multi-scale feature extraction module, configured to input the target pathological image into an image feature extraction network, extract features of different scales in the target pathological image, and obtain feature images of different scale levels;

[0048] A spatial feature enhancement module is used to input the feature images of different scale levels into the self-attention mechanism to perform spatial feature enhancement processing to obtain the final feature map;

[0049] The segmentation result generation module is used to input the final feature map into a baseline model based on a two-stage detector to generate a segmentation result.

[0050] The present invention also provides a terminal, comprising: a memory, a processor, and a multi-scale spatial feature enhancement program stored in the memory and runnable on the processor, wherein the multi-scale spatial feature enhancement program, when executed by the processor, implements the steps of the multi-scale spatial feature enhancement method described above.

[0051] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed to implement the steps of the multi-scale spatial feature enhancement method as described above.

[0052] Beneficial effects of the present invention: The embodiments of the present invention process the pathological image to be segmented to obtain a target pathological image; input the target pathological image into an image feature extraction network to extract features of different scales in the target pathological image to obtain feature images of different scale levels; input the feature images of different scale levels into a self-attention mechanism to perform spatial feature enhancement processing to obtain a final feature map; and input the final feature map into a baseline model based on a two-stage detector to generate a segmentation result. The present invention inputs the feature images of different scale levels into a self-attention mechanism to perform spatial feature enhancement processing, thereby extracting a feature map with richer information, which can better identify and locate targets on the image, avoids relying solely on pixels on the last feature map, and improves segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of a preferred embodiment of the multi-scale spatial feature enhancement method in the present invention.

[0054] Figure 2 This is a block diagram of the principle of cell nucleus instance segmentation in pathological section images based on multi-scale feature enhancement in the present invention.

[0055] Figure 3 Schematic diagram of information interaction at different levels in the spatial information enhancement module of the present invention.

[0056] Figure 4 Schematic diagram of the self-attention mechanism of the spatial information enhancement module in the present invention.

[0057] Figure 5 It is a functional principle block diagram of a preferred embodiment of the multi-scale spatial feature enhancement device in the present invention.

[0058] Figure 6 It is a functional principle block diagram of the terminal in the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] To effectively identify multi-scale object information in images, an image pyramid is generated for the same image. High-level semantic information corresponds to low-resolution images, and low-level semantic information corresponds to high-resolution images. This allows for the identification of objects of different scales at corresponding levels. For example, a low-level semantic cup can be identified in a high-resolution image, while a high-level semantic computer screen can be identified in a low-resolution image. However, because each image requires a convolutional neural network for recognition, the image pyramid exponentially increases the time required for the neural network. Feature pyramid networks are one of the most representative algorithms, but they also have some significant limitations.

[0061] Instance segmentation tasks with pixel-level labels often require contextual information at multiple scales. Small objects may only require the local contextual information of lower-level features for identification. However, for larger objects, the local contextual information of low-level features only contains a portion of the object's detailed features, and accurate discrimination requires the global contextual information of higher-level features. For example, a display screen with only a few detailed features could be mistaken for a landline phone screen. Using non-local convolution and self-attention mechanisms, non-local contextual information can be more explicitly modeled. This interaction of spatial features can capture more accurate information, overcoming the problem of the original feature pyramid network, which simply fuses information by pixel-by-pixel summation of feature maps at different scales.

[0062] This paper explores the information loss that occurs during image feature extraction and analyzes the problem of cell stacking occluding each other in cell images. By leveraging multiscale information, this paper proposes a multiscale feature-enhanced cell nucleus segmentation algorithm. By addressing the information loss that occurs when information is transferred between different feature levels in the feature pyramid, a multiscale spatial information enhancement module is established. This algorithm can preserve the internal information of features at different scales and explore richer semantic relationships between different levels. It fully utilizes the local and global semantic information in the multiscale feature map to achieve more accurate segmentation results.

[0063] See Figure 1 The multi-scale spatial feature enhancement method in the embodiment of the present invention includes the following steps:

[0064] Step S100: Process the pathological image to be segmented to obtain a target pathological image.

[0065] Specifically, when a pathological image to be segmented is received, preprocessing is first performed to obtain a target pathological image that meets the requirements.

[0066] In one implementation, step S100 specifically includes:

[0067] Step S110: receiving a pathological image to be segmented, and scaling the side length of the pathological image to be segmented to a preset range to obtain a scaled image;

[0068] Step S120 : randomly cropping the zoomed image to generate a target pathological image with a preset resolution.

[0069] Specifically, the preset range is [800, 1333], and the preset resolution is 256 × 256. That is, in this embodiment, the side length of the pathological image to be segmented is first rescaled to the range of [800, 1333], and the rescaled image is randomly cropped to generate a target pathological image with a resolution of 256 × 256, so as to facilitate inputting it into the convolutional neural network for processing.

[0070] like Figure 1 As shown, the multi-scale spatial feature enhancement method further includes the following steps:

[0071] Step S200: Input the target pathological image into an image feature extraction network to extract features of different scales in the target pathological image to obtain feature images of different scale levels.

[0072] Specifically, see Figure 2 , the present invention inputs the entire image into the convolutional neural network to extract the feature map of the image, so as to input the feature map into the subsequent RPN network.

[0073] In one implementation, step S200 specifically includes:

[0074] Step S210: inputting the target pathological image into an image feature extraction network, wherein the image feature extraction network uses a 50-layer network pre-trained on an image dataset as a basic feature extractor;

[0075] Step S220: After the target pathological image passes through each stage of the network, feature images of different scale levels are output.

[0076] Specifically, the image feature extraction network is a convolutional neural network. The target pathological image is fed into the network to obtain the corresponding feature map. The present invention uses a 50-layer network pre-trained on a large-scale image dataset as the base feature extractor to improve the computational accuracy of the image feature extraction network.

[0077] In one embodiment, the resolutions of the feature images at different scale levels are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the target pathology image, respectively, and the dimensions are 256, 512, 1024, and 2048, respectively.

[0078] Specifically, the output of each stage of the network (C2, C3, C4, C5) is used as the input of the feature pyramid network. The resolution of the feature map is 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, and its dimensions are 256, 512, 1024, and 2048 respectively.

[0079] In one implementation, the pre-training formula of the image feature extraction network is expressed as:

[0080] I=f resnet (V);

[0081] Wherein, I is the extracted feature image I={i1,i2,…,i n}, n represents the number of images in the image dataset, f resnet (.) is the image feature extraction network ResNet, and V is the image in the image dataset.

[0082] In the original feature pyramid network, C5 is first convolved with 1×1 to reduce the number of channels of the feature map to 256 dimensions, and the intermediate feature M5 is obtained. M5 is upsampled by nearest neighbor interpolation and then added pixel by pixel with the feature map after 1×1 convolution of C4 to obtain M4. Repeat the above process to obtain M3 and M2 respectively. Finally, (M2, M3, M4, M5) are all convolved with 3×3 to obtain the final (P2, P3, P4, P5) features. The present invention believes that non-local information interaction itself should occur at the corresponding scale of the interacting object (or part), rather than only at a uniform scale as in existing methods. Non-local information interaction performed only in a feature map of a uniform scale is not sufficient to fully represent contextual information.

[0083] like Figure 1 As shown, the multi-scale spatial feature enhancement method further includes the following steps:

[0084] Step S300: Input the feature images at different scale levels into the self-attention mechanism, perform spatial feature enhancement processing, and obtain the final feature map.

[0085] Specifically, this paper incorporates a spatial information enhancement module into the existing top-down architecture, proposing a new feature information interaction module based on the original feature pyramid network. The input and output dimensions of the attention-based spatial information enhancement module are identical, making it easily transferable to other instance segmentation networks that use a feature pyramid architecture. This paper embeds it into a common detection-based instance segmentation network.

[0086] In one implementation, step S300 specifically includes:

[0087] Step S310: constructing a feature pyramid structure based on feature images at different scale levels;

[0088] Step S320: obtaining a high-level feature map and a low-level feature map in the feature pyramid structure;

[0089] Step S330: converting and mapping the low-level feature map into a first feature map and a second feature map, and mapping the high-level feature map into a third feature map;

[0090] Step S340: Merge the coarse-grained information on the low-level feature map into the high-level feature map through the first feature map, the second feature map, and the third feature map to obtain a final feature map.

[0091] Specifically, see Figure 3 The spatial information enhancement module can present the fine-grained concept of high-level features by merging the coarse-grained information on the low-level feature map, and can realize the interactive fusion of feature information on feature images of different scale levels. The input of the spatial information enhancement module is C2 and C h , C h The corresponding high-level feature map is converted and mapped into the third feature map Q, and the low-level feature map corresponding to C2 is mapped into the first feature map V and the second feature map K. After obtaining Q and K, it is necessary to measure the dependency between each position of the feature maps of different feature levels, and then obtain the final feature map based on the dependency between each position of the feature maps of different feature levels.

[0092] In one embodiment, step S340 specifically includes:

[0093] Step S341: Input the second feature map into a global average pooling layer to obtain a weight;

[0094] Step S342: multiply the weight and the third feature map pixel by pixel to obtain weighted information;

[0095] Step S343: Pass the first feature map through convolution layers with different step sizes to obtain a low-scale feature map;

[0096] Step S344: Add the weighted information and the low-scale feature map pixel by pixel, and process them through a self-attention mechanism to obtain a final feature map.

[0097] Specifically, K first obtains a weight w through a global average pooling layer, and then weights Q by multiplying w pixel by pixel. att , and then a 3×3 convolution is performed on this Q att Fine-tune. V needs to reduce the scale of the low-level feature map through convolution layers with different strides to obtain V down , and finally V down With Q att The final output is obtained by pixel-by-pixel addition.

[0098] The specific calculation process includes:

[0099] w = GAP(K);

[0100] Q att =w·Q;

[0101] V down =f d (V);

[0102] f o =V down +f conv (Q att) .

[0103] The spatial information enhancement module proposed in the present invention interactively fuses the feature information of feature images at different scale levels through the above process.

[0104] In one embodiment, step S344 specifically includes:

[0105] Step S3441: Add the weighted information and the low-scale feature map pixel by pixel to obtain a superimposed feature map, and obtain a target feature map by passing the superimposed feature map through a 1×1 convolution layer;

[0106] Step S3442: Map the pixels in the target feature map to the feature space to obtain a final feature map that learns the dependency relationship between pixels.

[0107] Specifically, the spatial information enhancement module also uses the self-attention mechanism to perform global spatial feature fusion on feature images at the same scale level, that is, to obtain global context information by measuring the dependency between any two positions in the feature map. Figure 4 , First, a feature map V∈R is obtained through a 1×1 convolutional layer C×H×WThen, two different convolution operations are performed to obtain two feature maps θ(V) and φ(V), and the attention map A (Attention map) is calculated by θ(V) and φ(V). Finally, the feature map f of the dependency relationship between pixels is learned. o ' can be obtained by the following formula:

[0108] f o '=V+λ*A;

[0109]

[0110] Among them, g, θ, and φ represent three different convolution operations, Softmax is the activation function, and λ is used to control the self-attention mechanism used on the feature map to the final feature map f o 'In order to adaptively adjust the weight of the self-attention mechanism, λ is set as a learnable parameter.

[0111] like Figure 1 As shown, the multi-scale spatial feature enhancement method further includes the following steps:

[0112] Step S400: Input the final feature map into a baseline model based on a two-stage detector to generate a segmentation result.

[0113] Specifically, the present invention uses a classic instance segmentation method based on a two-stage detector as a baseline model to generate segmentation results, which can be specifically divided into: a region of interest alignment layer, a detection layer, and a segmentation layer.

[0114] In one implementation, step S400 specifically includes:

[0115] Step S410: input the final feature map and the region candidate box belonging to the foreground into the region of interest alignment layer, and map the region candidate box onto the final feature map to obtain a region feature map;

[0116] Step S420: scaling the region feature maps of different sizes to a preset uniform size to obtain a target region feature map; inputting the target region feature map into two full-volume layers with a dimension of 1024, and performing category prediction of the region of interest through an activation layer, and performing bounding box regression at the same time to obtain the position coordinates of the detection box;

[0117] Step S430: Input the target area feature map into a feature pyramid network with 4 convolutional layers, perform two-fold upsampling on the target area feature map according to the position coordinates of the detection frame, and perform instance mask prediction processing to obtain a segmentation result.

[0118] Specifically, in the region of interest alignment layer, the feature map and the region candidate box belonging to the foreground are input into this layer together. First, the region candidate box is mapped to the feature map through the region of interest Align layer to obtain the region feature map, and then the region feature maps of different sizes are scaled to a uniform size and sent to the subsequent branch prediction sub-network for discrimination.

[0119] In the detection layer, a 7×7 regional feature map is used to first pass through two full-volume layers with a dimension of 1024, and then through an activation layer to predict the category of the region of interest, and at the same time perform bounding box regression to obtain the final position coordinates of the detection box.

[0120] In the segmentation layer, after passing through a feature pyramid network with 4 convolutional layers, the regional feature map is upsampled twice and then the instance mask prediction task is performed.

[0121] In one embodiment, step S410 specifically includes:

[0122] Step S411: input the final feature map and the candidate box of the foreground region into the region of interest alignment layer;

[0123] Step S412: Identify the image coordinates on the final feature map, and convert the image coordinates into feature map coordinates through bilinear interpolation;

[0124] Step S413: converting the feature map coordinates into feature coordinates of the region of interest through bilinear interpolation;

[0125] Step S414: Map the region candidate box to the final feature map according to the feature coordinates of the region of interest to obtain a region feature map.

[0126] Specifically, when using a pooling layer (i.e., a ROI alignment layer), quantization must be performed twice: when converting image coordinates to feature map coordinates and when converting feature map coordinates to ROI feature coordinates. This introduces two quantization errors, causing a mismatch between pixels in the image and pixels in the ROI feature, impacting performance. ROI alignment solves this region mismatch problem by replacing quantization with bilinear interpolation, reducing errors in the coordinate conversion process.

[0127] The present invention proposes a cell instance segmentation method with multi-scale spatial information enhancement, constructs a spatial self-attention mechanism for modeling contextual information of cell features at different scales, interactively fuses feature information at different scales and spaces, and obtains a richer feature map. First, based on the contextual information and scale changes of image features, a feature pyramid network based on spatial information enhancement is proposed. The input of the spatial information enhancement module is the convolutional feature pyramid, and the output feature information is obtained after the original feature information is re-encoded across scales and spaces. The present invention introduces a conversion structure based on the attention mechanism, interactively fuses feature information at different scales and spaces, enhances spatial features, and obtains a richer feature map. The goal of the network is to better identify and locate targets on the image by extracting feature maps with richer information, and generate pixel-by-pixel prediction results, which significantly improves the accuracy of cell nucleus segmentation in pathological sections.

[0128] In one embodiment, if Figure 5 As shown, based on the above multi-scale spatial feature enhancement method, the present invention also provides a multi-scale spatial feature enhancement device, including:

[0129] A preprocessing module 100 is used to process the pathological image to be segmented to obtain a target pathological image;

[0130] A multi-scale feature extraction module 200 is configured to input the target pathological image into an image feature extraction network, extract features of different scales in the target pathological image, and obtain feature images of different scale levels;

[0131] The spatial feature enhancement module 300 is used to input the feature images of different scale levels into the self-attention mechanism to perform spatial feature enhancement processing to obtain the final feature map;

[0132] The segmentation result generation module 400 is used to input the final feature map into a baseline model based on a two-stage detector to generate a segmentation result.

[0133] In one embodiment, if Figure 6 As shown, based on the above multi-scale spatial feature enhancement method, the present invention also provides a terminal, including a processor 10 and a memory 20. Figure 6 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0134] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code for installing the terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a multi-scale spatial feature enhancement program 30 is stored on the memory 20, and the multi-scale spatial feature enhancement program 30 can be executed by the processor 10, thereby realizing the multi-scale spatial feature enhancement method in the present application.

[0135] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the multi-scale spatial feature enhancement method.

[0136] In one embodiment, when the processor 10 executes the multi-scale spatial feature enhancement program 30 in the memory 20, the following steps are implemented:

[0137] Processing the pathological image to be segmented to obtain the target pathological image;

[0138] Inputting the target pathological image into an image feature extraction network, extracting features of different scales in the target pathological image, and obtaining feature images of different scale levels;

[0139] The feature images at different scale levels are input into the self-attention mechanism to perform spatial feature enhancement processing to obtain the final feature map;

[0140] The final feature map is input into a baseline model based on a two-stage detector to generate segmentation results.

[0141] The step of processing the pathological image to be segmented to obtain a target pathological image includes:

[0142] receiving a pathological image to be segmented, and scaling the side length of the pathological image to be segmented to a preset range to obtain a scaled image;

[0143] The zoomed image is randomly cropped to generate a target pathological image with a preset resolution.

[0144] Inputting the target pathological image into an image feature extraction network, extracting features of different scales in the target pathological image, and obtaining feature images of different scale levels, including:

[0145] Inputting the target pathological image into an image feature extraction network, wherein the image feature extraction network uses a 50-layer network pre-trained on an image dataset as a basic feature extractor;

[0146] The target pathological image outputs feature images of different scale levels after passing through each stage of the network.

[0147] The resolutions of the feature images at different scale levels are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the target pathological image, and the dimensions are 256, 512, 1024, and 2048, respectively.

[0148] The pre-training formula of the image feature extraction network is expressed as:

[0149] I=f resnet (V);

[0150] Wherein, I is the extracted feature image I={i1,i2,…,i n}, n represents the number of images in the image dataset, f resnet (.) is the image feature extraction network ResNet, and V is the image in the image dataset.

[0151] The feature images at different scale levels are input into the self-attention mechanism to perform spatial feature enhancement processing to obtain the final feature map, including:

[0152] Construct a feature pyramid structure based on feature images at different scale levels;

[0153] Obtaining a high-level feature map and a low-level feature map in the feature pyramid structure;

[0154] Converting and mapping the low-level feature map into a first feature map and a second feature map, and mapping the high-level feature map into a third feature map;

[0155] The coarse-grained information on the low-level feature map is merged into the high-level feature map through the first feature map, the second feature map, and the third feature map to obtain a final feature map.

[0156] Merging the coarse-grained information on the low-level feature map into the high-level feature map through the first feature map, the second feature map, and the third feature map to obtain a final feature map, including:

[0157] Input the second feature map into a global average pooling layer to obtain a weight;

[0158] Multiplying the weight and the third feature map pixel by pixel to obtain weighted information;

[0159] Pass the first feature map through convolution layers with different strides to obtain a low-scale feature map;

[0160] The weighted information and the low-scale feature map are added pixel by pixel and processed through a self-attention mechanism to obtain a final feature map.

[0161] The weighted information and the low-scale feature map are added pixel by pixel and processed through a self-attention mechanism to obtain a final feature map, including:

[0162] Adding the weighted information and the low-scale feature map pixel by pixel to obtain a superimposed feature map, and passing the superimposed feature map through a 1×1 convolution layer to obtain a target feature map;

[0163] The pixels in the target feature map are mapped to a feature space to obtain a final feature map that learns the dependency relationship between pixels.

[0164] The final feature map is input into a baseline model based on a two-stage detector to generate segmentation results, including:

[0165] Inputting the final feature map and the region candidate box belonging to the foreground into the region of interest alignment layer, mapping the region candidate box onto the final feature map to obtain a region feature map;

[0166] Scaling the regional feature maps of different sizes to a preset uniform size to obtain the target region feature map; inputting the target region feature map into two full-volume layers with a dimension of 1024, and performing category prediction of the region of interest through an activation layer, and performing bounding box regression at the same time to obtain the position coordinates of the detection box;

[0167] The target area feature map is input into a feature pyramid network with 4 convolutional layers. According to the position coordinates of the detection box, the target area feature map is upsampled twice, and the instance mask is predicted to obtain a segmentation result.

[0168] Inputting the final feature map and the region candidate box belonging to the foreground into the region of interest alignment layer, mapping the region candidate box onto the final feature map, and obtaining a region feature map, including:

[0169] Input the final feature map and the candidate box of the region belonging to the foreground into the region of interest alignment layer;

[0170] Identifying image coordinates on the final feature map, and converting the image coordinates into feature map coordinates through bilinear interpolation;

[0171] Converting the feature map coordinates into feature coordinates of the region of interest through bilinear interpolation;

[0172] The region candidate frame is mapped onto the final feature map according to the feature coordinates of the region of interest to obtain a region feature map.

[0173] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed to implement the steps of the multi-scale spatial feature enhancement method as described above.

[0174] In summary, the present invention discloses a multi-scale spatial feature enhancement method and device, which includes: processing a pathology image to be segmented to obtain a target pathology image; inputting the target pathology image into an image feature extraction network to extract features of different scales in the target pathology image to obtain feature images of different scale levels; inputting the feature images of different scale levels into a self-attention mechanism to perform spatial feature enhancement processing to obtain a final feature map; and inputting the final feature map into a baseline model based on a two-stage detector to generate a segmentation result. By inputting the feature images of different scale levels into a self-attention mechanism and performing spatial feature enhancement processing, the present invention extracts a feature map with richer information, which can better identify and locate targets on the image, avoids relying solely on pixels on the last feature map, and improves segmentation accuracy.

[0175] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-scale spatial feature enhancement method, characterized in that: include: Processing the pathological image to be segmented to obtain the target pathological image; Inputting the target pathological image into an image feature extraction network, extracting features of different scales in the target pathological image, and obtaining feature images of different scale levels; The feature images at different scale levels are input into the self-attention mechanism to perform spatial feature enhancement processing to obtain the final feature map; Input the final feature map into a baseline model based on a two-stage detector to generate a segmentation result; The feature images at different scale levels are input into the self-attention mechanism to perform spatial feature enhancement processing to obtain the final feature map, including: Construct a feature pyramid structure based on feature images at different scale levels; Obtaining a high-level feature map and a low-level feature map in the feature pyramid structure; Converting and mapping the low-level feature map into a first feature map and a second feature map, and mapping the high-level feature map into a third feature map; Merging the coarse-grained information on the low-level feature map into the high-level feature map through the first feature map, the second feature map, and the third feature map to obtain a final feature map; Merging the coarse-grained information on the low-level feature map into the high-level feature map through the first feature map, the second feature map, and the third feature map to obtain a final feature map, including: Input the second feature map into a global average pooling layer to obtain a weight; Multiplying the weight and the third feature map pixel by pixel to obtain weighted information; Pass the first feature map through convolution layers with different strides to obtain a low-scale feature map; Adding the weighted information and the low-scale feature map pixel by pixel, and processing them through a self-attention mechanism to obtain a final feature map; The final feature map is input into a baseline model based on a two-stage detector to generate segmentation results, including: Input the final feature map and the region candidate box belonging to the foreground into the region of interest Align layer, and map the region candidate box onto the final feature map to obtain a region feature map; Scaling the regional feature maps of different sizes to a preset uniform size to obtain the target region feature map; inputting the target region feature map into two full-volume layers with a dimension of 1024, and performing category prediction of the region of interest through an activation layer, and performing bounding box regression at the same time to obtain the position coordinates of the detection box; The target area feature map is input into a feature pyramid network with 4 convolutional layers. According to the position coordinates of the detection box, the target area feature map is upsampled twice, and the instance mask is predicted to obtain a segmentation result.

2. The multi-scale spatial feature enhancement method according to claim 1, characterized in that: The step of processing the pathological image to be segmented to obtain a target pathological image includes: receiving a pathological image to be segmented, and scaling the side length of the pathological image to be segmented to a preset range to obtain a scaled image; The zoomed image is randomly cropped to generate a target pathological image with a preset resolution.

3. The multi-scale spatial feature enhancement method according to claim 1, characterized in that: Inputting the target pathological image into an image feature extraction network, extracting features of different scales in the target pathological image, and obtaining feature images of different scale levels, including: Inputting the target pathological image into an image feature extraction network, wherein the image feature extraction network uses a 50-layer network pre-trained on an image dataset as a basic feature extractor; The target pathological image outputs feature images of different scale levels after passing through each stage of the network.

4. The multi-scale spatial feature enhancement method according to claim 3, characterized in that: The resolutions of the feature images at different scale levels are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the target pathological image, and the dimensions are 256, 512, 1024, and 2048, respectively.

5. The multi-scale spatial feature enhancement method according to claim 3, characterized in that: The pre-training formula of the image feature extraction network is expressed as: ； Among them, the is the extracted feature image , n represents the number of images in the image dataset, For image feature extraction network ResNet, is an image in the image dataset.

6. The multi-scale spatial feature enhancement method according to claim 1, characterized in that: The weighted information and the low-scale feature map are added pixel by pixel and processed through a self-attention mechanism to obtain a final feature map, including: The weighted information and the low-scale feature map are added pixel by pixel to obtain a superimposed feature map. The convolution layer obtains the target feature map; The pixels in the target feature map are mapped to a feature space to obtain a final feature map that learns the dependency relationship between pixels.

7. The multi-scale spatial feature enhancement method according to claim 1, characterized in that: Inputting the final feature map and the region candidate box belonging to the foreground into the region of interest Align layer, mapping the region candidate box onto the final feature map, and obtaining a region feature map, including: Input the final feature map and the candidate box of the foreground region into the region of interest Align layer; Identifying image coordinates on the final feature map, and converting the image coordinates into feature map coordinates through bilinear interpolation; Converting the feature map coordinates into feature coordinates of the region of interest through bilinear interpolation; The region candidate frame is mapped onto the final feature map according to the feature coordinates of the region of interest to obtain a region feature map.

8. A multi-scale spatial feature enhancement device, characterized in that: The device comprises: A preprocessing module is used to process the pathological image to be segmented to obtain a target pathological image; a multi-scale feature extraction module, configured to input the target pathological image into an image feature extraction network, extract features of different scales in the target pathological image, and obtain feature images of different scale levels; A spatial feature enhancement module is used to input the feature images of different scale levels into the self-attention mechanism to perform spatial feature enhancement processing to obtain the final feature map; A segmentation result generation module, configured to input the final feature map into a baseline model based on a two-stage detector to generate a segmentation result; The feature images at different scale levels are input into the self-attention mechanism to perform spatial feature enhancement processing to obtain the final feature map, including: Construct a feature pyramid structure based on feature images at different scale levels; Obtaining a high-level feature map and a low-level feature map in the feature pyramid structure; Converting and mapping the low-level feature map into a first feature map and a second feature map, and mapping the high-level feature map into a third feature map; Merging the coarse-grained information on the low-level feature map into the high-level feature map through the first feature map, the second feature map, and the third feature map to obtain a final feature map; Merging the coarse-grained information on the low-level feature map into the high-level feature map through the first feature map, the second feature map, and the third feature map to obtain a final feature map, including: Input the second feature map into a global average pooling layer to obtain a weight; Multiplying the weight and the third feature map pixel by pixel to obtain weighted information; Pass the first feature map through convolution layers with different strides to obtain a low-scale feature map; Adding the weighted information and the low-scale feature map pixel by pixel, and processing them through a self-attention mechanism to obtain a final feature map; The final feature map is input into a baseline model based on a two-stage detector to generate segmentation results, including: Input the final feature map and the region candidate box belonging to the foreground into the region of interest Align layer, and map the region candidate box onto the final feature map to obtain a region feature map; Scaling the regional feature maps of different sizes to a preset uniform size to obtain the target region feature map; inputting the target region feature map into two full-volume layers with a dimension of 1024, and performing category prediction of the region of interest through an activation layer, and performing bounding box regression at the same time to obtain the position coordinates of the detection box; The target area feature map is input into a feature pyramid network with 4 convolutional layers. According to the position coordinates of the detection box, the target area feature map is upsampled twice, and the instance mask is predicted to obtain a segmentation result.

9. A terminal, characterized in that: include: A memory, a processor, and a multi-scale spatial feature enhancement program stored in the memory and executable on the processor, wherein when the multi-scale spatial feature enhancement program is executed by the processor, the steps of the multi-scale spatial feature enhancement method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed to implement the steps of the multi-scale spatial feature enhancement method according to any one of claims 1 to 7.

Images