Image segmentation method and system based on residual reverse bottleneck and sparse attention

Abstract

The present invention relates to the field of image segmentation technology, and discloses an image segmentation method and system based on residual reverse bottleneck and sparse attention to improve the overall performance of segmentation. The method comprises: for the acquired medical image, both the encoder and the decoder use ESA modules and modules to perform corresponding downsampling and upsampling processing stage by stage, any RIB module is provided with two parallel connection paths, which are respectively a direct connection path for inserting a step convolution or a transposed convolution in a deep convolution layer to adjust the spatial resolution, and an indirect connection path for deep feature extraction and channel number adjustment, and the output feature map of each RIB module is obtained by adding the feature maps of the two parallel connection paths; any ESA module determines the attention relationship between regions by constructing a directed graph, obtains the routing index matrix of other regions that each region pays attention to, and performs operations and merges the routing regions in the routing index matrix.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to an image segmentation method and system based on residual reverse bottleneck and sparse attention. Background Art

[0002] Image segmentation is a core task in the field of computer vision, and its goal is to divide the input image into multiple regions, which usually correspond to actual objects or parts of interest in the image. In the image segmentation task, the computer system receives an image as input and then uses specific algorithms and models to mark the various regions or objects in the image. The result of image segmentation is usually a set of regions, each of which consists of a set of pixels that are similar or related in some sense, such as color, brightness, texture or depth. These regions can be further used for tasks such as object recognition, tracking and analysis. In order to achieve accurate image segmentation, it is necessary to select appropriate feature extraction methods, suitable segmentation models and sufficient training data.

[0003] Medical image segmentation refers to the precise regional division of medical images through computer vision and machine learning technology to assist doctors in diagnosing and treating diseases. The goal of this technology is to identify and mark specific structures or lesion areas in the image, such as organs, tumors, blood vessels, etc., to provide more accurate quantitative information and more intuitive visual effects. In the task of medical image segmentation, a series of preprocessing techniques and models, such as image enhancement, feature extraction, and deep learning models, are usually used to extract useful features from medical images and perform accurate segmentation of lesion areas. Medical image segmentation has important practical significance for doctors. It can help doctors accurately identify the location of lesions, quantify the size of lesions, evaluate disease progression and treatment effects, and thus provide patients with more accurate and personalized treatment plans.

[0004] Although traditional image segmentation methods can be applied to medical image segmentation, in medical image segmentation, the lesion areas and structures in medical images usually have complex shapes, sizes and textures. At the same time, factors such as noise, artifacts in the image and differences in various imaging devices may affect the accuracy of segmentation. For example, some lesions may be very small or irregular in shape, and traditional threshold-based or edge-based segmentation methods may not be able to accurately identify these lesions. In addition, since various noises and artifacts may be introduced in the process of acquiring medical images, these noises and artifacts may interfere with the segmentation algorithm and cause errors in the segmentation results. Summary of the invention

[0005] The present invention aims to disclose an image segmentation method and system based on residual reverse bottleneck and sparse attention to improve the overall performance of segmentation.

[0006] To achieve the above object, the method disclosed in the present invention comprises:

[0007] The feature encoder obtains the original image of the medical image to be detected, and generates feature maps of different scales corresponding to different stages through linear embedding layers and downsampling;

[0008] The feature decoder fuses the feature maps of different scales until they are upsampled to the original size of the medical image, and then processes them through a linear projection layer to output pixel-level segmentation predictions;

[0009] The feature encoder and the feature decoder both use an ESA (Efficient Sparse Attention) module and a RIB (Residual Inverted Bottleneck) module to perform downsampling and upsampling processing in stages, and any of the RIB modules is provided with two parallel connection paths, which are a direct connection path for inserting a strided convolution or a transposed convolution in a deep convolution layer to adjust the spatial resolution and an indirect connection path for deep feature extraction and channel number adjustment, and the output feature map of each of the RIB modules is obtained by adding the feature maps of the two parallel connection paths;

[0010] Any of the ESA modules first evenly divides the input feature map into at least four non-overlapping regions, then constructs a regional attention directed graph by calculating the attention weights between regions, and obtains sparse inter-regional connection relationships by top-k sampling of the directed graph, and finally selectively aggregates the regional features according to the sparse connection relationships to obtain an output feature map.

[0011] Preferably, in the direct connection path of the RIB module, the depth convolution layer into which the strided convolution or the transposed convolution is inserted adopts a 1x1 convolution layer with a stride of 2.

[0012] Preferably, in the indirect connection path of the RIB module, the input feature map first passes through a 3 x 3 depth convolution layer with a stride of 2, and then is normalized; then, the feature map passes through a 1x1 convolution layer to compress or expand the number of channels; then, the feature map passes through a nonlinear activation function to perform a nonlinear transformation; finally, the feature map passes through another 1x1 convolution layer to compress or expand the number of channels.

[0013] Preferably, the processing of the ESA module includes:

[0014] Assume that the input feature map is , the algorithm divides the feature map into non-overlapping regions, each of which contains feature vectors, Reshape ;in, is the number of image rows in the input feature map before reshaping, is the number of image columns of the input feature map before reshaping, is the number of channels of the feature map;

[0015] Transformation by linear projection , generate query, key, and value tensors ,get:

[0016] ;

[0017] in, are the projection weights of the query, key, and value tensors respectively;

[0018] Construct a directed graph to determine the attention relationship between regions. First, and The average is performed to obtain the region-level query and key, which is expressed as Then, by and transposed Perform matrix multiplication to obtain the adjacency matrix of the region-to-region affinity graph ,as follows:

[0019] ;

[0020] By retaining the previous connections to prune the affinity graph and Perform top-k operations in the row direction to obtain a routing index matrix ,Right now:

[0021] ;

[0022] Among them, the routing index matrix No. The line contains the The most relevant areas The index of the region, is a pre-set hyperparameter;

[0023] In the present invention, a gather operation is used according to the routing index matrix From the key tensor Sum value tensor Get the attention information of each region in For each query token in , look at all Indexed The key-value pairs in the union of the routing zones are combined into a new tensor ;Right now:

[0024] ;

[0025] in, The gather operation is to obtain the elements at the corresponding positions in the data tensor according to the index tensor and combine these elements into a new tensor, thus providing an efficient, index-based data aggregation method.

[0026] For each region, compute the query tensor All patches in this area and the collected key-value pairs The attention weights are then used to collect the values Perform weighted summation to obtain the output features of the area.

[0027] To achieve the above objectives, the present invention also discloses an image segmentation system based on residual reverse bottleneck and sparse attention, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0028] The present invention has the following beneficial effects:

[0029] On the one hand, by introducing a residual reverse bottleneck module to replace the conventional upsampling and downsampling blocks, the network retains rich semantic information at a lower spatial resolution.

[0030] On the other hand, the sparse attention module achieves more flexible computation distribution through hierarchical routing, exploiting sparsity to save computation and memory.

[0031] Moreover, based on the combination of the above two strategies, it not only effectively reduces the complexity of the model, but also ensures precise focus on the lesion area, thereby improving the algorithm's ability to segment medical images.

[0032] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0034] Figure 1 It is a network framework diagram of a medical image segmentation method based on residual reverse bottleneck and sparse attention disclosed in an embodiment of the present invention;

[0035] Figure 2 A network framework diagram of sparse attention disclosed in an embodiment of the present invention;

[0036] Figure 3 This is a network framework diagram of the residual reverse bottleneck module disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0038] Example 1

[0039] This embodiment uses an improved efficient sparse attention module based on Swin-Unet (Swin Unet, a Transformer-based deep learning model for processing visual tasks) pre-trained on ImageNet (a large visual database for visual object recognition software research) as the core of the feature encoder and feature decoder. Figure 1 First, the medical image to be detected (i.e., the original image) is obtained. The input medical image is first divided into non-overlapping blocks of size 4 × 4. After that, the feature dimension of each block is projected to the predetermined dimension by applying a linear embedding layer. The converted feature map is processed in stages by the sub-encoder of the feature encoder to generate feature maps of different scales at different stages. The obtained feature maps are then transmitted to the decoder. The reverse residual bottleneck module used between the feature encoders is down-sampled, which helps to extract higher-level and more abstract features in the medical image. The feature decoder starts from the feature map of the last stage, up-samples it through the designed reverse residual bottleneck module, and superimposes it with the feature map fused with the self-attention multi-scale feature. This process is also carried out in stages (the number of stages is the same as the number of stages of the encoder). After all feature maps are processed and fused, the reverse residual bottleneck module up-samples the feature maps of all scales to the size of the original image, processes them through the linear projection layer, and outputs the pixel-level segmentation prediction, thereby achieving accurate medical image segmentation. In this way, it is possible to focus on local areas (such as lesion areas) and capture more information features, thereby improving the accuracy of medical image segmentation.

[0040] In this embodiment, the linear embedding layer is used to convert the projected features into any dimension, and the linear projection layer is used to predict the final result. The linear embedding layer and the linear projection layer are concepts well known to those skilled in the art and will not be described in detail.

[0041] Reference Figure 3 ,In each stage of feature encoder and feature decoder, there are a RIB module and an ESA module distributed.

[0042] In the RIB module design, the design mainly contains two parallel connection paths, each path performs a series of operations on the input feature map, and then combines the results of the two paths at the end to obtain the final output feature map. This structure is a reverse residual bottleneck structure. The following is a detailed description of the two paths:

[0043] Direct connection path: In this path, the input feature map directly passes through a 1x1 convolution layer with a stride of 2. This convolution layer can be a strided convolution (used to reduce the spatial resolution, that is, reduce the height and width of the feature map), or a transposed convolution (used to increase the spatial resolution, that is, increase the height and width of the feature map). This path is designed to maintain the main information of the input feature map and make necessary adjustments to its spatial resolution.

[0044] Indirect connection path: In this path, the input feature map first passes through a 3×3 deep convolution layer with a stride of 2, and then passes through a GNorm layer for normalization. Next, the feature map passes through a 1×1 convolution layer to compress or expand the number of channels. Then, the feature map passes through a GELU nonlinear activation function for nonlinear transformation. Finally, the feature map passes through another 1×1 convolution layer to compress or expand the number of channels. This path is designed to perform deep feature extraction and channel number adjustment on the input feature map.

[0045] The results of these two paths will meet at the end, that is, through the element-level addition operation, the feature maps of the two paths are added to obtain the final output feature map. This design can ensure that the main information of the input feature map is retained, and at the same time, deeper features can be extracted from the input feature map. This structure is called the reverse residual bottleneck structure. The key is that it combines the transmission of main information with the extraction of deep features, so that the network can effectively extract deep features while maintaining the main information. Among them, the first deep convolution layer of the residual reverse bottleneck module inserts a strided convolution, a transposed convolution, or Figure 3 The depthwise convolution (DW) shown in Figure 1 is used to change the spatial resolution of the feature map.

[0046] This embodiment inserts a strided convolution or a transposed convolution in the first deep convolution layer of the downsampling and upsampling blocks to achieve a change in the spatial resolution of the feature map. Then, in the last compression layer of the upsampling or downsampling block, the channel dimension of the feature map is changed by increasing or decreasing the number of channels. The middle layer contains an overcomplete convolution layer whose output channel number is CR, where R is the expansion ratio, followed by a GELU activation function. A large R value allows the network to expand in width, while a 1×1 convolution kernel limits the growth of computational complexity. The importance of this level is that it decouples the width expansion from the expansion of the receptive field (i.e., the size of the convolution kernel) of the previous layer, which means that the width expansion of the network is no longer limited by the size of the receptive field.

[0047] Reference Figure 2 , in the ESA module (i.e. Figure 1 In the design of the ESRSeg module in , it is assumed that the input feature map is , the algorithm divides this feature map into non-overlapping regions, each of which contains feature vectors, Reshape ;in, is the number of image rows in the input feature map before reshaping, is the number of image columns of the input feature map before reshaping, is the number of channels. Transformed by linear projection , generate query, key, and value tensors Specifically, we get:

[0048] ;

[0049] in, are the projection weights of query, key, and value respectively. The input feature map refers to the intermediate representation in the model processing process, which contains the feature information of the input image at a specific processing stage.

[0050] Next, we design and construct a directed graph to determine the attention relationship between regions. The algorithm first compares and The average is performed to obtain the region-level query and key, which is expressed as ; Thus, by averaging the feature vectors in each region, the information in the region can be aggregated to obtain a vector that summarizes the overall characteristics of the region. This aggregation operation can be regarded as a dimensionality reduction, which is used to compress the local information in the region into a compact representation. Then, by and transposed Perform matrix multiplication to obtain the adjacency matrix of the region-to-region affinity graph ,as follows:

[0051] ;

[0052] In the adjacency matrix In , each element measures the correlation between two regions. Each element in Indicates Regions and The correlation between regions is measured by the product of the query vector and the key vector at the region level. If the query vector and key vector of two regions are very similar (i.e., their product is large), then the two regions are considered to be highly correlated.

[0053] To further optimize the proposed model, the algorithm retains the previous connections to prune the affinity graph. Perform top- Operation to obtain a routing index matrix ,as follows:

[0054] ;

[0055] Among them, the routing index matrix No. The line contains the The most relevant areas The index of the region, is a pre-set hyperparameter, and topkIndex is a function. For the adjacency matrix For each row (corresponding to each region), use the topk operation to find the most relevant The index of the element. That is: Used to return the maximum value of each row of the input matrix By retaining only the most relevant The original fully connected adjacency graph is "pruned" and becomes a sparser graph. This is because some connections are removed from the original fully connected graph, leaving only the most relevant ones. This reduces computational complexity while still preserving the most important information.

[0056] In this way, a directed graph is successfully constructed to find out which other areas each area should pay attention to. The area-to-area routing index matrix has been obtained , we can now implement a fine-grained token-to-token attention mechanism. Specifically, for the region For each query token in , look at all Indexed The key-value pairs in the union of the routing areas are used to gather the index, that is:

[0057] ;

[0058] in, is the collected key and value tensor. Then, we can apply the attention mechanism on the collected key-value pairs: The following are some examples:

[0059] As above, suppose there is a size Input feature map , after evenly dividing the feature map into non-overlapping regions, the size of each region will be Next, the feature vectors in each region are rearranged into a vector. This process can be regarded as "flattening" the feature vectors in each region (similar to the "linear projection transformation" mentioned above), so that each region will contain The length is The rearranged vector is called a "patch" of the region.

[0060] for For each row of and Take all the patches of the corresponding index area and connect them. Assume there are 4 areas, each with 16 patches, k=2. Then and The shape is 4×16×C. The shape of is 4×2. Assume The content is as follows: , which means: for the 0th region, we need to collect the key-value pairs of the 0th and 2nd regions, which will give us 2×16 = 32 patches; for the 1st region, we need to collect the key-value pairs of the 1st and 3rd regions, which will also give us 32 patches. And so on. After the gather operation, we get 4 regions, each with 32 patches.

[0061] In the above example, the gather operation yields and The shape of is 4×(2×16)×C, because each region now has 2×16=32 patches from 2 related regions, each with 16 patches.

[0062] Then, for each region, the query tensor is calculated All patches in this area and the collected key-value pairs The attention weights are then used to collect the values Perform a weighted sum to get the output feature of the region; in other words, apply the attention mechanism on the collected key and value tensors to get: .

[0063] In summary, those skilled in the art can obtain:

[0064] 1. The connection between regions is based on the local region cut from the feature map, rather than directly on the original image.

[0065] 2. After the region information is input into the main part of the model, in the processing of each layer, the region is first projected to the query, key, and value vectors, and then the attention mechanism is applied. These operations are performed on the cut region instead of the original image.

[0066] 3. When constructing the affinity graph between regions, the similarity between the cut regions is calculated, not the similarity between any two positions in the original image. Specifically: Use the region-level query vector and key vector To calculate the affinity matrix ,in and It is obtained by averaging or other aggregation operations in the cut area.

[0067] 4. Based on this affinity matrix, select the most relevant k regions for each region. Again, this selection process is performed at the level of the cut region. The selected k regions may come from different parts of the image, but they all correspond to the cut region, not any arbitrary position in the original image.

[0068] In addition, the following supplementary explanations are provided regarding the principles, functions, and technical implementation of the gather operation:

[0069] Principle: The basic principle of the gather operation is to select specific elements or slices from the input tensor according to the given index. It can be seen as a general indexing operation that allows us to obtain a specific part of the input tensor according to the index provided by another tensor. and Select the relevant area: , the gather operation here will and For indexing. For each row of and Select the regions with corresponding indexes and stitch them together.

[0070] Function: The main function of the gather operation is to select specific elements or slices from the input tensor and combine them into a new tensor based on the given index. The purpose here is to select relevant elements from the value tensor based on the attention weight.

[0071] Technical implementation: First, determine the shape of the output tensor based on the shapes of the input tensor and the index tensor. Then traverse each element of the output tensor and copy the data from the input tensor to the output tensor according to the corresponding index. The process can be expressed in pseudo code as follows:

[0072] for each index in output_tensor:

[0073] output_tensor[index] = input_tensor[index_tensor[index]]

[0074] The gather operation here needs to be based on Provided index, from and Copy the data to the corresponding location of and This process can be highly parallelized and can be done very efficiently using the GPU's indexing and copying operations.

[0075] In summary, the main function of the gather operation is to obtain the elements at the corresponding positions in the data tensor according to the index tensor and combine these elements into a new tensor, thus providing an efficient, index-based data aggregation method.

[0076] In a specific application, the trained network algorithm is used for medical image segmentation, including the following steps:

[0077] First, this application needs to preprocess the original image, including normalization, denoising, contrast enhancement and other steps to improve the accuracy of subsequent segmentation. Then, the preprocessed image is input into the trained network model. This model specifically uses efficient sparse attention and residual reverse bottleneck. Efficient sparse attention can help the model better capture long-distance dependencies in the image, while the residual reverse bottleneck can effectively improve the model's expression ability while reducing computational complexity. Through this model, key features can be automatically extracted from the image. Next, based on these extracted features, the network model will segment the image into multiple regions, each of which includes a category represented by a single or multiple pixels, such as normal tissue, diseased tissue, etc. After the segmentation is completed, this application also needs to perform some post-processing operations, such as removing small connected areas, filling holes, smoothing edges, etc., to further improve the quality of segmentation. Finally, the segmentation results need to be evaluated, usually by comparing with the results of manual segmentation by experts. Commonly used evaluation indicators include Dice coefficient, Jaccard coefficient, accuracy, recall rate, etc. The actual test proves that the method of this embodiment is superior to other traditional methods in many aspects such as segmentation accuracy and timeliness, thereby effectively reducing the complexity of the model and ensuring accurate focus on the lesion area. At the same time, the performance comparison data between other methods can be referred to the following Table 1:

[0078]

[0079] As shown in Table 1, the algorithm has a significant advantage in terms of efficiency. Specifically, the algorithm has 12.3M parameters and 11.78G computational complexity, which is significantly lower than most of the comparison methods. This is due to the introduction of the sparse attention mechanism, which can significantly reduce the model complexity while retaining key context information.

[0080] For multi-category semantic segmentation, the IoU (Intersection over Union) of each category is usually calculated separately. mIoU (mean Intersection over Union) is the average of the IoUs of all categories to evaluate the overall segmentation performance. mIoU is used to measure the degree of overlap between the predicted results and the actual situation, that is, to calculate the ratio of the intersection and union of the predicted results and the actual labels.

[0081] Example 2

[0082] Corresponding to the above embodiment, this embodiment discloses an image segmentation system based on residual reverse bottleneck and sparse attention, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following method when executing the computer program:

[0083] Step S1: The feature encoder obtains the original image of the medical image to be detected, and generates feature maps of different scales corresponding to different stages through a linear embedding layer and downsampling.

[0084] Optionally, the pixel size of the original image can be 256×256. In the specific implementation process of this step, the input medical image can be first divided into non-overlapping blocks of size 4×4. After that, by applying a linear embedding layer, the feature dimension of each block is projected to a predetermined dimension, and the converted feature map is processed by multiple efficient sparse attention mechanism units.

[0085] In this step, the main motivation for cutting is to reduce computational complexity. If the similarity between each pair of pixels is calculated directly on the original image, the amount of calculation will be very large, especially for high-resolution images. By cutting the image first and then calculating the affinity at the region level, the number of similarity pairs that need to be calculated can be greatly reduced: only the similarity between regions needs to be calculated, not the similarity between pixels, making the calculation more efficient.

[0086] Among them, the cutting method will affect the resolution of the upsampled feature map. For example, if the original image is cut into 4×4 regions and then the attention mechanism is applied at the region level, the resolution of the resulting feature map will be 1 / 16 of the original resolution. If we change to a 3×3 cutting method, the resolution of the feature map will be 1 / 9 of the original resolution.

[0087] At the same time, the cutting method will also affect the amount of upsampling calculations. Generally speaking, the more areas are cut, the greater the amount of upsampling calculations. For example, a 3×3 cutting method will produce more areas than a 4×4 cutting method, so more areas need to be processed during upsampling.

[0088] Similarly, the cutting method will affect the granularity of downsampling. For example, if we use a 4×4 cutting method and then do average or maximum pooling at the region level, this is equivalent to doing a 4×4 downsampling of the original image. If we change to a 3×3 cutting method, the granularity of downsampling will become 3×3.

[0089] At the same time, different downsampling granularities will affect the model's ability to capture features. Generally speaking, the larger the downsampling granularity, the coarser the features captured by the model, and some detailed information may be lost. On the contrary, the smaller the downsampling granularity, the model can capture finer-grained features.

[0090] In addition, the choice of cutting method will also affect other aspects of the model, such as:

[0091] Computational and memory efficiency: Different cutting methods will result in different numbers of regions, which will affect the computational and memory overhead of the model.

[0092] Receptive field: The cutting method affects the receptive field size of each area, which in turn affects the model's ability to capture global information.

[0093] Step S2: The feature decoder fuses feature maps of different scales until they are upsampled to the original size of the medical image, and then processes them through a linear projection layer to output pixel-level segmentation predictions.

[0094] Among them, the "segmentation" described in this step is a completely different concept from the "cutting" in step S1, which can be understood as classifying each pixel. Preferably, in this embodiment, red, green and blue can be used to distinguish the type of segmentation results of each pixel of the image. These three colors represent true positive (TP), false negative (FN) and false positive (FP), respectively. In CT images, taking the segmentation of the lesion area of ​​the lungs caused by viral pneumonia (wherein the area is composed of pixels, that is, whether each pixel is a lesion pixel of viral pneumonia based on the fused scale features) as an example, the aforementioned three terms come from the evaluation indicators of binary classification tasks, and the specific meanings are as follows:

[0095] True Positive (TP): The model correctly predicted the positive example. In this case, it means that the model correctly identified the areas of viral pneumonia lesions, which are marked in red in the image.

[0096] False Negative (FN): The model incorrectly predicted a negative example. In this case, it means that the model did not identify areas of viral pneumonia lesions, which actually exist. These areas are marked in green in the image.

[0097] False Positive (FP): The model incorrectly predicted a positive example. In this case, it means that the model incorrectly identified healthy lung tissue or other areas of non-viral pneumonia lesions as viral pneumonia lesions. These areas are marked in blue in the image.

[0098] Among them, the encoder and the decoder described in this embodiment both adopt the ESA module and the RIB module to perform downsampling and upsampling processing in stages, and any of the RIB modules is provided with two parallel connection paths, namely, a direct connection path for inserting strided convolution or transposed convolution in the deep convolution layer to adjust the spatial resolution, and an indirect connection path for deep feature extraction and channel number adjustment, and the output feature map of each RIB module is obtained by adding the feature maps of the two parallel connection paths.

[0099] In this embodiment, any of the ESA modules determines the attention relationship between regions by constructing a directed graph, obtains the routing index matrix of other regions that each region pays attention to, and performs operations to merge the routing regions within the routing index matrix. In other words, any of the ESA modules first evenly divides the input feature map into at least four (usually N2, N is a positive integer greater than or equal to 2) non-overlapping regions, then constructs a regional attention directed graph by calculating the attention weights between regions, and obtains a sparse connection relationship between regions by top-k sampling of the directed graph, and finally selectively aggregates the regional features according to the sparse connection relationship to obtain an output feature map.

[0100] Preferably, in this embodiment, in the direct connection path of the RIB module, the depth convolution layer in which the strided convolution or the transposed convolution is inserted adopts a 1×1 convolution layer with a stride of 2.

[0101] Preferably, in the indirect connection path of the RIB module of this embodiment, the input feature map first passes through a 3×3 depth convolution layer with a stride of 2, and then is normalized; then, the feature map passes through a 1×1 convolution layer to compress or expand the number of channels; then, the feature map passes through a nonlinear activation function to perform a nonlinear transformation; finally, the feature map passes through another 1×1 convolution layer to compress or expand the number of channels.

[0102] The processing process of the ESA module refers to the above embodiment and will not be described in detail.

[0103] In summary, the image segmentation method and system based on residual reverse bottleneck and sparse attention disclosed in the embodiments of the present invention have at least the following beneficial effects:

[0104] On the one hand, by introducing a residual reverse bottleneck module to replace the conventional upsampling and downsampling blocks, the network retains rich semantic information at a lower spatial resolution.

[0105] On the other hand, the sparse attention module achieves more flexible computation distribution through hierarchical routing, exploiting sparsity to save computation and memory.

[0106] Moreover, based on the combination of the above two strategies, it not only effectively reduces the complexity of the model, but also ensures precise focus on the lesion area, thereby improving the algorithm's ability to segment medical images.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image segmentation method based on residual reverse bottleneck and sparse attention, characterized in that: include: The feature encoder obtains the original image of the medical image to be detected, and generates feature maps of different scales corresponding to different stages through linear embedding layers and downsampling; The feature decoder fuses the feature maps of different scales until they are upsampled to the original size of the medical image, and then processes them through a linear projection layer to output pixel-level segmentation predictions; Wherein, the feature encoder and the feature decoder both use the ESA module and the RIB module to perform downsampling and upsampling processing in stages, and any of the RIB modules is provided with two parallel connection paths, which are a direct connection path for inserting a strided convolution or a transposed convolution in a deep convolution layer to adjust the spatial resolution and an indirect connection path for deep feature extraction and channel number adjustment, and the output feature map of each of the RIB modules is obtained by adding the feature maps of the two parallel connection paths; Any of the ESA modules first evenly divides the input feature map into at least four non-overlapping regions, then constructs a regional attention directed graph by calculating the attention weights between the regions, obtains a sparse inter-regional connection relationship by top-k sampling of the directed graph, and finally selectively aggregates the regional features according to the sparse connection relationship to obtain an output feature map; The processing of the ESA module includes: Assume that the input feature map is , the algorithm divides the feature map into non-overlapping regions, each of which contains feature vectors, Reshape ;in, is the number of image rows in the input feature map before reshaping, is the number of image columns of the input feature map before reshaping, is the number of channels of the feature map; Transformation by linear projection , generate query, key, and value tensors ,get: in, are the projection weights of the query, key, and value tensors respectively; Construct a directed graph to determine the attention relationship between regions. First, and The average is performed to obtain the region-level query and key, which is expressed as ; Then, by and transposed Perform matrix multiplication to obtain the adjacency matrix of the region-to-region affinity graph ,as follows: ； By retaining the previous connections to prune the affinity graph and Perform top- Operation to obtain a routing index matrix ,Right now: ； Among them, the routing index matrix No. The line contains the The most relevant areas The index of the region, is a pre-set hyperparameter; For Region For each query token in , look at all Indexed The key-value pairs in the union of the routing areas are used to gather the index, that is: in, are the collected key and value tensors; For each region, compute the query tensor All patches in this area and the collected key-value pairs The attention weights are then used to collect the values Perform weighted summation to obtain the output features of the area.

2. The method according to claim 1, characterized in that: In the direct connection path of the RIB module, the depthwise convolutional layer into which the strided convolution or transposed convolution is inserted adopts a 1×1 convolutional layer with a stride of 2.

3. The method according to claim 1, characterized in that: In the indirect connection path of the RIB module, the input feature map first passes through a 3×3 depth convolution layer with a stride of 2 and then is normalized; then, the feature map passes through a 1×1 convolution layer to compress or expand the number of channels; then, the feature map passes through a nonlinear activation function to perform a nonlinear transformation; finally, the feature map passes through another 1×1 convolution layer to compress or expand the number of channels.

4. An image segmentation system based on residual reverse bottleneck and sparse attention, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

Images