Image segmentation method and system based on local attention and detail enhancement

Abstract

The invention relates to an image segmentation method and system based on local attention and detail enhancement, and belongs to the field of image segmentation. The method comprises the following steps: inputting a target image into an encoder path for low-level feature extraction and resolution compression; performing advanced feature extraction on the features extracted by the encoder path through a decoder path, and gradually recovering the spatial resolution of the image; combining the high-level features extracted by each decoder stage with the low-level features extracted by the corresponding encoder stage by adopting an attention module and jump connection; 1 * 1 convolution is applied to the final feature combining the decoding and coding paths to generate a predicted binary image. Experiments carried out on two public and available data sets, namely INbreast and CBIS-DDSM, and a private data set, namely In-home, show that the method disclosed by the invention has relatively good performance, and the generalization of the model is improved.

Description

Technical Field

[0001] The invention relates to an image segmentation method and system based on local attention and detail enhancement, and belongs to the field of image segmentation. Background Art

[0002] The application of image segmentation technology in the field of emergency safety, especially in smoke recognition, has become one of the key technologies for ensuring public safety and improving emergency response capabilities. With the increasing demand for real-time monitoring and safety warnings in modern society, smoke, as an early signal of fire or other disasters, plays a vital role in timely warning and reducing disaster losses. Traditional smoke recognition methods rely on smoke sensors or manual monitoring, but these methods have some limitations, such as insufficient sensor sensitivity, high false alarm rate, and difficulty in coping with complex environmental changes. Although manual monitoring has high accuracy, it is difficult to effectively achieve large-scale real-time monitoring due to factors such as personnel fatigue and emotional fluctuations.

[0003] In recent years, with the rapid development of computer vision and deep learning technology, smoke recognition technology based on image segmentation has gradually become a focus of research and application. The visual characteristics of smoke are usually blurred, dynamically changing and transparent, which makes it difficult for traditional image processing methods to effectively extract its features. Image segmentation methods based on deep learning, especially convolutional neural networks (CNNs) and attention mechanisms, can adaptively learn and extract complex visual features, thereby improving the accuracy and robustness of smoke recognition. By training on a large amount of smoke image data, deep learning models can identify the texture, color changes and morphological characteristics of smoke, thereby providing timely warning information at the early stage of a fire.

[0004] At present, the application of image segmentation methods in smoke recognition faces some challenges. First, the visual characteristics of smoke will be affected by different lighting, environment and viewing angles. How to improve the adaptability and accuracy of the algorithm, especially in complex scenes, is still a difficult point of research. Secondly, how to improve the real-time and computational efficiency of the smoke recognition algorithm to meet the needs of large-scale monitoring systems is also an urgent problem to be solved. In addition, multimodal data fusion combining smoke recognition technology with other sensor data (such as temperature, humidity, gas concentration, etc.) can further improve the recognition accuracy and system robustness, and become an important direction for future development.

[0005] In addition to the field of emergency safety, image segmentation technology has also been gradually applied to smart healthcare, smart parks, smart cities and other fields. In the field of smart healthcare, accurate image segmentation methods can effectively assist doctors in extracting lesion areas in images and improve diagnostic efficiency and accuracy. For example, early diagnosis of breast cancer and tumor segmentation have important applications in medical image processing. Traditional breast cancer screening methods, such as mammography, are inexpensive and easy to operate, but the accurate segmentation of tumors still relies on manual judgment. Using image segmentation technology, the tumor area can be automatically extracted, reducing the interference of human factors and improving the efficiency and accuracy of diagnosis.

[0006] In summary, this image segmentation method has broad application potential in the fields of emergency safety and smart medical care. With the development of deep learning and multi-sensor fusion technology, this technology will be able to provide efficient and intelligent solutions for management and prevention in more fields in the future. Summary of the invention

[0007] In view of the deficiencies of the above-mentioned prior art, the present invention provides an image segmentation method and system based on local attention and detail enhancement. The present invention adopts an asymmetric encoder-decoder architecture and introduces a lightweight local attention module, aiming to early and accurately locate the fire smoke part and breast mass area. The present invention designs a new detail enhancement module to improve and enhance the learning of smoke and mass edge features, and improve the segmentation accuracy of the model. The image segmentation method of the present invention is better and has more accurate segmentation performance.

[0008] The present invention provides an image segmentation method based on local attention and detail enhancement, the method comprising:

[0009] Step 1: Input the target image into the encoder path for low-level feature extraction and resolution compression;

[0010] Step 2: The features extracted by the encoder path are used to extract high-level features through the decoder path, while gradually restoring the spatial resolution of the image;

[0011] Step 3: Use attention modules and skip connections to combine the high-level features extracted in each decoder stage with the low-level features extracted in the corresponding encoder stage.

[0012] Step 4: Apply 1x1 convolution to the final features of the combined decoding and encoding paths to generate a predicted binary image.

[0013] Furthermore, in the Step 1, the encoder path includes five encoder modules, and maximum pooling is applied after the first four modules; the encoder module is used to enhance the edge details of the image through a detail enhancement module (DEFRB).

[0014] Furthermore, the detail enhancement module (DEFRB) includes three convolutional layers, the first convolutional layer uses a 1x1 kernel to perform dimensional expansion without affecting the computational complexity and spatial dimension of the feature map; the second and third convolutional layers are detail enhancement convolutional modules, using a 3x3 kernel and using residual connections after the second convolutional layer; the output Y of the detail enhancement module DEFRB It is expressed as follows:

[0015]

[0016] Among them, x represents the input of the detail enhancement module, δ represents the ReLU function, W1, W2, W3 represent the weights of the first, second, and third convolutional layers respectively, b1, b2, b3 represent the bias of the first, second, and third convolutional layers respectively, and * represents the convolution operation.

[0017] Furthermore, the detail enhancement convolution module includes five parallel convolution layers with 3x3 kernels, namely, vanilla convolution, center difference convolution, angular difference convolution, horizontal difference convolution and vertical difference convolution; multiple convolutions are simplified into one standard convolution by using a reparameterization technique; considering that using multiple parallel convolution layers for feature extraction will increase parameters and inference time, the present invention uses the additivity of convolution layers to simplify multiple convolutions into one standard convolution; according to an effective property: when two-dimensional kernels of the same size produce outputs on the same input with the same stride and padding, adding their outputs is equivalent to summing these kernels at corresponding positions, thereby obtaining an equivalent kernel that produces the same final output, and this method is called a reparameterization technique; in addition, in order to further enhance the fusion effect of different convolution results, the present invention designs a simple strategy to give the detail enhancement convolution module the characteristic of dynamic weight adjustment; the detail enhancement convolution module processing process includes the following steps:

[0018] Step 1.1, the input is convolved five times in parallel to obtain the corresponding output, and then the output is summed to obtain the weight w1 and bias b1; the calculation formula of the weight w1 is as follows (for simplicity, the bias is omitted):

[0019]

[0020] In the formula, w i , i = 1:5, indicating the weights corresponding to five parallel convolutions of different types;

[0021] Step 1.2. Concatenate the obtained weights with the results of four differential convolutions along the channel dimension and use 1x1 convolution to reduce the dimension by a factor of 5 to obtain the weight w2 and bias of the original dimension; the calculation formula for the weight w2 of the original dimension is as follows:

[0022] w2=Conv1(Concat(w1,w2,w3,w4,w5)) (3)

[0023] In the formula, Conv1 represents a 1x1 convolution operation to reduce the channel dimension, and Concat represents connection along the channel dimension;

[0024] Step 1.3, apply the reparameterization technique to add the obtained results respectively, and calculate the weight factor a through sigmoid operation; the calculation formula of the weight factor a is as follows:

[0025] a=δ(Con(x,w1)+Con(x,w2)) (4)

[0026] Where a is the gating weight calculated using the sigmoid function, and Con is the reparameterization technique;

[0027] Step 1.4, apply corresponding weights to the reparameterized results, sum the outputs of these weights, and perform residual operations with the original input x to obtain the final output out; the calculation formula for the final output out is as follows:

[0028] out=x+a·Conv(x,w1)+(1-a)·Conv(x,w2) (5)

[0029] Among them, Conv represents the convolution operation.

[0030] Furthermore, in the Step 2, the decoder path includes four alternating decoder modules and four upsampling modules; the alternating decoder module recovers the feature map through the basic module, and the basic module consists of two consecutive 3x3 convolutions; the output Y of the basic module basic The calculation formula is as follows:

[0031] Y basic =δ(W2*δ(W1*x+b1)+b2) (6).

[0032] Furthermore, the upsampling module adopts an attention-guided density upsampling module to further enhance the fusion of the extracted high- and low-level features; the upsampling module adopts dense upsampling convolution and bilinear upsampling post-convolution layer to the high-level-low-resolution feature F high Upsample and get F duc and F buc ; Perform batch normalization and ReLU activation on the upsampled features; then, pass F duc With low-level features F low The sum is to add the two elements one by one to get the fused feature F sum ; F sumFurther processing: first pass through a convolutional layer and then bilinearly upsample the convolutional layer with another path to generate the feature F buc Splice to get F concat , thus promoting a smoother connection process; finally, channel attention is applied from F concat The intuitive meaning of the sum operation is that it is a simple way to integrate features from different sources while maintaining the same tensor shape (consistent number of channels and spatial dimensions).

[0033] Get F concat The concatenation operation here is done along the channel dimension. Unlike addition, it is directly stacked without superimposing the information. Purpose and significance:

[0034] This design intention is from F high and F low Extract as much important information as possible while reducing information loss.

[0035] F duc and F low The combination of (addition operation) emphasizes the direct fusion of information, while F concat It is hoped that more feature information can be retained before the final output for subsequent operation selection.

[0036] Furthermore, in the Step 3, the attention module adopts a local attention module; the local attention module is applied to the encoding and skip connection stages to achieve early and accurate positioning of the target area of ​​the target image; the local attention (LA) module includes a stripe pooling layer, a 1D convolutional layer with a kernel size of 5 or 7, a group normalization layer, and a sigmoid activation function; the local attention (LA) module processing process includes:

[0037] First, the stripe pooling layer is used to capture the feature vectors along the horizontal and vertical spatial dimensions respectively;

[0038] Subsequently, these feature vectors are subjected to one-dimensional convolution to enhance the position information;

[0039] After this, a group normalization layer normalizes the output of the convolution;

[0040] Finally, the sigmoid function is used to generate accurate position attention predictions.

[0041] Furthermore, in Step 3, the specific steps of the attention module processing process are as follows:

[0042] (1) Local attention uses the stripe pooling method to average pool each channel in the horizontal and vertical spatial dimensions, generating an output representation of channel c with a height of h and a width of w; the calculation formula is as follows:

[0043]

[0044] in, are the horizontal and vertical feature vectors of channel c, and It not only captures the global receptive field, but also encodes precise location information; H and W represent the height and width of the input feature map, respectively, x c (h,i) represents the input feature value of channel c at position (h,i); x c (j,w) represents the input feature value of channel c at position (j,w), i and j are index variables used to traverse the feature map in the horizontal and vertical directions, and h and w are the coordinate indexes of the input feature map in the horizontal and vertical directions;

[0045] (2) In order to effectively utilize these features, the present invention designs a simple processing method; considering the location information embedded and It is a sequential signal within the channel. Compared with two-dimensional convolution, one-dimensional convolution is more suitable for processing these sequential signals. Therefore, the local attention module adopts one-dimensional convolution with a kernel size of 5 or 7 to further enhance its feature extraction ability and enhance the interactive ability of position information embedding. This adjustment ensures that LA accurately recognizes the target area. The traditional method involves using a 7x7 kernel, which provides excellent performance despite being slightly heavier in terms of parameters. In order to strike a balance between performance and the number of parameters, the optimal group size of the one-dimensional convolution is selected as the plane size or the plane size divided by 8 to meet different needs;

[0046] (3) Considering that the group normalization layer can enhance the generalization of the model to a certain extent, the local attention module LA uses the group normalization layer to process the enhanced position information to obtain the representation of the position attention in the horizontal and vertical directions; the representation of the position attention in the horizontal and vertical directions is as follows:

[0047]

[0048] y w =δ(G n (F w (z w ))) (10)

[0049] Among them, y h and w Represent the position attention representation in the horizontal and vertical directions respectively, G n represents the group normalization technique, δ represents the operation of the sigmoid activation function;

[0050] Among them, z h yes Abbreviation of z w yes Abbreviation of F h Indicates processing z by a horizontal 1D convolution operation (kernel size is usually 5 or 7) h The feature obtained later, F w Indicates that z is processed by a 1D convolution operation in the vertical direction w The features obtained later, z h Specifically, the horizontal position information is embedded, which represents the average features of all columns at each height h, z w Specifically, the vertical position information is embedded, which represents the average features of all rows at each width w;

[0051] Finally, the output of the local attention module is derived, denoted as Y:

[0052] Y = x c ×y h ×y w (11)

[0053] Among them, x c Represents the input feature map of channel c, which is a matrix whose element x c (h,i) or x c (j,w) represents the input feature value of channel c at position (h,i) or (j,w).

[0054] The present invention also provides an image segmentation system based on local attention and detail enhancement, the system comprising: a module for executing the above-mentioned image segmentation method based on local attention and detail enhancement.

[0055] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned image segmentation method based on local attention and detail enhancement is implemented.

[0056] The beneficial effects of the present invention are:

[0057] 1. The local attention module LA combines 1D convolution and group normalization techniques to effectively encode position information without sacrificing channel size and improve model generalization;

[0058] 2. The local attention module LA is carefully designed to adapt to various visual tasks, and achieves accurate positioning of the entire region of interest through a simple and efficient architecture. It is applied to the encoding and skip connection stages of the model to achieve early and accurate positioning of the fire smoke part and breast mass area;

[0059] 3. The present invention designs a new detail enhancement module to improve and enhance the learning of smoke and mass edge features, and improve the segmentation accuracy of the model; the image segmentation method of the present invention is better and has more accurate segmentation performance;

[0060] 4. The LADE-Net network model corresponding to the method proposed in the present invention has good performance in segmentation accuracy, which is better than several other segmentation networks and improves the generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a technical flow chart of the present invention;

[0062] Figure 2 It is the overall structure diagram of LADE-Net of the present invention;

[0063] Figure 3 It is a structural diagram of the detail enhancement convolution module of the present invention;

[0064] Figure 4 It is the comparison result of LADE-Net of the present invention and the classic segmentation model in INbreast dataset;

[0065] Figure 5 It is the comparison result of LADE-Net of the present invention and the classic segmentation model in the CBIS-DDSM dataset;

[0066] Figure 6 It is the comparison result of LADE-Net of the present invention and the classic segmentation model in the In-house dataset;

[0067] Figure 7 It is the ablation result of LADE-Net of the present invention on the INbreast dataset;

[0068] Figure 8 is the ablation result of LADE-Net of the present invention on the CBIS-DDSM dataset;

[0069] Fig. 9 This is the ablation result of LADE-Net of the present invention on the In-house dataset. DETAILED DESCRIPTION

[0070] Example 1: Figure 1-Figure 9 As shown, an image segmentation method based on local attention and detail enhancement, the method comprising:

[0071] Step 1: Input the target mammographic image into the encoder path for low-level feature extraction and resolution compression;

[0072] Furthermore, in the Step 1, the encoder path includes five encoder modules, and maximum pooling is applied after the first four modules; the encoder module is used to enhance the edge details of the image through a detail enhancement module (DEFRB).

[0073] Furthermore, the detail enhancement module (DEFRB) includes three convolutional layers. The first convolutional layer uses a 1x1 kernel to perform dimensionality expansion without affecting the computational complexity and spatial dimension of the feature map. The second and third convolutional layers are detail enhancement convolutional modules, using a 3x3 kernel and a residual connection after the second convolutional layer; the output Y of the detail enhancement module DEFRB It is expressed as follows:

[0074]

[0075] Among them, x represents the input of the detail enhancement module, δ represents the ReLU function, W1, W2, W3 represent the weights of the first, second, and third convolutional layers respectively, b1, b2, b3 represent the bias of the first, second, and third convolutional layers respectively, and * represents the convolution operation.

[0076] Furthermore, the detail enhancement convolution module includes five parallel convolution layers with 3x3 kernels, namely, vanilla convolution, center difference convolution, angular difference convolution, horizontal difference convolution and vertical difference convolution; multiple convolutions are simplified into one standard convolution by using a reparameterization technique; considering that using multiple parallel convolution layers for feature extraction will increase parameters and inference time, the present invention uses the additivity of convolution layers to simplify multiple convolutions into one standard convolution; according to an effective property: when two-dimensional kernels of the same size produce outputs on the same input with the same stride and padding, adding their outputs is equivalent to summing these kernels at corresponding positions, thereby obtaining an equivalent kernel that produces the same final output, and this method is called a reparameterization technique; in addition, in order to further enhance the fusion effect of different convolution results, the present invention designs a simple strategy to give the detail enhancement convolution module the characteristic of dynamic weight adjustment; the detail enhancement convolution module processing process includes the following steps:

[0077] Step 1.1, the input is convolved five times in parallel to obtain the corresponding output, and then the output is summed to obtain the weight w1 and bias b1; the calculation formula of the weight w1 is as follows (for simplicity, the bias is omitted):

[0078]

[0079] In the formula, w i , i = 1:5, indicating the weights corresponding to five parallel convolutions of different types;

[0080] Step 1.2. Concatenate the obtained weights with the results of four differential convolutions along the channel dimension and use 1x1 convolution to reduce the dimension by a factor of 5 to obtain the weight w2 and bias of the original dimension; the calculation formula for the weight w2 of the original dimension is as follows:

[0081] w2=Conv1(Concat(w1,w2,w3,w4,w5)) (3)

[0082] In the formula, Conv1 represents a 1x1 convolution operation to reduce the channel dimension, and Concat represents connection along the channel dimension;

[0083] Step 1.3, apply the reparameterization technique to add the obtained results respectively, and calculate the weight factor a through sigmoid operation; the calculation formula of the weight factor a is as follows:

[0084] a=δ(Con(x,w1)+Con(x,w2)) (4)

[0085] Where a is the gating weight calculated using the sigmoid function, and Con is the reparameterization technique;

[0086] Step 1.4, apply corresponding weights to the reparameterized results, sum the outputs of these weights, and perform residual operations with the original input x to obtain the final output out; the calculation formula for the final output out is as follows:

[0087] out=x+a·Conv(x,w1)+(1-a)·Conv(x,w2) (5)

[0088] Among them, Conv represents the convolution operation.

[0089] Step 2: The features extracted by the encoder path are used to extract high-level features through the decoder path, while gradually restoring the spatial resolution of the image;

[0090] Furthermore, in the Step 2, the decoder path includes four alternating decoder modules and four upsampling modules; the alternating decoder module recovers the feature map through the basic module, and the basic module consists of two consecutive 3x3 convolutions; the output Y of the basic module basic The calculation formula is as follows:

[0091] Y basic =δ(W2*δ(W1*x+b1)+b2) (6).

[0092] Furthermore, the upsampling module adopts an attention-guided density upsampling module to further enhance the fusion of the extracted high- and low-level features; the upsampling module adopts dense upsampling convolution and bilinear upsampling post-convolution layer to the high-level-low-resolution feature Fhigh Upsample and get F duc and F buc ; Perform batch normalization and ReLU activation on the upsampled features; then, pass F duc With low-level features F low The sum is to add the two elements one by one to get the fused feature F sum ; F sum Further processing: first pass through a convolutional layer and then bilinearly upsample the convolutional layer with another path to generate the feature F buc Splice to get F concat , thus promoting a smoother connection process; finally, channel attention is applied from F concat The intuitive meaning of the addition operation is that it is a simple way to integrate features from different sources while keeping the same tensor shape (consistent number of channels and spatial dimensions).

[0093] Get F concat The concatenation operation here is done along the channel dimension. Unlike addition, it is directly stacked without superimposing the information. Purpose and significance:

[0094] This design intention is from F high and F low Extract as much important information as possible while reducing information loss.

[0095] F duc and F low The combination of (addition operation) emphasizes the direct fusion of information, while F concat It is hoped that more feature information can be retained before the final output for subsequent operation selection.

[0096] Step 3: Use attention modules and skip connections to combine the high-level features extracted in each decoder stage with the low-level features extracted in the corresponding encoder stage.

[0097] Furthermore, in the Step 3, the attention module adopts a local attention module; the local attention module is applied to the encoding and skip connection stages to achieve early and accurate positioning of the breast mass area; the local attention (LA) module includes a stripe pooling layer, a 1D convolutional layer with a kernel size of 5 or 7, a group normalization layer, and a sigmoid activation function; the local attention (LA) module processing process includes:

[0098] First, the stripe pooling layer is used to capture the feature vectors along the horizontal and vertical spatial dimensions respectively;

[0099] Subsequently, these feature vectors are subjected to one-dimensional convolution to enhance the position information;

[0100] After this, a group normalization layer normalizes the output of the convolution;

[0101] Finally, the sigmoid function is used to generate accurate position attention predictions.

[0102] Furthermore, in Step 3, the specific steps of the attention module processing process are as follows:

[0103] (1) Local attention uses the stripe pooling method to average pool each channel in the horizontal and vertical spatial dimensions, generating an output representation of channel c with a height of h and a width of w; the calculation formula is as follows:

[0104]

[0105] in, are the horizontal and vertical feature vectors of channel c, and It not only captures the global receptive field, but also encodes precise location information; H and W represent the height and width of the input feature map, respectively, x c (h,i) represents the input feature value of channel c at position (h,i); x c (j,w) represents the input feature value of channel c at position (j,w), i and j are index variables used to traverse the feature map in the horizontal and vertical directions, and h and w are the coordinate indexes of the input feature map in the horizontal and vertical directions;

[0106] (2) In order to effectively utilize these features, the present invention designs a simple processing method; considering the location information embedded and It is a sequential signal within the channel. Compared with two-dimensional convolution, one-dimensional convolution is more suitable for processing these sequential signals. Therefore, the local attention module adopts one-dimensional convolution with a kernel size of 5 or 7 to further enhance its feature extraction ability and enhance the interactive ability of position information embedding. This adjustment ensures that LA accurately recognizes the target area. The traditional method involves using a 7x7 kernel, which provides excellent performance despite being slightly heavier in terms of parameters. In order to strike a balance between performance and the number of parameters, the optimal group size of the one-dimensional convolution is selected as the plane size or the plane size divided by 8 to meet different needs;

[0107] (3) Considering that the group normalization layer can enhance the generalization of the model to a certain extent, the local attention (LA) module uses the group normalization layer to process the enhanced position information to obtain the representation of the position attention in the horizontal and vertical directions; the representation of the position attention in the horizontal and vertical directions is as follows:

[0108]

[0109] y w =δ(G n (F w (z w ))) (10)

[0110] Among them, y h and w Represent the position attention representation in the horizontal and vertical directions respectively, G n represents the group normalization technique, δ represents the operation of the sigmoid activation function;

[0111] Among them, z h yes Abbreviation of z w yes Abbreviation of F h Indicates processing z by a horizontal 1D convolution operation (kernel size is usually 5 or 7) h The feature obtained later, F w Indicates that z is processed by a 1D convolution operation in the vertical direction w The features obtained later, z h Specifically, the horizontal position information is embedded, which represents the average features of all columns at each height h, z w Specifically, the vertical position information is embedded, which represents the average features of all rows at each width w;

[0112] Finally, the output of the local attention module is derived, denoted as Y:

[0113] Y = x c ×y h ×y w (11)

[0114] Among them, x c Represents the input feature map of channel c, which is a matrix whose element x c (h,i) or x c (j,w) represents the input feature value of channel c at position (h,i) or (j,w).

[0115] LA combines 1D convolution and group normalization techniques to effectively encode location information without sacrificing channel size, thus improving model generalization. Secondly, LA generates four different versions, LA-S, LA-B, LA-S and LA-L, through three carefully designed hyperparameters to adapt to various visual tasks, and achieves precise positioning of the entire region of interest through a concise and efficient architecture. After preliminary experimental verification, the present invention selected the LA-S version and applied it to the encoding and jump connection stages of the model to achieve early and precise positioning of the breast mass area.

[0116] Step 4: Apply 1x1 convolution to the final features of the combined decoding and encoding paths to generate a predicted binary image.

[0117] The present invention also provides an image segmentation system based on local attention and detail enhancement, the system comprising:

[0118] The enhancement module is used to input the target image into the encoder path for low-level feature extraction and resolution compression;

[0119] The recovery module is used to extract high-level features from the features extracted by the encoder path through the decoder path, while gradually restoring the spatial resolution of the image;

[0120] A combining module for combining the high-level features extracted from each decoder stage with the low-level features extracted from the corresponding encoder stage using an attention module and skip connections;

[0121] The generation module is used to apply 1x1 convolution to the final features of the combined decoding and encoding paths to generate a predicted binary image.

[0122] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned image segmentation method based on local attention and detail enhancement is implemented.

[0123] In order to verify the effect of the present invention, the following experiments were performed:

[0124] Dataset preparation:

[0125] The proposed LADE-Net is validated using two publicly available datasets: INbreast, CBIS-DDSM and one private dataset: In-house. For each dataset, the present invention randomly divides it into training set, validation set and test set with a ratio of 8:1:1.

[0126] The INbreast dataset comes from a well-known foreign hospital and consists of digital mammography images with detailed segmentation annotations. It includes 410 images from 115 multi-view cases, showing various types of lesions such as masses, calcifications, asymmetries, and distortions, all of which are well annotated. Among them, 107 images have accurate massive segmentation masks. CBIS-DDSM is a subset of the Digital Database for Mammography Screening (DDSM) established by foreign medical institutions. It is a repository of annotated mammography images for segmentation and classification purposes. It includes 3,071 images distributed in mass and calcification cases, of which 1,591 contain masses, and 1,473 images are selected for use. In-house comes from the medical record database of a tumor hospital affiliated to a medical university. The dataset has been approved by the Ethics Review Committee of a tumor hospital affiliated to a medical university. Since the dataset involves the use of anonymous and retrospective data, the requirement for patient informed consent has been waived. Among them, 214 multi-field digital mammography images were collected, totaling 406 images, including 113 benign masses and 293 malignant masses, of which 203 images had accurate mass segmentation masks.

[0127] Parameters and experiments:

[0128] The code in this invention is implemented in the PyTorch framework, and the experiment uses the RTX 3090 GPU. No pre-trained model is used in the training process. The model is trained using the Adam optimizer with an initial learning rate of 0.0001, a minimum decay rate of 0.000001, a momentum parameter of 0.9, and a cosine annealing scheduler for learning rate decay. The batch size of the training is 4 and stops after 150 times. A combination of binary cross entropy loss, dice loss, and focal loss is used to supervise the training process.

[0129] L=L BCE +L Dice +L Focal

[0130] Where L is the composite loss function used in the training process. BCE , L Dice、 L Focal They are binary cross entropy loss, dice loss, and focal loss respectively.

[0131] Five common indicators are used to evaluate the performance of mass segmentation, including Dice coefficient, mean intersection over union (mIoU), accuracy (Acc), recall (Recall) and precision (Precision).

[0132] First, the proposed LADE-Net is compared with previous tumor segmentation results on two public datasets, INbreast and CBIS-DDSM. All results are cited from the original publications. As shown in Table 1, LADE-Net achieves the highest index on each dataset, greatly surpassing other works, except that the Recall index on the CBIS-DDSM dataset is slightly lower.

[0133] For the INbreast dataset, Dice, mIoU, Recall, Precision, and Acc are 91.69%, 85.70%, 97.47%, 96.99%, and 99.70%, respectively. Compared with the second-best works, there are significant improvements of 16.59%, 20.82%, 4.97%, 22.28%, and 1.05%, respectively. For the CBIS-DDSM dataset, Dice, mIoU, Recall, Precision, and Acc are 79.13%, 70.46%, 80.49%, 77.89%, and 99.62%, respectively. Compared with other top works, Dice, mIoU, Precision, and Acc are improved by 4.61%, 6.42%, 19.33%, and 0.19%, respectively; slightly behind YOLO-LOGO in the Recall indicator.

[0134] Table 1 Comparative experiments with previous tumor segmentation methods on two data sets

[0135]

[0136] Secondly, the LADE-Net segmentation network of the present invention and various classic segmentation models are tested and evaluated on three datasets: INBreast, CBIS-DDSM and In-house. All the results are reproduced by us, as shown in Table 2.

[0137] LADE-Net achieved Dice scores of 91.69%, 79.13%, and 76.48% on the three datasets, respectively, surpassing the most advanced methods. Compared with methods that only focus on a single metric, LADE-Net combines localization and boundary refinement, considers five evaluation metrics including Dice, mIoU, Acc, Recall, and Precision, and conducts a more comprehensive segmentation experimental comparison.

[0138] On the INbreast dataset, LADE-Net achieved the highest scores in all indicators, significantly outperforming other models: Dice, mIoU, recall, precision and accuracy were 91.69%, 85.70%, 97.47%, 96.99% and 99.70%, respectively, which were 4.14%, 5.72%, 13.58%, 4.88% and 0.15% higher than the second-ranked method. On the CBIS-DDSM dataset, LADE-Net's Dice, mIoU, recall, precision and accuracy were 79.13%, 70.46%, 80.49%, 77.89% and 99.62%, respectively, which were 1.17% and 1.15% higher than the second-ranked method in terms of key indicators Dice and mIoU. On the in-house dataset, LADE-Net achieved Dice, mIoU, recall, precision and accuracy of 76.48%, 67.88%, 76.85%, 76.12% and 99.53% respectively. Compared with the second-ranked method, it achieved the best results in Dice, mIoU and recall, with improvements of 3.91%, 2.14% and 2.42% respectively.

[0139] The above experimental results show that LADE-Net has strong robustness and effectiveness. The present invention provides four examples on each dataset, covering the main challenges in breast mass segmentation, such as small mass ratio, blurred boundaries and rare shapes, etc. Figure 4 , Figure 5 , Figure 6 For these challenging scenes, LADE-Net shows good performance in both mass region localization and boundary delineation.

[0140] Table 2 Comparative experiments with classic segmentation models on three datasets

[0141]

[0142] Finally, a series of ablation experiments were conducted to verify the effectiveness of the LA and DEFRB modules in the breast X-ray image mass segmentation task. First, the settings of each experiment were introduced. Default: remove LA and DEFRB from LADE-Net; DE-Net: remove LA from LADE-Net; LA-Net: remove DEFRB from LADE-Net. Removing DEFRB means replacing it with the basic module.

[0143] The experimental results are shown in Table 3. First, it can be observed that the introduction of LA improves the performance of Default, with Dice improved by 3.65% and 0.1% on the INbreast and In-house datasets, respectively. The designed DEFRB improves the Dice score by 0.77% on the INbreast dataset and by 1.06% on the CBIS-DDSM dataset. When LA and DEFRB are used simultaneously, the network performance reaches its peak, with Dice improved by 4.14%, 1.17%, and 3.91% over the baseline on the INbreast, CBIS-DDSM, and In-house datasets, respectively. This shows that LADE-Net can solve the current challenges of breast mass segmentation to a certain extent. The present invention also provides a comparison chart of the ablation experiment, as shown in Figure 7 , Figure 8 , Fig. 9 As shown in the figure, it can be clearly seen that the introduction of LA and DEFRB has improved the baseline to a certain extent in terms of the accuracy of tumor area positioning and tumor edge delineation.

[0144] Table 3 Ablation experiment results

[0145]

[0146]

[0147] The results of a series of comparison and ablation experiments on multiple data sets show that the LADE-Net network proposed in the present invention has better performance in segmentation accuracy, outperforming several other segmentation networks and improving the generalization of the model.

[0148] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0149] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An image segmentation method based on local attention and detail enhancement, characterized in that: The method comprises: Step 1: Input the target image into the encoder path for low-level feature extraction and resolution compression; Step 2: The features extracted by the encoder path are used to extract high-level features through the decoder path, while gradually restoring the spatial resolution of the image; Step 3: Use attention modules and skip connections to combine the high-level features extracted in each decoder stage with the low-level features extracted in the corresponding encoder stage. Step 4: Apply 1x1 convolution to the final features of the combined decoding and encoding paths to generate a predicted binary image.

2. The image segmentation method based on local attention and detail enhancement according to claim 1, characterized in that: In the Step 1, the encoder path includes five encoder modules, and the maximum pooling is applied after the first four modules; the encoder module is used to enhance the edge details of the image through the detail enhancement module.

3. The image segmentation method based on local attention and detail enhancement according to claim 2, characterized in that: The detail enhancement module includes three convolutional layers. The first convolutional layer uses a 1x1 kernel to perform dimensional expansion. The second and third convolutional layers are detail enhancement convolutional modules, using a 3x3 kernel and a residual connection after the second convolutional layer. The output Y of the detail enhancement module DEFRB It is expressed as follows: Among them, x represents the input of the detail enhancement module, δ represents the ReLU function, W1, W2, W3 represent the weights of the first, second, and third convolutional layers respectively, b1, b2, b3 represent the bias of the first, second, and third convolutional layers respectively, and * represents the convolution operation.

4. The image segmentation method based on local attention and detail enhancement according to claim 3, characterized in that: The detail enhancement convolution module includes five parallel convolution layers with 3x3 kernels, namely, vanilla convolution, center difference convolution, angular difference convolution, horizontal difference convolution and vertical difference convolution; multiple convolutions are simplified into one standard convolution by using reparameterization technology; the detail enhancement convolution module processing process includes the following steps: Step 1.1, the input is convolved five times in parallel to get the corresponding output, and then the output is summed to get the weight w1 and bias b1; the calculation formula of weight w1 is as follows: In the formula, w i , i = 1:5, indicating the weights corresponding to five parallel convolutions of different types; Step 1.

2. Concatenate the obtained weights with the results of four differential convolutions along the channel dimension and use 1x1 convolution to reduce the dimension by a factor of 5 to obtain the weight w2 and bias of the original dimension; the calculation formula for the weight w2 of the original dimension is as follows: w2=Conv1(Concat(w1,w2,w3,w4,w5)) (3) In the formula, Conv1 represents a 1x1 convolution operation to reduce the channel dimension, and Concat represents connection along the channel dimension; Step 1.3, apply the reparameterization technique to add the obtained results respectively, and calculate the weight factor a through sigmoid operation; the calculation formula of the weight factor a is as follows: a=δ(Con(x,w1)+Con(x,w2)) (4) Where a is the gating weight calculated using the sigmoid function, and Con is the reparameterization technique; Step 1.4, apply corresponding weights to the reparameterized results, sum the outputs of these weights, and perform residual operations with the original input x to obtain the final output out; the calculation formula for the final output out is as follows: out=x+a·Conv(x,w1)+(1-a)·Conv(x,w2) (5) Among them, Conv represents the convolution operation.

5. The image segmentation method based on local attention and detail enhancement according to claim 1, characterized in that: In Step 2, the decoder path includes four alternating decoder modules and four upsampling modules; the alternating decoder module recovers the feature map through the basic module, and the basic module consists of two consecutive 3x3 convolutions; the output Y of the basic module basic The calculation formula is as follows: Y basic =δ(W2*δ(W1*x+b1)+b2) (6)。 6. The image segmentation method based on local attention and detail enhancement according to claim 5, characterized in that: The upsampling module adopts an attention-guided density upsampling module to further enhance the fusion of the extracted high- and low-level features; the upsampling module uses dense upsampling convolution and bilinear upsampling post-convolution layers to ensemble the high-level and low-resolution features F. high Upsample and get F duc and F buc ; Perform batch normalization and ReLU activation on the upsampled features; then, pass F duc With low-level features F low The sum is the fused feature F sum ; F sum Further processing: first pass through a convolutional layer and then bilinearly upsample the convolutional layer with another path to generate the feature F buc Splice to get F concat , thus promoting a smoother connection process; finally, channel attention is applied from F concat Select Important Information.

7. The image segmentation method based on local attention and detail enhancement according to claim 1, characterized in that: In the Step 3, the attention module adopts a local attention module; the local attention module is applied to the encoding and skip connection stages to achieve early and accurate positioning of the target area of ​​the target image; the local attention module includes a stripe pooling layer, a 1D convolutional layer with a kernel size of 5 or 7, a group normalization layer, and a sigmoid activation function; the local attention module processing process includes: First, the stripe pooling layer is used to capture the feature vectors along the horizontal and vertical spatial dimensions respectively; Subsequently, these feature vectors are subjected to one-dimensional convolution to enhance the position information; After this, a group normalization layer normalizes the output of the convolution; Finally, the sigmoid function is used to generate accurate position attention predictions.

8. The image segmentation method based on local attention and detail enhancement according to claim 1, characterized in that: In Step 3, the specific steps of the attention module processing process are as follows: (1) Local attention uses the stripe pooling method to average pool each channel in the horizontal and vertical spatial dimensions, generating an output representation of channel c with a height of h and a width of w; the calculation formula is as follows: in, are the feature vectors in the horizontal and vertical directions of channel c, H and W represent the height and width of the input feature map respectively, x c (h,i) represents the input feature value of channel c at position (h,i); x c (j,w) represents the input feature value of channel c at position (j,w), i and j are index variables used to traverse the feature map in the horizontal and vertical directions, and h and w are the coordinate indexes of the input feature map in the horizontal and vertical directions; (2) The local attention module uses 1D convolution with a kernel size of 5 or 7 to further enhance its feature extraction capability and enhance the interactive capability of position information embedding; (3) The local attention module uses the group normalization layer to process the enhanced position information to obtain the representation of the position attention in the horizontal and vertical directions; the representation of the position attention in the horizontal and vertical directions is as follows: y w =δ(G n (F w (z w ))) (10) Among them, y h and w Represent the position attention representation in the horizontal and vertical directions respectively, G n represents the group normalization technique, δ represents the operation of the sigmoid activation function; Among them, z h yes Abbreviation of z w yes Abbreviation of F h Indicates processing z by a horizontal 1D convolution operation h The feature obtained later, F w Indicates that z is processed by a 1D convolution operation in the vertical direction w The features obtained later, z h Specifically, the horizontal position information is embedded, which represents the average features of all columns at each height h, z w Specifically, the vertical position information is embedded, which represents the average features of all rows at each width w; Finally, the output of the local attention module is derived, denoted as Y: Y=x c ×y h ×y w (11) Among them, x c Represents the input feature map of channel c, which is a matrix whose element x c (h,i) or x c (j,w) represents the input feature value of channel c at position (h,i) or (j,w).

9. An image segmentation system based on local attention and detail enhancement, characterized in that: The system comprises: a module for executing an image segmentation method based on local attention and detail enhancement as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, an image segmentation method based on local attention and detail enhancement as described in any one of claims 1 to 8 is implemented.

Images