Image processing method, device, equipment and readable storage medium

Abstract

The present application provides an image processing method, apparatus, device, and readable storage medium, which relate to the field of image processing technology and, to a certain extent, solve the problem of inaccurate image segmentation results in existing image processing methods. The method includes obtaining an image to be processed; processing the image to be processed using a trained image segmentation model to obtain a segmented image. In the image segmentation model, M first encoding feature layers and M first decoding feature layers are connected one-to-one via an attention mechanism module, and N second encoding feature layers and N second decoding feature layers are connected one-to-one via a self-attention mechanism module. The attention mechanism module is used to perform feature enhancement processing on the low-level features output by the corresponding first encoding feature layer to obtain target area features; and the self-attention mechanism module is used to extract global context information from the high-level semantic features output by the corresponding second encoding feature layer.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an image processing method, apparatus, device and readable storage medium. Background Art

[0002] Image segmentation technology can divide an image into several specific regions with unique properties and extract the target region from these regions. Image segmentation technology is widely used in fields such as medicine, military, remote sensing, and meteorology. For example, in the medical field, image segmentation technology can be used to segment the subthalamic nucleus and red nucleus in brain magnetic resonance imaging, thereby determining the implantation location of stimulating electrodes in deep brain stimulation (DBS) of the subthalamic nucleus.

[0003] Currently, image segmentation models based on deep learning segmentation networks (U-Nets) are commonly used. The image to be processed is passed through an "encoder-bottleneck layer-decoder" structure, where multiple convolution operations such as downsampling and upsampling are performed. This extracts low-level features and high-level semantic features from the image to be processed, and outputs a segmentation result based on the extracted features. However, in existing image segmentation models, the low-level and high-level semantic features extracted by the encoder often suffer from information loss, resulting in deviations in the semantic information extraction of the image to be processed and insufficient correlation between different parts. Due to this, the decoder further amplifies this deviation during the decoding process. Furthermore, the insufficient correlation between different parts of the image significantly affects blurred targets, limiting the segmentation performance of the image segmentation model. This is especially true when segmenting images of small objects with variable shapes and blurred boundaries. False positive areas are common, leading to inaccurate image segmentation results. Summary of the Invention

[0004] In view of this, embodiments of the present application provide an image processing method, apparatus, device, and readable storage medium to solve the problem of inaccurate image segmentation results in existing image processing methods.

[0005] A first aspect of an embodiment of the present application provides an image processing method, the method comprising: obtaining an image to be processed; processing the image to be processed using a trained image segmentation model to obtain a segmented image; wherein the image segmentation model comprises M first encoding feature layers, N second encoding feature layers, N second decoding feature layers and M first decoding feature layers connected in sequence, M≥1, N≥1; the M first encoding feature layers correspond one-to-one to the M first decoding feature layers, an attention mechanism module is provided between the first encoding feature layer and the corresponding first decoding feature layer, the attention mechanism module is used to perform feature enhancement processing on the low-level features output by the corresponding first encoding feature layer to obtain target area features, and input the target area features into the corresponding first decoding feature layer; the N second encoding feature layers correspond one-to-one to the N second decoding feature layers, a self-attention mechanism module is provided between the second encoding feature layer and the corresponding second decoding feature layer, the self-attention mechanism module is used to extract global context information from the high-level semantic features output by the corresponding second encoding feature layer, and input the global context information into the corresponding second decoding feature layer.

[0006] In combination with the first aspect, in a first possible implementation of the first aspect, the attention mechanism module is an attention gate structure module; and the self-attention mechanism module is a Transformer structure module.

[0007] In combination with the first aspect, in a second possible implementation of the first aspect, inputting the target area feature into the corresponding first decoding feature layer includes: performing a dot product of the target area feature and the input information of the corresponding first decoding feature layer and inputting the resultant data into the first decoding feature layer, where the input information is output information of a previous layer of the first decoding feature layer.

[0008] In combination with the first aspect, in a third possible implementation of the first aspect, the global context information is input into the corresponding second decoding feature layer, including: adding the global context information to the input information of the corresponding second decoding feature layer and inputting the resultant information into the second decoding feature layer, where the input information is the output information of the previous layer of the second decoding feature layer.

[0009] In combination with the first aspect, in a fourth possible implementation manner of the first aspect, the image to be processed includes a brain magnetic resonance image, and the segmented image is an image including segmentation results of the subthalamic nucleus and the red nucleus.

[0010] In combination with the first aspect, in a fifth possible implementation manner of the first aspect, the method further includes: determining the target position coordinates based on the segmented image.

[0011] In combination with the first aspect, in a sixth possible implementation of the first aspect, the image segmentation model is trained in the following manner: obtaining training set images, where the training images are images marked with target areas; inputting the training set images into the image segmentation model to be trained, and training the image segmentation model based on a loss function, where the loss function is determined based on the sum of the cross entropy loss and the Dice loss.

[0012] A second aspect of an embodiment of the present application provides an image processing device, which includes: an acquisition unit for acquiring an image to be processed; a processing unit for processing the image to be processed through a trained image segmentation model to obtain a segmented image; wherein the image segmentation model includes M first encoding feature layers, N second encoding feature layers, N second decoding feature layers and M first decoding feature layers connected in sequence, M≥1, N≥1; the M first encoding feature layers correspond one-to-one to the M first decoding feature layers, and an attention mechanism module is provided between the first encoding feature layer and the corresponding first decoding feature layer, and the attention mechanism module is used to perform feature enhancement processing on the low-level features output by the corresponding first encoding feature layer to obtain target area features, and input the target area features into the corresponding first decoding feature layer; the N second encoding feature layers correspond one-to-one to the N second decoding feature layers, and a self-attention mechanism module is provided between the second encoding feature layer and the corresponding second decoding feature layer, and the self-attention mechanism module is used to extract global context information from the high-level semantic features output by the corresponding second encoding feature layer, and input the global context information into the corresponding second decoding feature layer.

[0013] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in any one of the first aspects are implemented.

[0014] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0015] Compared with the prior art, the embodiments of the present application have the following advantages: Based on the image processing method, apparatus, device, and readable storage medium provided by the present application, the method segments the image to be processed based on an image segmentation model to obtain a segmented image. The image segmentation model is an encoder-decoder structure, wherein the M first encoding feature layers in the encoder are connected to the M first decoding feature layers in the decoder in a one-to-one correspondence via an attention mechanism module, and the N second encoding feature layers in the encoder are connected to the N second decoding feature layers in the decoder via a self-attention mechanism module. The attention mechanism module is used to perform feature enhancement processing on the low-level features output by the corresponding first encoding feature layer to obtain target region features, and input the target region features into the corresponding first decoding feature layer, so that the decoder generates a first decoding feature map based on the target region features and corresponding input information; the self-attention mechanism module is used to extract global context information from the high-level semantic features output by the corresponding second encoding feature layer, and input the global context information into the corresponding second decoding feature layer, so that the decoder generates a second decoding feature map based on the global context information and corresponding input information. This method can perform targeted processing on features at different levels guided by the hierarchical attention mechanism, thereby improving image segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 Schematic diagram of the segmentation results of the subthalamic nucleus and red nucleus in a brain MRI image provided in an embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of a traditional U-Net-based image segmentation model provided in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of an image segmentation model provided by one embodiment of the present application;

[0020] Figure 4 is a schematic diagram of the processing process of the attention gate structure provided in an embodiment of the present application;

[0021] Figure 5 This is a schematic diagram of the processing process of the Transformer structure provided in an embodiment of the present application;

[0022] Figure 6This is a schematic flow chart of an image segmentation method provided by one embodiment of the present application;

[0023] Figure 7 This is a schematic diagram of the process of obtaining a segmented image through an image segmentation model provided in an embodiment of the present application;

[0024] Figure 8 This is a diagram showing some of the segmentation results provided in the embodiments of the present application;

[0025] Figure 9 Schematic diagram of the target location method provided in the embodiment of the present application;

[0026] Figure 10 Schematic diagram of the positioning process of the target positioning method provided in the embodiment of the present application;

[0027] Figure 11 is a schematic diagram of an image segmentation device provided in an embodiment of the present application;

[0028] Figure 12 Schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0030] The technical solution provided in this application is explained in detail below with reference to specific embodiments.

[0031] Currently, the image segmentation model based on U-Net has been widely used in the field of medical image segmentation, but it has the problem of inaccurate segmentation results. Figure 1 Taking the red nucleus and subthalamic nucleus in the brain (Magnetic Resonance Imaging, MRI) image (a) as an example, the following can be obtained by manual delineation by clinical experts: Figure 1 The annotation results shown in (b) can be obtained by processing the brain MRI image based on the U-Net image segmentation model. Figure 1 The annotation results shown in (c) in Figure 1 show that compared with manual annotation, the annotation results obtained by the U-Net-based image segmentation model contain false positive areas, that is, areas detected by the model as target areas but are actually non-target areas.

[0032] See also Figure 2 The appearance of false positive areas is due to the fact that when performing image segmentation, the image segmentation model based on U-Net usually performs multi-level convolution operations such as downsampling and upsampling on the image to be processed through the "encoder-bottleneck layer-decoder" structure to extract low-level features and high-level semantic features in the image to be processed, and outputs the segmentation result based on the extracted features. However, in this model, the low-level features and high-level semantic features extracted by the encoder usually lose information, resulting in deviations in the extraction of semantic information of the image to be processed and insufficient correlation between the various parts. Based on this, the decoder will continue to amplify this deviation during the decoding process. On the other hand, the insufficient correlation between the various parts of the image has a greater impact on blurred targets, which limits the segmentation performance of the image segmentation model. In particular, when segmenting small targets with variable shapes and blurred boundaries in the image to be processed, the problem of false positive areas is common, resulting in inaccurate image segmentation results.

[0033] Based on this, an embodiment of the present application provides an image processing method. This method is based on an image segmentation model. After obtaining an image to be processed, the image to be processed is processed through an image segmentation model (HAU-Net) equipped with a hierarchical attention mechanism to obtain a segmented image. This method can perform targeted processing on features at different levels (low-level features and high-level semantic features) guided by the hierarchical attention mechanism, thereby improving the segmentation accuracy of the image to be processed. The hierarchical attention mechanism includes: in the image segmentation model, the low-level features and high-level semantic features of the image to be processed are hierarchically processed in the model according to their respective characteristics.

[0034] Figure 3 Schematic diagram of an image segmentation model provided by an embodiment of the present application. Figure 3 As shown in the figure, according to the image processing process, the image segmentation model includes an input end, an encoder, a bottleneck layer, a decoder and an output end in sequence.

[0035] The input end is used to input an image to be processed to the encoder. In one example, the image to be processed is a brain magnetic resonance image.

[0036] The encoder includes M first encoding feature layers (also referred to as M shallow encoding feature layers) and N second encoding feature layers (also referred to as N deep encoding feature layers) that are connected in sequence near the input end of the image segmentation model. In the direction from the input end of the encoder to the bottleneck layer, in the M first encoding feature layers and the N second encoding feature layers, after the respective input information (such as the input image) is downsampled by convolution operations performed in each encoding feature layer, the size of the resulting encoding feature map gradually decreases, thereby outputting encoding feature maps of different sizes (including the first encoding feature map and the second encoding feature map). In one example of the present application, the convolution kernel size of each encoding feature layer is consistent, and illustratively, the size of the convolution kernel is 3*3.

[0037] Exemplarily, in the encoder, a multi-level convolution operation is first performed on the image to be processed through M first coding feature layers to perform dimensionality reduction and downsampling, and low-level features in the image to be processed are extracted, and then a first coding feature map of corresponding size is generated. Taking the image to be processed as a brain MRI image with a size of 512*512 as an example, the size of the first coding feature map output after the brain MRI image passes through the first coding feature layer may be 128*128. The obtained first coding feature map is then further subjected to dimensionality reduction and downsampling through N second coding feature layers to extract high-level semantic features in the feature map, and a high-level semantic feature map of corresponding size is obtained, that is, the second coding feature map output by each second coding feature layer.

[0038] It should be noted that low-level features of the image being processed include meaningful features such as the color, outline, and specific location of the target region in the image being processed. High-level semantic features include the meaning of the target region in the image being processed. They are semantic abstractions of each target region in the image being processed and reflect the neural network's semantic understanding of each target region in the image being processed.

[0039] The bottleneck layer is the connection layer between the encoder and the decoder, see Figure 3 As shown in Figure 1, this is the convolutional layer with the smallest output feature map in the image segmentation model. The bottleneck layer is used to perform a convolution operation on the second encoded feature map obtained by the second encoded feature layer in the encoder, extract high-level semantic features from the second encoded feature map, generate a bottleneck layer feature map, and then input this bottleneck layer feature map into the decoder.

[0040] The decoder includes M first decoding feature layers (also referred to as M shallow decoding feature layers) and N second decoding feature layers (also referred to as N deep decoding feature layers) that are connected in sequence near the output end of the image segmentation model. The N second decoding feature layers are connected to the N second encoding feature layers through a bottleneck layer. In the direction from the bottleneck layer to the output end of the decoder, in the N second decoding feature layers and the M first decoding feature layers, after the respective input information (such as the input image) is upsampled by convolution operations performed by each decoding feature layer in sequence, the size of the resulting decoding feature map gradually increases, thereby outputting decoding feature maps of different sizes (including the first decoding feature map and the second decoding feature map). In one example of the present application, the convolution kernel size of each decoding feature layer is consistent. For example, the size of the convolution kernel is 3*3. It should be noted that, in the decoder, the decoding feature map output by each of the M first decoding feature layers is the same size as the encoding feature map output by each of the corresponding M first encoding feature layers, and the decoding feature map output by each of the N second decoding feature layers is the same size as the encoding feature map output by each of the corresponding N second encoding feature layers.

[0041] For example, see Figure 3 As shown, the image segmentation model in this embodiment follows the U-Net structure as a whole. The encoder of the image segmentation model includes five downsampling operations to form six encoding feature maps of different scales. Correspondingly, the decoder also includes five upsampling operations to form six decoding feature maps of different scales. In this model, the features of the image to be processed processed by the three scale convolution layers close to the input and output ends of the model are regarded as low-level features, and the features of the image to be processed processed by the other three scale convolution layers are regarded as high-level semantic features. The two are used to construct a hierarchical attention mechanism to process different types of features in a hierarchical manner.

[0042] The skip connection includes a first skip connection with a one-to-one correspondence between the M first encoding feature layers and the M first decoding feature layers; and a second skip connection with a one-to-one correspondence between the N second encoding feature layers and the N second decoding feature layers.

[0043] In this embodiment, an attention mechanism module is provided in the first jump connection. The attention mechanism module is used to perform feature enhancement processing on the low-level features output by the corresponding first encoding feature layer to obtain target area features, and input the target area features into the corresponding first decoding feature layer. In the feature enhancement processing provided by this embodiment, since the image segmentation model performs low-level feature extraction, feature contours similar to the feature contours of the target area may exist in other areas of the image to be processed (i.e., areas unrelated to the target area or areas outside the target area). Therefore, when extracting the target area, the segmentation error of the image segmentation model for the target area can be reduced by strengthening the target features in the image to be processed. For example, during the training of the image segmentation model, based on the annotation results of the target area, the weight of the non-target area outside the target area in the image to be processed is reduced in the image segmentation model, so that its influence on the segmentation result is reduced, thereby reducing the segmentation error of the target area.

[0044] In this embodiment, a self-attention mechanism module is provided in the second jump connection, which is used to extract global context information from the high-level semantic features output by the corresponding second encoding feature layer, and input the global context information into the corresponding second decoding feature layer.

[0045] In some embodiments, the attention mechanism includes an attention gate (AG) structure; the self-attention mechanism includes a Transformer structure. In this embodiment, the AG is embedded in the jump connection between the first encoding feature layer and the first decoding feature layer to enhance the target features in the image to be processed. The Transformer structure is embedded in the jump connection between the second encoding feature layer and the second decoding feature layer to extract the global context information of the high-level semantic features in the image to be processed. The image segmentation model with a hierarchical attention mechanism provided in this embodiment utilizes the difference between the AG for pixel-level attention mechanism and the Transformer for the self-attention mechanism for building global context associations, and can effectively mine corresponding valuable information for features with different characteristics.

[0046] In this embodiment, the purpose of using the attention gate structure AG is to perform product weighting on each pixel-level feature of the input image to be processed, so as to achieve the purpose of strengthening the effective features, such as Figure 4 As shown in the figure, the input of AG module With weight Perform pixel-by-pixel multiplication to obtain the weighted output result. The core of AG is to generate attention weights. Figure 4 As shown, enter The feature labels in the decoder of adjacent small scales are ,Will and Apply 1×1×1 convolution operations respectively, then add the convolution results and input the ReLU activation function and Sigmoid function. The obtained weight matrix is ​​resampled by the interpolation algorithm (Resampler) to obtain the same value as the input Scale-consistent attention weights The essence of the ReLU activation function is to output features with eigenvalues ​​greater than 0 as is, and to return features with eigenvalues ​​less than 0 to zero, thereby filtering out features with smaller eigenvalues. The Sigmoid function is a normalization function used to normalize the product features to 0-1, resulting in a probability value of a weighted matrix.

[0047] For example, in the field of medical image processing, the attention gate structure AG can automatically learn to focus on target regions of various shapes and sizes. Image segmentation models incorporating this attention gate structure can highlight specific image feature regions.

[0048] In this embodiment, Figure 5 As shown in the figure, the Transformer structure is a structure based on the self-attention mechanism, which can extract the global contextual relationship information between the features in each image to be processed. The specific implementation method includes: first, the high-level semantic features generated by the encoder are represented as , converting it into a two-dimensional sequence ,in , C represents the number of feature channels, D, H, and W represent the depth, height, and width of the input, respectively. To encode the spatial position of the image to be processed, a learnable parameter matrix with the same shape as E is added to the Transformer structure to represent the positional relationship between the elements in the sequence. This parameter matrix is ​​called position embedding (PE). In the Transformer structure, the position encoding PE is directly added to the two-dimensional sequence E to obtain the final two-dimensional sequence. Then the two-dimensional sequence The extraction of global context information is achieved through the Multi-head Attention module (MSA) and the Multi-layer Perceptron (MLP). ,The MSA module first performs linear projection (Linear Projection) through a multi-layer perceptron, and uses three linear mapping layers to obtain Q, K, V, as shown in formula (1).

[0049] (1)

[0050] In formula (1), are the learnable parameters of the three linear layers, and the self-attention module can be expressed as:

[0051] (2)

[0052] In formula (2), are the i-th row of Z and Q respectively, Represents the attention map, which indicates the similarity between each spatial voxel and other voxels. The higher the similarity, the stronger the connection between the two points. Then, matrix multiplication is performed on V and S to obtain the attention enhancement feature. MSA is an extension of SA, which includes multiple SA operations. The results are combined together and the MSA result is obtained by linear mapping. The process is shown in the following formula (3).

[0053] (3)

[0054] In formula (3), , , m is the number of heads in MSA, the output of MSA will be input into MLP, the whole process can be expressed as follows,

[0055] (4)

[0056] It should be noted that the features in the Transformer structure are normalized by layer norm before being input into MSA and MLP, ultimately obtaining high-level semantic features that can be output by the Transformer structure for feature weighting.

[0057] In this embodiment, the attention mechanism module inputs the target region feature into the corresponding first decoding feature layer, including: performing a dot product between the target region feature and the input information of the corresponding first decoding feature layer and inputting the result into the first decoding feature layer, wherein the input information is the output information of the previous layer of the first decoding feature layer. For example, see Figure 3 As shown, the input information of the first decoding feature layer e is the output information of the first decoding feature layer d; the input information of the first decoding feature layer f is the output information of the first decoding feature layer e.

[0058] In this embodiment, the self-attention mechanism module inputs the global context information into the corresponding second decoding feature layer, including: adding the global context information to the input information of the corresponding second decoding feature layer and then inputting it into the second decoding feature layer, where the input information is the output information of the previous layer of the second decoding feature layer. For example, see Figure 3 As shown, the input information of the second decoding feature layer b is the output information of the first decoding feature layer a; the input information of the second decoding feature layer c is the output information of the first decoding feature layer b.

[0059] In this embodiment, when generating the corresponding second decoding feature map, the second decoding feature layer is generated by merging the global context information extracted by the self-attention mechanism module in the corresponding second jump connection with the input information of the corresponding second decoding feature layer. In other words, the global context relationship of the target area is constructed in the second encoding feature map through the self-attention mechanism module, which enables the decoder to accurately obtain the target features of the target area with blurred boundaries when performing feature merging. When generating the corresponding first decoding feature map, the first decoding feature layer is generated by merging the target area features obtained by the attention mechanism module in the corresponding first jump connection with the input information of the corresponding first decoding feature layer.

[0060] In this embodiment, when the decoder performs feature merging through the "upsampling-feature merging-convolution" operation, it needs to be performed continuously multiple times until the size of the output first decoded feature map is consistent with the size of the input image to be processed.

[0061] In this embodiment, the first decoding feature map output by the decoder is a segmented image generated by the high-level semantic features of the target area determined by the self-attention mechanism module (the meaning of the target area, for example, the target area is the subthalamic nucleus or the red nucleus) and the low-level features of the target area determined by the self-attention mechanism module (the specific position of the target area in the image to be processed).

[0062] The output terminal is configured to perform a convolution operation on the first decoded feature map using a convolution layer with a 1×1 convolution kernel and a softmax function, classify the features in the first decoded feature map, obtain a segmentation result, and output a segmented image. Exemplarily, the segmented image includes the segmentation results of the subthalamic nucleus and the red nucleus.

[0063] Figure 6 This is a schematic flow chart of an image segmentation method provided in one embodiment of the present application, which is applied to electronic devices. Figure 6 As shown, the method includes the following steps S601-S602.

[0064] S601: The electronic device obtains an image to be processed.

[0065] In this embodiment, the images to be processed include all images used for segmentation operations in various fields (such as medicine, military, remote sensing, and meteorology). For example, in the medical field, MRI images of various parts of the human body (such as brain MRI images) obtained by a magnetic resonance imaging device are used.

[0066] In some embodiments, when the electronic device acquires the image to be processed, it can acquire the image through a second device for acquiring images to be segmented. For example, the second device can be a brain magnetic resonance imaging device for acquiring brain MRI images.

[0067] In some embodiments, the electronic device and the second device may be the same device or different devices.

[0068] S602: The electronic device processes the image to be processed through an image segmentation model to obtain a segmented image.

[0069] In this embodiment, the electronic device processes the image to be processed using an image segmentation model to obtain a segmented image based on the target area.

[0070] For example, see Figure 7 As shown in FIG, it is a schematic diagram of the process of an electronic device obtaining a segmented image through an image segmentation model. Figure 7 As shown, the electronic device receives a medical brain MRI image as input. The image segmentation model segments the subthalamic nucleus and red nucleus in the brain MRI image, producing a segmented image. This segmented image clearly highlights the location, shape, and size of the subthalamic nucleus and red nucleus.

[0071] The image segmentation model provided in this application can be applied to various fields such as medical image segmentation, and can also be applied to any technology that requires segmentation of target areas in the image to be processed.

[0072] The following takes the segmentation task of the subthalamic nucleus and red nucleus in brain MRI images in the field of medical image segmentation as an example. Through three parts: (i) selection of training sample sets, (ii) training process of image segmentation models, and (iii) feasibility verification of image segmentation models, the training process and effect of the image segmentation model provided in this application are exemplified.

[0073] (1) Selection of training sample sets

[0074] In this example, brain MRI images of all subjects diagnosed with Parkinson's disease are used as training samples. All images in the training samples are T2 modality images acquired by a 3T MRI scanner, with a slice thickness of 2 mm and a resolution of , the data size is . The subthalamic nucleus and red nucleus in each training sample image were manually outlined by two radiologists with more than 6 years of experience in neuroradiology. In this embodiment, a total of 99 MRI image samples and corresponding labels were selected, of which 80 were selected as training sample sets and the remaining 19 were used as test sample sets. Five cross-validations were performed on the training sample set, and the image segmentation model validated each time was used to obtain the segmentation results on the test sample set, and the average results on the test sample set were used to evaluate the performance of the image segmentation model.

[0075] (2) Training process of image segmentation model

[0076] In this embodiment, before training the image segmentation model, all training sample images are resampled to the same spatial resolution and cropped to [192, 192, 48] as the input images of the image segmentation model. During the training of the image segmentation model, data augmentation can be used to expand the data in the training sample set, where the data augmentation methods include random rotation, elastic deformation, Gaussian noise, mirror transformation, and scaling. The angle of random rotation is , the scaling range is (0.85, 1.25).

[0077] During the training phase of the image segmentation model, the sum of cross-entropy loss and Dice loss was used as the loss function, using the Stochastic Gradient Descent (SGD) optimizer with a learning rate of 0.01, momentum of 0.99, and weight decay of 3e-5. For example, the entire training process of this image segmentation model can be implemented in Python, using the PyTorch 1.8.0 framework, and trained and tested on an NVIDIA GeForce GTX 3090 GPU. The training batch size was set to 2, and all models were trained using the nnU-Net framework for 150 epochs, with 250 batches per epoch.

[0078] It should be noted that the method for performing image segmentation using the trained image segmentation model and the method for training the image segmentation model can be performed by the same electronic device or by different electronic devices. The electronic device may include but is not limited to various smartphones, portable notebooks, tablets, smart wearable devices, computers, robots, etc.

[0079] (3) Feasibility Verification of Image Segmentation Model

[0080] In this example, the segmented image results obtained using the image segmentation method provided in this example were compared with those obtained using traditional U-Net, Attention U-Net, R2U-Net, CS2-Net, and a Fully Convolutional Network (FCN). The results were compared on 19 test sample sets. The comparison metrics included the Dice Similarity Coefficient (DSC), Jaccard Coefficient (JA), Sensitivity (SEN), and 95% Hausdorff Distance (HD95). These metrics are used to assess the similarity between the segmentation results of the model network and the standard segmentation results. A larger Dice Coefficient (DSC), Jaccard Coefficient (JA), and Sensitivity (SEN), and a smaller HD95, indicate a higher similarity and better fitting performance.

[0081] The comparison results are shown in Table 1. As shown in Table 1, the image segmentation method provided by this embodiment outperforms other methods in all indicators. Specifically, for the subthalamic nucleus and red nucleus, the Dice coefficients reached 88.20% and 92.36%, respectively, which are 2.94% and 3.20% higher than the baseline method U-Net, respectively. The image segmentation method provided by this embodiment has a greater advantage in the Jaccard coefficient, which is 4.9% and 5.55% higher than the baseline method in the subthalamic nucleus and red nucleus, respectively. Compared with Attention U-Net, the image segmentation method provided by this embodiment also has a performance improvement of 3.57% and 4.75% on the two targets, respectively. These improvements indicate that the HAU-Net proposed in this embodiment has better learning and generalization capabilities for the subthalamic nucleus segmentation task.

[0082] Table 1 Experimental results of different methods

[0083]

[0084] See also Figure 8 The middle part of the segmentation results is shown, and the key and difficult areas of segmentation are marked by borders, such as Figure 8 As shown, the method in this embodiment has higher consistency and greater effectiveness than the annotation by manual segmentation.

[0085] In addition, this embodiment also conducted ablation experiments on the Transformer structure and attention gate structure added to the image segmentation model (HAU-Net) provided in this embodiment to explore their impact on the experimental results. The experimental results are shown in Table 2. Removing the Transformer structure or the attention gate structure from the image segmentation model (HAU-Net) resulted in a decrease in model performance. As shown in Table 2, the Dice coefficient for the subthalamic nucleus and red nucleus decreased by 2.11% / 1.03% and 2.17% / 0.89%, respectively.

[0086] Table 2 Ablation experiment results

[0087]

[0088] It can be seen that by adding the Transformer structure and the attention gate structure to the image segmentation model (HAU-Net) provided in this embodiment, the segmentation performance of the image segmentation model can be effectively improved.

[0089] The image segmentation model (HAU-Net) provided in this embodiment can be applied in the field of medical image segmentation, for example, in target location during electrode implantation for deep brain stimulation (DBS). Based on this, this embodiment also provides a target location method, see Figure 9 As shown, the process includes the following steps S901-S902.

[0090] S901: The electronic device obtains a segmented image.

[0091] In this embodiment, the electronic device obtains a segmented image by inputting an image to be processed (eg, a brain MRI image) into the image segmentation model provided in the above embodiment to perform image segmentation.

[0092] S902: The electronic device determines the target position coordinates based on the segmented image.

[0093] The electronic device measures the position coordinates of the target point in the segmented image and then marks the position coordinates of the target point in the original image (i.e., the brain MRI image). Figure 10 , which is a schematic diagram of the positioning process of the target positioning method provided in this embodiment.

[0094] It should be noted that the electronic device that executes the target positioning method in this embodiment and the electronic device that executes the image segmentation model (HAU-Net) training process and the image segmentation method can be the same electronic device or different electronic devices.

[0095] The image processing method provided in the embodiment of the present application performs targeted processing on features at different levels (low-level features and high-level semantic features) guided by a hierarchical attention mechanism, utilizes an attention gating mechanism and a self-attention-based Transformer structure to improve the extraction efficiency of low-level and high-level features in the neural network model, and more efficiently mines local features and global context information of the neural network model, thereby improving the image segmentation accuracy. The image segmentation model provided in this method can realize automatic feature extraction of the subthalamic nucleus and red nucleus in brain MRI images, realize accurate segmentation of the subthalamic nucleus and red nucleus, and locate the position coordinates of the target point in deep brain stimulation DBS of the subthalamic nucleus based on the segmented image, thereby determining the implantation position of the stimulation electrode, which can improve the efficiency of the operation.

[0096] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0097] Figure 11 A schematic diagram of an image segmentation device provided in an embodiment of the present application is shown in FIG. Figure 11 As shown, the device includes: an acquisition unit for acquiring an image to be processed; a processing unit for processing the image to be processed through a trained image segmentation model to obtain a segmented image; wherein the image segmentation model includes an encoder and a decoder, and the encoder and the decoder are correspondingly connected through a jump connection; the encoder is used to encode the image to be processed and generate a first encoding feature map and a second encoding feature map in sequence; the jump connection is configured with an attention mechanism module and a self-attention mechanism module, the attention mechanism module is used to perform feature enhancement processing on the low-level features of the first encoding feature map, and send the processed low-level features to the decoder, wherein the feature enhancement processing includes strengthening the target area features in the low-level features; the self-attention mechanism module is used to extract the global context information of the high-level semantic features in the second encoding feature map, and send the high-level semantic features and the global context information to the decoder; the decoder is used to determine the segmented image based on the processed low-level features, high-level semantic features and global context information.

[0098] Figure 12 Schematic diagram of an electronic device provided by an embodiment of the present application. Figure 12As shown, the electronic device 12 of this embodiment includes: a processor 120, a memory 121, and a computer program 122 stored in the memory 121 and executable on the processor 120, such as an image segmentation program. When the processor 120 executes the computer program 122, the steps of the above-described image segmentation method embodiments are implemented. Alternatively, when the processor 120 executes the computer program 122, the functions of the modules / units in the above-described device embodiments are implemented.

[0099] Exemplarily, the computer program 122 may be divided into one or more modules / units, which are stored in the memory 121 and executed by the processor 120 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 122 in the electronic device 12.

[0100] The electronic device 12 may be a computing device such as a tablet computer, a tablet computer, a desktop computer, a notebook computer, a PDA, a cloud server, etc. The electronic device may include, but is not limited to, a processor 120 and a memory 121. It will be understood by those skilled in the art that Figure 12 It is only an example of the electronic device 12 and does not constitute a limitation of the electronic device 12. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0101] The processor 120 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0102] The memory 121 may be an internal storage unit of the electronic device 12, such as a hard drive or memory of the electronic device 12. The memory 121 may also be an external storage device of the electronic device 12, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 12. Furthermore, the memory 121 may include both an internal storage unit of the electronic device 12 and an external storage device. The memory 121 is used to store the computer program and other programs and data required by the electronic device. The memory 121 may also be used to temporarily store data that has been output or is about to be output.

[0103] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0104] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0105] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0108] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0109] If the integrated module / unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments through hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0110] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Get the image to be processed; Processing the image to be processed by a trained image segmentation model to obtain a segmented image; The image segmentation model includes M first encoding feature layers, N second encoding feature layers, N second decoding feature layers and M first decoding feature layers connected in sequence, M≥1, N≥1; There is a one-to-one correspondence between the M first encoding feature layers and the M first decoding feature layers. An attention mechanism module is provided between the first encoding feature layer and the corresponding first decoding feature layer. The attention mechanism module is an attention gate structure. The attention mechanism module is used to perform feature enhancement processing on the low-level features output by the corresponding first encoding feature layer to obtain target area features, and input the target area features into the corresponding first decoding feature layer; There is a one-to-one correspondence between the N second encoding feature layers and the N second decoding feature layers. A self-attention mechanism module is provided between the second encoding feature layer and the corresponding second decoding feature layer. The self-attention mechanism module is a Transformer structure. The self-attention mechanism module is used to extract global context information from the high-level semantic features output by the corresponding second encoding feature layer, and input the global context information into the corresponding second decoding feature layer. Inputting the target area feature into the corresponding first decoding feature layer includes: Performing a dot product between the target area feature and the corresponding input information of the first decoding feature layer and inputting the resultant information into the first decoding feature layer, wherein the input information is the output information of the previous layer of the first decoding feature layer; The inputting the global context information into the corresponding second decoding feature layer includes: The global context information is added to the input information of the corresponding second decoding feature layer and then inputted into the second decoding feature layer, where the input information is the output information of a previous layer of the second decoding feature layer.

2. The method according to claim 1, characterized in that The image to be processed includes a brain magnetic resonance image, and the segmented image is an image including segmentation results of the subthalamic nucleus and the red nucleus marked thereon.

3. The method according to claim 2, characterized in that The method further comprises: The target point position coordinates are determined based on the segmented image.

4. The method according to any one of claims 1 to 3, characterized in that The image segmentation model is trained in the following way: Obtaining training set images, wherein the training images are images marked with target areas; The training set images are input into the image segmentation model to be trained, and the image segmentation model is trained based on a loss function, where the loss function is determined according to the sum of the cross entropy loss and the Dice loss.

5. An image processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire an image to be processed; a processing unit, configured to process the image to be processed using a trained image segmentation model to obtain a segmented image; The image segmentation model includes M first encoding feature layers, N second encoding feature layers, N second decoding feature layers and M first decoding feature layers connected in sequence, M≥1, N≥1; There is a one-to-one correspondence between the M first encoding feature layers and the M first decoding feature layers. An attention mechanism module is provided between the first encoding feature layer and the corresponding first decoding feature layer. The attention mechanism module is an attention gate structure. The attention mechanism module is used to perform feature enhancement processing on the low-level features output by the corresponding first encoding feature layer to obtain target area features, and input the target area features into the corresponding first decoding feature layer; There is a one-to-one correspondence between the N second encoding feature layers and the N second decoding feature layers. A self-attention mechanism module is provided between the second encoding feature layer and the corresponding second decoding feature layer. The self-attention mechanism module is a Transformer structure. The self-attention mechanism module is used to extract global context information from the high-level semantic features output by the corresponding second encoding feature layer, and input the global context information into the corresponding second decoding feature layer. Inputting the target area feature into the corresponding first decoding feature layer includes: Performing a dot product between the target area feature and the corresponding input information of the first decoding feature layer and inputting the resultant information into the first decoding feature layer, wherein the input information is the output information of the previous layer of the first decoding feature layer; The inputting the global context information into the corresponding second decoding feature layer includes: The global context information is added to the input information of the corresponding second decoding feature layer and then inputted into the second decoding feature layer, where the input information is the output information of a previous layer of the second decoding feature layer.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images