Image segmentation method, training method, device, equipment and medium

Abstract

The application provides an image segmentation method, a training method, a device, equipment and a medium. In a skip connection between encoding and decoding, feature map division is performed based on a window mechanism, first feature map calculation of fused local features in a window is performed, second feature map calculation of fused global features between windows is performed, the first feature map and the second feature map are spliced to obtain a fused feature map for decoding output. In this way, low-overhead intra-window and inter-window feature calculation is adopted, calculation of feature similarity between each pair of graph nodes is avoided, overall calculation complexity is reduced, and applicability is expanded.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image segmentation method, a training method, a device, an apparatus and a medium. BACKGROUND

[0002] Abdominal organ segmentation plays a crucial role in medical image processing and is an indispensable part of computer-aided diagnosis, surgical navigation, visual enhancement, radiotherapy, and biomarker measurement systems. Accurate segmentation results can provide key information such as organ size, position, boundary characteristics, and their spatial relationships, which are of great significance for image analysis, surgical planning, clinical decision support, and subsequent follow-up. In particular, in the process of cancer and tumor radiotherapy, precise segmentation of dangerous areas not only helps to improve treatment effectiveness, but also effectively reduces the impact on surrounding healthy tissues. However, due to the complex abdominal anatomical structure, diverse organ shapes, and various pathological conditions, it is a challenging and time-consuming task for clinicians to accurately analyze computed tomography (CT) and magnetic resonance (MR) abdominal image data, and errors are prone to occur. Therefore, developing efficient and reliable abdominal organ segmentation technology has become an important direction to improve the quality and efficiency of diagnosis and treatment.

[0003] In order to improve the accuracy of image interpretation, optimize the clinical decision-making process, and enhance the quality of patient care, there is an urgent need to develop efficient and robust automatic abdominal image segmentation technology to replace the time-consuming and error-prone traditional manual delineation method. In recent years, with the advancement of deep learning technology, medical image segmentation methods based on deep learning have shown significant advantages. By learning complex features from a large amount of medical image data, the accuracy and efficiency of the segmentation task have been greatly improved. However, existing mainstream models such as convolutional neural networks (CNN) and Transformer models still face some challenges in processing medical image segmentation. Traditional CNN relies on the pixel grid structure in Euclidean space and has limitations in capturing long-distance dependencies and complex spatial relationships. Although Transformer has made some progress by dividing the image into small blocks and using self-attention mechanisms to identify long-range dependencies, it still has room for improvement in handling image details and edge detection. In addition, the Transformer model usually has high computational complexity, and the attention mechanism used by Transformer needs to consider all other nodes when updating the current node representation, which may introduce redundant information in the feature extraction process, thereby affecting the accuracy and efficiency of the segmentation results.

[0004] Recently, graph-based modeling methods have been applied to image tasks due to their flexibility and versatility. These methods have been shown to be effective in modeling irregular structures in images and have achieved success in computer vision and medical imaging. Graph-based modeling methods not only can represent irregular topological structures, thus flexibly modeling irregular shapes and regions with large contrast changes in medical images, but also can selectively fuse features through the connection relationship of edges, avoiding the capture of excessive redundant information.

[0005] However, existing graph modeling methods usually divide an image into multiple patches, and each patch is regarded as a graph node, and then the edge connection relationship between nodes is established through the K-neighbor algorithm. When establishing the edge connection relationship, the feature similarity between each pair of nodes needs to be calculated, resulting in expensive computational complexity. This limitation hinders the applicability of graph modeling methods, especially in dense prediction tasks such as image segmentation. SUMMARY

[0006] The present application provides an image segmentation method, a training method, an apparatus, a device and a medium, which can solve one of the problems in the background art.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] In a first aspect, a training method of an image segmentation model is provided, and the training method comprises:

[0009] obtaining training data, the training data comprising: an image to be segmented and an image segmentation result; and

[0010] training the image segmentation model using the training data,

[0011] wherein the image segmentation model comprises: an encoding part, a decoding part, an output part located at the processing back end of the decoding part, and a skip connection part located between the encoding part and the decoding part,

[0012] the encoding part is configured to obtain a multi-scale feature map from the image to be segmented;

[0013] the skip connection part is configured to perform first division on at least one scale feature map with a predetermined size window to obtain a first-level feature map, perform second division on the first-level feature map to obtain a second-level feature map, perform self-attention calculation on the second-level feature map to obtain a first feature map fused with local features, perform graph convolution operation on the first-level feature map to obtain a second feature map fused with global features, and concatenate the first feature map and the second feature map to obtain a fused feature map for output to the decoding part.

[0014] Based on the above technical scheme, in the skip connection between encoding and decoding, based on the window mechanism, the feature map is divided, the first feature map calculation of the local feature fusion in the window is performed, and the second feature map calculation of the global feature fusion between windows is performed, the first feature map and the second feature map are spliced to obtain the fusion feature map for decoding output. In this way, the intra-window and inter-window feature calculation with low overhead is adopted, the calculation of the feature similarity between each pair of graph nodes is avoided, and the overall calculation complexity is reduced, and the applicability is expanded.

[0015] In a possible design of the first aspect, the skip connection part is further configured to:

[0016] The fusion feature maps corresponding to the scale feature maps are spliced in the same channel dimension and in the spatial dimension to obtain a node set and a node feature map in the spatial dimension;

[0017] The node set corresponds to a graph representation, and the relationship between adjacent scale nodes is taken as an edge.

[0018] The graph representation is subjected to a graph convolution operation to obtain a node feature representation reflecting the correlation between the fusion feature maps of different scales; and

[0019] The node feature representation is restored at each scale and spliced with the corresponding fusion feature map to obtain a spliced feature map output to the decoding part.

[0020] In a possible design of the first aspect, the graph representation is subjected to a graph convolution operation to obtain a node feature representation reflecting the correlation between the fusion feature maps of different scales, and specifically, the node feature map is subjected to an edge convolution operation to obtain the node feature representation.

[0021] In a possible design of the first aspect, the encoding part includes a first layer encoder, a second layer encoder, a third layer encoder, a fourth layer encoder, and a fifth layer encoder connected by pooling, the first layer encoder inputs the image to be segmented, the decoding part includes a first layer decoder, a second layer decoder, a third layer decoder, a fourth layer decoder, and a fifth layer decoder connected by up-sampling, the fifth layer encoder is correspondingly output to the first layer decoder, the skip connection part includes a first skip connection between the first layer encoder and the fifth layer decoder, a second skip connection between the second layer encoder and the fourth layer decoder, a third skip connection between the third layer encoder and the third layer decoder, and a fourth skip connection between the fourth layer encoder and the second layer decoder, and the second skip connection, the third skip connection, and the fourth skip connection are used to obtain the fusion feature map corresponding to the corresponding scale.

[0022] In a possible design of the first aspect, the image to be segmented is a computed tomography (CT) abdominal image or a magnetic resonance (MR) abdominal image.

[0023] In a second aspect, an image segmentation method is provided, and the method includes:

[0024] obtaining an image to be processed;

[0025] processing the image to be processed by using the image segmentation model trained as described above, to obtain an image processing result.

[0026] In a third aspect, a training apparatus of an image segmentation model is provided, and the training apparatus includes:

[0027] a first obtaining unit configured to obtain training data, the training data including an image to be segmented and an image segmentation result; and

[0028] a training unit configured to train the image segmentation model by using the training data,

[0029] wherein the image segmentation model includes an encoding part, a decoding part, an output part located at a processing back end of the decoding part, and a skip connection part located between the encoding part and the decoding part,

[0030] the encoding part is configured to obtain multi-scale feature maps from the image to be segmented;

[0031] the skip connection part is configured to perform first division on at least one scale feature map by using a predetermined size window, to obtain a first-level feature map; perform second division on the first-level feature map, to obtain a second-level feature map; perform self-attention calculation on the second-level feature map, to obtain a first feature map that fuses local features; perform graph convolution operation on the first-level feature map, to obtain a second feature map that fuses global features; and splice the first feature map and the second feature map, to obtain a fusion feature map to be output to the decoding part.

[0032] In a fourth aspect, an image segmentation apparatus is provided, and the segmentation apparatus includes:

[0033] a second obtaining unit configured to obtain an image to be processed; and

[0034] a segmentation unit configured to process the image to be processed by using the image segmentation model trained as described above, to obtain an image processing result.

[0035] In a fifth aspect, an electronic device is provided, and the electronic device includes a processor and a memory coupled to the processor,

[0036] The memory is configured to store a computer program.

[0037] The processor is configured to execute the computer program stored in the memory, so that the electronic device executes the training method or executes the segmentation method.

[0038] In a sixth aspect, a computer readable storage medium is provided, and the computer readable storage medium includes a computer program or instructions, which, when executed on a computer, cause the computer to execute the training method or execute the segmentation method.

[0039] The beneficial effects of the second aspect to the sixth aspect are as described in the first aspect, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or related technical descriptions will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 is a schematic diagram of the abdominal multi-organ segmentation network provided by the embodiments of the present application;

[0042] Figure 2 is a schematic diagram of the window graph convolution module provided by the embodiments of the present application;

[0043] Figure 3 is a schematic diagram of window division provided by the embodiments of the present application, which generates local feature maps within the window and window feature maps through the change of feature dimensions;

[0044] Figure 4 is a schematic diagram of the construction of the multi-scale feature static graph in the hierarchical architecture provided by the embodiments of the present application, according to the relationship between the upper and lower layers in the hierarchical structure, the edges from the upper layer nodes to the lower layer nodes are constructed, so as to construct the static graph, and realize the modeling of the hierarchical features and the multi-scale features;

[0045] Figure 5 is a schematic diagram of the abdominal multi-organ segmentation result provided by the embodiments of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0047] It is to be understood that even though the method steps, acts, and events of the embodiments are described in a particular, sequential order, unless otherwise specified, some steps, acts or events can be performed concurrently, in parallel, or in any order. The terms "first," "second," "third," etc. are used herein to denote different stages in a process, and do not connote an ordinal nature unless otherwise indicated.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for describing the embodiments only and is not intended to be limiting.

[0049] This embodiment will design a multi-organ segmentation method of the abdomen superior to the existing mainstream method of mixed attention and graph convolution. In view of the problem that the existing segmentation method based on graph neural network has high computational complexity in the graph construction process, this embodiment mainly divides the feature map based on the window mechanism, and effectively learns the local and distant dependencies by combining the attention mechanism and the graph representation learning method. In addition, this embodiment also designs a static graph, which realizes the modeling of multi-scale information in the hierarchical architecture.

[0050] In summary, this embodiment effectively models the medical image and the multi-scale information in the hierarchical architecture by means of the graph, which can accurately capture the complex and irregular topological structure in the abdominal image to achieve better and more accurate results.

[0051] Next, the symbol explanation and the corresponding technical background of the method will be introduced in Part 1, the algorithm flow proposed in this embodiment will be introduced in detail in Part 2, and the results and advantages of the proposed algorithm will be stated in Part 3.

[0052] 1. Symbol explanation and background introduction

[0053] 1.1 Definition of graph

[0054] A graph is composed of a finite number of nodes and edges connecting two nodes, which can be represented as G=(V,E). Where V represents the node set, and |V|=N represents the number of nodes in the graph as N, E represents the edge set. In graph machine learning and graph neural networks, the node feature matrix is used to represent the node set V, and d represents the dimension of the node feature; the weighted adjacency matrix is used to represent the connection relationship between two nodes, and is used to represent the edge set E. If there is an edge from node i to node j, then A i,j is set to 1 or the weight of the edge, otherwise it is set to 0.

[0055] 1.2 K-Nearest Neighbor (KNN) graph construction

[0056] KNN composition is a composition method based on K-Nearest Neighbor algorithm, which is used to compose an image into a graph structure. Specifically, given an image of size H x W, the image is first divided into N patches. By converting each patch into a node vector We can get a node feature matrix X = (x1, x2,..., x N ). In this way, a set of nodes V is obtained. Then, we find the K nearest neighbors of each node in the feature space and add edges to them by K-Nearest Neighbor algorithm, so as to obtain a set of edges E. Specifically, the distance between each two nodes needs to be calculated, usually using Euclidean distance, as shown in formula (1):

[0057]

[0058] where x ik and x jk represent the value of the i-th and j-th node at the k-th feature. In this way, a distance matrix D ij is obtained, which represents the distance between the i-th and j-th nodes. Then, the Top-k nearest neighbor nodes are selected for each column of D, and the corresponding position value in the column is set to 1, and the rest of the positions are set to 0. In this way, an adjacency matrix

[0059] Through the above process, the image structure can be converted into a graph structure, so that the image can be processed using related methods of graph.

[0060] 1.3 Graph Convolution

[0061] In traditional CNN, convolution operation is performed on regular grid data (such as images), while graph convolution is designed to handle irregular data structure - graph, where the connection between nodes can be arbitrary. The goal of graph convolution is to extract features from nodes and their neighbors while preserving the graph structure information. Graph convolution usually aggregates the information of a node and its adjacent nodes through the normalized adjacency matrix , and converts it through linear transformation and nonlinear activation function to get new node feature representation. A simple graph convolution layer can be described as follows:

[0062]

[0063] Here is a degree matrix, which is a diagonal matrix and its i-th diagonal element represents the degree of node i, that is, the number of edges connected to node i. is the adjacency matrix of the graph plus the identity matrix (i.e., each node is connected to itself), indicating the existence of self-loops. (l) is the trainable weight matrix of the layer, and σ is the activation function. (l) and X (l+1) are the node feature matrices of the current and next layers, respectively.

[0064] 2. Method

[0065] The embodiment proposes a method for abdominal multi-organ segmentation by combining attention and graph convolution. While reducing the computational complexity of the graph construction process, the method uses graph modeling of images and multi-scale information in a hierarchical architecture to accurately capture the complex and irregular topological structure present in the abdominal region, thereby achieving accurate abdominal multi-organ segmentation.

[0066] 2.1 Construction of abdominal multi-organ segmentation network

[0067] For the specific task of abdominal multi-organ segmentation, we build a U-shaped architecture as shown in Figure 1 . It is composed of five layers of encoders, five layers of decoders, and skip connections.

[0068] Each layer of the encoder includes an encoder module and a pooling layer. The encoder module is composed of two layers of 3x3 convolution layers and one layer of 1x1 convolution layer, and each layer of convolution contains an activation function layer and a normalization layer. The pooling layer is used to maintain the hierarchical architecture of the U-shaped network and mainly performs 2 times down-sampling.

[0069] In the skip connection part, a window graph convolution module is designed, which mainly identifies basic features through attention modules and graph modules more effectively.

[0070] After that, the hierarchical multi-scale features generated by the network are captured through the multi-scale recombination graph block (RGCB) to capture the local and global correlations between different scale features.

[0071] Then, the information of different scales will be input into the decoder part. The decoder up-samples the features from the previous layer of the decoder, and concatenates the output of the same layer in the skip connection in the feature dimension, and then fuses them through a layer of 3x3 convolution layer and a layer of 1x1 convolution, and each layer of convolution also contains an activation function layer and a normalization layer.

[0072] Finally, the output of the decoder part is passed through a segmentation head (including a 1x1 convolution layer and a sigmoid activation function) to generate pixel-level class labels, thereby obtaining the segmentation result.

[0073] 2.2 Window graph convolution module

[0074] To reduce the complexity of the composition process, we design a window graph convolution (WGC) module, as shown in Figure 2 This module introduces a window mechanism and combines attention mechanism and graph convolution method to effectively learn local features and long-distance dependencies.

[0075] (1) Window partition

[0076] We first partition the feature map into windows to reduce the computational complexity of the attention mechanism and the composition process. It should be noted that the proposed network does not partition the window in the last two layers, but directly processes it through the graph convolution module in Figure 1 This is mainly because the feature map that has been through four pooling in the U-shaped architecture (the size has become of the original feature map) is already relatively small, and the computational overhead it brings is relatively low.

[0077] Specifically, for an input feature map of HxWxC and a window size of h x w, where C is the channel dimension, the input feature map is first converted to indicates that the windowed partition is performed on the HxW feature map with a h x w window to facilitate the generation of windowed feature maps and windowed feature maps through subsequent feature dimension conversion and combination. Then, through feature conversion, a windowed local feature map of size and a windowed feature map of size are obtained, as shown in Figure 3 The former fuses local features through local window attention calculation. The latter selectively fuses global features through windowed graph convolution operation. This process can be expressed as

[0078] X in ,X out = window_patition(X) (3)

[0079] where X ∈ R H×W×C represents the input feature map, represents the windowed local feature map, represents the windowed feature map. Wherein, L = h x w C' = h x w x C.

[0080] It should be noted that HxWxC is the size of the original image and has not been processed by the WGC module. Mainly considering the problem of computational overhead and effect. For the input original feature map, each point in the feature map represents a pixel, and the performance improvement brought by directly processing it is limited. At the same time, even after windowed processing, the time overhead brought by processing it is huge, so it is not processed by WGC.

[0081] (2) Self-attention computation within local window

[0082] After the partition of the feature map by windowing, we perform self-attention computation within the local window. The reason for limiting this computation to a smaller window is to reduce the quadratic complexity globally. Meanwhile, considering the small number of nodes within the window, we directly model it as a fully connected graph and efficiently extract local features through the self-attention mechanism. Why can the attention mechanism be understood as a fully connected graph? The main reason is that the attention mechanism needs to compute the attention matrix, which is actually a feature similarity matrix between nodes. This process is consistent with the logic of adding edges in the K- nearest neighbor algorithm in graph modeling. However, the difference between the two lies in the way of updating the features. The attention mechanism needs to consider the features of all other nodes when updating the current node. This can be understood as that the current node has an edge with every other node, which is a fully connected graph. On the other hand, the graph limits the feature update to the nearest few nodes rather than considering all nodes, so the graph convolution can be described as "selectively fusing". This is also the reason why the background technology mentioned that the Transformer may introduce redundant information, because it needs to consider all nodes.

[0083] Specifically, for the local feature map within the window of input , we consider dimension 1 as the batch and dimension 2 as the sequence and perform feature extraction through the Attention module. The Attention module, also known as the attention module, is actually modeled in the form of a sequence, so it requires the input to have the form (B, N, C). In the attention module, B is the batch and N is the length of the sequence. This module can be described as:

[0084] Q = W1 · X in K = W2 · X in V = W3 · X in

[0085]

[0086] where are learnable parameters, and Q, K, and V represent queries, keys, and values, respectively. They are all calculated through linear layers based on the feature map X in . Then we get the feature map that fuses local features

[0087] (3) Graph convolution operation between windows

[0088] The graph convolution-based feature extraction module is designed to extract global features. For the input window feature map, each window is treated as a node, resulting in a node set V = {v1, v2, ..., v...}. N} and node feature matrix Considering that C′ has a large feature dimension, we first reduce the feature dimension from C′ to d using convolution. Then, we use the K-nearest neighbor method to perform convolution on each node v. i Find the K nearest neighbors N(v) of ∈V i For all v j ∈N(v i Add a line from v j to v i The edge e ji In this way, an edge set E is constructed, where e ji ∈E, thus obtaining a graph representation of the window features. Then, a graph convolutional layer is used to exchange node information by aggregating features from its neighboring nodes. The graph convolution operation can be described as:

[0089] G′(X)=Update(Aggregate(G,W agg ),W update (5)

[0090] Where Aggregate represents the aggregation operation, Update represents the update operation, G represents the input window feature map, G′ represents the output of the graph convolution operation, and W agg and W update These are learnable parameters in aggregation and update operations.

[0091] Graph convolution can be further described as updating the feature representation of a node by aggregating the features of its neighboring nodes. Here, edge convolution is used, thus the above process can be simplified to...

[0092] X′ out =EdgeConv(X out ·W4)·W5 (6)

[0093] Where EdgeConv represents edge convolution. For window feature map, These are learnable parameters. After obtaining the updated node features... Subsequently, feature maps incorporating long-distance dependencies are obtained through similar feature dimension transformations.

[0094] (4) Fusion of local features and remote dependencies.

[0095] After obtaining the feature map X″ of fused local features in ∈RH×W×C and fuse the remote dependency relationship into the feature map X" out ∈R H×W×C After that, we concatenate the two feature maps along the channel dimension and fuse the local features and remote dependencies by convolution operation. This process can be formulated as

[0096] X out =Mixer(X" in ,X" out )=BN(Contact(X" in ,X" out )·W6) (7)

[0097] where Contact(·) denotes feature concatenation along the channel dimension, is a learnable parameter, and BN is a normalization layer, is the feature map that fuses the local features and remote dependency relationship.

[0098] 2.3 Multi-scale Re-mixed Graph Block

[0099] We explore the method of using graph modeling to integrate multi-scale information in the hierarchical architecture and design a multi-scale Re-mixed Graph Block (RGCB) module. This module integrates the local and global correlations between different scale feature maps by utilizing detailed features and high-level semantic features including global dependencies and local contexts.

[0100] For an input image of HxWxC, we can get multi-scale features i is the number of layers in the network. First, we flatten and rearrange it to have the same channel dimension C, so that we get the converted multi-scale features Then we concatenate all the multi-scale features of the layers in the spatial dimension (dimension 1) and regard them as nodes from the spatial dimension, so that we get the node set V Bridge and the node feature map The specific calculation is as follows: input an image of HxWxC, according to the figure Figure 1 , the feature maps input into this module are After flattening and rearranging the five inputs, we get Regarding the first dimension as a node, there are nodes.

[0101] Then based on the upper and lower layer relationship existing in the hierarchical architecture, we add edges to the nodes of adjacent layers to construct the edge set E Bridge , as shown in Figure 4As shown in Fig. 2. Specifically, considering that the pooling operation used in the encoder will pool the 2x2 feature block of the upper layer into the 1x1 feature block of the lower layer, we add edges from the upper layer feature block to the lower layer feature block according to this relationship, so as to obtain the edge set E Bridge and the hierarchical structure of the static graph representation G Bridge = (V Bridge ,E Bridge ). Then the constructed static graph representation is input into the graph convolution module represented by formula (5) to capture the correlation between different scale feature maps. Since edge convolution is adopted, this process can be simplified as:

[0102] X′ Bridge = EdgeConv(X Bridge ) (8)

[0103] where EdgeConv represents edge convolution, and X Bridge is the node feature map of the multi-scale feature.

[0104] Finally, the spliced multi-scale feature is restored to the feature map of different scales and input into the decoder of the corresponding layer. Specifically, for the obtained node feature map of the multi-scale feature we split it into feature maps of different scales in the first dimension in order Then, after the transformation of the feature dimension, we get five outputs. These five outputs will be input into the decoder of the corresponding layer.

[0105] 3. Advantages of the method

[0106] (1) The obtained results on the public abdominal multi-organ segmentation dataset are better than those of existing methods. Table 1 shows the experimental results of different segmentation methods on the abdominal multi-organ segmentation dataset Synapse. We use Dice consistency loss (DSC) and Hausdorff distance (HD) as two indicators to evaluate the proposed method. In addition, Table 1 records the Dice consistency loss of different methods on different organs.

[0107] (2) The proposed method can better balance performance and model complexity. Table 2 shows the performance, parameter quantity, computational quantity (FLOPs), and model training time of different segmentation methods on the abdominal multi-organ segmentation dataset Synapse. Compared with AHGNN which is also a graph modeling method, the proposed method has fewer parameters and computational quantity. Compared with other methods of the same size, the proposed method can obtain better results.

[0108] (3) Figure 5This paper presents a qualitative comparison between our method and other segmentation methods. The comparison results show that our method is more sensitive to organ boundary information and can capture and generate more accurate organ edge information.

[0109] Table 1. Performance Comparison with Representative Segmentation Methods

[0110]

[0111] Table 2 compares the performance, number of parameters, computational cost (FLOPs), and training time of representative segmentation methods.

[0112]

[0113]

[0114] This application embodiment also provides a training device for an image segmentation model, the training device comprising:

[0115] The first acquisition unit is used to acquire training data, the training data including: the image to be segmented and the image segmentation result; and

[0116] The training unit is used to train the image segmentation model using the training data.

[0117] The image segmentation model includes: an encoding part, a decoding part, an output part located at the processing end of the decoding part, and a skip connection part located between the encoding part and the decoding part.

[0118] The encoding portion is used to obtain a multi-scale feature map from the image to be segmented;

[0119] The skip connection portion is used to first divide the feature map of at least one scale into a window of a predetermined size to obtain a first-level feature map; second divide the first-level feature map into a second-level feature map; perform self-attention calculation on the second-level feature map to obtain a first feature map that fuses local features; perform graph convolution operation on the first-level feature map to obtain a second feature map that fuses global features; and concatenate the first feature map and the second feature map to obtain a fused feature map for output to the decoding portion.

[0120] This application embodiment also provides an image segmentation apparatus, the segmentation apparatus comprising:

[0121] The second acquisition unit is used to acquire the image to be processed; and

[0122] The segmentation unit is used to process the image to be processed using the image segmentation model trained as described above, and obtain the image processing result.

[0123] The embodiments of the present application also provide an electronic device, comprising: a processor, and a memory coupled with the processor, the memory being configured to store a computer program; and the processor being configured to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of the preceding embodiments.

[0124] The electronic device can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The electronic device can include, but is not limited to, a processor and a memory.

[0125] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is a control center of the electronic device, and connects all parts of the electronic device through various interfaces and lines.

[0126] The memory can be configured to store the computer program, and the processor can be configured to realize various functions of the electronic device by running or executing the computer program stored in the memory and calling data stored in the memory.

[0127] The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required by a function, and the like; and the data storage area can store data created according to the use of the electronic device, and the like. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0128] The embodiment of the present application further provides a storage medium, which is a computer readable storage medium, and a computer program is stored in the computer readable storage medium. When the computer program is executed by a processor, steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0129] The embodiment of the present application further provides a computer program product, which includes a computer program or instructions, when the computer program or instructions run on a computer, make the computer execute the method of any possible implementation manner described above.

[0130] The above is the preferred embodiment of the present application. It should be pointed out that, for those skilled in the art, without departing from the principle of the present application, a number of improvements and refinements can be made, which are also considered to be within the protection scope of the present application.

Claims

1. A training method for an image segmentation model, characterized in that, The training method includes: The training data includes: the image to be segmented and the image segmentation results; and The image segmentation model is trained using the training data. The image segmentation model includes: an encoding part, a decoding part, an output part located at the processing end of the decoding part, and a skip connection part located between the encoding part and the decoding part. The encoding portion is used to obtain a multi-scale feature map from the image to be segmented; The skip connection portion is used to perform a first division of the feature map at least one scale into a window of a predetermined size to obtain a first-level feature map; perform a second division of the first-level feature map to obtain a second-level feature map; perform self-attention calculation on the second-level feature map to obtain a first feature map that fuses local features; perform graph convolution operation on the first-level feature map to obtain a second feature map that fuses global features; and concatenate the first feature map and the second feature map to obtain a fused feature map for output to the decoding portion. The jump connection portion is also used for: For each scale feature map, the corresponding fused feature map is spliced ​​together in the spatial dimension with the same channel dimension to obtain a node set and node feature map in the spatial dimension. A graph representation corresponding to the node set is constructed using the relationships between adjacent scale nodes as edges; Performing a graph convolution operation on the graph representation yields node feature representations reflecting the correlations between the fused feature maps at different scales; and The node feature representations are restored at various scales and then concatenated with the corresponding fused feature maps to obtain a concatenated feature map, which is then output to the decoding section.

2. The training method as described in claim 1, characterized in that, Graph convolution is performed on the graph representation to obtain node feature representations that reflect the correlation between the fused feature maps at different scales. Specifically, edge convolution is performed on the node feature maps to obtain the node feature representations.

3. The training method as described in claim 1, characterized in that, The encoding part includes a pooled-connected first-layer encoder, a second-layer encoder, a third-layer encoder, a fourth-layer encoder, and a fifth-layer encoder. The first-layer encoder takes the image to be segmented as input. The decoding part includes an upsampled-connected first-layer decoder, a second-layer decoder, a third-layer decoder, a fourth-layer decoder, and a fifth-layer decoder. The fifth-layer encoder outputs to the first-layer decoder. The skip connection part includes a first skip connection between the first-layer encoder and the fifth-layer decoder, a second skip connection between the second-layer encoder and the fourth-layer decoder, a third skip connection between the third-layer encoder and the third-layer decoder, and a fourth skip connection between the fourth-layer encoder and the second-layer decoder. The second, third, and fourth skip connections are used to obtain fused feature maps corresponding to the respective scales.

4. The training method according to any one of claims 1 to 3, characterized in that, The image to be segmented is a computed tomography (CT) abdominal image or a magnetic resonance imaging (MRI) abdominal image.

5. An image segmentation method, characterized in that, The segmentation method includes: Obtain the image to be processed; The image segmentation model trained according to any one of claims 1 to 4 is used to process the image to be processed to obtain the image processing result.

6. A training device for an image segmentation model, characterized in that, The training device includes: The first acquisition unit is used to acquire training data, the training data including: the image to be segmented and the image segmentation result; and The training unit is used to train the image segmentation model using the training data. The image segmentation model includes: an encoding part, a decoding part, an output part located at the processing end of the decoding part, and a skip connection part located between the encoding part and the decoding part. The encoding portion is used to obtain a multi-scale feature map from the image to be segmented; The skip connection portion is used to perform a first division of the feature map at least one scale into a window of a predetermined size to obtain a first-level feature map; perform a second division of the first-level feature map to obtain a second-level feature map; perform self-attention calculation on the second-level feature map to obtain a first feature map that fuses local features; perform graph convolution operation on the first-level feature map to obtain a second feature map that fuses global features; and concatenate the first feature map and the second feature map to obtain a fused feature map for output to the decoding portion. The jump connection portion is also used for: For each scale feature map, the corresponding fused feature map is spliced ​​together in the spatial dimension with the same channel dimension to obtain a node set and node feature map in the spatial dimension. A graph representation corresponding to the node set is constructed using the relationships between adjacent scale nodes as edges; Performing a graph convolution operation on the graph representation yields node feature representations reflecting the correlations between the fused feature maps at different scales; and The node feature representations are restored at various scales and then concatenated with the corresponding fused feature maps to obtain a concatenated feature map, which is then output to the decoding section.

7. An image segmentation apparatus, characterized in that, The segmentation device includes: The second acquisition unit is used to acquire the image to be processed; and A segmentation unit is used to process the image to be processed using the image segmentation model trained according to any one of claims 1 to 4, and obtain an image processing result.

8. An electronic device, characterized in that, The electronic device includes: a processor, and a memory coupled to the processor. The memory is used to store computer programs; and The processor is configured to execute the computer program stored in the memory, such that the electronic device performs the training method as described in any one of claims 1 to 4, or performs the segmentation method as described in claim 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program or instructions that, when executed on a computer, cause the computer to perform the training method as described in any one of claims 1 to 4, or the segmentation method as described in claim 5.

Images