A multi-organ instance segmentation method and system

Abstract

The present invention provides a multi-organ instance segmentation method and system, belonging to the field of medical image processing technology. The method of the present invention includes constructing a training data set: obtaining a medical image to be segmented, performing preprocessing, segmenting and annotating each part on each preprocessed medical image, and integrating to obtain an instance segmentation data set as a training data set; constructing a training model: constructing an image segmentation network based on a U‑Net network and a Transformer model, and then using the training data set to train the image segmentation network to obtain a U‑NET‑Transformer model; obtaining an image segmentation result: using the trained U‑NET‑Transformer model to segment the intracranial image to be segmented, and obtaining a segmentation result. The present invention can display the mutual influence relationship of various parts in a medical image on one image, allowing doctors to diagnose the condition more clearly and conveniently.

Description

Technical Field

[0001] The present invention relates to an image segmentation method, and in particular to a multi-organ instance segmentation method and system. Background Art

[0002] In recent years, deep learning has achieved significant results in basic image tasks such as image recognition, target detection, semantic segmentation, and advanced image vision tasks such as posture recognition and face recognition. In many tasks, the effect of deep learning has exceeded that of manual work, greatly saving manpower and material resources. In the medical field, the most common application scenario of deep learning computer vision tasks is medical image processing. Among them, image segmentation is an important research direction in medical image processing tasks. Although many image segmentation methods have been developed, including traditional segmentation methods and segmentation methods based on deep learning models, the development of the field of medical image analysis has been relatively slow due to the particularity of various diseases, the long training cycle of doctors, and the low fault tolerance rate. Therefore, the performance of computer-aided diagnosis technology based on medical image analysis needs to be improved.

[0003] At present, there are two main problems with domestic medical resources: one is the overall uneven distribution; the other is the large regional differences. Computer-aided diagnosis technology can effectively improve these two problems. Accurate and efficient computer diagnosis technology can improve the accuracy of doctors' diagnosis and assist doctors in lesion detection, achieve early prevention and screening of diseases, and greatly improve the efficiency of doctors' diagnosis and treatment. With the advantage of deep learning feature extraction, the rapidly developing computer diagnosis technologies include lung nodule detection, diabetic retinopathy screening, and skin cancer identification. By analyzing and processing the patient's CT images or other imaging files and combining the patient's medical history, auxiliary diagnosis and early warning are carried out. Among all these computer-aided diagnosis technologies, accurate and efficient segmentation technology plays a vital role, which is directly related to the reliability of subsequent diagnosis results.

[0004] At present, the medical image segmentation technology that can be applied in clinical practice is usually semantic segmentation, that is, the region of interest (ROI) is segmented in the mode of foreground and background pixels according to the category of the object in the image. However, this brings two types of problems. First, when multiple objects of the same category are adjacent or overlapped, it is impossible to segment a single object, which is not conducive to the doctor's later diagnosis; second, in the actual human body, the image formed by medical scanning does not only contain the required parts. For example, when taking a scan of the lung image, the image may not only show the lung nodules you want to observe, but also other organs and other parts of the lungs, which will affect the segmentation effect. In addition, doctors do not rely solely on observing the image of one part to make a diagnosis. They usually need to observe a series of adjacent and related images to make a comprehensive diagnosis. This type of single-type segmentation method cannot handle such problems well.

[0005] In the process of intracranial segmentation, since various parts of the skull are cross-stacked and interrelated with each other, it is difficult to obtain accurate segmentation results using a single semantic segmentation method. In addition, every tiny part and every tiny nerve in the skull may cause errors in the doctor's diagnosis, which is extremely unfavorable for the doctor's further diagnosis.

[0006] Each part of the skull is segmented separately to obtain a large number of images. However, if doctors need to observe so many images one by one, it is extremely labor-intensive, and the relationship between each part cannot be well reflected, which may lead to errors in diagnosis. Therefore, a method of instance segmentation that can be performed on various complex parts of the skull is needed, which is more conducive to doctors judging the status of each part, understanding the mutual influence between each part, and conducting in-depth diagnosis and treatment, greatly improving doctors' work efficiency and reducing manpower consumption. Summary of the invention

[0007] In order to solve the problems in the prior art, the present invention provides a multi-organ instance segmentation method and system.

[0008] The multi-organ instance segmentation method of the present invention comprises the following steps:

[0009] S1: Constructing a training data set: Obtain the medical images to be segmented, perform preprocessing, segment and annotate each part on each preprocessed medical image, and integrate them to obtain an instance segmentation data set as the training data set;

[0010] S2: Constructing a training model: constructing an image segmentation network based on a U-Net network and a Transformer model, and then using the training data set to train the image segmentation network to obtain a U-NET-Transformer model. The U-NET-Transformer model is as follows: first, a hybrid convolutional neural network and a Transformer model are used for feature extraction, and then each layer of data information is densely connected and input into a hidden feature layer with the same resolution, and then segmented through a U-Net network. At the same time, the input image is directly position-encoded and input into the Transformer model to learn the position information and the attention weight of the feature, and the obtained attention weight is returned to the U-NET network, and the attention weight is spliced ​​and added to the feature image to be spliced, and finally the prediction result is returned through upsampling of the U-NET network;

[0011] S3: Obtain image segmentation results: Use the trained U-NET-Transformer model to segment the intracranial image to obtain the segmentation results.

[0012] The present invention is further improved and also includes step S4: post-processing the segmentation result to obtain a more accurate segmentation result, and the post-processing method includes: one or more of morphological operation method, image grayscale processing method, grayscale threshold segmentation method, region segmentation method, and edge detection segmentation method.

[0013] The present invention is further improved. When the post-processing includes multiple methods, multiple semantic segmentation results are obtained through multiple post-processing methods, and then the multiple semantic segmentation results are fused back into one image to obtain a more accurate segmentation result.

[0014] The present invention is further improved, the morphological operation method is a morphological operation of first corroding and then dilating, and the operation mode of corroding the segmentation result is:

[0015] For sets A and B, it is represented as The corrosion formula of B to A is: z=(f,g), where f and g are the coordinate values ​​of the points in the plane coordinate system. In this formula, the erosion of A by B is a set of all points z contained in A by translating the point z along the x and y directions;

[0016] The expansion operation mode is:

[0017] For sets A and B, the expansion formula of B to A, expressed as A⊕B, is: In this formula, B' is the image of B about its origin, and then based on the translation of the image B' by point z, the expansion of B to A is the set of all displacement points z.

[0018] The present invention is further improved, the method of the image grayscale processing method is: different grayscale values ​​are set for each segmentation result according to the weight ratio, and the more important the part, the larger the grayscale value is set;

[0019] After the grayscale image is obtained by image fusion through the image grayscale processing method, the fused image is segmented again using the threshold segmentation method or the edge detection segmentation method to calibrate the segmentation boundary and obtain a more accurate result.

[0020] The present invention is further improved. In step S1, the image preprocessing method includes one or more of the following: scaling, resizing, contrast enhancement, grayscale color images and then binarization, scaling and rotating data to achieve data set expansion.

[0021] The present invention is further improved. In step S1, the method for image preprocessing is: different parts of the same original medical image need to be marked and segmented for information marking processing to facilitate subsequent instance segmentation. The preprocessing is to convert the original medical image into a fixed size of 512*512, and perform contrast enhancement processing. After graying the color image, the grayscale pixels are converted to between 0-1 to reduce the scale of the input features. After preprocessing, it also includes an integration step: the preprocessed data set is integrated with the original input image into a training data set.

[0022] The present invention is further improved. In step S2, the U-Net network model includes an upsampling layer, a downsampling layer, a skip connection structure, and feature fusion between layers and a pruning operation is added. The Transformer model is entirely composed of an attention mechanism, and a convolutional neural network is used to mix the Transformer model to extract features of different levels of the signal at the pixel level.

[0023] The present invention is further improved. In step S2, in the process of training the U-NET-Transformer model, the BCE Diceloss cross entropy loss function is introduced:

[0024]

[0025] Among them, c is the label 0 or 1, a is the output of the model after the activation function, ranging from (0,1), n ​​is the number of samples, and each time the network outputs a value, the BCE Diceloss cross entropy loss function gives a loss based on the value and the corresponding label. x is one of the samples, and its value is [1, n]. When c is 0, ylna is eliminated, leaving -ln(1-a). a needs to be close to 0 to reduce the loss value loss. When c is 1, (1-c)ln(1-a) is eliminated, leaving -lna. At this time, a needs to be close to 1 to reduce the loss value loss.

[0026] Based on the above, the present invention makes further optimization and introduces a binary classification foreground and background loss function:

[0027] L(b)=max(0,1-t*b)

[0028] Among them, b is the predicted value, which ranges from -1 to 1, and t is the target value 1 or -1, which means that the value of b can be between -1 and 1, and |b|>1 is not encouraged, that is, a sample can be correctly classified, and overconfidence of the classifier is not encouraged. When the distance between the sample and the dividing line exceeds 1, there will be no reward.

[0029] The present invention also provides a system for implementing the multi-organ instance segmentation method, comprising:

[0030] Constructing a training data set module: It is used to obtain the medical images to be segmented, perform preprocessing, segment and annotate each part on each preprocessed medical image, and integrate them to obtain an instance segmentation data set as a training data set;

[0031] Constructing a training model module: constructing an image segmentation network based on a U-Net network and a Transformer model, and then using the training data set to train the image segmentation network to obtain a U-NET-Transformer model. The U-NET-Transformer model is as follows: first, a hybrid convolutional neural network and a Transformer model are used for feature extraction, and then each layer of data information is densely connected and input into a hidden feature layer with the same resolution, and then segmented through a U-Net network. At the same time, the input image is directly position-encoded and input into the Transformer model to learn the position information and the attention weight of the feature, and the obtained attention weight is returned to the U-NET network, and the attention weight is spliced ​​and added to the feature image to be spliced, and finally the prediction result is returned through upsampling of the U-NET network;

[0032] Image segmentation result acquisition module: used to use the trained U-NET-Transformer model to segment the intracranial image to be segmented and obtain the segmentation result.

[0033] Compared with the prior art, the beneficial effects of the present invention are: it can display the mutual influence relationship of various parts in the medical image on one image, so that doctors can diagnose the disease more clearly and conveniently; it integrates the U-NET network and the transformer model, and has a better instance segmentation effect on complex structures. With the help of the unique encoding and decoding structure and jump connection characteristics of the U-NET network, high-level features and low-level features can be better integrated. This efficient feature extraction method can use fewer data sets to train the model to achieve the final effect; using the transformer model to learn the importance of position encoding can more effectively grasp the global information, and combining with the CNN network for feature extraction can obtain higher-quality feature information, making the segmentation result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the method of the present invention;

[0035] Figure 2 It is a schematic diagram of the network structure of the existing Transformer Layer;

[0036] Figure 3 It is a schematic diagram of the existing Transformer Layer2 network structure;

[0037] Figure 4 It is a schematic diagram of the existing U-Net network structure;

[0038] Figure 5 This is a schematic diagram of the structure of a U-NET-Transformer model that combines a Transformer model and a U-Net network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0040] like Figure 1 As shown, the present invention is applicable to the segmentation of organs in medical images of various parts of the body, especially in the case of overlapping organs such as the intracranial region, where the existing technology cannot segment a single organ and adjacent organs. The present invention has high accuracy, which is more conducive to doctors to judge the status of each part, understand the mutual influence between various parts, conduct in-depth diagnosis and treatment, greatly improve the work efficiency of doctors, and reduce manpower consumption.

[0041] Specifically, Figure 1 As shown, the multi-organ instance segmentation method of the present invention comprises the following steps:

[0042] Step S1: Construct a training dataset.

[0043] In this example, the medical images to be segmented are obtained and preprocessed. Each part is segmented and labeled on each preprocessed medical image, and the initial segmentation training label is marked. The corresponding instance segmentation data set is obtained by integrating each part as the training data set.

[0044] In this step, the method of obtaining the training data set by data preprocessing includes a series of preprocessing such as image scaling, size conversion, bilinear difference, contrast enhancement, grayscale color image and then binarization, and scaling and rotating the data to achieve data set expansion. One or more of them can also be used for processing, and other preprocessing can also be used to achieve the acquisition of a data set that meets the training requirements.

[0045] In this case, the linear difference method can be used as follows:

[0046] When R1=(x,y2),

[0047]

[0048] When R2=(x,y2),

[0049]

[0050] The interpolation value f(P) in the y direction can be deduced as:

[0051]

[0052] Among them, Q 11 , Q 12 , Q 21 , Q 22 are the vertices of the square surrounding point P, and R1 and R2 are the points on the straight line passing through point P and perpendicular to the x-axis that intersect with the square formed by the four points Q.

[0053] The binarization method in this example is as follows:

[0054] (1) Find the maximum grayscale value and the minimum grayscale value of the image, record them as Rmax and Rmin respectively, and set the threshold T1 = (Rmax + Rmin) / 2;

[0055] (2) Divide the average gray value of the image into two groups Ra and Rb according to the threshold T1;

[0056] (3) Calculate the average grayscale values ​​μ1 and μ2 of the two groups respectively;

[0057] (4) Calculate the new threshold value T2 = (μ1 + μ2) / 2.

[0058] (5) According to the new threshold T2, the values ​​greater than the threshold T2 are set to 255, and the values ​​less than the threshold T are set to 0.

[0059] Preferably, in this example, different parts of the same original medical image used for marking and segmentation need to be processed for information marking to facilitate subsequent instance segmentation. The preprocessing is to convert the original medical image into a fixed size of 512*512, and perform contrast enhancement processing. After graying the color image, the grayscale pixels are converted to between 0-1 to reduce the scale of the input features.

[0060] Step S2: Build a training model.

[0061] This example integrates the training models of U-Net network and Transformer model. The U-Net network model includes upsampling layer, downsampling layer, skip connection structure, feature fusion between layers and pruning operation. The Transformer model is composed of attention mechanism, which can better grasp the global information, but it is easy to ignore the low-resolution image details, which will cause errors in the decoding and recovery process. Therefore, CNN (convolutional neural network) is used to mix Transformer network to extract the features of different levels of the signal at the pixel level, which can grasp the global information while retaining the low-resolution detail information.

[0062] The following is a description of the U-Net network and Transformer model used in this example.

[0063] like Figure 2 As shown in the figure, the Transformer model structure in this example can be roughly divided into four parts: input, encoding, decoding and output, including positional encoding, forward propagation, self-attention, add & normalize and hybrid multi-head attention module.

[0064] The visual attention mechanism of computer vision allows computer systems to ignore irrelevant information through learning and focus on more important key information. It includes various attention methods such as self-attention and multi-head attention that neural networks can learn autonomously.

[0065] Positional encoding: Enhance the model input by injecting word order information. One method is to use sine and cosine functions of different frequencies for direct calculation; the other is to learn a positional embedding, that is, broadcast it in the batch dimension. This example Transformer model uses the first method, and the calculation formula is:

[0066]

[0067]

[0068] Among them, pos represents position, l represents dimension, dmodel represents position vector dimension = embedding dimension = 512 (trans) or 768 (bert). This method can encode and represent any position, and can learn longer sequences, and the requirements for sequences become broader.

[0069] The three matrices Q (Query), K (Key), and V (Value) in the self-attention module all come from the same input. First, the dot product of Q and K is calculated, and in order to prevent the product from being too large, it needs to be divided by the scale dk is the dimension of a query and key vector. Then the result is normalized by the Softmax function and multiplied by the matrix V to get the weight:

[0070]

[0071] The Transformer network model requires two types of masks, Padding mask and Sequencemask. Padding mask adds 0 after the shorter sequence in each batch of input sequence or cuts off the left side of the longer sequence and discards the excess part to align multiple sequences of different lengths. Sequence mask is used to prevent the decoder from receiving future information. The output decoder should only rely on the output before reading and cannot perform output after time. This example achieves this effect by generating an upper triangular matrix and setting it to 0.

[0072] Specifically: CNN hybrid Transformer is used here for feature extraction, and downsampling is performed through the CNN network to obtain feature information at different resolutions.

[0073] CNN network is a convolutional neural network. Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning.

[0074] The network structure diagram of Transformer Layer2 of the Transformer model is as follows Figure 3As shown, the input here adopts a similar approach to Vit, slicing the 2D image into a 1D sequence for input, and then position encoding is performed and input into the Transformer network to calculate the weights of its feature parts, and the calculation method is the same as above.

[0075]

[0076] like Figure 4 As shown in the figure, the U-Net network model of this example adopts the classic encoding and decoding structure of U-Net, including down-sampling, up-sampling and skip connection. The skip connection can fuse the features obtained by each down-sampling of the encoder part with the up-sampling results of the decoder part through cascade and convolution, so as to make the segmentation result more accurate. Down-sampling is mainly to increase the robustness of the model, slight rotation invariance, reduce training parameters and increase the network receptive field. Up-sampling is to gradually restore the extracted features to the label prediction that is consistent with the original image size. Here, it is mainly for the category prediction of pixel points, so the input image and its corresponding label need to be of the same size. Figure 4 In the figure, input is input, output is output, Max Pooling is maximum pooling, and Block copled is cropping and splicing.

[0077] The convolution layer is mainly composed of convolution calculations and activation functions. Its purpose is to extract different features of the input. The first convolution layer may only be able to extract some low-level features such as edges, lines, and corners. More layers of the network can iteratively extract more complex features from low-level features.

[0078] The pooling layer is used to reduce the feature vector output by the convolutional layer and improve the results so that the structure is not prone to overfitting.

[0079] Each node in the fully connected layer is connected to all the nodes in the previous layer, which is used to integrate the features extracted previously, that is, to map the learned features to the sample label space.

[0080] When the segmentation label images corresponding to each part after data preprocessing and the original images are used as training sets, the output obtained through network forward propagation is the probability of the category to which each pixel belongs, which is processed to obtain the segmentation model.

[0081] The improved U-NET-Transformer model in this example is:

[0082] First, a hybrid convolutional neural network and Transformer model are used to extract features. Then, the data information of each layer is densely connected and input into the hidden feature layer with the same resolution, and then segmented through the U-Net network. At the same time, the input image is directly position-encoded and input into the Transformer model to learn the position information and the attention weight of the feature. The obtained attention weight is returned to the U-NET network, and the attention weight is spliced ​​and added to the feature image to be spliced. Finally, the prediction result is returned through upsampling of the U-NET network.

[0083] As a preferred embodiment of the present invention, this example introduces the BCE Diceloss (Sigmoid and log loss) cross entropy loss function in the process of training the U-NET-Transformer model:

[0084]

[0085] Among them, c is the label 0 or 1, a is the output of the model after the activation function, ranging from (0,1), n ​​is the number of samples, and each time the network outputs a value, the BCE Diceloss cross entropy loss function gives a loss based on the value and the corresponding label. x is one of the samples, and its value is [1, n]. When c is 0, ylna is eliminated, leaving -ln(1-a). a needs to be close to 0 to reduce the loss value loss. When c is 1, (1-c)ln(1-a) is eliminated, leaving -lna. At this time, a needs to be close to 1 to reduce the loss value loss. This loss takes into account the cases of 0 and 1 and is usually used for binary classification.

[0086] Based on the above, the present invention makes further optimization and also introduces the binary foreground and background loss functions:

[0087] L(b)=max(0,1-t*b)

[0088] Among them, b is the predicted value, which ranges from -1 to 1, and t is the target value 1 or -1, which means that the value of b can be between -1 and 1, and |b|>1 is not encouraged, that is, a certain sample can be correctly classified, and the classifier is not encouraged to be overconfident. When the distance between the sample and the dividing line exceeds 1, there will be no reward. The purpose is to make the classifier more focused on the overall classification error.

[0089] Step S3: Obtain image segmentation results

[0090] Use the trained U-NET-Transformer model to segment the intracranial images that need to be segmented and obtain the segmentation results.

[0091] Step S4: Post-processing the segmentation results

[0092] In order to obtain more accurate segmentation results, this example performs post-processing on the obtained segmentation results. The post-processing methods in this example include: one or more of morphological operation method, image grayscale processing method, grayscale threshold segmentation method, region segmentation method, and edge detection segmentation method.

[0093] In order to make the boundary of the final prediction result smoother and eliminate the influence of small prediction noise, this example performs a morphological operation of first erosion and then dilation on the final prediction result.

[0094] Here, morphology refers to the content of mathematical morphology, which is used as a tool to extract useful image components that express and depict the shape of the region from the image, such as boundaries, skeletons, and convex hulls.

[0095] The operation method for corroding the segmentation result is:

[0096] For sets A and B, it is represented as The corrosion formula of B to A is: z=(f,g), where f and g are the coordinate values ​​of the points in the plane coordinate system. In this formula, the erosion of A by B is a set of all points z contained in A by translating the point z along the x and y directions;

[0097] The expansion operation mode is:

[0098] For sets A and B, it is represented as The expansion formula of B to A is: In this formula, B' is the image of B about its origin, and then based on the translation of the image B' by point z, the expansion of B to A is the set of all displacement points z.

[0099] As a preferred embodiment of the present invention, in order to make the segmentation result smoother and more complete, the image is post-processed as follows:

[0100] Image grayscale processing: Set different grayscale values ​​for each segmentation result according to the weight ratio. The more important the part, the larger the grayscale value.

[0101] At the same time, after image fusion to obtain a grayscale image, the fused image can be segmented again using the threshold segmentation method or edge detection segmentation method to calibrate the segmentation boundaries and obtain more accurate results, which is exactly what medical images require.

[0102] Grayscale threshold segmentation method: First, a basic threshold needs to be determined, and then each pixel is judged. The threshold processing formula is:

[0103]

[0104] f(i,j) represents the pixel with abscissa i and ordinate j, and g(i,j) represents each pixel after threshold segmentation. The part greater than the threshold is set to 1 (white), and the part less than the threshold is set to 0 (black), so that the image is divided into different regions according to the threshold. Similar to this is the region segmentation method, such as the region growing method, which uses the similarity of the grayscale values ​​of pixels in the region to merge similar regions and separate dissimilar regions. The specific steps are: first find a pixel point in each region to be segmented as the seed point of the region; then, according to certain connectivity rules, the pixels around the seed point with similar properties are merged into the region of the seed point according to certain criteria; finally, these new pixels are used as new seed points to continue the above steps until the segmentation of the entire image is completed.

[0105] The condition for reaching the maximum value is the edge detection segmentation method: the edge of an object is the part of the image where the local brightness changes most significantly. Based on this, the edge points are first detected, and then the edge points are connected into edge lines according to a certain method to segment the image area. Since the edge line neighborhood on the image is an area where the grayscale changes more dramatically in the image, the two most effective features to measure this change are the rate of change and the direction of change. Mathematically speaking, it is the amplitude and direction of the gradient vector. Therefore, for an image f(x,y), finding the local maximum value and direction of its gradient is edge detection. It is known that the gradient of f(x,y) along r in the q direction is defined as follows:

[0106]

[0107] gradient The condition for reaching the maximum value is Right now:

[0108] -f x sinθ g +f y cosθ g =0 to get

[0109] Maximum value of gradient (gradient modulus):

[0110]

[0111] The gradient modulus operator is isotropic and displacement invariant, and is suitable for edge detection. The direction of grayscale change, that is, the boundary direction, can be determined by θ g =arctanθf y / f x Calculated.

[0112] After completing the above steps, you can perform semantic segmentation through multiple training models, obtain multiple semantic segmentation results, and then fuse them back into one image to obtain results similar to instance segmentation.

[0113] The present invention also provides a system for implementing the multi-organ instance segmentation method, comprising:

[0114] Constructing a training data set module: used to obtain the medical images to be segmented, perform preprocessing, segment and annotate each part on each preprocessed medical image, and obtain an instance segmentation data set after integration as a training data set; Constructing a training model module: constructing an image segmentation network based on a U-Net network and a Transformer model, and then using the training data set to train the image segmentation network to obtain a U-NET-Transformer model. The U-NET-Transformer model is as follows: first, a hybrid convolutional neural network and a Transformer model are used for feature extraction, and then each layer of data information is densely connected and input into the hidden feature layer with the same resolution, and then segmented through a U-Net network. At the same time, the input image is directly position-encoded and input into the Transformer model to learn the position information and the attention weight of the feature, and the obtained attention weight is returned to the U-NET network, and the attention weight is spliced ​​and added to the feature image to be spliced, and finally the prediction result is returned through upsampling of the U-NET network;

[0115] Image segmentation result acquisition module: used to segment the intracranial image to be segmented using the trained U-NET-Transformer model and obtain the segmentation result;

[0116] Post-processing module: used to post-process the segmentation results to obtain more accurate segmentation results.

[0117] The beneficial effects of the present invention include at least:

[0118] (1) It can display the mutual influence relationship of various parts in medical images on one image, allowing doctors to diagnose the disease more clearly and conveniently;

[0119] (2) The U-NET network and the transformer model are integrated to achieve better instance segmentation results for complex structures. With the unique encoding and decoding structure and skip connection characteristics of the U-NET network, high-level features and low-level features can be better integrated. This efficient feature extraction method can use fewer data sets to train the model to achieve the final effect.

[0120] (3) Using the transformer model to learn the importance of position encoding can more effectively grasp the global information. Combining it with the CNN network for feature extraction can obtain better feature information and make the segmentation results more accurate.

[0121] (4) The segmentation effect of the existing instance segmentation method is weaker than the existing semantic segmentation result. Using a method that fuses multiple semantic segmentation results can obtain more accurate segmentation results.

[0122] The specific implementation modes described above are preferred implementation modes of the present invention, and are not intended to limit the specific implementation scope of the present invention. The scope of the present invention includes but is not limited to the specific implementation modes, and all equivalent changes made according to the present invention are within the protection scope of the present invention.

Claims

1. A multi-organ instance segmentation method, characterized in that: The steps include: S1: Constructing a training data set: Obtain the medical images to be segmented, perform preprocessing, segment and annotate each part on each preprocessed medical image, and integrate them to obtain an instance segmentation data set as the training data set; S2: Constructing a training model: constructing an image segmentation network based on a U-Net network and a Transformer model, and then using the training data set to train the image segmentation network to obtain a U-NET-Transformer model. The U-NET-Transformer model is as follows: first, a hybrid convolutional neural network and a Transformer model are used for feature extraction, and then each layer of data information is densely connected and input into a hidden feature layer with the same resolution, and then segmented through a U-Net network. At the same time, the input image is directly position-encoded and input into the Transformer model to learn the position information and the attention weight of the feature, and the obtained attention weight is returned to the U-NET network, and the attention weight is spliced ​​and added to the feature image to be spliced, and finally the prediction result is returned through upsampling of the U-NET network; S3: Obtain image segmentation results: Use the trained U-NET-Transformer model to segment the intracranial image to obtain the segmentation results.

2. The multi-organ instance segmentation method according to claim 1, characterized in that: The method also includes step S4: post-processing the segmentation result to obtain a more accurate segmentation result. The post-processing method includes one or more of a morphological operation method, an image grayscale processing method, a grayscale threshold segmentation method, a region segmentation method, and an edge detection segmentation method. When the post-processing method includes multiple methods, multiple semantic segmentation results are obtained through multiple post-processing methods, and then the multiple semantic segmentation results are fused back into one image to obtain a more accurate segmentation result.

3. The multi-organ instance segmentation method according to claim 2, characterized in that: The morphological operation method is a morphological operation of first corroding and then dilating. The operation method of corroding the segmentation result is: For sets A and B, it is represented as The corrosion formula of B to A is: z=(f,g), where f and g are the coordinate values ​​of the points in the plane coordinate system. In this formula, the erosion of A by B is a set of all points z contained in A by translating the point z along the x and y directions; The expansion operation mode is: For sets A and B, the expansion formula of B to A, expressed as A⊕B, is: z=(f,g), in which B' is the image of B about its origin, and then based on the translation of the image B' based on the point z, the expansion of B to A is the set of all displacement points z.

4. The multi-organ instance segmentation method according to claim 2, characterized in that: The method of the image grayscale processing method is: setting different grayscale values ​​for each segmentation result according to the weight ratio, and the more important the part, the larger the grayscale value set; After the grayscale image is obtained by image fusion through the image grayscale processing method, the fused image is segmented again using the threshold segmentation method or the edge detection segmentation method to calibrate the segmentation boundary and obtain a more accurate result.

5. The method for segmenting multiple organ instances according to any one of claims 1 to 4, characterized in that: In step S1, the image preprocessing method includes one or more of the following: scaling, resizing, contrast enhancement, grayscale color images and then binarization, scaling, and rotating data to achieve data set expansion.

6. The multi-organ instance segmentation method according to claim 5, characterized in that: In step S1, the method for image preprocessing is: different parts of the same original medical image need to be marked and segmented for information marking processing to facilitate subsequent instance segmentation. The preprocessing is to convert the original medical image into a fixed size of 512*512, and perform contrast enhancement processing. After graying the color image, the grayscale pixels are converted to between 0-1 to reduce the scale of the input features. After preprocessing, it also includes an integration step: the preprocessed data set is integrated with the original input image into a training data set.

7. The method for segmenting multiple organ instances according to any one of claims 1 to 4, characterized in that: In step S2, the U-Net network model includes an upsampling layer, a downsampling layer, a skip connection structure, and feature fusion between layers and a pruning operation is added. The Transformer model is entirely composed of an attention mechanism, and a convolutional neural network is used to mix the Transformer model to extract features of different levels of the signal at the pixel level.

8. The multi-organ instance segmentation method according to claim 7, characterized in that: In step S2, during the training of the U-NET-Transformer model, the BCE Diceloss cross entropy loss function is introduced: Among them, c is the label 0 or 1, a is the output of the model after the activation function, ranging from (0,1), n ​​is the number of samples, and each time the network outputs a value, the BCE Diceloss cross entropy loss function gives a loss based on the value and the corresponding label. x is a specific sample, and its value is [1, n]. When c is 0, ylna is eliminated, leaving -ln(1-a). a needs to be close to 0 to reduce the loss value loss. When c is 1, (1-c)ln(1-a) is eliminated, leaving -lna. At this time, a needs to be close to 1 to reduce the loss value loss.

9. The multi-organ instance segmentation method according to claim 8, characterized in that: It also includes binary foreground and background loss functions: L(b)=max(0,1-t*b) Among them, b is the predicted value, which ranges from -1 to 1, and t is the target value 1 or -1, which means that the value of b can be between -1 and 1, and |b|>1 is not encouraged, that is, a sample can be correctly classified, and overconfidence of the classifier is not encouraged. When the distance between the sample and the dividing line exceeds 1, there will be no reward.

10. A system for implementing the multi-organ instance segmentation method according to any one of claims 1 to 9, characterized in that: include: Constructing a training data set module: It is used to obtain the medical images to be segmented, perform preprocessing, segment and annotate each part on each preprocessed medical image, and integrate them to obtain an instance segmentation data set as a training data set; Constructing a training model module: constructing an image segmentation network based on a U-Net network and a Transformer model, and then using the training data set to train the image segmentation network to obtain a U-NET-Transformer model. The U-NET-Transformer model is as follows: first, a hybrid convolutional neural network and a Transformer model are used for feature extraction, and then each layer of data information is densely connected and input into a hidden feature layer with the same resolution, and then segmented through a U-Net network. At the same time, the input image is directly position-encoded and input into the Transformer model to learn the position information and the attention weight of the feature, and the obtained attention weight is returned to the U-NET network, and the attention weight is spliced ​​and added to the feature image to be spliced, and finally the prediction result is returned through upsampling of the U-NET network; Image segmentation result acquisition module: used to use the trained U-NET-Transformer model to segment the intracranial image to be segmented and obtain the segmentation result.

Images