Multi-task medical image segmentation method and system supporting multi-doctor collaborative supervision

Abstract

The present invention discloses a multi-task medical image segmentation method and system supporting multi-doctor collaborative supervision, the method comprising: obtaining at least one medical image set containing annotation information; sorting each medical image in the same medical image set according to the size of the difference coefficient to obtain at least one medical image sequence; using the at least one medical image sequence and the image segmentation result corresponding to the at least one medical image sequence as the input and output of a hybrid neural network based on improved convolution and attention, iteratively training the hybrid neural network to obtain an image segmentation model; inputting the medical image containing annotation information obtained in real time into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result. Multi-task processing based on multiple decoders is realized, and this design enhances the ability of the encoder to extract image features, thereby improving the model segmentation performance and accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a multi-task medical image segmentation method and system supporting collaborative supervision by multiple doctors. Background Art

[0002] Medical image analysis has gradually evolved from traditional image processing techniques to advanced analysis methods based on machine learning. In early medical image analysis research, many techniques were based on traditional machine learning methods, such as support vector machines (SVMs) and random forests. These methods have achieved good results on certain specific academic datasets, but are not so ideal for deployment in actual medical environments.

[0003] One of the main reasons is that academic research usually tends to "brush indicators" on specific, often standardized, single-modality, and limited data sets. Although such research has its value in academia, the data situation in real medical environments is much more complicated. For example, the same hospital may use different equipment, different parameters, and different operators, which leads to a large difference between the actual images obtained and the academic data sets, which in turn hinders the progress of artificial intelligence technology (AI) from academic research to practical application. More importantly, most traditional machine learning and deep learning methods are not designed with multi-doctor labeling in mind. In a real medical environment, different doctors may have different interpretations and annotations of the same medical image (such as Figure 1 This difference may be due to the doctor's professional background, experience, or understanding of certain diseases. Traditional AI methods usually assume that there is a "standard" annotated data set for training and testing, rather than taking into account the diversity of annotations. Summary of the invention

[0004] The present invention provides a multi-task medical image segmentation method, system and readable storage medium that support collaborative supervision by multiple doctors, which are used to solve the technical problem that traditional AI methods usually use the annotation information of a single doctor as the basic standard to build a data set and perform AI model training, which often ignores the rich information annotated by multiple doctors and cannot handle the subtle differences in annotations between doctors.

[0005] In a first aspect, the present invention provides a multi-task medical image segmentation method supporting multi-doctor collaborative supervision, comprising:

[0006] Acquire at least one medical image set containing annotation information, wherein one medical image set contains the same medical image with different annotation information, and the annotation information is information annotated by a user on a target area in the medical image;

[0007] Calculating the difference coefficients between the medical images in the same medical image set according to a preset difference function, and sorting the medical images in the same medical image set according to the magnitude of the difference coefficients to obtain at least one medical image sequence, wherein a target medical image sequence only includes the same medical images with different annotation information;

[0008] The at least one medical image sequence and the image segmentation result corresponding to the at least one medical image sequence are used as the input and output of a hybrid neural network based on improved convolution and attention, and the hybrid neural network is iteratively trained to obtain an image segmentation model, wherein the at least one medical image sequence is updated according to a preset dynamic update strategy during the iterative training of the hybrid neural network;

[0009] The medical image containing the annotation information acquired in real time is input into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result.

[0010] In a second aspect, the present invention provides a multi-task medical image segmentation system supporting multi-doctor collaborative supervision, comprising:

[0011] an acquisition module configured to acquire at least one medical image set containing annotation information, wherein one medical image set contains the same medical image with different annotation information, and the annotation information is information annotated by a user on a target area in the medical image;

[0012] a calculation module configured to calculate a difference coefficient between each medical image in the same medical image set according to a preset difference function, and sort each medical image in the same medical image set according to the magnitude of the difference coefficient to obtain at least one medical image sequence, wherein a target medical image sequence only contains the same medical image with different annotation information;

[0013] A training module, configured to use the at least one medical image sequence and the image segmentation result corresponding to the at least one medical image sequence as the input and output of a hybrid neural network based on improved convolution and attention, iteratively train the hybrid neural network to obtain an image segmentation model, wherein the at least one medical image sequence is updated according to a preset dynamic update strategy during the iterative training of the hybrid neural network;

[0014] The output module is configured to input the medical image containing annotation information acquired in real time into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result.

[0015] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the multi-task medical image segmentation method supporting multi-doctor collaborative supervision according to any embodiment of the present invention.

[0016] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor executes the steps of a multi-task medical image segmentation method supporting multi-doctor collaborative supervision according to any embodiment of the present invention.

[0017] The multi-task medical image segmentation method and system supporting multi-doctor collaborative supervision of the present application has the following beneficial effects:

[0018] 1. The present invention incorporates a self-attention mechanism into the underlying part of the network, which enables the model to efficiently learn high-dimensional features and automatically focus on the key parts of the image, helping to more deeply capture and understand the subtle differences between multiple doctors' annotations;

[0019] 2. The present invention uses dilated convolution and deformable convolution in the encoder and dual decoder. Dilated convolution can increase the receptive field of the network without increasing the computational cost; while deformable convolution can make the network better adapt to the deformation in the image and improve the adaptability of the model to the differences in annotations of different doctors;

[0020] 3. The present invention proposes a multi-decoder structure that supports multiple tasks. While one decoder supports outputting image segmentation prediction, another decoder supports outputting the geometric center of the target area, thereby enhancing the feature extraction capability of the encoder.

[0021] 4. The present invention proposes an adaptive training strategy, which actively removes potential incorrectly labeled training data by learning the annotation differences between different doctors. It can integrate the professional knowledge and skills of each doctor, which not only helps to generate images with more advanced semantic features, but also optimizes the results of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1A flowchart of a multi-task medical image segmentation method supporting multi-doctor collaborative supervision provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of a model training process in the case of multiple medical staff annotating data is provided for one embodiment of the present invention;

[0025] Figure 3 A diagram of an encoder-dual decoder image segmentation neural network architecture based on a self-attention mechanism and a novel convolution kernel according to a specific embodiment of an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of the details of the bottom architecture of an image segmentation neural network according to a specific embodiment of the present invention is provided;

[0027] Figure 5 A detailed schematic diagram of an image segmentation neural network encoder-decoder is provided for one embodiment of the present invention;

[0028] Figure 6 A curve diagram showing the relationship between the amount of data with incorrect labels removed and the training process according to a specific embodiment of the present invention is provided;

[0029] Figure 7 A simplified schematic diagram of a traditional convolution kernel and a proposed improved convolution kernel according to a specific embodiment of the present invention is provided;

[0030] Figure 8 A structural block diagram of a multi-task medical image segmentation system supporting multi-doctor collaborative supervision provided by an embodiment of the present invention;

[0031] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0033] In the description of this specification, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps is not limited and can be appropriately adjusted as needed.

[0034] In the image target region segmentation task in medical image analysis, various types of medical images, such as ultrasound, CT, MRI and other images, need to be accurately segmented for organs and lesions to provide accurate diagnostic information. However, due to the objective complexity of medical images and the subjective differences in doctors' annotations, accurate medical image segmentation is extremely challenging. Past AI methods usually use the annotation information of a single doctor as the basic standard to build a data set and train AI models. However, this often ignores the rich information of multiple doctors' annotations and cannot handle the subtle differences in annotations between doctors. Therefore, how to effectively use multi-doctor annotations to train artificial neural networks for medical image segmentation has become an important issue.

[0035] The following will introduce in detail a multi-task medical image segmentation method supporting multi-doctor collaborative supervision provided by an embodiment of the present invention in conjunction with the accompanying drawings.

[0036] See also Figure 1 , which shows a flowchart of a multi-task medical image segmentation method of the present application that supports multi-doctor collaborative supervision.

[0037] like Figure 1 As shown, the multi-task medical image segmentation method supporting multi-doctor collaborative supervision specifically includes steps S101 to S104.

[0038] Step S101, obtaining at least one medical image set containing annotation information, wherein one medical image set contains the same medical image with different annotation information, and the annotation information is information annotated by a user on a target area in the medical image.

[0039] In this step, annotations from multiple doctors are received as input. These annotations can be regarded as feedback judgments from multiple raters with different levels of expertise on the same medical image. For each doctor, the network learns their specific annotation style and professional level during the training process. The purpose of this step is to enable the neural network to understand and learn the structure and features of the image from the perspective of multiple doctors. The neural network implements parameter initialization, and the training process is optimized using the gradient descent method. For each image, the model accepts annotations from multiple doctors, and all the doctors' annotations are added to the subsequent training to initialize the parameters of the model, so that the model can be preliminarily trained on medical imaging data. Although it is usually easy for multiple doctors to disagree on the boundaries of the target area, the center position of the target area is often consistent. Therefore, in the data preprocessing stage, the regional geometric center is additionally generated based on the regional boundaries annotated by multiple doctors, which is recorded as additional auxiliary annotation information and added to the subsequent training.

[0040] Step S102, calculating the difference coefficient between each medical image in the same medical image set according to a preset difference function, and sorting each medical image in the same medical image set according to the size of the difference coefficient to obtain at least one medical image sequence, wherein a target medical image sequence only contains the same medical image with different annotation information.

[0041] In this step, by comparing the target region segmentation annotations of different doctors, doctors calculate the difference between the same medical image, so as to perceive the difference between doctors and quantify the difference. Specifically, the difference calculation is based on the Dice-Coefficient, which ranges from 0 to 1, where 1 indicates perfect similarity and 0 indicates no similarity. It should be noted that the expression of the difference function is:

[0042]

[0043] Where Dice is the coefficient of difference, X represents the annotation information of the first doctor, Y represents the annotation information of the second doctor, |X∩Y| represents the number of pixels in the same target area annotated by the two doctors, and |X|+|Y / represents the total number of pixels in the same target area annotated by the two doctors.

[0044] Furthermore, according to the progress of subsequent model training and the current fitting status, the labeled data of doctors to be used are dynamically adjusted to optimize the training process. The model training process can be roughly divided into underfitting and overfitting. In the early stage of model training, due to parameter initialization, the model is in an underfitting state. At this stage, we hope to use as much data as possible to accelerate the learning of the model. In the later stage of model training, when the model performance is already good, we hope that the model can identify low-quality or wrong annotations and remove them, so as to avoid overfitting the model to these possible mislabelings.

[0045] In order to implement this dynamic strategy, the use of doctor annotation data is quantified. Specifically, the difference between each doctor's annotation and other doctors' annotations is calculated. Through the difference function, the quality and credibility of each doctor's annotation can be quantified. These difference data will be sorted from large to small. A large difference may mean that the doctor's annotation is greatly deviated from the majority of doctors, and there is a potential possibility of incorrect annotation.

[0046] For example, there are pictures A, B, C, and D. The difference values ​​of picture A and picture B, picture A and picture C, picture A and picture D, picture B and picture C, picture B and picture D, and picture C and picture D are calculated. For picture A, the difference between picture A and picture B, picture C, and picture D can be calculated and summed. Similarly, the difference between picture B and picture A, picture C, and picture D can be calculated and summed. Similarly, the difference between picture C and picture A, picture B, and picture D can be calculated and summed. Similarly, the difference between picture D and picture A, picture B, and picture C can be calculated and summed. Then the differences between pictures A, picture B, picture C, and picture D are sorted.

[0047] Step S103, using the at least one medical image sequence and the image segmentation result corresponding to the at least one medical image sequence as the input and output of a hybrid neural network based on improved convolution and attention, iteratively training the hybrid neural network to obtain an image segmentation model, wherein during the iterative training of the hybrid neural network, the at least one medical image sequence is updated according to a preset dynamic update strategy.

[0048] In this step, after the initial training, the strategy of "the minority obeys the majority" is applied in the subsequent training to identify and reduce subjective or objective "erroneous noise annotations". Specifically, when a doctor's annotation is different from that of all other doctors, the network will actively remove the doctor's annotation. The purpose of this step is to avoid the impact of a single doctor's misjudgment on the entire neural network training. In each training round, for the same image, the doctors' annotations are tested for differences in pairs and sorted from high to low. The greater the difference between a doctor's annotation and other doctors, the more potential there is for erroneous annotations, and they are removed. The determination of the number of eliminations is related to the training process. Fewer images are eliminated in the early stage of training, and more images are eliminated in the later stage of training. The reason is that the initial goal of hybrid neural network training is the initial training and initialization parameters of the model. Even if there are "erroneous noise annotations" in all doctors' annotations, they are meaningful for the initialization training of the hybrid neural network. In the later stage of training, when the hybrid neural network needs to achieve higher performance, "erroneous noise annotations" have a greater impact on the training of the hybrid neural network. Therefore, the Ramp Up function is used to determine the specific number of images to be eliminated in different training stages, such as Figure 6 As shown, the expression of the Ramp Up function is: y = e mx , where m is a positive real number, x is the model training progress, and y is the number of eliminated samples; at least one medical image sequence is updated according to the number of eliminated images.

[0049] This strategy ensures that all data are utilized to the maximum extent in the early stages of hybrid neural network training, while in the later stages, I focus on labeled data with higher quality and smaller differences, thereby improving the robustness and stability of the hybrid neural network.

[0050] Step S104, inputting the medical image containing the annotation information acquired in real time into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result.

[0051] In this step, in the training generated by the image segmentation decoder, the hybrid loss function contains the following three segmentation loss functions:

[0052] Cross-Entropy Loss: Used to calculate the difference between the model’s prediction and the doctor’s annotation at each pixel.

[0053] Dice Loss: Measures the similarity between the model's prediction and the doctor's annotation in terms of size area.

[0054] Boundary-Aware Loss: A loss function based on the model’s predictions and the topological structure of the doctor’s annotations.

[0055] By combining these losses, the model is able to fully learn and accurately segment images based on the given dataset. The novel hybrid loss function can be expressed as:

[0056]

[0057]

[0058]

[0059] At the same time, in the prediction result generated by another geometric center decoder, the distance loss function is taken, and the expression is:

[0060] Loss 距离 =(X 目标 -X 标注 ) 2 +(Y 目标 -Y 标注 ) 2 ,

[0061] Where X is the abscissa of the geometric center of the target area, and Y is the ordinate of the geometric center of the target area;

[0062] The loss function of the encoder-dual decoder is obtained as follows:

[0063] Loss=Loss 分割 +Loss 距离 ,

[0064] In the formula, β and γ are the weights of the cross entropy loss function, the Dice loss function, and the boundary loss function, respectively. M is the number of categories in the target area, c is the specific category, y is the label, p is the model prediction, and Loss 分割 is the segmentation loss function, Loss 距离 is the distance loss function, D represents the distance transformation, S is the predicted boundary, g is the labeled boundary, and X 目标 is the horizontal coordinate of the geometric center of the target area, X 标注 is the horizontal coordinate of any point in the target area, Y 目标 is the ordinate of the geometric center of the target area, Y 标注 is the ordinate of any point in the target area, yc is the doctor's annotation that this pixel belongs to category c, and p c The model predicts the probability that this annotation point belongs to class c, s c is the predicted boundary belonging to category c, g c is the labeled boundary belonging to category c.

[0065] After obtaining the loss function Loss, we use optimizers including but not limited to Gradient Descent, Stochastic Gradient Descent (SGD), Adagrad, RMSprop, or Adam (Adaptive Moment Estimation) to update and calculate the weights and parameters in the network to reduce the loss function. Their purpose is to find the minimum value of the loss function during the training process, thereby improving the accuracy and efficiency of the model. Each optimizer has its applicable scenarios and specific types of network architectures. In practical applications, which optimizer to choose usually depends on the requirements of the specific problem, the type of data, and the complexity of the model to be optimized. Sometimes, in order to achieve better training results, researchers and engineers can try and adjust the optimizer.

[0066] It should also be pointed out that while proposing collaborative training of multi-doctor annotation, innovation is made in the hybrid neural network itself. The present invention proposes an encoder-dual decoder architecture, and at the same time introduces a method of mixing a self-attention mechanism (Self-Attention) and a convolutional neural network (CNN) in the design of each layer of the neural network to better learn and extract effective information from multi-doctor annotations. The purpose is to further improve the learning ability and adaptability of the network. An encoder-dual decoder design is proposed, which includes an image general feature encoding process, an upsampled image segmentation decoding process, and a target center coordinate decoding process based on the VGG model. In the image general feature encoding process, the model extracts and learns features from the input image; in the target center coordinate decoding process, the model will realize the task of predicting the geometric center of the target area; in the upsampled image segmentation decoding process, the model will map the learned features back to the pixel level, thereby completing image segmentation. Transformer is a model widely used in natural language processing, and its core is the self-attention mechanism (Self-Attention). The self-attention mechanism enables the model to better capture the relationship between the parts of the input data, rather than just local information. This makes Transformer have superior performance when processing sequence data, such as text. In this new model, the self-attention mechanism is introduced into the encoder-dual decoder (such as Figure 2 As shown in Figure 2, the model can not only consider the local information of the image, but also the global information of the image when processing the medical image segmentation task, so as to better learn and extract effective information from multiple doctors' annotations. This hybrid method of self-attention mechanism and convolutional neural network improves the learning ability and adaptability of the network, making the results of medical image segmentation more accurate and efficient. It can be divided into the following stages (such as Figure 3 shown):

[0067] Input stage: The network receives a medical image as input. This image may be a CT, MRI or other types of medical images.

[0068] Feature extraction and encoding stage: This stage uses the convolutional neural network layer to downsample the input image and gradually extract the features of the image. Each convolution operation will halve the size of the image and increase the number of feature maps. This process encodes the spatial information of the image into deep features while retaining the position information of the original image.

[0069] Self-attention stage: After the downsampling stage, we apply the self-attention mechanism to the extracted features. The purpose of this stage is to identify the intrinsic correlation between the features in the image, even if these features are spatially separated. In this way, the model is able to capture more complex and global information than just the local convolution window.

[0070] Upsampling image segmentation decoding stage: This stage uses an inverse convolution (or deconvolution) neural network to expand the feature map back to the size of the original image to complete the image reconstruction. During this process, the network will classify each pixel to determine whether it belongs to a specific tissue or structure. At the same time, the decoding stage will also jump to the corresponding layer of the encoding stage to retain more detailed information.

[0071] Geometric center decoding stage: This stage adopts the classic convolutional network architecture based on VGG to fit the geometric centers of different target areas in high-dimensional features.

[0072] Output stage: Finally, the network outputs the segmented medical image. Each pixel is classified into a specific category (such as tumor, normal tissue, etc.), forming a segmentation map.

[0073] This encoder structure with a self-attention mechanism can better learn and extract effective information from multiple doctors' annotations, further improving the accuracy and efficiency of medical image segmentation. In the dual decoder architecture, this design is particularly suitable for scenarios where multiple doctors collaborate in training. Although different doctors may have certain differences in their annotations of boundaries, they often have a high degree of consistency in the geometric center of the target area. Therefore, by simultaneously training the model to optimize the two tasks of image segmentation and geometric center positioning, the encoder's feature extraction capability can be effectively improved, thereby enhancing the performance of the overall model.

[0074] Neural network encoder, dual decoder module (such as Figure 4). The neural network encoder contains four downsampling stages, each consisting of two consecutive convolutional neural network layers. These convolution operations are performed to gradually extract image features and gradually increase the depth of the feature map. As the network depth increases, the spatial dimension gradually decreases, but the feature information becomes richer. After each downsampling step, a maximum pooling operation is usually used to reduce the spatial size of the feature map to capture a wider range of contextual information. The image segmentation decoder is the opposite of the encoder and contains four upsampling stages. Similarly, each upsampling stage consists of two consecutive convolutional neural network layers, but here, transposed convolution or deconvolution is used to increase the spatial size of the feature map. In order to retain more detail information in the decoding stage, we insert a skip connection after each upsampling step to pass the feature map of the corresponding depth directly from the encoder. This skip connection ensures that the image segmentation decoder can use the original image details captured by the encoder to reconstruct the image more accurately. Specifically, the feature extraction process can be expressed as follows:

[0075] First, from the input feature map X i-1 Initially, its size is C in ×H×W. Among them, C in is the number of channels of the input feature map, while H and W represent the height and width of the feature map respectively.

[0076] In the traditional convolution operation, it can be expressed as:

[0077] X i = Dropout(BatchNorm(W*X i-1 )),

[0078] In the formula, * represents the convolution operation, BatchNorm is the batch normalization operation, Dropout is the random inactivation operation, and X i is the output feature obtained after being processed by a layer of convolutional neural network.

[0079] It is proposed that in the encoder-dual decoder network (such as Figure 5 As shown in Figure 2, a new type of convolution operation is used. Dilated CNN introduces a dilation factor r to increase the receptive field of the convolution kernel. The spacing between each element in the convolution kernel is multiplied by the dilation rate r. When the dilation rate r is 1, the dilated convolution is equivalent to the standard convolution. When the dilation rate r increases, the input feature map area covered by the convolution kernel will expand, but the number of elements actually involved in the operation will not increase.

[0080] In addition, in order to further capture complex geometric deformations, the flexibility of the model is further increased by introducing deformable convolution (Deformable CNN). Unlike traditional convolution, variable convolution can slightly offset the position of each element in the convolution kernel to capture more complex geometric structures. It adds learnable offsets on the basis of traditional convolution, which allow each element of the convolution kernel to move adaptively on the input feature map. This means that the convolution kernel is no longer fixed on a regular grid, but can automatically adjust its shape according to the image content. Mathematically, this can be represented by adding an offset field ΔP, which is learned by the network and is used to adjust the position of each convolution kernel element. Specifically, for deformable convolution, each convolution kernel position P0 will adjust its position according to the learned offset ΔP, and the calculation formula is as follows:

[0081]

[0082] Where, X i is the output feature of the i-th layer of the neural network, Dropout is a common network regularization technique, DilatedConv(W, r) is a dilated convolution, X i-1 is the output feature of the i-1th layer of the neural network, Deformable(W, r) is the deformable convolution, BatchNorm is the batch normalization, * is the convolution operation, W(P n ) is the convolution kernel at position P n The weight of P0 is the convolution kernel position, P n is a position in R, R is the regular grid area covered by the convolution kernel, ΔP n For position P n The corresponding offset, X i-1 (P0+P n +ΔP n ) is the value of the input feature map at the adjusted position.

[0083] Therefore, combining dilated convolution and deformable convolution, the two-layer convolution operation in the encoder and dual decoder can be more generally expressed as:

[0084] X i =Dropout(BatchNorm(DeformableConv(DilatedConv(W, r)*X i-1 ))),

[0085] In this formula, the dilated convolution kernel, deformable convolution kernel, and traditional convolution kernel used in the encoder-dual decoder are Figure 7 shown.

[0086] Finally, we get the output feature map Xi The size is C out ×H×W. Here C out is the number of output channels and is determined by the depth and design of the convolutional network, while H and W represent the height and width of the feature map, respectively.

[0087] At the bottom of the encoder, we introduce a self-attention layer (e.g. Figure 4 This is the innovation of this architecture. The self-attention mechanism enables the model to focus on various parts of the input feature map and assign weights according to their importance. This mechanism allows the model to better capture long-range dependencies, so that all parts in the high-dimensional feature space can be properly handled. The specific mathematical expression can be expressed as:

[0088] First, we have an input feature matrix X i-1 , where each row is a feature vector.

[0089] X i-1 =[x1, x2, ..., x n ] T ,

[0090] Where n is the sequence length. Secondly, it is processed by layer normalization and multi-head self-attention mechanism.

[0091] X i =Multi-head self-attention mechanism (layer normalization (X i-1 ))+X i ;

[0092] Next, the feature information enters the multi-layer perceptron.

[0093] X i+1 = Multilayer Perceptron (layer normalization (X i ))+X i ;

[0094] Final output X i+1 for:

[0095] X i-1 =[x1, x2, ..., x n ] T .

[0096] In summary, the method of the present application, taking into account the challenging multi-doctor collaborative labeling situation proposed, innovatively proposes a multi-task learning framework, which simultaneously performs image segmentation and geometric center prediction through an encoder and two decoders. In medical image segmentation and labeling, the controversial parts of multi-doctor labeling are usually the boundary areas around target areas such as tumors and organs, while the geometric center of the target area is often undisputed. Therefore, the geometric center information is fully utilized to propose a multi-task training framework, so that the decoder of the model can learn to predict the geometric center of the target area while predicting the segmentation of the target area, thereby realizing multi-task processing based on multiple decoders. This design enhances the encoder's ability to extract image features, thereby improving the model segmentation performance and accuracy; and combines the advantages of CNN and self-attention mechanism to build an encoder-dual decoder artificial neural network, and proposes a novel adaptive training strategy, which provides a more powerful and flexible tool for medical image processing, especially when processing complex, multi-doctor labeled medical image data.

[0097] Please refer to Figure 8 , which shows a structural block diagram of a multi-task medical image segmentation system supporting multi-doctor collaborative supervision in the present application.

[0098] like Figure 8 As shown, the multi-task medical image segmentation system 200 includes an acquisition module 210 , a calculation module 220 , a training module 230 and an output module 240 .

[0099] Among them, the acquisition module 210 is configured to acquire at least one medical image set containing annotation information, wherein a medical image set contains the same medical image with different annotation information, and the annotation information is the information that the user annotates the target area in the medical image; the calculation module 220 is configured to calculate the difference coefficient between each medical image in the same medical image set according to a preset difference function, and sort the medical images in the same medical image set according to the size of the difference coefficient to obtain at least one medical image sequence, wherein a target medical image sequence only contains the same medical image with different annotation information; the training module 230 is configured to use the at least one medical image sequence and the image segmentation result corresponding to the at least one medical image sequence as the input and output of the hybrid neural network based on the improved convolution and attention, iteratively train the hybrid neural network, and obtain an image segmentation model, wherein during the iterative training of the hybrid neural network, the at least one medical image sequence is updated according to a preset dynamic update strategy; the output module 240 is configured to input the medical image containing annotation information obtained in real time into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result.

[0100] It should be understood that Figure 8 Modules and references documented in Figure 1 Therefore, the operations and features described above for the method and the corresponding technical effects are also applicable to Figure 8 The modules in it will not be described in detail here.

[0101] In some other embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor executes a multi-task medical image segmentation method supporting multi-doctor collaborative supervision in any of the above method embodiments;

[0102] As an implementation mode, the computer-readable storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows:

[0103] Acquire at least one medical image set containing annotation information, wherein one medical image set contains the same medical image with different annotation information, and the annotation information is information annotated by a user on a target area in the medical image;

[0104] Calculating the difference coefficients between the medical images in the same medical image set according to a preset difference function, and sorting the medical images in the same medical image set according to the magnitude of the difference coefficients to obtain at least one medical image sequence, wherein a target medical image sequence only includes the same medical images with different annotation information;

[0105] The at least one medical image sequence and the image segmentation result corresponding to the at least one medical image sequence are used as the input and output of a hybrid neural network based on improved convolution and attention, and the hybrid neural network is iteratively trained to obtain an image segmentation model, wherein the at least one medical image sequence is updated according to a preset dynamic update strategy during the iterative training of the hybrid neural network;

[0106] The medical image containing the annotation information acquired in real time is input into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result.

[0107] The computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of a multi-task medical image segmentation system supporting collaborative supervision by multiple doctors, etc. In addition, the computer-readable storage medium may include a high-speed random access memory, and may also include a memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the multi-task medical image segmentation system supporting collaborative supervision by multiple doctors via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] Fig. 9 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the device includes: a processor 310 and a memory 320. The electronic device may also include: an input device 330 and an output device 340. The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Fig. 9 The example of connection via bus is taken. The memory 320 is the above-mentioned computer-readable storage medium. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the multi-task medical image segmentation method supporting multi-doctor collaborative supervision of the above-mentioned method embodiment is realized. The input device 330 can receive input digital or character information, and generate key signal input related to user settings and function control of the multi-task medical image segmentation system supporting multi-doctor collaborative supervision. The output device 340 may include display devices such as display screens.

[0109] The electronic device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0110] As an implementation mode, the electronic device is applied to a multi-task medical image segmentation system supporting multi-doctor collaborative supervision, and is used for a client, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can:

[0111] Acquire at least one medical image set containing annotation information, wherein one medical image set contains the same medical image with different annotation information, and the annotation information is information annotated by a user on a target area in the medical image;

[0112] Calculating the difference coefficients between the medical images in the same medical image set according to a preset difference function, and sorting the medical images in the same medical image set according to the magnitude of the difference coefficients to obtain at least one medical image sequence, wherein a target medical image sequence only includes the same medical images with different annotation information;

[0113] The at least one medical image sequence and the image segmentation result corresponding to the at least one medical image sequence are used as the input and output of a hybrid neural network based on improved convolution and attention, and the hybrid neural network is iteratively trained to obtain an image segmentation model, wherein the at least one medical image sequence is updated according to a preset dynamic update strategy during the iterative training of the hybrid neural network;

[0114] The medical image containing the annotation information acquired in real time is input into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-task medical image segmentation method supporting multi-doctor collaborative supervision, characterized in that: include: Acquire at least one medical image set containing annotation information, wherein one medical image set contains the same medical image with different annotation information, and the annotation information is information annotated by a user on a target area in the medical image; Calculating the difference coefficients between the medical images in the same medical image set according to a preset difference function, and sorting the medical images in the same medical image set according to the magnitude of the difference coefficients to obtain at least one medical image sequence, wherein a target medical image sequence only includes the same medical images with different annotation information; The at least one medical image sequence and the image segmentation result corresponding to the at least one medical image sequence are respectively used as the input and output of a hybrid neural network based on improved convolution and attention, and the hybrid neural network is iteratively trained to obtain an image segmentation model, wherein the at least one medical image sequence is updated according to a preset dynamic update strategy during the iterative training of the hybrid neural network; the hybrid loss function of the image segmentation model is: ， , , , , In the formula, are the weights of the cross entropy loss function, the Dice loss function, and the boundary loss function, respectively. M is the number of categories in the target area, c is the specific category, y is the label, and p is the model prediction. is the segmentation loss function, is the distance loss function, D represents the distance transform, which is used to calculate the distance from each pixel in the image to the nearest boundary pixel, S is the predicted boundary, and g is the labeled boundary. is the horizontal coordinate of the geometric center of the target area, is the horizontal coordinate of any point in the target area, is the ordinate of the geometric center of the target area, is the ordinate of any point in the target area, The doctor is labeled that this pixel belongs to category C. The model predicts the probability that this annotation point belongs to class c, is the predicted boundary belonging to category c, is the labeled boundary belonging to category c; The image segmentation model includes a neural network encoder, an image segmentation decoder, and a target center coordinate decoder based on a VGG network; Inputting the medical image containing the annotation information acquired in real time into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result, wherein the inputting the medical image containing the annotation information acquired in real time into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result comprises: Input stage: receiving a medical image as input; Feature extraction and encoding stage: Use the convolutional neural network layer to downsample the input medical image, extract the features of the medical image, and encode the spatial information of the medical image into deep features; Self-attention mechanism stage: After downsampling, the correlation between the features in the medical image is identified based on the self-attention mechanism; Upsampling image segmentation decoding stage: Use the inverse convolutional neural network to expand the feature map back to the size of the original image to complete the image reconstruction. In the image reconstruction process, each pixel is classified to determine whether each pixel belongs to a specific tissue or structure. At the same time, the decoding stage is jump-connected to the corresponding layer of the encoding stage to retain more detail information; Geometric center decoding stage: using the classic convolutional network architecture based on VGG to fit the geometric centers of different target areas in high-dimensional features; Output stage: The network outputs the segmented medical image, where each pixel is classified into a specific category, forming a segmentation map.

2. According to claim 1, a multi-task medical image segmentation method supporting multi-doctor collaborative supervision is characterized in that: The neural network encoder comprises four downsampling stages, each of which consists of two consecutive convolutional neural network layers, and the image segmentation decoder comprises four upsampling stages, each of which consists of two consecutive convolutional neural network layers; Among them, the expression of data feature processing of the convolutional neural network layer is: ， ， In the formula, is the output feature of the i-th layer of the neural network, is a general network regularization technique, is a dilated convolution, is the output feature of the i-1th layer of the neural network, is batch normalization, * is the convolution operation, is the convolution kernel at position The weight of is the convolution kernel position, is a position in R, is the regular grid area covered by the convolution kernel, For location The corresponding offset, is the value of the input feature map at the adjusted position.

3. According to claim 1, a multi-task medical image segmentation method supporting multi-doctor collaborative supervision is characterized in that: The expression of the difference function is: ， In the formula, is the coefficient of difference, X represents the annotation information of the first doctor, and Y represents the annotation information of the second doctor. It indicates the number of pixels that are identical for the same target area marked by two doctors. It is represented by the total number of target areas annotated by two doctors.

4. The multi-task medical image segmentation method supporting multi-doctor collaborative supervision according to claim 1, characterized in that: The updating of the at least one medical image sequence according to a preset dynamic update strategy during the iterative training of the hybrid neural network comprises: The Ramp Up function is used to determine the specific number of images to be eliminated in different training stages, wherein the expression of the Ramp Up function is: ， In the formula, is a positive real number, x is the model training progress, and y is the number of samples to be removed; The at least one medical image sequence is updated according to the number of discarded images.

5. A multi-task medical image segmentation system supporting multi-doctor collaborative supervision according to any one of the methods described in claims 1-4, characterized in that: include: an acquisition module configured to acquire at least one medical image set containing annotation information, wherein one medical image set contains the same medical image with different annotation information, and the annotation information is information annotated by a user on a target area in the medical image; a calculation module configured to calculate a difference coefficient between each medical image in the same medical image set according to a preset difference function, and sort each medical image in the same medical image set according to the magnitude of the difference coefficient to obtain at least one medical image sequence, wherein a target medical image sequence only contains the same medical image with different annotation information; A training module, configured to use the at least one medical image sequence and the image segmentation result corresponding to the at least one medical image sequence as the input and output of a hybrid neural network based on improved convolution and attention, iteratively train the hybrid neural network to obtain an image segmentation model, wherein the at least one medical image sequence is updated according to a preset dynamic update strategy during the iterative training of the hybrid neural network; The output module is configured to input the medical image containing annotation information acquired in real time into the trained image segmentation model, and the image segmentation model outputs the final medical image segmentation result.

6. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images