A knee cartilage defect segmentation method, electronic equipment and medium

Abstract

The application discloses a knee cartilage defect segmentation method, an electronic device and a medium, and the method comprises the steps: inputting an image with a cartilage defect label into a student model, wherein the student model comprises a Trans U-Net network and an Attention U-Net network, and first and second labeled prediction results are respectively outputted; inputting an image without a cartilage defect label into the student model and a teacher model respectively, wherein the teacher model adopts the Attention U-Net network, and first, second and third unlabeled prediction results are outputted; setting a supervised loss and a semi-supervised loss; taking a weighted sum of the supervised loss and the semi-supervised loss as a total loss function to train the student model; performing an exponential moving average on network weights of the Attention U-Net network in the student model to update the network weights in the teacher model, and obtaining an optimized teacher model; and taking the optimized teacher model as a knee cartilage defect segmentation model, which is used for confirming a knee cartilage defect mask image according to a knee cartilage defect image to be segmented.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image detection, in particular to a knee cartilage defect segmentation method, an electronic device and a medium. BACKGROUND

[0002] Cartilage is a tissue that covers the surface of the bone in the joint, which helps to reduce friction between joints and protect bones. However, the number of people with joint cartilage disease is gradually increasing, and there are many types, of which the most common is cartilage defect. The treatment methods for cartilage defect include injection treatment, surgical treatment and the currently developing robot treatment method. The position, shape and size of the cartilage defect are necessary information for the above-mentioned treatment methods, so obtaining accurate segmentation results of the knee cartilage defect is crucial for the treatment of knee cartilage defects.

[0003] The size of the cartilage tissue varies, and the type, color, size and shape of the cartilage defect are also very different. Even if a professional doctor observes accurately by the naked eye, it is extremely difficult to make an accurate defect judgment, and it is difficult to achieve cartilage defect segmentation using traditional computer vision methods.

[0004] In addition, the acquisition of knee cartilage defect samples needs to meet certain medical and ethical requirements, and the acquisition of samples has certain difficulty. The annotation task of the sample needs to be carried out under the complex and difficult medical diagnosis standard of the cartilage defect, and a large amount of accurate defect annotation of the sample requires a certain amount of time and labor cost, so the high-quality annotated knee cartilage defect dataset is relatively small in quantity. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application provides a knee cartilage defect segmentation method, an electronic device and a medium.

[0006] In a first aspect, an embodiment of the present application provides a knee cartilage defect segmentation method, which comprises:

[0007] inputting an image with a cartilage defect label into a student model, the student model comprising a Trans U-Net network and an Attention U-Net network, and outputting a first labeled prediction result and a second labeled prediction result respectively;

[0008] inputting an image without a cartilage defect label into the student model and a teacher model respectively, the teacher model adopting an Attention U-Net network; and outputting a first unlabeled prediction result, a second unlabeled prediction result and a third unlabeled prediction result;

[0009] Set a supervised loss and a semi-supervised loss; take a weighted sum of the supervised loss and the semi-supervised loss as a total loss function; train the student model by using the total loss function;

[0010] Exponentially moving average the network weights of the Attention U-Net network in the student model to update the network weights in the teacher model, and obtain an optimized teacher model; taking the optimized teacher model as a knee cartilage defect segmentation model; wherein the knee cartilage defect segmentation model is used to confirm a knee cartilage defect mask image of a knee cartilage defect image to be segmented according to the knee cartilage defect image.

[0011] In a second aspect, an electronic device is provided, including a memory and a processor, the memory is coupled with the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the above-mentioned knee cartilage defect segmentation method.

[0012] In a third aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the program is executed by a processor to realize the above-mentioned knee cartilage defect segmentation method.

[0013] In a fourth aspect, a computer program product is provided, and the computer program product includes computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the above-mentioned knee cartilage defect segmentation method.

[0014] Compared with the prior art, the present application has the following beneficial effects:

[0015] The present application discloses a knee cartilage defect segmentation method, which trains a student model by setting a supervised loss and a semi-supervised loss, and updates the network weights in the teacher model by exponentially moving average the network weights of the Attention U-Net network in the student model, and takes the optimized teacher model as a knee cartilage defect segmentation model, realizes a diagnosis method that multiple experts comprehensive interpretation can improve the reliability of diagnosis and produce less false negative diagnosis, and realizes accurate and reliable cartilage defect segmentation effect. At the same time, the problem of high cost of cartilage defect data set is solved, and the labeling cost is effectively reduced. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1A schematic diagram of a knee cartilage defect segmentation method provided by an embodiment of the present application is shown in the figure.

[0018] Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0020] It should be noted that the features in the following embodiments and implementation manners can be combined with each other without conflict.

[0021] As shown in the figure, Figure 1 the present application provides a knee cartilage defect segmentation method, which comprises:

[0022] Step S1, inputting an image X gt with a cartilage defect label Y l to a student model, wherein the student model comprises a TransU-Net network and an Attention U-Net network; outputting a first labeled prediction result, a second labeled prediction result;

[0023] inputting an image X u without a cartilage defect label to the student model and a teacher model respectively, wherein the teacher model adopts an Attention U-Net network; outputting a first unlabeled prediction result, a second unlabeled prediction result, and a third unlabeled prediction result;

[0024] setting a supervised loss and a semi-supervised loss; taking a weighted sum of the supervised loss and the semi-supervised loss as a total loss function; and training the student model by using the total loss function.

[0025] Further, the Attention U-Net network and the Trans U-Net network in the student model adopt different initialization weights to provide different perspectives, and the initialization weight of the Attention U-Net network in the teacher model adopts the same initialization weight as the Attention U-Net network in the student model.

[0026] Further, in the data processing aspect of the present embodiment, the image X uThe noise is added to the input data, and the noise has the same matrix form as the input data, and the value is between-0.2 and 0.2; by adding noise, the network of the teacher model can better adapt to the data without real labels, improve the generalization ability of the model, thereby reducing the risk of overfitting, and making it more robust.

[0027] The supervised loss and the semi-supervised loss include:

[0028] According to the image X gt with the cartilage defect label Y l The first labeled prediction result output by the Attention U-Net network f1(θ1) in the student model is compared with the cartilage defect label Y gt The first supervised loss loss_sup1 is set, and the expression is as follows:

[0029] loss_sup1=CE(Y gt ,f1(X l |θ1))+Dice(Y gt ,f1(X l |θ1)

[0030] In the formula, CE(.) is the cross-entropy loss, Dice(.) is the dice coefficient loss, Y gt is the real label, X l is the data with real label, θ1 is the parameter of the Attention U-Net network in the student model, and f1(X l |θ1) is the first labeled prediction result output by the Attention U-Net network f1(θ1) in the student model.

[0031] According to the image X gt with the cartilage defect label Y l The second labeled prediction result output by the Trans U-Net network f1(θ1) in the student model is compared with the cartilage defect label Y gt The second supervised loss loss_sup2 is set, and the expression is as follows:

[0032] loss_sup2=CE(Y gt ,f2(X l |θ2))+Dice(Y gt ,f2(X l |θ2)

[0033] In the formula, θ2 is the parameter of the Trans U-Net network in the student model, and f2(X2|θ2) is the second labeled prediction result output by the Trans U-Net network f2(θ2) in the student model.

[0034] According to the first unlabeled prediction result Y p 1u , the second unlabeled prediction result Y p 2u The first semi-supervised loss loss semi1 and the second semi-supervised loss loss semi2 are respectively set based on the Dice coefficient loss, and the expressions are as follows:

[0035] loss semi1 = Dice (f1 (X u | θ1), f2 (X u | θ2))

[0036] loss semi2 = Dice (f2 (X u | θ2), f1 (X u | θ1))

[0037] It should be noted that loss semi1 is the first unlabeled prediction result Y u through the Attention U-Net network f1 (θ1) in the student model p 1u and the image X u through the second unlabeled prediction result Y p 2u The semi-supervised loss calculated based on the Dice coefficient loss; loss semi2 is the second unlabeled prediction result Y u through the Trans U-Net network f2 (θ2) in the student model p 2u and the image X u through the first unlabeled prediction result Y p 1u based on the Dice coefficient loss; loss semi1 and loss semi2 are used for cross supervision of the Trans U-Net network and the Attention U-Net network in the student model.

[0038] According to the first unlabeled prediction result Y p 1u , the third unlabeled prediction result Y p 3u The third semi-supervised loss loss semi3 is set based on the Dice coefficient loss, and the expression is as follows:

[0039] loss_semi3 = Dice(f1(X u |θ1), f3(X u |θ3))

[0040] According to the second no-label prediction result Y p 2u , the third no-label prediction result Y p 3u The fourth semi-supervised loss loss_semi4 is set based on the dice coefficient loss, and the expression is as follows:

[0041] loss_semi4 = Dice(f2(X u |θ2), f3(X u |θ3))

[0042] It should be noted that loss_semi3 is a semi-supervised loss based on the first no-label prediction result of the Attention U-Net network f1(θ1) in the student model and the image X u without cartilage defect label, and the third no-label prediction result of the Attention U-Net network f3(θ3) in the teacher model is based on the dice coefficient loss. u loss_semi4 is a semi-supervised loss based on the second no-label prediction result of the Trans U-Net network f2(θ2) in the student model and the image X u without cartilage defect label, and the third no-label prediction result of the Attention U-Net network f3(θ3) in the teacher model is based on the dice coefficient loss. u loss_semi3 and loss_semi4 are used to control the consistency between the teacher model and the student model.

[0043] The total loss function is obtained by weighted summation of the supervised loss and the semi-supervised loss, and the total loss function formula is: Loss_train = loss_sup1 + loss_sup2 + λ1(loss_semi1 + loss_semi2) + λ2(loss_semi3 + loss_semi4)

[0044] In the formula, λ1 and λ2 represent weight coefficients. In this example, λ1 = 7 × 0.1 × (epoch / Epoch); λ2 = 0.1 × (epoch / Epoch); epoch is the current training round, and Epoch is the total number of training rounds. The reason why λ1 is set to be larger than λ2 here is that the effect of setting cross-supervision is greater than the effect of controlling consistency.

[0045] Step S2, exponentially moving average the network weights of the Attention U-Net network in the student model to update the network weights in the teacher model, to obtain an optimized teacher model; the optimized teacher model is used as a knee cartilage defect segmentation model; wherein the knee cartilage defect segmentation model is used to confirm the knee cartilage defect mask image of the knee cartilage defect image to be segmented according to the knee cartilage defect image to be segmented.

[0046] wherein, exponentially moving average (EMA) the network weights of the Attention U-Net network in the student model to update the network weights in the teacher model includes:

[0047]

[0048] wherein, represents the network weights of the Attention U-Net network in the student model, represents the network weights of the Attention U-Net network in the teacher model, and t is the training round.

[0049] It should be noted that a increases with the increase of the training round, and approaches to 1. When in the initial stage of the training process, that is, t is small, the weight (1-a) is large, and the Attention U-Net network in the student model updates the parameter θ2 t The update of the parameter θ3 t of the teacher model is greatly affected. When t is large, from the derivation of EMA, θ3 t-1 is the weighted summation result of the parameters θ2 t (t=1, 2, 3... epoch-1) of the Attention U-Net network in the student model updated in the training rounds 1 to epoch-1. Although the weight (1-a) becomes small, compared with the composition of θ3 t-1 , the θ2 t (t=1, 2, 3... epoch-1) obtained from the past training, the θ2 t (t=epoch) obtained in the current training round plays a dominant role in θ3 t (t=epoch) in the current training round t=epoch. Through this way, the test index is improved and the model robustness is increased.

[0050] Further, the method further includes: using Target IOU, mIOU, Recall, Precision and F1-score to evaluate the accuracy and reliability of the teacher model.

[0051] Further, the model weight of the teacher model trained through the network needs to be verified for its performance through experiments. The experiments are performed under the condition of using different proportions of annotated data in the entire data set, i.e., 25%, 50%, 75%, and 100%. In the experiment, the classical semi-supervised network needs to be reproduced. In the comparison experiment of the semi-supervised network, the reproduced semi-supervised network includes MT, UAMT, and CPS, all of which use Attention U-Net as the backbone segmentation network. Through the experiment, the classical semi-supervised network is compared to verify that the teacher model trained in the present example can guarantee the accurate cartilage defect semantic segmentation effect while reducing the high-quality annotation cost. MT refers to the Mean Teacher network, UAMT refers to the Uncertainty-Aware Mean Teacher network, and CPS refers to the cross pseudo supervision network.

[0052] Further, the experimental aspect of the present example obtains the following results:

[0053] Table 1: Experimental results of different semi-supervised networks (using 25% of the annotation data)

[0054] Network Target-IOU mIOU Recall Precision F1-score MT 74.3 86.38 88.32 82.39 85.25 UAMT 74.01 86.2 90.64 80.13 85.06 CPS 73.39 85.97 80.08 89.78 84.65 Morenet (Ours) 77.18 87.93 88.4 85.88 87.12

[0055] As shown in Table 1 above, when only 25% of the annotation data is used, the overall performance of Morenet is the best among all semi-supervised networks, in which the mIOU, Target-IOU, and F1-score achieve the best scores, which are 87.93, 77.18, and 87.12, respectively. Although the scores of Recall and Precision are not the highest, they also achieve the second best scores. The above data results can indicate that the comprehensive effect of high accuracy and low false detection rate of the Morenet network is the best.

[0056] The above knee cartilage defect segmentation method based on semi-supervised semantic segmentation is aimed at the cartilage defect data set in the present task. The teacher model network can achieve accurate and reliable cartilage defect segmentation effect, and the designed double-view collaborative training strategy can improve the diagnostic reliability and produce fewer false negative diagnoses by using multiple expert comprehensive explanations. In addition, the semi-supervised deep learning strategy is adopted to solve the problem of high cost of cartilage defect data set. The experimental results show that the present application can effectively reduce the annotation cost while maintaining high accuracy of cartilage defect semantic segmentation.

[0057] As Figure 2As shown, the embodiment of the present application provides an electronic device, which comprises a memory 101 for storing one or more programs; a processor 102. When the one or more programs are executed by the processor 102, the method of any one of the above first aspects is implemented.

[0058] Further comprising a communication interface 103, the memory 101, the processor 102 and the communication interface 103 are electrically connected with each other directly or indirectly to realize the transmission or interaction of data. For example, these elements can be electrically connected with each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.

[0059] Among them, the memory 101 can be, but is not limited to, a random access memory 101 (RAM), a read-only memory 101 (ROM), a programmable read-only memory 101 (PROM), an erasable programmable read-only memory 101 (EPROM), an electrically erasable programmable read-only memory 101 (EEPROM) and the like.

[0060] The processor 102 can be an integrated circuit chip with signal processing capability. The processor 102 can be a general-purpose processor 102, including a central processing unit 102 (CPU), a network processor 102 (NP) and the like; it can also be a digital signal processor 102 (DSP), an application specific integrated circuit 102 (ASIC), a field programmable gate array 102 (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0061] In the embodiments of the present application, it should be understood that the disclosed method and system can also be implemented by other means. The above described method and system embodiments are only illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the method and system, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which includes one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0062] In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0063] On the other hand, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor 102 to implement the method of any one of the above first aspect. The functions, if implemented in the form of software function modules and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory 101 (ROM, Read-Only Memory), a random access memory 101 (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0064] The above examples are only used to illustrate the design idea and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and the protection scope of the present application is not limited to the above examples. Therefore, any equivalent changes or modifications made according to the principles and design ideas disclosed by the present application are within the protection scope of the present application.

Claims

1. A method for segmenting a knee cartilage defect, characterized in that: The method comprises: Inputting the image with the cartilage defect label into the student model, the student model includes a Trans U-Net network and an Attention U-Net network, and outputting a first labeled prediction result and a second labeled prediction result respectively; Input the image without cartilage defect label to the student model and the teacher model respectively, wherein the teacher model adopts the Attention U-Net network; output the first unlabeled prediction result, the second unlabeled prediction result, and the third unlabeled prediction result; Set supervised loss and semi-supervised loss; take the weighted sum of supervised loss and semi-supervised loss as the total loss function; use the total loss function to train the student model; The network weights of the Attention U-Net network in the student model are exponentially moved averaged to update the network weights in the teacher model to obtain an optimized teacher model; the optimized teacher model is used as a knee cartilage defect segmentation model; wherein the knee cartilage defect segmentation model is used to confirm its knee cartilage defect mask image based on the knee cartilage defect image to be segmented.

2. The method for segmenting a knee cartilage defect according to claim 1, wherein: The Trans U-Net network and Attention U-Net network in the student model have different initialization parameters.

3. The method for segmenting a knee cartilage defect according to claim 1, wherein: The initialization parameters of the Attention U-Net network in the student model are the same as those in the teacher model.

4. The method for segmenting a knee cartilage defect according to claim 1, wherein: Setting the supervised loss includes: According to the cartilage defect label Y gt Image X l The first labeled prediction result output by the Attention U-Net network f1(θ1) in the student model is compared with the cartilage defect label Y gt Set the first supervised loss loss_sup1, the expression is as follows: loss_sup1=CE(Y gt ,f1(X l |θ1))+Dice(Y gt ,f1(X l |θ1)) Where CE(.) is the cross entropy loss, Dice(.) is the dice coefficient loss, and Y gt is the true label, X l is the data with real labels, θ1 is the parameter of the Attention U-Net network in the student model, f1(X l |θ1) is the first labeled prediction result output by the AttentionU-Net network f1(θ1) in the student model; According to the cartilage defect label Y gt Image X l The second labeled prediction result output by the Trans U-Net network f2(θ2) in the student model is compared with the cartilage defect label Y gt Set the second supervision loss loss_sup2, the expression is as follows: loss_sup2=CE(Y gt ,f2(X l |θ2))+Dice(Y gt ,f2(X l |θ2)) Where θ2 is the parameter of the Trans U-Net network in the student model, f2(X l |θ2) is the second labeled prediction result output by the Trans U-Net network f2(θ2) in the student model.

5. The method for segmenting a knee cartilage defect according to claim 4, wherein: Setting the semi-supervised loss includes: According to the image X without cartilage defect label u The first unlabeled prediction result of the Attention U-Net network f1(θ1) in the student model is compared with the image X without the cartilage defect label u The first semi-supervised loss is calculated based on the dice coefficient loss through the second unlabeled prediction result of the Trans U-Net network f2(θ2) in the student model, and the expression is as follows: loss_semi1=Dice(f1(X u |θ1),f2(X u |θ2)) According to the image X without cartilage defect label u The second unlabeled prediction result of the Trans U-Net network f2(θ2) in the student model is compared with the image X without the cartilage defect label u The second semi-supervised loss is calculated based on the dice coefficient loss through the first unlabeled prediction result of the Attention U-Net network f1(θ1) in the student model, and the expression is as follows: loss_semi2=Dice(f2(X u |θ2),f1(X u |θ1)) According to the image X without cartilage defect label u The first unlabeled prediction result of the Attention U-Net network f1(θ1) in the student model is compared with the image X without the cartilage defect label u The third semi-supervised loss is calculated based on the dice coefficient loss through the third unlabeled prediction result of the Attention U-Net network f3(θ3) in the teacher model, and the expression is as follows: loss_semi3=Dice(f1(X u |θ1),f3(X u |θ3)) According to the image X without cartilage defect label u The second unlabeled prediction result of the Trans U-Net network f2(θ2) in the student model is compared with the image X without the cartilage defect label u The fourth semi-supervised loss is calculated based on the dice coefficient loss through the third unlabeled prediction result of the Attention U-Net network f3(θ3) in the teacher model, and the expression is as follows: loss_semi4=Dice(f2(X u |θ2),f3(X u |θ3)) Where, f1(X u |θ1) represents the first unlabeled prediction result, f2(X u |θ2) represents the second unlabeled prediction result, f3(X u |θ3) represents the third unlabeled prediction result.

6. The method for segmenting a knee cartilage defect according to claim 5, wherein: The weighted sum of the supervised loss and the semi-supervised loss is used as the total loss function, and the expression of the total loss function is as follows: Loss_train=loss_sup1+loss_sup2+λ1(loss_semi1+loss_semi2)+λ2(loss_semi3+loss_semi4) Where λ1 and λ2 represent weight coefficients.

7. The method for segmenting a knee cartilage defect according to claim 1, wherein: The exponential moving average of the network weights of the Attention U-Net network in the student model is used to update the network weights in the teacher model. Where, Represents the network weight of the Attention U-Net network in the student model, represents the network weight of the Attention U-Net network in the teacher model, and t is the training round.

8. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the knee cartilage defect segmentation method described in one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the knee cartilage defect segmentation method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for segmenting knee cartilage defects according to any one of claims 1 to 7 is implemented.

Images