Knee joint cartilage defect segmentation method, electronic equipment and medium

Abstract

The invention discloses a knee joint cartilage defect segmentation method, electronic equipment and a medium, and the method comprises the steps: inputting an image with a cartilage defect label into a student model which comprises a Trans U-Net network and an Attention U-Net network, and outputting a first prediction result with a label and a second prediction result with a label respectively; the images without the cartilage defect labels are input into a student model and a teacher model respectively, and the teacher model adopts an Attention U-Net network to output first, second and third label-free prediction results; setting supervised loss and semi-supervised loss; taking the weighted sum of the supervised loss and the semi-supervised loss as a total loss function to train a student model; performing index moving average on the network weight of the Attention U-Net network in the student model so as to update the network weight in the teacher model, and obtaining an optimized teacher model; and taking the optimized teacher model as a knee joint cartilage defect segmentation model for confirming a knee joint cartilage defect mask image according to a to-be-segmented knee joint cartilage defect image.

Description

Technical Field

[0001] The present invention relates to the field of image detection, and in particular to a knee cartilage defect segmentation method, electronic equipment and medium. Background Art

[0002] Cartilage is the tissue covering the surface of bones in joints, which helps reduce friction between joints and protect bones. However, the number of people suffering from articular cartilage diseases is gradually increasing, and there are many types. The most common one is cartilage defect. The treatments for cartilage defects include injection therapy, surgical treatment, and robotic treatment, which is currently in the development stage. The physical information of the cartilage defect, such as the location, shape, and size, is necessary for the above-mentioned treatments. Therefore, obtaining accurate segmentation results of knee cartilage defects is crucial for the treatment of knee cartilage defects.

[0003] The shape and size of cartilage tissue vary. The type, color, shape and size of cartilage defects are also very different. Even for professional doctors, it is extremely difficult to make an accurate defect judgment through human eye observation. At the same time, it is difficult to achieve the cartilage defect segmentation task using traditional computer vision methods.

[0004] In addition, since the acquisition of knee cartilage defect samples needs to meet certain medical and ethical requirements, it is difficult to obtain samples. In addition, the sample annotation task needs to be carried out under the complex and difficult medical diagnostic standards of cartilage defects. It takes a certain amount of time and manpower to complete the accurate defect annotation of samples on a large scale. Therefore, there are relatively few high-quality annotated knee cartilage defect data sets. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a knee cartilage defect segmentation method, electronic equipment, and medium.

[0006] In a first aspect, an embodiment of the present invention provides a method for segmenting a knee cartilage defect, the method comprising:

[0007] Inputting the image with the cartilage defect label into the student model, the student model includes a Trans U-Net network and an Attention U-Net network, and outputting a first labeled prediction result and a second labeled prediction result respectively;

[0008] Input the images without cartilage defect labels into the student model and the teacher model respectively, wherein the teacher model adopts the Attention U-Net network; output the first unlabeled prediction result, the second unlabeled prediction result, and the third unlabeled prediction result;

[0009] Set supervised loss and semi-supervised loss; take the weighted sum of supervised loss and semi-supervised loss as the total loss function; use the total loss function to train the student model;

[0010] The network weights of the Attention U-Net network in the student model are exponentially moved averaged to update the network weights in the teacher model to obtain an optimized teacher model; the optimized teacher model is used as a knee cartilage defect segmentation model; wherein the knee cartilage defect segmentation model is used to confirm the knee cartilage defect mask image according to the knee cartilage defect image to be segmented.

[0011] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned knee cartilage defect segmentation method.

[0012] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the above-mentioned knee cartilage defect segmentation method is implemented.

[0013] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the above-mentioned knee cartilage defect segmentation method when executed by a processor.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] The present invention discloses a method for segmenting cartilage defects in the knee joint. The supervised loss and semi-supervised loss are set to collaboratively train the student model, and the network weights of the Attention U-Net network in the student model are exponentially moved averaged to update the network weights in the teacher model. The optimized teacher model is used as the segmentation model for cartilage defects in the knee joint, and a diagnosis method that can improve the reliability of diagnosis and produce fewer false negatives by comprehensive interpretation of multiple experts is realized, achieving an accurate and reliable cartilage defect segmentation effect. At the same time, the problem of high cost of cartilage defect data sets is solved, and the annotation cost is effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0017] Figure 1A schematic diagram of a knee cartilage defect segmentation method provided by an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0021] like Figure 1 As shown, an embodiment of the present invention provides a method for segmenting a knee cartilage defect, the method comprising:

[0022] Step S1, label the cartilage defect gt Image X l Input to the student model, the student model includes a TransU-Net network and an Attention U-Net network; output a first labeled prediction result and a second labeled prediction result;

[0023] The image X without cartilage defect label u Input them into the student model and the teacher model respectively, wherein the teacher model adopts the Attention U-Net network; output the first unlabeled prediction result, the second unlabeled prediction result, and the third unlabeled prediction result;

[0024] Set supervised loss and semi-supervised loss; take the weighted sum of supervised loss and semi-supervised loss as the total loss function; use the total loss function to train the student model.

[0025] Furthermore, the Attention U-Net network and Trans U-Net network in the student model use different initialization weights to provide different perspectives, and the initialization weights of the Attention U-Net network in the teacher model use the same initialization weights as the Attention U-Net network in the student model.

[0026] Furthermore, in the data processing of this embodiment, in the input data, the image X without the cartilage defect label uNoise is added to the dataset. The noise has the same matrix form as the input data and its value is between -0.2 and 0.2. By adding noise, the network of the teacher model can be better adapted to data without real labels, and the generalization ability of the model can be improved, thereby reducing the risk of overfitting and making it more robust.

[0027] Among them, supervised loss and semi-supervised loss include:

[0028] According to the cartilage defect label Y gt Image X l The first labeled prediction result output by the Attention U-Net network f1(θ1) in the student model is compared with the cartilage defect label Y gt Set the first supervision loss loss_sup1, the expression is as follows:

[0029] loss_sup1=CE(Y gt ,f1(X l |θ1))+Dice(Y gt ,f1(X l |θ1))

[0030] In the formula, CE(.) is the cross entropy loss, Dice(.) is the dice coefficient loss, and Y gt is the true label, X l is the data with real labels, θ1 is the parameter of the Attention U-Net network in the student model, f1(X l |θ1) is the first labeled prediction result output by the Attention U-Net network f1(θ1) in the student model;

[0031] Cartilage defect label Y gt Image X l The second labeled prediction result output by the Trans U-Net network f1(θ1) in the student model is compared with the cartilage defect label Y gt Set the second supervision loss loss_sup2, the expression is as follows:

[0032] loss_sup2 = CE(Y gt ,f2(X l |θ2))+Dice(Y gt ,f2(X l |θ2))

[0033] Where θ2 is the parameter of the Trans U-Net network in the student model, and f2(X2|θ2) is the second labeled prediction result output by the Trans U-Net network f2(θ2) in the student model.

[0034] According to the first unlabeled prediction result Y p 1u , the second unlabeled prediction result Y p 2u Based on the dice coefficient loss, the first half of the supervised loss loss_semi1 and the second half of the supervised loss loss_semi2 are set respectively. The expressions are as follows:

[0035] loss_semi1=Dice(f1(X u |θ1),f2(X u |θ2))

[0036] loss_semi2=Dice(f2(X u |θ2),f1(X u |θ1))

[0037] It should be noted that loss_semi1 is the image X without the cartilage defect label u The first unlabeled prediction result Y of the Attention U-Net network f1(θ1) in the student model p 1u Image X without cartilage defect label u The second unlabeled prediction result Y through the Trans U-Net network f2(θ2) in the student model p 2u Semi-supervised loss calculated based on dice coefficient loss; loss_semi2 is the image X without cartilage defect label u The second unlabeled prediction result Y through the Trans U-Net network f2(θ2) in the student model p 2u Image X without cartilage defect label u The first unlabeled prediction result Y of the AttentionU-Net network f1(θ1) in the student model p 1u Semi-supervised loss calculated based on dice coefficient loss; loss_semi1 and loss_semi2 are used for cross-supervision of the Trans U-Net network and Attention U-Net network in the student model.

[0038] According to the first unlabeled prediction result Y p 1u , the third unlabeled prediction result Y p 3u The third semi-supervised loss loss_semi3 is set based on the dice coefficient loss. The expression is as follows:

[0039] loss_semi3=Dice(f1(X u |θ1),f3(X u |θ3))

[0040] According to the second unlabeled prediction result Y p 2u , the third unlabeled prediction result Y p 3u The fourth semi-supervised loss loss_semi4 is set based on the dice coefficient loss. The expression is as follows:

[0041] loss_semi4=Dice(f2(X u |θ2),f3(X u |θ3))

[0042] It should be noted that loss_semi3 is based on the image X without the cartilage defect label u The first unlabeled prediction result of the Attention U-Net network f1(θ1) in the student model is compared with the image X without the cartilage defect label u The third unlabeled prediction result of the Attention U-Net network f3(θ3) in the teacher model is a semi-supervised loss based on the dice coefficient loss; loss_semi4 is based on the image X without the cartilage defect label u The second unlabeled prediction result of the Trans U-Net network f2(θ2) in the student model is compared with the image X without the cartilage defect label u The semi-supervised loss is calculated based on the Dice coefficient loss through the third unlabeled prediction result of the Attention U-Net network f3(θ3) in the teacher model. loss_semi3 and loss_semi4 are used to control the consistency between the teacher model and the student model.

[0043] The total loss function is obtained by weighted summation of supervised loss and semi-supervised loss. The total loss function formula is: Loss_train = loss_sup1 + loss_sup2 + λ1 (loss_semi1 + loss_semi2) + λ2 (loss_semi3 + loss_semi4)

[0044] Where λ1 and λ2 represent weight coefficients. In this example, λ1 = 7 × 0.1 × (epoch / Epoch); λ2 = 0.1 × (epoch / Epoch); epoch is the current training round, and Epoch is the total number of training rounds. The reason why λ1 is set larger than λ2 here is that the effect of setting cross-supervision is more influential than the effect of controlling consistency.

[0045] Step S2, performing exponential moving average on the network weights of the Attention U-Net network in the student model to update the network weights in the teacher model to obtain an optimized teacher model; using the optimized teacher model as a knee cartilage defect segmentation model; wherein the knee cartilage defect segmentation model is used to confirm its knee cartilage defect mask image according to the knee cartilage defect image to be segmented.

[0046] Among them, the exponential moving average (EMA) of the network weights of the Attention U-Net network in the student model is used to update the network weights in the teacher model, including:

[0047]

[0048] In the formula, represents the network weights of the Attention U-Net network in the student model, represents the network weight of the Attention U-Net network in the teacher model, and t is the training round.

[0049] It should be noted that α approaches 1 as the number of training rounds increases. When in the initial stage of the training process, that is, when t is small, the weight (1-α) is large, and the parameter θ2 updated by the Attention U-Net network in the student model in this training generation t=epoch is t The parameter θ3 of the teacher model t The update of θ3 has a greater impact. When t is large, it can be seen from the derivation of EMA that θ3 t-1 The Attention U-Net network in the student model updates the parameter θ2 from training round 1 to epoch-1 t The weighted summation result of (t=1,2,3...epoch-1) is that although the weight (1-α) becomes smaller, it is still smaller than θ3 t-1 The θ2 obtained from previous training in the composition t (t=1,2,3...epoch-1), θ2 obtained in this training round t (t = epoch) for θ3 t The influence of (t = epoch) plays a leading role in this training round t = epoch. This method is used to improve the test index and increase the robustness of the model.

[0050] Furthermore, the method also includes: using Target IOU, mIOU, Recall, Precision and F1-score to evaluate the accuracy and reliability of the teacher model.

[0051] Furthermore, the model weights of the teacher model obtained through network training are obtained, and experiments are needed to verify its performance. The experiment is conducted under the conditions of using data sets with different proportions of labeled data to all data, which are 25%, 50%, 75%, and 100% respectively. In this experiment, the classic semi-supervised network needs to be reproduced. In this comparative semi-supervised network experiment, the semi-supervised networks used for reproduction include: MT, UAMT, and CPS, all of which use Attention U-Net as the backbone segmentation network. Through this experiment, it is compared with the classic semi-supervised network to verify that the teacher model trained in this example can ensure the accurate semantic segmentation effect of cartilage defects while reducing the cost of high-quality annotation. Among them, MT refers to the Meanteacher network, UAMT refers to the Uncertainty-Aware Mean Teacher network, and CPS refers to the crosspseudo supervision network.

[0052] Furthermore, the experimental results of this example are as follows:

[0053] Table 1: Experimental results using different semi-supervised networks (using 25% of the annotation data)

[0054] network Target-IOU mIOU Recall Precision F1-score MT 74.3 86.38 88.32 82.39 85.25 UAMT 74.01 86.2 90.64 80.13 85.06 CPS 73.39 85.97 80.08 89.78 84.65 Morenet(Ours) 77.18 87.93 88.4 85.88 87.12

[0055] As shown in Table 1 above, when only 25% of the annotated data is used, the overall performance of Morenet is the best among all semi-supervised networks, among which mIOU, Target-IOU, and F1-score achieve the best scores, which are 87.93, 77.18, and 87.12 respectively. Although the scores of Recall and Precision are not the highest, they also achieve the second best scores. The above data results show that the comprehensive effect of high precision and low false positive rate of the Morenet network is the best.

[0056] The above-mentioned knee cartilage defect segmentation method based on semi-supervised semantic segmentation can achieve accurate and reliable cartilage defect segmentation effects for the cartilage defect dataset in this task. At the same time, the designed dual-view collaborative training strategy realizes the comprehensive interpretation of multiple experts to improve the diagnostic reliability and produce fewer false negatives. The semi-supervised deep learning strategy is adopted to solve the problem of high cost of cartilage defect datasets. The experimental results show that the present invention can effectively reduce the annotation cost while maintaining a high accuracy of cartilage defect semantic segmentation.

[0057] like Figure 2As shown, an embodiment of the present application provides an electronic device, which includes a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, any method in the first aspect described above is implemented.

[0058] The memory 101, the processor 102 and the communication interface 103 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0059] Among them, the memory 101 can be, but is not limited to, a random access memory 101 (Random Access Memory, RAM), a read only memory 101 (Read Only Memory, ROM), a programmable read-only memory 101 (Programmable Read-Only Memory, PROM), an erasable read-only memory 101 (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable read-only memory 101 (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0060] The processor 102 may be an integrated circuit chip with signal processing capability. The processor 102 may be a general-purpose processor 102, including a central processing unit 102 (CPU), a network processor 102 (NP), etc.; it may also be a digital signal processor 102 (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0061] In the embodiments provided in the present application, it should be understood that the disclosed method and system can also be implemented in other ways. The method and system embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the method and system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0062] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0063] On the other hand, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by the processor 102, a method as in any one of the first aspects described above is implemented. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory 101 (ROM, Read-Only Memory), a random access memory 101 (RAM, RandomAccess Memory), a disk or an optical disk.

[0064] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for segmenting a knee cartilage defect, characterized in that: The method comprises: Inputting the image with the cartilage defect label into the student model, the student model includes a Trans U-Net network and an Attention U-Net network, and outputting a first labeled prediction result and a second labeled prediction result respectively; Input the images without cartilage defect labels into the student model and the teacher model respectively, wherein the teacher model adopts the Attention U-Net network; output the first unlabeled prediction result, the second unlabeled prediction result, and the third unlabeled prediction result; Set supervised loss and semi-supervised loss; take the weighted sum of supervised loss and semi-supervised loss as the total loss function; use the total loss function to train the student model; The network weights of the Attention U-Net network in the student model are exponentially moved averaged to update the network weights in the teacher model to obtain an optimized teacher model; the optimized teacher model is used as a knee cartilage defect segmentation model; wherein the knee cartilage defect segmentation model is used to confirm the knee cartilage defect mask image according to the knee cartilage defect image to be segmented.

2. A method for segmenting a knee cartilage defect according to claim 1, characterized in that: The Trans U-Net network and Attention U-Net network in the student model have different initialization parameters.

3. A method for segmenting a knee cartilage defect according to claim 1, characterized in that: The initialization parameters of the Attention U-Net network in the student model are the same as those in the teacher model.

4. A method for segmenting a knee cartilage defect according to claim 1, characterized in that: Setting the supervised loss includes: According to the cartilage defect label Y gt Image X l The first labeled prediction result output by the Attention U-Net network f1(θ1) in the student model is compared with the cartilage defect label Y gt Set the first supervision loss loss_sup1, the expression is as follows: loss_sup1=CE(Y gt ,f1(X l |θ1))+Dice(Y gt ,f1(X l |θ1)) In the formula, CE(.) is the cross entropy loss, Dice(.) is the dice coefficient loss, and Y gt is the true label, X l is the data with real labels, θ1 is the parameter of the Attention U-Net network in the student model, f1(X l |θ1) is the first labeled prediction result output by the AttentionU-Net network f1(θ1) in the student model; According to the cartilage defect label Y gt Image X l The second labeled prediction result output by the Trans U-Net network f2(θ2) in the student model is compared with the cartilage defect label Y gt Set the second supervision loss loss_sup2, the expression is as follows: loss_sup2=CE(Y gt ,f2(X l |θ2))+Dice(Y gt ,f2(X l |θ2)) Where θ2 is the parameter of the Trans U-Net network in the student model, and f2(X2|θ2) is the second labeled prediction result output by the Trans U-Net network f2(θ2) in the student model.

5. The method for segmenting a knee cartilage defect according to claim 1, characterized in that: Setting the semi-supervised loss includes: According to the image X without cartilage defect label u The first unlabeled prediction result of the Attention U-Net network f1(θ1) in the student model is compared with the image X without the cartilage defect label u The first semi-supervised loss is calculated based on the dice coefficient loss through the second unlabeled prediction result of the Trans U-Net network f2(θ2) in the student model, and the expression is as follows: loss_semi1=Dice(f1(X u |θ1),f2(X u |θ2)) According to the image X without cartilage defect label u The second unlabeled prediction result of the Trans U-Net network f2(θ2) in the student model is compared with the image X without the cartilage defect label u The second semi-supervised loss is calculated based on the dice coefficient loss through the first unlabeled prediction result of the Attention U-Net network f1(θ1) in the student model, and the expression is as follows: loss_semi2=Dice(f2(X u |θ2),f1(X u |θ1)) According to the image X without cartilage defect label u The first unlabeled prediction result of the Attention U-Net network f1(θ1) in the student model is compared with the image X without the cartilage defect label u The third semi-supervised loss is calculated based on the dice coefficient loss through the third unlabeled prediction result of the Attention U-Net network f3(θ3) in the teacher model, and the expression is as follows: loss_semi3=Dice(f1(X u |θ1),f3(X u |θ3)) According to the image X without cartilage defect label u The second unlabeled prediction result of the Trans U-Net network f2(θ2) in the student model is compared with the image X without the cartilage defect label u The fourth semi-supervised loss is calculated based on the dice coefficient loss through the third unlabeled prediction result of the Attention U-Net network f3(θ3) in the teacher model, and the expression is as follows: loss_semi4=Dice(f2(X u |θ2),f3(X u |θ3)) In the formula, f1(X u |θ1) represents the first unlabeled prediction result, f2(X u |θ2) represents the second unlabeled prediction result, f3(X u |θ3) represents the third unlabeled prediction result.

6. A method for segmenting a knee cartilage defect according to claim 4 or 5, characterized in that: The weighted sum of the supervised loss and the semi-supervised loss is used as the total loss function, and the expression of the total loss function is as follows: Loss_train=loss_sup1+loss_sup2+λ1(loss_semi1+loss_semi2)+λ2(loss_semi3+loss_semi4) Where λ1 and λ2 represent weight coefficients.

7. A method for segmenting a knee cartilage defect according to claim 1, characterized in that: The exponential moving average of the network weights of the Attention U-Net network in the student model is used to update the network weights in the teacher model, including: In the formula, represents the network weights of the Attention U-Net network in the student model, represents the network weight of the Attention U-Net network in the teacher model, and t is the training round.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the knee cartilage defect segmentation method described in one of claims 1-7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the knee cartilage defect segmentation method as described in any one of claims 1-7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for segmenting a knee cartilage defect as described in any one of claims 1 to 7 is implemented.

Images