Ultrasonic image segmentation model training method, ultrasonic image segmentation method, electronic device and storage medium

Abstract

The embodiments of the present invention provide an ultrasound image segmentation model training method, an ultrasound image segmentation method, an electronic device and a storage medium, and relate to the field of medical image segmentation technology. The labeled original ultrasound image and the unlabeled original ultrasound image are subjected to strong or weak perturbations respectively, and the perturbed ultrasound image is input into the student segmentation network and the teacher segmentation network, and the pixel-level supervision loss information and the pixel-level consistency loss information, as well as the perception-level supervision loss information and the perception-level consistency loss information are calculated, and then the total loss information is calculated, and the parameters of the student segmentation network are iteratively optimized according to the total loss information, and the teacher segmentation network is iteratively optimized according to the parameters of the updated student segmentation network. In this way, the model can be constrained by the pixel-level loss information and the perception-level loss information, so as to improve the recognition ability of the teacher segmentation network for the nodule boundaries and shape features in the ultrasound image.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to an ultrasound image segmentation model training method, an ultrasound image segmentation method, an electronic device and a storage medium. Background Art

[0002] Thyroid nodules refer to solid or cystic lesions in the thyroid gland, which are quite common in clinical practice. Ultrasound examination is the main means of detecting thyroid nodules. It is widely used because of its non-invasiveness, ease of operation and high sensitivity. Ultrasound images are obtained through ultrasound examination, and then the ultrasound images are segmented using automated segmentation technology.

[0003] Existing segmentation technologies use algorithms based on supervised learning. However, supervised learning is highly dependent on the support of a large amount of labeled data, which usually needs to be manually labeled by experienced clinicians, which is time-consuming and laborious.

[0004] Therefore, the existing technology uses a semi-supervised learning method to solve the above problem. The semi-supervised learning method includes a method based on consistency learning and a method based on pseudo-label generation. The method based on consistency learning constrains the prediction consistency of the same ultrasound image under different disturbances. This method usually uses two models with different weights to implement the training framework.

[0005] However, due to the high similarity between the target and the background in ultrasound images, as well as the unclear boundaries of the target, methods that enforce consistency constraints at the pixel level usually produce poor segmentation contours and background mis-segmentation problems, and fail to fully capture small but critical structural differences. Summary of the invention

[0006] In view of this, an object of an embodiment of the present invention is to provide an ultrasound image segmentation model training method, an ultrasound image segmentation method, an electronic device and a storage medium to at least partially improve the above-mentioned problem.

[0007] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a method for training an ultrasound image segmentation model, the method comprising:

[0009] Acquire an original ultrasonic image with annotations and an original ultrasonic image without annotations; the original ultrasonic image with annotations includes real annotations;

[0010] Performing strong perturbations on the labeled original ultrasound image to obtain a plurality of labeled strongly perturbed images; performing strong perturbations and weak perturbations on the unlabeled original ultrasound image to obtain a plurality of unlabeled strongly perturbed images and a plurality of unlabeled weakly perturbed images;

[0011] Input each of the labeled strongly perturbed images and each of the unlabeled strongly perturbed images into a student segmentation network, respectively, to obtain a plurality of corresponding labeled student prediction results and a plurality of corresponding unlabeled student prediction results; input each of the unlabeled weakly perturbed images into a teacher segmentation network, to obtain a plurality of corresponding unlabeled teacher prediction results;

[0012] Calculate pixel-level supervision loss information and pixel-level consistency loss information according to the labeled student prediction results, the true annotations, the unlabeled student prediction results, and the unlabeled teacher prediction results;

[0013] Calculate the perception-level supervision loss information and the perception-level consistency loss information according to the labeled student prediction results, the real annotations, the unlabeled student prediction results, the unlabeled teacher prediction results, the labeled original ultrasound image and the unlabeled original ultrasound image;

[0014] Calculating total loss information according to the pixel-level supervision loss information, the pixel-level consistency loss information, the perception-level supervision loss information, and the perception-level consistency loss information;

[0015] The parameters of the student segmentation network are iteratively optimized according to the total loss information, the teacher segmentation network is iteratively optimized according to the updated parameters of the student segmentation network, and the optimized teacher segmentation network is used as an ultrasound image segmentation model.

[0016] Optionally, the calculating of the perceptual-level supervision loss information and the perceptual-level consistency loss information according to the labeled student prediction results, the real annotations, the unlabeled student prediction results, the unlabeled teacher prediction results, the labeled original ultrasound image and the unlabeled original ultrasound image includes:

[0017] Performing matrix multiplication on the labeled original ultrasound image, the labeled student prediction result and the real annotation, respectively, to obtain a labeled student mask image and a labeled real mask image;

[0018] Performing matrix multiplication on the unlabeled original ultrasound image, the unlabeled student prediction result and the unlabeled teacher prediction result, respectively, to obtain an unlabeled student mask image and an unlabeled teacher mask image;

[0019] Inputting the labeled student mask image and the labeled real mask image into a first feature extractor respectively, and calculating the perceptual level supervision loss information;

[0020] The unlabeled student mask image and the unlabeled teacher mask image are respectively input into a second feature extractor to calculate perception-level consistency loss information.

[0021] Optionally, the first feature extractor includes a plurality of first convolutional layers, and the labeled student mask image and the labeled real mask image are respectively input into the first feature extractor to calculate the perceptual level supervision loss information, including:

[0022] The labeled student mask image and the labeled real mask image are respectively input into the first feature extractor, and after each layer of the first convolution layer, the sub-perception information of each layer of the first convolution layer is calculated; the calculation formula of the sub-perception information is:

[0023] φ i (I) = ReLU(W i *ReLU(W i-1 *…ReLU(W1*I+b1)…+b i-1 )+b i )

[0024] Where I is the labeled student mask image or the labeled real mask image, W i is the convolution kernel of the first convolutional layer of the i-th layer, b i is the bias of the first convolutional layer in the i-th layer, * is the convolution operation;

[0025] The perception-level supervision loss information is calculated according to each of the sub-perception information; the calculation formula of the perception-level supervision loss information is:

[0026]

[0027] Among them, I gs is the labeled student mask image, φ i (I gs ) is the sub-perception information of each layer of the first convolutional layer corresponding to the labeled student mask image, I gt is the annotated true mask image, φ i (I gt ) is the sub-perception information of the first convolutional layer of each layer corresponding to the labeled true mask image.

[0028] Optionally, the second feature extractor includes a plurality of second convolutional layers, and the unlabeled student mask image and the unlabeled teacher mask image are respectively input into the second feature extractor to calculate the perceptual level consistency loss information, including:

[0029] Inputting the unlabeled student mask image and the unlabeled teacher mask image into a second feature extractor respectively, and passing through each layer of the second convolution layer to obtain a plurality of first student feature maps and a plurality of first teacher feature maps;

[0030] Upsampling each of the first student feature maps and each of the first teacher feature maps to perform feature alignment to obtain a plurality of second student feature maps and second teacher feature maps with consistent spatial sizes;

[0031] Performing feature fusion on each of the second student feature graphs and each of the second teacher feature graphs to obtain a third student feature graph and a third teacher feature graph respectively;

[0032] According to the third student feature map and the third teacher feature map, the perceptual level consistency loss information is calculated; the calculation formula of the perceptual level consistency loss information is:

[0033] l2=∥φ(I s )-φ(I t )∥ 2

[0034] Among them, I s is the third student characteristic graph, I t For the third teacher characteristic graph, the φ formula is expressed as:

[0035] φ(I)=Conv(U(L1(I)),U(L2(L1(I))),...,U(L n (L n-1 (...L1(I)...))))

[0036] Among them, I is the input image, U is the upsampling operation, Conv is a 1×1 convolution operation, and L n It is the second convolutional layer of the nth layer.

[0037] Optionally, the calculating pixel-level supervision loss information and pixel-level consistency loss information according to each of the labeled student prediction results, each of the real annotations, each of the unlabeled student prediction results, and each of the unlabeled teacher prediction results includes:

[0038] Calculate pixel-level supervision loss information according to the labeled student prediction results and the real annotations; the pixel-level supervision loss information includes cross entropy loss and Dice loss;

[0039] According to the unlabeled student prediction results and the unlabeled teacher prediction results, pixel-level consistency loss information is calculated; the pixel-level consistency loss information includes mean square error loss.

[0040] Optionally, the iterative optimization of the teacher segmentation network according to the updated parameters of the student segmentation network includes:

[0041] According to the updated parameters of the student segmentation network, the parameters of the teacher segmentation network are optimized by exponential moving average; the formula of the exponential moving average is:

[0042] θ′ t =αθ′ t-1 +(1-α)θ t

[0043] Among them, θ′ t-1 is the current parameter of the teacher segmentation network, θ′ t is the updated parameter of the teacher segmentation network, θ t is the parameter of the student segmentation network, and α is a hyperparameter that controls the parameter update amplitude of the teacher segmentation network.

[0044] Optionally, the strong disturbance includes Rayleigh noise and CutOut; the weak disturbance includes rotation, translation, scaling and cropping.

[0045] In a second aspect, an embodiment of the present invention provides an ultrasound image segmentation method, the method comprising:

[0046] Acquiring an ultrasound image to be segmented;

[0047] The ultrasound image to be segmented is input into an ultrasound image segmentation model to obtain a segmented image; the ultrasound image segmentation model is trained by the method described in any one of the first aspects above.

[0048] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the first aspects above is implemented; and / or the method described in the second aspect above is implemented.

[0049] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any one of the first aspect above; and / or the method described in the second aspect above.

[0050] Compared with the prior art, the embodiments of the present invention provide an ultrasound image segmentation model training method, an ultrasound image segmentation method, an electronic device and a storage medium. By performing strong or weak perturbations on the labeled original ultrasound image and the unlabeled original ultrasound image, the perturbed ultrasound image is input into the student segmentation network and the teacher segmentation network, and pixel-level supervision loss information and pixel-level consistency loss information, as well as perception-level supervision loss information and perception-level consistency loss information are calculated, and then the total loss information is calculated. The parameters of the student segmentation network are iteratively optimized according to the total loss information, and the teacher segmentation network is iteratively optimized according to the parameters of the updated student segmentation network. Thus, the model can be constrained by the pixel-level loss information and the perception-level loss information, and the recognition ability of the teacher segmentation network for the nodule boundary and shape features in the ultrasound image can be improved.

[0051] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a flow chart of an ultrasound image segmentation model training method provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a flow chart of perception-level loss information calculation provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of perceptual feature extraction provided by an embodiment of the present invention;

[0057] Figure 5 A schematic flow chart of an ultrasound image segmentation method provided by an embodiment of the present invention.

[0058] Icon: 100 - electronic device; 101 - memory; 102 - communication interface; 103 - processor; 104 - bus. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0062] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0063] In the prior art, the methods for training ultrasound image segmentation models mainly include:

[0064] 1. Based on the supervised learning algorithm, the model is trained with the support of a large amount of labeled data.

[0065] 2. Semi-supervised learning methods, consistency learning-based methods, and pseudo-label generation-based methods. The consistency learning-based method constrains the prediction consistency of the same ultrasound image under different perturbations. This method usually uses two models with different weights to implement the training framework.

[0066] However, due to the high similarity between the target and the background in ultrasound images, as well as the unclear boundaries of the target, methods that enforce consistency constraints at the pixel level usually produce poor segmentation contours and background mis-segmentation problems, and fail to fully capture small but critical structural differences.

[0067] Based on the above situation, the embodiment of the present invention provides an ultrasound image segmentation model training method, an ultrasound image segmentation method, an electronic device and a storage medium, by performing strong or weak perturbations on the labeled original ultrasound image and the unlabeled original ultrasound image, respectively, and inputting the perturbed ultrasound image into the student segmentation network and the teacher segmentation network, calculating the pixel-level supervision loss information and the pixel-level consistency loss information, as well as the perception-level supervision loss information and the perception-level consistency loss information, and then calculating the total loss information, iteratively optimizing the parameters of the student segmentation network according to the total loss information, and iteratively optimizing the teacher segmentation network according to the updated parameters of the student segmentation network. Thus, the model can be constrained by the pixel-level loss information and the perception-level loss information, and the recognition ability of the teacher segmentation network for the nodule boundaries and shape features in the ultrasound image can be improved.

[0068] To implement the process steps and functions of each example of the present invention, please refer to Figure 1 , Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0069] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical server, but may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, a Windows system, a Linux system, etc.

[0070] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0071] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).

[0072] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the embodiment of the present invention can be completed by the hardware integrated logic circuit in the processor 103 or the instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0073] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0074] The following is an exemplary description of the ultrasound image segmentation model training method provided by the present invention. Specifically, Figure 2 A flow chart of a method for training an ultrasound image segmentation model provided by an embodiment of the present invention is shown in FIG. Figure 2 , the execution subject of this method can be the above Figure 1 The electronic device 100 shown in FIG. 1 includes: Figure 2The following steps are shown:

[0075] S210: Acquire an annotated original ultrasound image and an unannotated original ultrasound image; wherein the annotated original ultrasound image includes a real annotation.

[0076] The acquired labeled original ultrasound images and unlabeled original ultrasound images are both large batches of data. In one training iteration cycle, a batch of ultrasound images may be processed, for example, 50 labeled original ultrasound images and 50 unlabeled original ultrasound images, or only two ultrasound images may be processed, that is, 1 labeled original ultrasound image and 1 unlabeled original ultrasound image. For ease of understanding, this embodiment uses 1 labeled original ultrasound image and 1 unlabeled original ultrasound image for explanation.

[0077] The actual annotation may be an annotation of a nodule, for example, an annotation of a thyroid nodule in a thyroid ultrasound image.

[0078] S220: performing strong perturbation on the labeled original ultrasound image to obtain a plurality of labeled strongly perturbed images; performing strong perturbation and weak perturbation on the unlabeled original ultrasound image to obtain a plurality of unlabeled strongly perturbed images and a plurality of unlabeled weakly perturbed images.

[0079] A strongly perturbed original labeled ultrasound image is subjected to strong perturbation to obtain a plurality of strongly perturbed images with labels, which may be a preset number, for example, 5 images. A strongly perturbed and weakly perturbed original unlabeled ultrasound image are subjected to strong perturbation and weak perturbation respectively to obtain 5 strongly perturbed images with no labels and 5 weakly perturbed images with no labels. That is, through perturbation processing, 1 labeled original ultrasound image is transformed into 5 strongly perturbed images, and 1 unlabeled original ultrasound image is transformed into 5 strongly perturbed images and 5 weakly perturbed images. If a batch of ultrasound images are processed, similarly, each ultrasound image is processed to obtain the corresponding perturbed image.

[0080] Optionally, strong perturbations may include Rayleigh noise and CutOut; weak perturbations may include rotation, translation, scaling, and cropping.

[0081] S230: Input each labeled strongly perturbed image and each unlabeled strongly perturbed image into the student segmentation network respectively, and obtain a plurality of corresponding labeled student prediction results and a plurality of corresponding unlabeled student prediction results respectively; input each unlabeled weakly perturbed image into the teacher segmentation network, and obtain a plurality of corresponding unlabeled teacher prediction results.

[0082] In this embodiment, the 5 labeled strongly perturbed images and 5 unlabeled strongly perturbed images obtained above are input into the student segmentation network to obtain 5 labeled student prediction results and 5 unlabeled student prediction results. For the 5 unlabeled weakly perturbed images, the teacher segmentation network is input to obtain 5 unlabeled teacher prediction results.

[0083] Optionally, the student segmentation network and the teacher segmentation network can be basic segmentation networks such as UNet and VNet.

[0084] S240: Calculate pixel-level supervision loss information and pixel-level consistency loss information based on the labeled student prediction results, the true labels, the unlabeled student prediction results, and the unlabeled teacher prediction results.

[0085] S250: Calculate the perception-level supervision loss information and the perception-level consistency loss information according to the labeled student prediction results, the true labels, the unlabeled student prediction results, the unlabeled teacher prediction results, the labeled original ultrasound image and the unlabeled original ultrasound image.

[0086] S260: Calculate the total loss information based on the pixel-level supervision loss information, the pixel-level consistency loss information, the perception-level supervision loss information, and the perception-level consistency loss information.

[0087] For the total loss information, the following formula can be used:

[0088] l=l sup +l con

[0089] l sup =α1l sup1 +β1l sup2

[0090] l con =α2l con1 +β2l con2

[0091] Among them, l is the total loss information, l sup To monitor loss information, l con is the consistency loss information, l sup1 is the pixel-level supervision loss information, l sup2 is the perceptual-level supervision loss information, l con1 is the pixel-level consistency loss information, l con2 is the perceptual level consistency loss information, and α1, β1, α2, β2 are hyperparameters for balancing pixel constraints and perceptual constraints.

[0092] S270: Iteratively optimize the parameters of the student segmentation network according to the total loss information, iteratively optimize the teacher segmentation network according to the updated parameters of the student segmentation network, and use the optimized teacher segmentation network as the ultrasound image segmentation model.

[0093] In the ultrasound image segmentation model training method, whether it is an annotated image or an unannotated image, the student segmentation network receives strongly perturbed data as input, prompting the student segmentation network to learn different states of the same image. The teacher segmentation network receives weakly perturbed unannotated data, so that the teacher segmentation network provides more accurate guidance to the student segmentation network during the training process. Constraints between the predictions of the student segmentation network and the true annotations, and between the predictions of the student segmentation network and the predictions of the teacher segmentation network. During the entire training process, the pixel-level constraints ensure that the student segmentation network is consistent with the true annotations and the prediction results of the teacher segmentation network at the detail level. This constraint can effectively improve the accuracy and robustness of the model. The constraints at the perception level further enhance the student segmentation network's ability to understand and capture high-level semantic information by utilizing multi-level features. Finally, through the dual constraints at the pixel and perception levels, the model can effectively avoid the mis-segmentation of areas similar to the target in ultrasound images, thereby improving the performance and stability of the model.

[0094] There are many ways to implement pixel-level loss information, such as cross entropy loss, mean square error loss, etc. In one possible implementation, the above step S240 may include the following steps:

[0095] S241: Calculate pixel-level supervision loss information based on the labeled student prediction results and the true labels; the pixel-level supervision loss information includes cross entropy loss and Dice loss.

[0096] Specifically, the pixel-level supervision loss information can be expressed by the following formula:

[0097] l sup1 = l dice +l ce

[0098] Among them, l ce is the cross entropy loss function, l dice is the Dice loss function.

[0099] S242: Calculate pixel-level consistency loss information based on the unlabeled student prediction results and the unlabeled teacher prediction results; wherein the pixel-level consistency loss information includes mean square error loss.

[0100] Specifically, the calculation formula for pixel-level consistency loss information can be:

[0101]

[0102] Among them, y i Prediction results for unlabeled teachers, Predict results for unlabeled students.

[0103] For the constraints at the perception level, the feature extractor can be used to extract the prediction results and then calculate the loss information. In one possible implementation, see Figure 3 , Figure 3 A schematic diagram of a flow chart of a perception level loss information calculation provided by an embodiment of the present invention, wherein the above step S250 may include the following steps: Figure 3 The following steps are shown:

[0104] S251: Perform matrix multiplication on the labeled original ultrasound image, the labeled student prediction result and the true annotation respectively, to obtain a labeled student mask image and a labeled true mask image respectively.

[0105] In this embodiment, there is one annotated original ultrasound image, five annotated student prediction results and one true annotation. After matrix multiplication, five annotated student mask images and one annotated true mask image are obtained. The obtained mask image only retains the area in the original image that is related to the true annotation or prediction result, and the other areas are replaced with a black background.

[0106] S252: Perform matrix multiplication on the unlabeled original ultrasound image, the unlabeled student prediction result and the unlabeled teacher prediction result, respectively, to obtain an unlabeled student mask image and an unlabeled teacher mask image.

[0107] In this embodiment, it includes 1 unlabeled original ultrasound image, 5 unlabeled student prediction results and 5 unlabeled teacher prediction results. After matrix multiplication, 5 unlabeled student mask images and 5 unlabeled teacher mask images are obtained.

[0108] S253: Input the labeled student mask image and the labeled real mask image into the first feature extractor respectively to calculate the perception-level supervision loss information.

[0109] S254: Input the unlabeled student mask image and the unlabeled teacher mask image into the second feature extractor respectively, and calculate the perception-level consistency loss information.

[0110] The first feature extractor and the second feature extractor may be the same or different. Now, steps S253 and S254 are described in the case where the first feature extractor and the second feature extractor are different.

[0111] The first feature extractor may include multiple layers of first convolutional layers, and step S253 may include the following steps:

[0112] S2531: Input the labeled student mask image and the labeled real mask image into the first feature extractor respectively, and calculate the sub-perception information of each first convolution layer after passing through each first convolution layer; the calculation formula of the sub-perception information is:

[0113] φ i (I) = ReLU(W i *ReLU(W i-1 *…ReLU(W1*I+b1)…+b i-1 )+b i )

[0114] Among them, I is a labeled student mask image or a labeled real mask image, W i is the convolution kernel of the first convolutional layer of the i-th layer, b i is the bias of the first convolutional layer of the i-th layer, and * is the convolution operation.

[0115] The first feature extractor can be a VGG19 encoder, including 19 first convolutional layers. By calculating through the above-mentioned sub-perception information calculation formula, 19 sub-perception information corresponding to the labeled student mask image and 19 sub-perception information corresponding to the labeled real mask image can be obtained.

[0116] S2532: Calculate the perception-level supervision loss information according to each sub-perception information; the calculation formula of the perception-level supervision loss information is:

[0117]

[0118] Among them, I gs is the labeled student mask image, φ i (I gs ) is the sub-perception information of the first convolutional layer of each layer corresponding to the labeled student mask image, I gt is the annotated true mask image, φ i (I gt ) is the sub-perception information of the first convolutional layer of each layer corresponding to the labeled true mask image.

[0119] After calculating multiple sub-perceptual information of the labeled student mask image and the labeled true mask image, the loss of each first convolutional layer can be calculated, and then the perceptual level supervision loss information can be calculated. That is, the perceptual information is obtained by extracting features from the predicted image through different levels of the first feature extractor, and the final perceptual loss is obtained by superimposing the losses between all layers of the first feature extractor.

[0120] The second feature extractor may include multiple layers of second convolutional layers, and step S254 may include the following steps:

[0121] S2541: Input the unlabeled student mask image and the unlabeled teacher mask image into the second feature extractor respectively, and obtain multiple first student feature maps and multiple first teacher feature maps after passing through each second convolution layer.

[0122] S2542: Perform feature alignment on up-sampling of each first student feature map and each first teacher feature map to obtain a plurality of second student feature maps and second teacher feature maps with consistent spatial sizes.

[0123] S2543: Perform feature fusion on each second student feature map and each second teacher feature map to obtain a third student feature map and a third teacher feature map respectively.

[0124] S2544: Calculate the perceptual level consistency loss information according to the third student feature map and the third teacher feature map; the calculation formula of the perceptual level consistency loss information is:

[0125] l con2 =∥φ(I s )-φ(I t )∥ 2

[0126] Among them, I s is the third student characteristic diagram, I t is the third teacher characteristic diagram, and the φ formula is expressed as:

[0127] φ(I)=Conv(U(L1(I)),U(L2(L1(I))),...,U(L n (L n-1 (...L1(I)...))))

[0128] Among them, I is the input image, U is the upsampling operation, Conv is a 1×1 convolution operation, and L n It is the second convolutional layer of the nth layer.

[0129] The unlabeled student mask image and the unlabeled teacher mask image are used as input images. After each layer of convolution operation, the feature map is passed layer by layer to simultaneously obtain the low-level semantic features and high-level semantic features in the input image. After each layer of feature extraction, an upsampling operation is performed so that the feature maps of different layers have the same spatial dimension. Specifically, the input image I passes through the second convolution layer L of the nth layer. n Then we get the feature map F n ,Right now:

[0130] F n =L n (F n-1 )

[0131] Where F0=I is the input image, F2, ..., F n , the feature map is aligned with the size of F1 through upsampling. The upsampled feature map is fused through 1×1 convolution.

[0132] Please refer to Figure 4 , Figure 4 A schematic diagram of perceptual feature extraction provided by an embodiment of the present invention. In this embodiment, the second feature extractor is illustrated using 4 second convolutional layers. For the unlabeled student mask image, the unlabeled student mask image is input into the second feature extractor, and 4 first student feature maps are obtained after passing through the second convolutional layers L1, L2, L3, and L4. L2, L3, and L4 are upsampled to obtain 3 second student feature maps of the same size as F1. L1 is fused with the 3 second student feature maps to obtain 1 third student feature map. Of course, L1 can also be upsampled to obtain the second student feature map, but its size remains unchanged.

[0133] Similarly, the third teacher feature map can be obtained. Finally, the perceptual level consistency loss information can be calculated through the calculation formula of the perceptual level consistency loss information.

[0134] It can be understood that the calculation method for the perception-level supervision loss information and the perception-level consistency loss information is not fixed. In addition to the method described above, it can also include: both the perception-level supervision loss information and the perception-level consistency loss information are calculated using the method of step S253; or both are calculated using the method of step S254; or, the perception-level supervision loss information is calculated using the method of step S254, and the perception-level consistency loss information is calculated using the method of step S253.

[0135] In a possible implementation, the iterative optimization of the teacher segmentation network according to the parameters of the student segmentation network may be performed by: the teacher segmentation network updates its own parameters by exponentially moving the parameters of the student segmentation network. Specifically, the iterative optimization of the teacher segmentation network according to the updated parameters of the student segmentation network may include:

[0136] According to the updated parameters of the student segmentation network, the parameters of the teacher segmentation network are optimized by exponential moving average. The formula of exponential moving average is:

[0137] θ′ t =αθ′ t-1 +(1-α)θ t

[0138] Among them, θ′ t-1 is the current parameter of the teacher segmentation network, θ′ t is the updated parameter of the teacher segmentation network, θ tis the parameter of the student segmentation network, and α is a hyperparameter that controls the parameter update amplitude of the teacher segmentation network.

[0139] Further, based on the above ultrasound image segmentation model training method, see Figure 5 , Figure 5 The present invention provides a flow chart of an ultrasonic image segmentation method according to an embodiment of the present invention. The present invention also provides an ultrasonic image segmentation method, which includes: Figure 5 The following steps are shown:

[0140] S310: Acquire the ultrasound image to be segmented;

[0141] S320: Input the ultrasound image to be segmented into the ultrasound image segmentation model to obtain a segmented image. The ultrasound image segmentation model is trained by the ultrasound image segmentation model training method.

[0142] The ultrasound image may be an ultrasound image of a target such as a thyroid gland, a skin disease, or a breast, which is close to the background area. By using the ultrasound image segmentation method, the nodules in the ultrasound image can be accurately identified.

[0143] In summary, an ultrasonic image segmentation model training method, an ultrasonic image segmentation method, an electronic device and a storage medium provided by an embodiment of the present invention, whether it is an annotated image or an unannotated image, the student segmentation network receives strongly disturbed data as input, prompting the student segmentation network to learn different states of the same image. The teacher segmentation network accepts weakly disturbed unannotated data, so that the teacher segmentation network provides more accurate guidance for the student segmentation network during the training process. Constraints between the prediction of the student segmentation network and the true annotation, and constraints between the prediction of the student segmentation network and the prediction of the teacher segmentation network. During the entire training process, the constraints at the pixel level ensure that the student segmentation network is consistent with the prediction results of the true annotation and the teacher segmentation network at the detail level. This constraint can effectively improve the accuracy and robustness of the model. The constraints at the perception level further enhance the student segmentation network's ability to understand and capture high-level semantic information by utilizing multi-level features. Finally, through the dual constraints of the pixel and perception levels, the model can effectively avoid the mis-segmentation of areas similar to the target in the ultrasonic image, thereby improving the performance and stability of the model. Using the trained ultrasonic image segmentation model to segment the ultrasonic image, nodules in the ultrasonic image can be accurately identified.

[0144] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0146] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0147] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0148] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for training an ultrasound image segmentation model, characterized in that: The method comprises: Acquire an original ultrasound image with annotations and an original ultrasound image without annotations; the original ultrasound image with annotations includes real annotations; Performing strong perturbations on the labeled original ultrasound image to obtain a plurality of labeled strongly perturbed images; performing strong perturbations and weak perturbations on the unlabeled original ultrasound image to obtain a plurality of unlabeled strongly perturbed images and a plurality of unlabeled weakly perturbed images; Input each of the labeled strongly perturbed images and each of the unlabeled strongly perturbed images into a student segmentation network, respectively, to obtain a plurality of corresponding labeled student prediction results and a plurality of corresponding unlabeled student prediction results; input each of the unlabeled weakly perturbed images into a teacher segmentation network, to obtain a plurality of corresponding unlabeled teacher prediction results; Calculate pixel-level supervision loss information and pixel-level consistency loss information according to the labeled student prediction results, the true annotations, the unlabeled student prediction results, and the unlabeled teacher prediction results; Calculate the perception-level supervision loss information and the perception-level consistency loss information according to the labeled student prediction results, the real annotations, the unlabeled student prediction results, the unlabeled teacher prediction results, the labeled original ultrasound image and the unlabeled original ultrasound image; Calculating total loss information according to the pixel-level supervision loss information, the pixel-level consistency loss information, the perception-level supervision loss information, and the perception-level consistency loss information; The parameters of the student segmentation network are iteratively optimized according to the total loss information, the teacher segmentation network is iteratively optimized according to the updated parameters of the student segmentation network, and the optimized teacher segmentation network is used as an ultrasound image segmentation model.

2. The method according to claim 1, characterized in that The method of calculating the perceptual level supervision loss information and the perceptual level consistency loss information according to the labeled student prediction results, the real annotations, the unlabeled student prediction results, the unlabeled teacher prediction results, the labeled original ultrasound image and the unlabeled original ultrasound image includes: Performing matrix multiplication on the labeled original ultrasound image, the labeled student prediction result and the real annotation, respectively, to obtain a labeled student mask image and a labeled real mask image; Performing matrix multiplication on the unlabeled original ultrasound image, the unlabeled student prediction result and the unlabeled teacher prediction result, respectively, to obtain an unlabeled student mask image and an unlabeled teacher mask image; Inputting the labeled student mask image and the labeled real mask image into a first feature extractor respectively, and calculating the perceptual level supervision loss information; The unlabeled student mask image and the unlabeled teacher mask image are respectively input into a second feature extractor to calculate perception-level consistency loss information.

3. The method according to claim 2, characterized in that The first feature extractor includes a plurality of first convolutional layers, and the labeled student mask image and the labeled real mask image are respectively input into the first feature extractor to calculate the perceptual level supervision loss information, including: The labeled student mask image and the labeled real mask image are respectively input into the first feature extractor, and after each layer of the first convolution layer, the sub-perception information of each layer of the first convolution layer is calculated; the calculation formula of the sub-perception information is: φ i (I)=ReLU(W i *ReLU(W i-1 *…ReLU(W1*I+b1)…+b i-1 )+b i ) Where I is the labeled student mask image or the labeled real mask image, W i is the convolution kernel of the first convolutional layer of the i-th layer, b i is the bias of the first convolutional layer in the i-th layer, * is the convolution operation; The perception-level supervision loss information is calculated according to each of the sub-perception information; the calculation formula of the perception-level supervision loss information is: Among them, I gs is the labeled student mask image, φ i (I gs ) is the sub-perception information of each layer of the first convolutional layer corresponding to the labeled student mask image, I gt is the annotated true mask image, φ i (I gt ) is the sub-perception information of the first convolutional layer of each layer corresponding to the labeled true mask image.

4. The method according to claim 2, characterized in that: The second feature extractor includes a plurality of second convolutional layers, and the unlabeled student mask image and the unlabeled teacher mask image are respectively input into the second feature extractor to calculate the perceptual level consistency loss information, including: Inputting the unlabeled student mask image and the unlabeled teacher mask image into a second feature extractor respectively, and passing through each layer of the second convolution layer to obtain a plurality of first student feature maps and a plurality of first teacher feature maps; Upsampling each of the first student feature maps and each of the first teacher feature maps to perform feature alignment to obtain a plurality of second student feature maps and second teacher feature maps with consistent spatial sizes; Performing feature fusion on each of the second student feature graphs and each of the second teacher feature graphs to obtain a third student feature graph and a third teacher feature graph respectively; According to the third student feature map and the third teacher feature map, the perceptual level consistency loss information is calculated; the calculation formula of the perceptual level consistency loss information is: l con2 =∥φ(I s )-φ(I t )∥ 2 Among them, I s is the third student characteristic graph, I t For the third teacher characteristic graph, the φ formula is expressed as: φ(I)=Conv(U(L1(I)),U(L2(L1(I))),...,U(L n (L n-1 (...L1(I)...)))) Among them, I is the input image, U is the upsampling operation, Conv is a 1×1 convolution operation, and L n It is the second convolutional layer of the nth layer.

5. The method according to claim 1, characterized in that The pixel-level supervision loss information and the pixel-level consistency loss information are calculated according to the labeled student prediction results, the real annotations, the unlabeled student prediction results and the unlabeled teacher prediction results, including: Calculate pixel-level supervision loss information according to the labeled student prediction results and the real annotations; the pixel-level supervision loss information includes cross entropy loss and Dice loss; According to the unlabeled student prediction results and the unlabeled teacher prediction results, pixel-level consistency loss information is calculated; the pixel-level consistency loss information includes mean square error loss.

6. The method according to claim 1, characterized in that The iterative optimization of the teacher segmentation network according to the updated parameters of the student segmentation network includes: According to the updated parameters of the student segmentation network, the parameters of the teacher segmentation network are optimized by exponential moving average; the formula of the exponential moving average is: θ′ t =θ′ t-1 +(1-α)θ t Among them, θ′ t-1 is the current parameter of the teacher segmentation network, θ′ t is the updated parameter of the teacher segmentation network, θ t is the parameter of the student segmentation network, and α is a hyperparameter that controls the parameter update amplitude of the teacher segmentation network.

7. The method according to claim 1, characterized in that The strong disturbances include Rayleigh noise and CutOut; the weak disturbances include rotation, translation, scaling and cropping.

8. An ultrasonic image segmentation method, characterized in that: The method comprises: Acquiring an ultrasound image to be segmented; The ultrasound image to be segmented is input into an ultrasound image segmentation model to obtain a segmented image; the ultrasound image segmentation model is trained by the method according to any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 and / or claim 8 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 and / or claim 8 is implemented.

Images