A plaque segmentation network training method and device, an electronic device, and a storage medium

Abstract

The application provides a plaque segmentation network training method and device, electronic equipment and a storage medium, wherein the plaque segmentation network training method comprises: inputting a real ultrasonic image of a carotid plaque into a generator subnetwork in a generative adversarial network, and obtaining a generated ultrasonic image output by the generator subnetwork; inputting the generated ultrasonic image into a plaque segmentation network, and obtaining a generated plaque label output by the plaque segmentation network; jointly training the plaque segmentation network and the generative adversarial network, and obtaining a trained plaque segmentation network. The above scheme jointly trains the generative adversarial network comprising the generator subnetwork, an image discriminator, a structure discriminator and a matching discriminator, and the plaque segmentation network, the structure discriminator in the generative adversarial network can stimulate the generator subnetwork to generate more diversified generated ultrasonic images, the plaque segmentation network is trained by using the diversified generated ultrasonic images, and this is beneficial to improving the segmentation accuracy of the plaque segmentation network.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a patch segmentation network training method, apparatus, electronic device, and storage medium. Background Technology

[0002] Ultrasound imaging technology plays a vital role in medical applications, its non-invasive and real-time nature making it a common diagnostic tool. With technological advancements, ultrasound imaging has been widely used to examine organs such as the heart, liver, and kidneys, especially in the cardiovascular field. Ultrasound imaging provides high-resolution images, helping doctors assess vascular structure and abnormalities. Carotid artery ultrasound, as a non-invasive and low-cost examination method, has seen widespread use in recent years. However, the interpretation and analysis of carotid artery ultrasound images still face some challenges. Plaque segmentation in the images has become a focus of research, as accurate plaque segmentation is crucial for diagnosing diseases such as atherosclerosis.

[0003] Currently, most carotid plaque segmentation schemes in related technologies use segmentation models to segment plaques in ultrasound images. However, the segmentation models used in these technologies often experience mode collapse, which reduces the accuracy of plaque segmentation in carotid plaque segmentation models. Summary of the Invention

[0004] The purpose of this application is to provide a plaque segmentation network training method, apparatus, electronic device, and storage medium to improve the accuracy of carotid artery plaque segmentation.

[0005] In a first aspect, embodiments of this application provide a method for training a plaque segmentation network. The method includes: inputting a real ultrasound image of a carotid artery plaque into a generator subnetwork of a generative adversarial network (GAN) to obtain a generated ultrasound image output by the generator subnetwork; inputting the generated ultrasound image into a plaque segmentation network to obtain generated plaque annotations output by the plaque segmentation network; and jointly training the plaque segmentation network and the GAN based on the real ultrasound image, the real plaque annotations of the real ultrasound image, the generated ultrasound image, and the generated plaque annotations to obtain a trained plaque segmentation network. The GAN includes the generator subnetwork, an image discriminator for identifying whether the input image is the real ultrasound image, a structure discriminator for identifying whether the input annotation is the real plaque annotation, and a matching discriminator for determining whether the generated ultrasound image and the generated plaque annotation match.

[0006] In the implementation of the above scheme, a generative adversarial network (GAN) containing a generator sub-network, an image discriminator, a structure discriminator, and a matching discriminator is trained together with a patch segmentation network. The structure discriminator in the GAN can incentivize the generator sub-network to generate more diverse generated ultrasound images. Using diverse generated ultrasound images to train the patch segmentation network helps improve the segmentation accuracy of the patch segmentation network.

[0007] In one implementation of the first aspect, the joint training of the patch segmentation network and the generative adversarial network includes: training the image discriminator with the real ultrasound image and the generated ultrasound image as input and minimizing a first loss function as the training objective; wherein the first loss function is used to characterize the image discriminator's ability to distinguish between the real ultrasound image and the generated ultrasound image.

[0008] In the implementation of the above scheme, the image discriminator is trained using a first loss function that characterizes the discriminator's ability to distinguish between real and generated ultrasound images. This incentivizes the generator subnetwork to generate ultrasound images that are closer to the distribution of real ultrasound images. Using more realistic generated ultrasound images in conjunction with real ultrasound images to train the patch segmentation network helps improve the segmentation accuracy of the patch segmentation network.

[0009] In one implementation of the first aspect, the joint training of the patch segmentation network and the generative adversarial network includes: training the structure discriminator with the real patch labels and the generated patch labels as inputs and minimizing a second loss function as the training objective; wherein the second loss function is used to characterize the discriminator's ability to distinguish between the real patch labels and the generated patch labels.

[0010] In the implementation of the above scheme, a second loss function representing the ability of the structure discriminator to distinguish between real patch annotations and generated patch annotations is used to train the structure discriminator. This makes the generated patch annotations extracted by the patch segmentation network conform to the real distribution of real patch annotations, thereby increasing the diversity of generated ultrasound images generated by the generator subnetwork. Using generated ultrasound images with diversity in combination with real ultrasound images to train the patch segmentation network is beneficial to improving the patch segmentation accuracy of the patch segmentation network.

[0011] In one implementation of the first aspect, the joint training of the patch segmentation network and the generative adversarial network includes: training the matching discriminator with the real ultrasound image, the real patch annotation, the generated ultrasound image, and the generated patch annotation as input, and minimizing a third loss function as the training objective; wherein the third loss function is used to characterize the discriminator's ability to distinguish between a first matching relationship and a second matching relationship; the first matching relationship is used to characterize the matching relationship between the real ultrasound image and the real patch annotation; and the second matching relationship is used to characterize the matching relationship between the generated ultrasound image and the generated patch annotation.

[0012] In the implementation of the above scheme, by introducing a matching discriminator, the generative adversarial network can focus on the matching relationship between the generated ultrasound image and the generated patch label, thereby making the generated ultrasound image generated by the generator sub-network more closely matched with the generated patch label, which is beneficial to improving the practicality and reliability of the generated ultrasound image. Using the generated ultrasound image generated by the generator sub-network in conjunction with the real ultrasound image to train the matching discriminator is beneficial to improving the patch segmentation accuracy of the patch segmentation network.

[0013] In one implementation of the first aspect, the joint training of the patch segmentation network and the generative adversarial network includes: training the generator sub-network with the real ultrasound image as input and minimizing a fourth loss function as the training objective; wherein the fourth loss function is used to characterize the difference between the real ultrasound image and the generated ultrasound image.

[0014] In the implementation of the above scheme, a fourth loss function that characterizes the difference between real ultrasound images and generated ultrasound images is used to train the generator sub-network, so that the generated ultrasound images generated by the generator sub-network are close to the real ultrasound images. Using the generated ultrasound images generated by the generator sub-network in conjunction with real ultrasound images to train the patch segmentation network is beneficial to improving the patch segmentation accuracy of the patch segmentation network.

[0015] In one implementation of the first aspect, the joint training of the patch segmentation network and the generative adversarial network includes: training the patch segmentation network with the real ultrasound image, the real patch annotations, the generated ultrasound image, and the generated patch annotations as inputs, and minimizing a fifth loss function as the training objective; wherein the fifth loss function is a loss function calculated based on the first loss function of the image discriminator, the second loss function of the structure discriminator, the third loss function of the matching discriminator, the fourth loss function of the generator sub-network, and the difference between the output annotations and the input annotations of the patch segmentation network.

[0016] In the implementation of the above scheme, a fifth loss function, which integrates the first, second, third, and fourth loss functions as well as the difference between the input and output labels of the patch segmentation network, is used to train the patch segmentation network. This allows the patch segmentation network and the generative adversarial network to be trained together. On the one hand, this makes the generated ultrasound images produced by the generative adversarial network more closely resemble real ultrasound images. On the other hand, it makes the generated patch labels extracted by the patch segmentation network from the generated ultrasound images more closely resemble real patch labels. This is beneficial to improving the training effect of the patch segmentation network, thereby improving the patch segmentation accuracy of the patch segmentation network.

[0017] Secondly, embodiments of this application provide a plaque segmentation method, the method comprising: acquiring an ultrasound image of a carotid plaque to be segmented; inputting the ultrasound image of the carotid plaque to be segmented into a pre-trained plaque segmentation network to obtain a carotid plaque segmentation result of the ultrasound image of the carotid plaque to be segmented; wherein the pre-trained plaque segmentation network is trained based on the method provided in the first aspect or any possible implementation of the first aspect.

[0018] Thirdly, embodiments of this application provide a patch segmentation network training device, the device comprising:

[0019] An ultrasound image acquisition module is used to input a real ultrasound image of a carotid artery plaque into a generator subnetwork in a generative adversarial network, and acquire the generated ultrasound image output by the generator subnetwork.

[0020] A patch annotation acquisition module is used to input the generated ultrasound image into a patch segmentation network and obtain the generated patch annotations output by the patch segmentation network;

[0021] The training module is used to jointly train the patch segmentation network and the generative adversarial network based on the real ultrasound image, the real patch annotation of the real ultrasound image, the generated ultrasound image and the generated patch annotation, so as to obtain the trained patch segmentation network.

[0022] The generative adversarial network includes a generator subnetwork, an image discriminator for identifying whether the input image is the real ultrasound image, a structure discriminator for identifying whether the input label is the real patch label, and a matching discriminator for determining whether the generated ultrasound image and the generated patch label match.

[0023] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other through the communication bus; the memory stores computer program instructions that can be executed by the processor, and the computer program instructions are read and executed by the processor to perform the method provided by the first aspect or any possible implementation of the first aspect or the second aspect or any possible implementation of the second aspect.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided by the first aspect or any possible implementation thereof, or the second aspect or any possible implementation thereof.

[0025] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic flowchart illustrating the patch segmentation network training method provided in this application embodiment;

[0028] Figure 2 This is a schematic diagram of the structure of a generative adversarial network provided in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of the generator subnetwork provided in an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of the structure of the patch segmentation network provided in an embodiment of this application;

[0031] Figure 5 A schematic diagram of the architecture of the image discriminator, structure discriminator, and matching discriminator provided in the embodiments of this application;

[0032] Figure 6A schematic diagram illustrating the training process of the patch segmentation network provided in an embodiment of this application;

[0033] Figure 7 A schematic flowchart illustrating the patch segmentation method provided in an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of the structure of the patch segmentation network training device provided in the embodiments of this application;

[0035] Figure 9 This is a schematic diagram of the structure of the patch segmentation device provided in the embodiments of this application;

[0036] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of this application, and are therefore merely examples and should not be used to limit the scope of protection of this application.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0039] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0040] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0041] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0042] Pattern collapse refers to the phenomenon where, during the training of a generative adversarial network (GAN), the generator can only produce samples with limited diversity, failing to cover all patterns in the data distribution. This results in the generator's output samples being either too similar or lacking diversity. Due to the existence of pattern collapse, the plaque segmentation accuracy of carotid plaque segmentation models decreases.

[0043] Based on this, this application provides a patch segmentation network training method. This method trains a patch segmentation network together with a generative adversarial network (GAN) that includes a generator sub-network, an image discriminator, a structure discriminator, and a matching discriminator. The structure discriminator in the GAN can incentivize the generator sub-network to generate more diverse generated ultrasound images. Using diverse generated ultrasound images to train the patch segmentation network helps improve the segmentation accuracy of the patch segmentation network.

[0044] The training method for the patch segmentation network described above is explained in detail below. Please refer to [link / reference needed]. Figure 1 This application provides a method for training a patch segmentation network, the method comprising:

[0045] Step S110: Input the real ultrasound image of the carotid artery plaque into the generator subnetwork of the generative adversarial network, and obtain the generated ultrasound image output by the generator subnetwork;

[0046] Step S120: Input the generated ultrasound image into the patch segmentation network and obtain the generated patch annotations output by the patch segmentation network;

[0047] Step S130: Based on real ultrasound images, real patch annotations of real ultrasound images, generated ultrasound images, and generated patch annotations, jointly train the patch segmentation network and the generative adversarial network to obtain the trained patch segmentation network.

[0048] First, the generative adversarial network used in the embodiments of this application will be described in detail:

[0049] Generative Adversarial Networks (GANs) are deep learning models consisting of two networks: a generator network and a discriminator network. The generator network aims to generate data similar to the real data distribution (the data generated by the generator network can also be called pseudo-data), while the discriminator network aims to distinguish between the generated data and the real data. During the training process of a GAN, the generator and discriminator train against each other; that is, the generator attempts to deceive the discriminator, making it unable to distinguish between generated and real data, while the discriminator attempts to accurately distinguish between the generated and real data. Through continuous iterative training, the generator gradually learns how to generate more realistic data, while the discriminator gradually becomes more accurate.

[0050] The generative adversarial network structure used in the embodiments of this application is as follows: Figure 2 As shown, it includes:

[0051] The generator subnetwork takes a real ultrasound image of a carotid artery plaque as input and outputs a generated ultrasound image. The goal of the generator subnetwork is to generate multi-domain, diverse generated ultrasound images based on the input real ultrasound image x. And it will not seriously damage the content information in the real ultrasound image x;

[0052] Image discriminator D img Its inputs are a real ultrasound image x and a generated ultrasound image. (i.e. Figure 2 (fakeimage in D) img Used to determine the authenticity of the input image, that is, to determine whether the input image is a real ultrasound image or a generated ultrasound image;

[0053] Structure discriminator D strct Its inputs are the real patch annotation z and the generated patch annotation. (i.e. Figure 2 (fake structure in D) strct Used to determine the authenticity of input annotations, that is, to determine whether the input image annotations are real image annotations or generated image annotations;

[0054] Matching Discriminator D pair Its input is a real ultrasound image - real plaque annotation pair<x,z> Generate ultrasound images - Generate patch annotation pairs D pair Used for judgment Generating ultrasound images With generated patch annotation Does it match?

[0055] Additionally, it should be noted that the input to the aforementioned generative adversarial network includes a real ultrasound image x and its corresponding real patch annotation z, and the generated patch annotation... This is achieved by using a patch segmentation network to generate ultrasound images. Obtained after processing.

[0056] Both the generator subnetwork and the patch segmentation network described above can be constructed using a U-net with skip connections. The structures of the generator subnetwork and the patch segmentation network are as follows: Figure 3 and Figure 4 As shown, because the generator subnetwork is designed to modify the appearance of an image, it has a lighter structural weight and fewer convolutional and pooling layers compared to the patch segmentation network. Image discriminator D img Structure discriminator D strct Matching discriminator D pair Can be shared, such as Figure 5 The same classification structure design is shown, except that the matching discriminant D... pair It is a dual-channel input.

[0057] The following section provides a detailed explanation of step S130, taking into account the structures of the generative adversarial network and the patch segmentation network described above.

[0058] The co-training in step S130 refers to a training method that combines the Generative Adversarial Network (GAN) and the patch segmentation network. In this method, the GAN generates generated ultrasound images that are similar in distribution to real ultrasound images, while the patch segmentation network segments these generated ultrasound images to obtain annotated images. Through co-training, the GAN and the patch segmentation network can learn from and optimize each other. The GAN can provide realistic generated ultrasound images for the patch segmentation network to train on, while the patch segmentation network can provide annotated generated patches to help the GAN better understand the structure and features of the images.

[0059] The image discriminator D is discussed below. img Structure discriminator D strct Matching discriminator D pair The training methods will be introduced as follows:

[0060] As an optional implementation of the above-mentioned patch segmentation network training method, step S130 includes: training the image discriminator with real ultrasound images and generated ultrasound images as inputs and minimizing a first loss function as the training objective; wherein, the first loss function is used to characterize the image discriminator's ability to distinguish between real ultrasound images and generated ultrasound images.

[0061] The calculation method for the first loss function mentioned above is as follows:

[0062]

[0063] Among them, E x (·) denotes the expectation operation on the probability distribution; D img (x) represents the image discriminator D. img The real ultrasound image x is determined to be the true probability distribution; The discriminator will generate ultrasound images. The probability distribution is judged to be true; log(·) is the log likelihood function.

[0064] The first loss function mentioned above consists of the following two parts:

[0065] (1)E x [log D img (x)], this part relates to D img The log likelihood of (x) can be obtained by performing an expectation operation on all possible x, thus yielding the image discriminator D. img The focus of determining the distribution of the log likelihood of a real ultrasound image x as a true probability distribution is the true distribution of the real ultrasound image x.

[0066] (2) This section represents the image discriminator D. img Determine the generation of ultrasound images The negative log-likelihood of the true probability distribution is the focus of generating ultrasound images. The degree of similarity between the actual distribution of the real ultrasound image x in Part (1) above.

[0067] It is understandable that the smaller the value of the first loss function mentioned above, the better the image discriminator D is. img The stronger the ability to distinguish between real and fake images, the more realistic the generated ultrasound images produced by the generator subnetwork will be.

[0068] The above scheme uses a first loss function that characterizes the ability of the image discriminator to distinguish between real and generated ultrasound images to train the image discriminator. This incentivizes the generator subnetwork to generate ultrasound images that are closer to the distribution of real ultrasound images. Using more realistic generated ultrasound images in conjunction with real ultrasound images to train the patch segmentation network helps improve the segmentation accuracy of the patch segmentation network.

[0069] As an optional implementation of the above-mentioned patch segmentation network training method, step S130 includes: training the structure discriminator with real patch labels and generated patch labels as input and minimizing the second loss function as the training objective; wherein, the second loss function is used to characterize the ability of the structure discriminator to distinguish between real patch labels and generated patch labels.

[0070] The calculation method for the second loss function mentioned above is as follows:

[0071]

[0072] Among them, E z (·) denotes the expectation operation on the probability distribution; D strct (z) represents the structure discriminator D. strct The actual patch is labeled z and judged as the true probability distribution; Represents the structure discriminator D strct Generate patch annotations The probability distribution is judged to be true; log(·) is the log likelihood function.

[0073] The second loss function mentioned above consists of the following two parts:

[0074] (1)E z [log D strct (z)], this part relates to D strct The expectation operation on the log likelihood of (z) can be obtained by weighted summation over all possible z, thus yielding the structure discriminator D. strct Determining the distribution of the log likelihood of the true probability distribution of the real patch label z as the overall distribution focuses on the true distribution of the real ultrasound label z.

[0075] (2) This section represents the structure discriminator D. strct Determine the generation of patch annotations The negative log-likelihood of the true probability distribution is the focus of generating patch annotations. The degree of similarity between the actual distribution of the actual patch label z in Part (1) above.

[0076] It is understandable that the smaller the value of the second loss function mentioned above, the better the structure discriminator D is. strct The weaker the ability to distinguish between genuine and fake annotations, the more realistic the generated patch annotations will be.

[0077] The above scheme uses a second loss function to train the structure discriminator, which represents the ability of the structure discriminator to distinguish between real patch annotations and generated patch annotations. This makes the generated patch annotations extracted by the patch segmentation network conform to the real distribution of real patch annotations, thereby increasing the diversity of generated ultrasound images generated by the generator subnetwork. Using generated ultrasound images with diversity in combination with real ultrasound images to train the patch segmentation network is beneficial to improving the patch segmentation accuracy of the patch segmentation network.

[0078] As an optional implementation of the above-mentioned patch segmentation network training method, step S130 includes: training the matching discriminator with real ultrasound images, real patch annotations, generated ultrasound images and generated patch annotations as inputs and minimizing the third loss function as the training objective.

[0079] The third loss function is used to characterize the ability of the matching discriminator to distinguish between the first matching relationship and the second matching relationship; the first matching relationship is used to characterize the matching relationship between the real ultrasound image and the real patch annotation; and the second matching relationship is used to characterize the matching relationship between the generated ultrasound image and the generated patch annotation.

[0080] The calculation method for the third loss function mentioned above is as follows:

[0081]

[0082] Among them, E ＜x,z＞ D represents the expectation operation on the probability distribution; pair (<x,z> ) represents the matching discriminator D pair Real<x,z> The probability distribution that is judged to be true; D represents the matching discriminator. pair Will generate The probability distribution that is judged as true; log(·) is the log likelihood function.

[0083] The third loss function mentioned above consists of the following two parts:

[0084] (1)E ＜x,z＞ [log D pair (<x,z> This part concerns D. pair (<x,z> The expected operation on the log likelihood of ) can be performed by considering all possible log likelihoods.<x,z> We obtain the matching discriminant D by performing a weighted summation. pair Judging the truth<x,z> The focus is on the overall distribution of the log-likelihood, which is the true probability distribution.<x,z> The true distribution of pairs;

[0085] (2) This section represents the matching discriminator D. pairJudgment generation For the negative log-likelihood that is the true probability distribution, the focus is on generating... Regarding the true content in Part (1) above<x,z> The degree of similarity between the true distributions of pairs.

[0086] Understandably, the smaller the value of the third loss function mentioned above, the better the matching discriminator D is. pair For the truth<x,z> Pairs and generation The weaker the ability to distinguish correct values, the less likely it is to be generated. The more realistic the matching relationship between them.

[0087] The above scheme introduces a matching discriminator, enabling the generative adversarial network to focus on the matching relationship between the generated ultrasound image and the generated patch label. This makes the generated ultrasound image generated by the generator sub-network more closely matched with the generated patch label, which is beneficial to improving the practicality and reliability of the generated ultrasound image. Using the generated ultrasound image generated by the generator sub-network in conjunction with the real ultrasound image to train the matching discriminator is beneficial to improving the patch segmentation accuracy of the patch segmentation network.

[0088] The training method for the generator subnetwork is described below:

[0089] As an optional implementation of the above-mentioned patch segmentation network training method, step S130 includes: training the generator sub-network with a real ultrasound image as input and minimizing the fourth loss function as the training objective; wherein the fourth loss function is used to characterize the difference between the real ultrasound image and the generated ultrasound image.

[0090] The goal of the generator subnetwork is to generate multi-domain diverse generated ultrasound images based on real ultrasound images x. Therefore, the generator subnetwork can be trained using Manhattan distance (i.e., L1 distance) as the generation loss.

[0091] The calculation method for the fourth loss function mentioned above is as follows:

[0092]

[0093] Where x represents the actual ultrasound image; This indicates the generation of ultrasound images.

[0094] In the implementation of the above scheme, a fourth loss function that characterizes the difference between real ultrasound images and generated ultrasound images is used to train the generator sub-network, so that the generated ultrasound images generated by the generator sub-network are close to the real ultrasound images. Using the generated ultrasound images generated by the generator sub-network in conjunction with real ultrasound images to train the patch segmentation network is beneficial to improving the patch segmentation accuracy of the patch segmentation network.

[0095] The training method for the patch segmentation network is described below:

[0096] like Figure 6 As shown, as an optional implementation of the above-mentioned patch segmentation network training method, step S130 includes: training the patch segmentation network with real ultrasound images, real patch annotations, generated ultrasound images and generated patch annotations as inputs and minimizing the fifth loss function as the training objective.

[0097] The fifth loss function is a loss function calculated by combining the first loss function of the image discriminator, the second loss function of the structure discriminator, the third loss function of the matching discriminator, the fourth loss function of the generator sub-network, and the difference between the output and input labels of the patch segmentation network.

[0098] Understandably, since convolutional networks are built on a limited receptive field, they may fail at the boundaries of ultrasound images, defective regions (such as ultrasound shadow regions), etc. Therefore, cross-entropy loss can be used to obtain shape supervision based on Dice coefficients, thereby solving the boundary defect problem.

[0099] The calculation method for the fifth loss function is as follows:

[0100]

[0101] in, and l geb These are the first loss function, the second loss function, the third loss function, and the fourth loss function, respectively; seg denoted as the cross-entropy loss between the output and input labels of the patch segmentation network; β, γ, δ, θ, and α are the weights of the first, second, third, fourth, and cross-entropy losses, respectively.

[0102] Additionally, it is understandable that the patch segmentation network could use ImageNet initial weights before training it.

[0103] The above scheme employs a fifth loss function, which integrates the first, second, third, and fourth loss functions, as well as the difference between the input and output labels of the patch segmentation network, to train the patch segmentation network. This allows the patch segmentation network and the generative adversarial network to be trained together. On the one hand, this makes the generated ultrasound images produced by the generative adversarial network more closely resemble real ultrasound images. On the other hand, it makes the generated patch labels extracted by the patch segmentation network from the generated ultrasound images more closely resemble real patch labels. This improves the training effect of the patch segmentation network and thus improves the patch segmentation accuracy of the patch segmentation network.

[0104] Understandably, when jointly training the patch segmentation network and the generative adversarial network, the training of the generator sub-network, image discriminator, structure discriminator, matching discriminator, and patch segmentation network in each round of joint training follows a certain order. This order can be explained by considering the data required for training each of these components:

[0105] (1) Generator subnetwork;

[0106] Training the generator subnetwork requires both real and generated ultrasound images. Therefore, the generator subnetwork can be trained after it has acquired the generated ultrasound images.

[0107] (2) Image discriminator D img ;

[0108] Image discriminator D img Training also requires both real and generated ultrasound images; therefore, the image discriminator D... img Training can be performed after the generator subnetwork acquires the generated ultrasound images.

[0109] (3) Structure discriminator D strct ;

[0110] Structure discriminator D strxt Training the structure discriminator D requires both real patch annotations and generated patch annotations. strct Training can be performed after the patch segmentation network obtains the generated patch annotations.

[0111] (4) Matching Discriminator D pair ;

[0112] Matching Discriminator D pair Training requires real ultrasound images, real patch annotations, generated ultrasound images, and generated patch annotations. Therefore, for the matching discriminator D... pair Training can be performed after the patch segmentation network obtains the generated patch annotations.

[0113] (5) Patch segmentation network;

[0114] Training a patch segmentation network requires real ultrasound images, real patch annotations, generated ultrasound images, and generated patch annotations. Therefore, training the patch segmentation network can be performed after the patch segmentation network has obtained the generated patch annotations.

[0115] It should be noted that the above image discriminator D img Both the generator subnetwork and the generator subnetwork can be processed after the generator subnetwork acquires and generates the ultrasound image; the order between these two processes is not important. The aforementioned structure discriminator D... strct Matching discriminator D pair Both the patch segmentation network and the patch segmentation network can be used after the patch segmentation network generates patch annotations, and the order of these three processes does not need to be set.

[0116] It is understandable that by training the generator sub-network, image discriminator, structure discriminator, matching discriminator, and patch segmentation network in the above training sequence, the generative adversarial network and the patch segmentation network can be trained together, which is beneficial for optimizing the generative adversarial network and the patch segmentation network at the same time.

[0117] Please see Figure 7 Based on the same inventive concept, this application also provides a patch segmentation method, which includes:

[0118] Step S210: Acquire ultrasound images of the carotid artery plaque to be segmented;

[0119] Step S220: Input the ultrasound image of the carotid plaque to be segmented into the pre-trained plaque segmentation network to obtain the carotid plaque segmentation result of the ultrasound image of the carotid plaque to be segmented;

[0120] The pre-trained patch segmentation network is obtained by training based on any of the above-mentioned patch segmentation network training methods.

[0121] Please see Figure 8 Based on the same inventive concept, embodiments of this application also provide a patch segmentation network training device 300, which includes:

[0122] The ultrasound image acquisition module 310 is used to input a real ultrasound image of a carotid artery plaque into a generator subnetwork in a generative adversarial network, and acquire the generated ultrasound image output by the generator subnetwork.

[0123] The generated patch annotation acquisition module 320 is used to input the generated ultrasound image into the patch segmentation network and acquire the generated patch annotations output by the patch segmentation network;

[0124] Training module 330 is used to jointly train the patch segmentation network and the generative adversarial network based on the real ultrasound image, the real patch annotation of the real ultrasound image, the generated ultrasound image and the generated patch annotation, so as to obtain the trained patch segmentation network.

[0125] The generative adversarial network includes a generator subnetwork, an image discriminator for identifying whether the input image is the real ultrasound image, a structure discriminator for identifying whether the input label is the real patch label, and a matching discriminator for determining whether the generated ultrasound image and the generated patch label match.

[0126] As an optional implementation of the above-mentioned patch segmentation network training device, the training module 330 includes:

[0127] The image discriminator training unit is used to train the image discriminator by taking the real ultrasound image and the generated ultrasound image as inputs and minimizing the first loss function as the training objective.

[0128] The first loss function is used to characterize the image discriminator's ability to distinguish between the real ultrasound image and the generated ultrasound image.

[0129] As an optional implementation of the above-mentioned patch segmentation network training device, the training module 330 includes:

[0130] The structure discriminator training unit is used to train the structure discriminator by taking the real patch annotations and the generated patch annotations as inputs and minimizing the second loss function as the training objective.

[0131] The second loss function is used to characterize the ability of the structure discriminator to distinguish between the real patch labels and the generated patch labels.

[0132] As an optional implementation of the above-mentioned patch segmentation network training device, the training module 330 includes:

[0133] The matching discriminator training unit is used to train the matching discriminator by taking the real ultrasound image, the real patch annotation, the generated ultrasound image and the generated patch annotation as inputs and minimizing the third loss function as the training objective.

[0134] The third loss function is used to characterize the ability of the matching discriminator to distinguish between the first matching relationship and the second matching relationship; the first matching relationship is used to characterize the matching relationship between the real ultrasound image and the real patch annotation; and the second matching relationship is used to characterize the matching relationship between the generated ultrasound image and the generated patch annotation.

[0135] As an optional implementation of the above-mentioned patch segmentation network training device, the training module 330 includes:

[0136] The generator subnetwork training unit is used to train the generator subnetwork by taking the real ultrasound image as input and minimizing the fourth loss function as the training objective.

[0137] The fourth loss function is used to characterize the difference between the real ultrasound image and the generated ultrasound image.

[0138] As an optional implementation of the above-mentioned patch segmentation network training device, the training module 330 includes:

[0139] The patch segmentation network training unit is used to train the patch segmentation network by taking the real ultrasound image, the real patch annotation, the generated ultrasound image and the generated patch annotation as inputs and minimizing the fifth loss function as the training objective.

[0140] The fifth loss function is a loss function calculated by combining the first loss function of the image discriminator, the second loss function of the structure discriminator, the third loss function of the matching discriminator, the fourth loss function of the generator sub-network, and the difference between the output label and the input label of the patch segmentation network.

[0141] Please see Figure 9 Based on the same inventive concept, embodiments of this application also provide a patch segmentation device 400, which includes:

[0142] Carotid plaque ultrasound image acquisition module 410 is used to acquire ultrasound images of carotid plaques to be segmented.

[0143] The plaque segmentation module 420 is used to input the ultrasound image of the carotid plaque to be segmented into a pre-trained plaque segmentation network to obtain the carotid plaque segmentation result of the ultrasound image of the carotid plaque to be segmented.

[0144] The pre-trained patch segmentation network is obtained by training based on any of the patch segmentation network training methods described above.

[0145] Figure 10 This is a schematic diagram of an electronic device provided in an embodiment of this application. (Refer to...) Figure 10 The electronic device 500 includes a processor 510, a memory 520, and a communication interface 530. These components are interconnected and communicate with each other via a communication bus 540 and / or other forms of connection mechanism (not shown).

[0146] The memory 520 includes one or more (only one is shown in the figure), which may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The processor 510 and other possible components may access the memory 520 to read and / or write data therein.

[0147] Processor 510 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 510 described above can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a special-purpose processor, including a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0148] Communication interface 530 includes one or more (only one is shown in the figure) that can be used to communicate directly or indirectly with other devices to exchange data. For example, communication interface 530 can be an Ethernet interface; it can be a mobile communication network interface, such as an interface for 3G, 4G, or 5G networks; or it can be other types of interfaces with data transmission and reception capabilities.

[0149] One or more computer program instructions may be stored in the memory 520. The processor 510 may read and run these computer program instructions to implement the patch segmentation network training method or patch segmentation method provided in the embodiments of this application, as well as other desired functions.

[0150] Understandable. Figure 10 The structure shown is for illustrative purposes only; the electronic device 500 may also include more than [other components]. Figure 10 The more or fewer components shown, or having the same Figure 10 The different configurations shown. Figure 10 The components shown can be implemented using hardware, software, or a combination thereof. For example, electronic device 500 can be a single server (or other device with computing power), a combination of multiple servers, a cluster of a large number of servers, etc., and can be either a physical device or a virtual device.

[0151] This application also provides a computer-readable storage medium storing computer program instructions. These instructions are read and executed by a computer's processor to perform the patch segmentation network training method or patch segmentation method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 10 The memory 520 in the electronic device 500.

[0152] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0153] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0155] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for training a patch segmentation network, characterized in that, The method includes: The real ultrasound image of the carotid artery plaque is input into the generator subnetwork in the generative adversarial network, and the generated ultrasound image output by the generator subnetwork is obtained. The generated ultrasound image is input into a patch segmentation network to obtain the generated patch annotations output by the patch segmentation network; Based on the real ultrasound image, the real patch annotation of the real ultrasound image, the generated ultrasound image and the generated patch annotation, the patch segmentation network and the generative adversarial network are jointly trained to obtain the trained patch segmentation network. The generative adversarial network includes a generator subnetwork, an image discriminator for identifying whether the input image is the real ultrasound image, a structure discriminator for identifying whether the input label is the real patch label, and a matching discriminator for determining whether the generated ultrasound image and the generated patch label match.

2. The patch segmentation network training method according to claim 1, characterized in that, The joint training of the patch segmentation network and the generative adversarial network includes: The image discriminator is trained using the real ultrasound image and the generated ultrasound image as input, with the goal of minimizing the first loss function. The first loss function is used to characterize the image discriminator's ability to distinguish between the real ultrasound image and the generated ultrasound image.

3. The patch segmentation network training method according to claim 1, characterized in that, The joint training of the patch segmentation network and the generative adversarial network includes: The structure discriminator is trained using the real patch annotations and the generated patch annotations as inputs and minimizing the second loss function as the training objective. The second loss function is used to characterize the ability of the structure discriminator to distinguish between the real patch labels and the generated patch labels.

4. The patch segmentation network training method according to claim 1, characterized in that, The joint training of the patch segmentation network and the generative adversarial network includes: The matching discriminator is trained using the real ultrasound image, the real patch annotation, the generated ultrasound image, and the generated patch annotation as inputs, and minimizing the third loss function as the training objective. The third loss function is used to characterize the ability of the matching discriminator to distinguish between the first matching relationship and the second matching relationship; the first matching relationship is used to characterize the matching relationship between the real ultrasound image and the real patch annotation; and the second matching relationship is used to characterize the matching relationship between the generated ultrasound image and the generated patch annotation.

5. The patch segmentation network training method according to claim 1, characterized in that, The joint training of the patch segmentation network and the generative adversarial network includes: The generator subnetwork is trained using the real ultrasound image as input and minimizing the fourth loss function as the training objective. The fourth loss function is used to characterize the difference between the real ultrasound image and the generated ultrasound image.

6. The patch segmentation network training method according to claim 1, characterized in that, The joint training of the patch segmentation network and the generative adversarial network includes: The patch segmentation network is trained using the real ultrasound image, the real patch annotation, the generated ultrasound image, and the generated patch annotation as inputs, and minimizing the fifth loss function as the training objective. The fifth loss function is a loss function calculated by combining the first loss function of the image discriminator, the second loss function of the structure discriminator, the third loss function of the matching discriminator, the fourth loss function of the generator sub-network, and the difference between the output label and the input label of the patch segmentation network.

7. A patch segmentation method, characterized in that, The method includes: Acquire ultrasound images of the carotid artery plaque to be segmented; The ultrasound image of the carotid plaque to be segmented is input into a pre-trained plaque segmentation network to obtain the carotid plaque segmentation result of the ultrasound image of the carotid plaque to be segmented. The pre-trained patch segmentation network is obtained by training based on the patch segmentation network training method as described in any one of claims 1 to 6.

8. A patch segmentation network training device, characterized in that, The device includes: An ultrasound image acquisition module is used to input a real ultrasound image of a carotid artery plaque into a generator subnetwork in a generative adversarial network, and acquire the generated ultrasound image output by the generator subnetwork. A patch annotation acquisition module is used to input the generated ultrasound image into a patch segmentation network and obtain the generated patch annotations output by the patch segmentation network; The training module is used to jointly train the patch segmentation network and the generative adversarial network based on the real ultrasound image, the real patch annotation of the real ultrasound image, the generated ultrasound image and the generated patch annotation, so as to obtain the trained patch segmentation network. The generative adversarial network includes a generator subnetwork, an image discriminator for identifying whether the input image is the real ultrasound image, a structure discriminator for identifying whether the input label is the real patch label, and a matching discriminator for determining whether the generated ultrasound image and the generated patch label match.

9. An electronic device, characterized in that, include: A processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1 to 6 or claim 7 by calling the program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method as claimed in any one of claims 1 to 6 or 7.

Images