Contrast agent CT liver vascular enhancement generation method based on adversarial learning and related device

Abstract

The present application discloses a method and related device for generating contrast agent CT liver vascular enhancement based on adversarial learning. The method includes: obtaining a medical original CT image and a corresponding medical vascular contrast agent CTA image as a sample pair, and dividing them into a training data set and a test data set; performing enhancement preprocessing on the training set and the test set data to obtain an enhanced preprocessed CT image and an enhanced preprocessed CTA image; constructing a deep learning network model based on an adversarial learning framework; constructing a composite loss function for training the model; training the training data set after enhancement preprocessing to generate a deep learning network model training based on the adversarial learning framework; inputting the original CT image of the test data set into the trained model to generate the corresponding CTA image.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image generation, and in particular to a contrast agent CT liver blood vessel enhancement generation method based on adversarial learning and a related device. Background Art

[0002] Current clinical liver CT vascular imaging is highly dependent on the injection of iodine-containing contrast agents, but there are problems such as allergic risks, nephrotoxicity and high costs. Although traditional enhancement algorithms (such as histogram equalization and multi-scale filtering) can locally improve contrast, it is difficult to accurately distinguish between blood vessels and surrounding tissue textures, which can easily lead to artifacts or loss of details, especially for small vascular branches and low-contrast areas. The enhancement effect is limited. The deep model based on supervised learning needs to rely on a large amount of accurately labeled vascular data for training, but the high cost of medical image annotation and poor cross-device generalization restrict its clinical application. Most existing devices are based on a fixed algorithm framework, which is difficult to adapt to the imaging parameter differences of different CT devices, and the computational efficiency is low, making it difficult to meet the needs of real-time diagnosis. In response to the above pain points, it is urgent to develop non-invasive, low-cost intelligent enhancement technology to achieve high-quality generation of vascular structures of non-contrast CT through adversarial learning, while being compatible with multi-source device data to improve the visualization accuracy and diagnostic efficiency of liver vessels. Summary of the invention

[0003] The present invention relates to a method for generating contrast agent CT liver blood vessel enhancement based on adversarial learning and related devices, which include but are not limited to three-dimensional reconstruction equipment, electronic devices, computer-readable storage media and computer program products.

[0004] A first aspect provides a method for generating contrast agent CT liver blood vessel enhancement based on adversarial learning, the method comprising: Obtaining original medical CT images and corresponding medical vascular contrast agent CTA images as sample pairs, and dividing them into a training data set and a test data set; Perform enhancement preprocessing on the training set and the test set data to obtain enhanced preprocessed CT images and enhanced preprocessed CTA images; Constructing a deep learning network model based on an adversarial learning framework, wherein the deep learning network model includes a dynamic deformation compensation model, a student mapping model, and a teacher authentication model; The dynamic deformation compensation model is a residual connection encoder and residual connection decoder structure, the residual connection encoder includes a plurality of large-size shallow variable convolution layer blocks and a small-size deep variable convolution layer block, the residual connection decoder and the residual connection encoder blocks are mirror-symmetric and share parameters, and the number of the residual connection encoder blocks is the same as the number of the residual connection decoder blocks; The teacher authentication model is a convolutional neural network, which includes an asymmetric dilated convolution group to form a cross-scale feature pyramid, and each layer output is connected to a dual-branch authentication head; The deep learning network model is trained using the training data set, and then the compound L1 loss function, adversarial loss function, and zero-center penalty loss function of the deep learning network model with a ratio of 3:2:1 are used to perform regularization penalty, and the model parameters of the deep learning network model are adjusted by annealing dynamic cosine variable learning rate scheduling reverse gradient propagation, and then the trained model is verified and tested using the test data set; The trained deep learning network model is used to generate a contrast agent CT liver vascular image for the original CT image to be processed, and a medical vascular contrast agent CTA image corresponding to the original CT image is obtained.

[0005] In combination with any embodiment of the present application, the training set and test set data are subjected to enhanced preprocessing, and the enhanced preprocessing includes -270 to 330 HU value visual range capture, gama enhancement and contrast-limited adaptive histogram equalization.

[0006] In combination with any embodiment of the present application, the residual connection encoder block and the residual connection decoder block of the dynamic deformation compensation model, the residual connection encoder block includes: input layer, downsampling convolution layer, normalization, self-attention mechanism, feature fusion, the residual connection decoder block includes: upsampling transposed convolution layer, skip connection, self-attention mechanism, and a dynamic weight distribution layer is set between the output end of the self-attention mechanism and the feature fusion module.

[0007] In combination with any embodiment of the present application, the 2D encode block and 2D decode block of the student mapping model, the 2D encode block includes: an input layer, a downsampling convolution layer, a downsampling deformable convolution, the 2D decode block includes: a convolution layer, a variable convolution layer, a two-dimensional deconvolution, and the offset of the deformable convolution is constrained in the range of [-Δ, +Δ], where Δ=3×σ, and σ is the standard deviation of the blood vessel diameter.

[0008] In combination with any implementation manner of the present application, the convolutional neural network of the teacher authentication model includes: a convolutional layer, a spatial attention mechanism, and a maximum pooling layer.

[0009] In combination with any embodiment of the present application, the composite loss function includes an L1 loss function, an adversarial loss function and a zero-center penalty loss function, and the composite loss function is an L1 loss function, an adversarial loss function, a zero-center penalty loss function and a weight ratio of 3:2:1; The L1 loss function is as follows: , Where: N is the number of samples, is the i-th original CT image, is the medical angiography CTA image generated from the i-th original CT image, is the absolute error of the i-th generated medical angiography contrast agent CTA image; The adversarial loss function is as follows: , , in, is the judgment result of the teacher authentication model on the original CT image x, It is a medical vascular contrast agent CTA image generated from the original CT image corresponding to the student mapping model; The zero-center penalty loss function is as follows: , , in , is the regularization coefficient related to the HU value distribution characteristics of the CT image, is the gradient of the teacher authentication model to the original CT data x, is the data of medical vascular contrast agent CTA images generated by the student mapping model, is the gradient of the teacher authentication model to the data of the generated medical vascular contrast agent CTA image.

[0010] In a second aspect, an electronic device is provided, comprising: a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is used to store computer program code, wherein the program code includes computer instructions. When the processor executes these instructions, the electronic device will execute the method described in the second aspect and any embodiment thereof.

[0011] In a third aspect, a computer-readable storage medium is provided, wherein a computer program is stored, wherein the program includes program instructions. When these instructions are executed by a processor, the processor will execute the method described in the second aspect and any embodiment thereof.

[0012] In a fourth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions. When the computer program or instructions are run on a computer, the computer will execute according to the method described in the second aspect and any of its embodiments.

[0013] It should be understood that the above general description and the following detailed description are only used as examples and explanations and do not impose any limitations on the present application.

[0014] In this application, compared with the prior art, the present invention adopts deformable convolution and anatomical constrained attention collaborative mechanism to extract vascular features, realizes the maintenance of vascular topological structure through multi-scale feature pyramid fusion module, constructs a composite loss function including zero center gradient penalty for adversarial training optimization, and finally generates high-fidelity vascular enhanced images, which are specifically manifested as follows: in the feature extraction stage, deformable convolution kernels are used to replace traditional fixed convolution kernels to make the perception field adaptive to vascular morphological changes and structure preservation mechanism, introduce liver segment anatomical spatial attention and training optimization strategy into the student mapping model, innovatively propose zero center gradient penalty term, and combine cosine annealing learning rate scheduling to improve the convergence speed of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0016] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0017] Figure 1 A schematic diagram of a flow chart of a method for generating contrast agent CT liver blood vessel enhancement based on adversarial learning provided in an embodiment of the present application; Figure 2 A schematic diagram of a process for generating CTA images based on an adversarial learning deep neural network provided in an embodiment of the present application; Figure 3 A schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to allow professionals in the technical field to more fully understand the technical solution of the present application, the technical solution of the present application will be explained in detail and clearly through the accompanying drawings. It should be particularly pointed out that the described embodiments are only some examples of the present application and do not represent all. Based on these embodiments, those skilled in the art can directly derive all other possible implementation schemes without creative thinking, which are also included in the protection scope of the present application.

[0019] In the specification, claims and related drawings of this application, the terms "first", "second" and the like are used only to distinguish different elements and do not imply any particular order. At the same time, the use of "including" and "having" and their variations indicates non-exclusive inclusion. This means that if a process, method, system, product or device includes a series of steps or components, it means that the process, method, system, product or device is not limited to the listed steps or components, and may also include other steps or components not listed, or other steps or units inherent thereto.

[0020] The "embodiment" mentioned in this article refers to any instance in which a particular feature, structure or characteristic is combined, and these instances may belong to at least one embodiment of the present application. The "embodiment" mentioned in the text does not necessarily refer to the same specific case, nor does it mean that they are independent or exclusive alternatives. It should be understood by those skilled in the art that the embodiments described in the text can be used together with other embodiments. It should be clear that in the present application, "at least one" includes one or more instances, "multiple" means two or more instances, and "at least two" refers to two or more instances.

[0021] It should be understood that the method embodiment of the present application can also be implemented by a processor executing computer program code. The embodiment of the present application is described below in conjunction with the drawings in the embodiment of the present application.

[0022] See also Figure 1 , Figure 1 A flowchart of a method for generating contrast agent CT liver vascular enhancement based on adversarial learning provided in an embodiment of the present application.

[0023] 101. Obtain a medical original CT image and a corresponding medical vascular contrast agent CTA image as a sample pair, and divide them into a training data set and a test data set.

[0024] In the embodiment of the present application, the original medical CT image is an image obtained by performing a CT scan on the target object. The medical CT image includes the target object, wherein the image area of ​​the target object includes a liver blood vessel target object.

[0025] 102. Perform enhancement preprocessing on the training set and the test set data to obtain an enhanced preprocessed CT image and an enhanced preprocessed CTA image.

[0026] In the embodiment of the present application, the training set and the test set data are subjected to enhancement preprocessing, and the enhancement preprocessing includes -270 to 330 HU value visual range capture, gama enhancement and contrast-limited adaptive histogram equalization.

[0027] 103. Build a deep learning network model based on adversarial learning architecture.

[0028] In an embodiment of the present application, the deep learning network model includes a dynamic deformation compensation model, a student mapping model, and a teacher authentication model, wherein the residual connection encoding block and the residual connection decoding block of the dynamic deformation compensation model include: an input layer, a downsampling convolution layer, normalization, a self-attention mechanism, and feature fusion, and the residual connection decoding block includes: an upsampling transposed convolution layer, a jump connection, and a self-attention mechanism. A dynamic weight distribution layer is set between the output end of the self-attention mechanism and the feature fusion module, and the 2D encode block and the 2D decode block of the student mapping model include: an input layer, a downsampling convolution layer, and a downsampling deformable convolution. The 2D decode block includes: a convolution layer, a variable convolution layer, and a two-dimensional deconvolution. The offset of the deformable convolution is constrained to be in the range of [-Δ, +Δ], where Δ=3×σ, and σ is the standard deviation of the blood vessel diameter. The convolutional neural network of the teacher authentication model includes: a convolution layer, a spatial attention mechanism, and a maximum pooling layer.

[0029] 104. Construct a composite loss function for training the model.

[0030] The loss function in the embodiment of the present application is a composite loss function including an L1 loss function, an adversarial loss function and a zero-center penalty loss function, wherein the composite loss function is an L1 loss function, an adversarial loss function, a zero-center penalty loss function and a weight ratio of 3:2:1; The L1 loss function is as follows: , Where: N is the number of samples, is the i-th original CT image, is the medical angiography CTA image generated from the i-th original CT image, is the absolute error of the i-th generated medical angiography contrast agent CTA image; The adversarial loss function is as follows: , , in, is the judgment result of the teacher authentication model on the original CT image x, It is the medical vascular contrast agent CTA image generated from the original CT image corresponding to the student mapping model; The zero-center penalty loss function is as follows: , , in , is the regularization coefficient related to the HU value distribution characteristics of the CT image, is the gradient of the teacher authentication model to the original CT data x, is the data of medical vascular contrast agent CTA images generated by the student mapping model, is the gradient of the teacher authentication model to the data of the generated medical vascular contrast agent CTA image.

[0031] 105. Enhance the preprocessed training data set to generate deep learning network model training using adversarial learning architecture.

[0032] In an embodiment of the present application, a deep learning network model is generated according to the adversarial learning framework, two deep learning network models are trained using a training set, regularization penalty is performed using a composite loss function of the deep learning network model, the deep learning network model parameters are adjusted using a cosine annealing learning rate, and the trained deep learning network model is verified using a test data set.

[0033] See also Figure 2 , Figure 2 A schematic diagram of a process for generating CTA images based on an adversarial learning deep neural network provided in an embodiment of the present application. Figure 2 As shown in the flowchart, the original CT image is subjected to the student mapping model to generate an image, and then the dynamic deformation compensation model is used to correct the respiratory deformation of the generated image. Finally, the teacher authentication model is used to determine whether the image is a CTA image corresponding to the original CT image, and a synthesized CTA image is obtained.

[0034] 106. Input the original CT image of the test data set into the trained model to generate the corresponding CTA image.

[0035] In an embodiment of the present application, the original CT image of the test data set is generated according to the trained deep neural network to obtain a CTA image corresponding to the original CT image.

[0036] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0037] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0038] Figure 3A schematic diagram of the hardware architecture of an electronic device described in an embodiment of the present application is shown. The electronic device 2 is mainly composed of a processor 21 and a memory 22. In addition, the device may also include an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are interconnected through connection components, and these connection components can be various interfaces, data lines or communication buses, etc., which are not specifically specified in the embodiments of the present application. It should be clear that in multiple embodiments of the present application, the so-called connection refers to the mutual connection achieved by a specific method, which can be a direct connection or an indirect connection through other devices, such as connecting through various interfaces, data lines, communication buses, etc.

[0039] The processor 21 may be one or more graphics processing units (GPUs). If the processor 21 is a GPU, the GPU may be single-core or multi-core. As an option, the processor 21 may also be a processor group consisting of multiple GPUs, which are interconnected via one or more buses. In addition, the processor may also be other types of processors, which are not specifically limited in the embodiments of the present application.

[0040] The memory 22 is designed to store computer program instructions and various program codes required to execute the solution of the present application. As an option, the memory may include but is not limited to a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a compact disc read-only memory (CD-ROM), which are used to store relevant instructions and data.

[0041] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0042] It should be appreciated that in the embodiment of the present application, the memory 22 can store not only related instructions but also related data. The embodiment of the present application does not specifically specify the specific data content stored in the memory.

[0043] You should understand that Figure 3 Only a simplified design of an electronic device is shown. In actual use, the electronic device may also include other necessary components, such as different numbers of input / output devices, processors, memories, etc. All electronic devices that can implement the embodiments of the present application belong to the protection scope of the present application.

[0044] Those skilled in the art should recognize that, according to the components and algorithm steps of each example described in the embodiments disclosed herein, these functions can be implemented by electronic hardware or by combining computer software and electronic hardware. Whether these functions are performed by hardware or software will be determined based on the specific application requirements and design limitations of the technical solution. Technicians can adopt different implementation methods according to the needs of each specific application, but such implementation methods should not be considered to exceed the scope of protection of this application.

[0045] Professionals should understand that, for the convenience of description and simplification, the specific operation process of the above-mentioned systems, devices and components can refer to the corresponding steps in the previous method embodiments, and will not be repeated here. At the same time, professionals should also understand that each embodiment in this application has its own focus. For the convenience of description and simplification, the same or similar content may not be repeated in different embodiments. Therefore, if a part is not mentioned or described in detail in a certain embodiment, it can refer to the relevant description of other embodiments.

[0046] In several embodiments provided in the present application, it should be recognized that the disclosed systems, devices and methods can also be implemented in other ways. For example, the described device embodiments are only exemplary, wherein the division of the units is only a division of logical functions, and there may be different division methods in actual implementation. For example, multiple units or components may be merged or integrated into another system, or certain features may be omitted, or certain steps may not be performed. In addition, the connections between each other shown or discussed, whether direct or indirect, whether coupled or communicatively connected, may be implemented in electrical, mechanical or other forms through interfaces, devices or units.

[0047] The units described as independent components may or may not actually be physically separated; the parts presented as units may or may not be physical entities, that is, they may be concentrated in one location or dispersed across multiple network nodes. Depending on actual needs, some or all of these units may be selected to achieve the objectives of this embodiment.

[0048] In addition, in various embodiments of the present application, each functional unit can be integrated into one processing unit, or can exist physically independently, or two or more units can be merged into one unit. In the aforementioned embodiments, the relevant functions can be fully or partially implemented by software, hardware, firmware, or any combination thereof. If software implementation is selected, it can be carried out in whole or in part in the form of a computer program product. This computer program product contains one or more computer instructions. When these instructions are loaded and executed on a computer, a process or function that fully or partially complies with the description of the embodiment of the present application will be generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. These computer instructions can be stored in a computer-readable storage medium or transmitted through these media. Computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, DSL) or wireless (e.g., infrared, wireless, microwave, etc.). Computer-readable storage media can be any computer-accessible available medium, or a data storage facility such as a server or data center that integrates one or more available media. These available media may include magnetic media (such as floppy disks, hard disks, tapes), optical media (such as DVDs), semiconductor media (such as SSDs), etc. Those skilled in the art should understand that all or part of the processes for implementing the above-mentioned embodiments can be completed by computer program instructions related to hardware, and these programs can be stored in computer-readable storage media. When these programs are executed, they will contain the processes of the above-mentioned method embodiments. The above-mentioned storage media include, but are not limited to, various media that can store program codes, such as read-only memory (ROM), random access memory (RAM), disk or optical disk.

Claims

1. A method for generating contrast agent CT liver vascular enhancement based on adversarial learning, characterized in that: The method comprises: Obtaining original medical CT images and corresponding medical vascular contrast agent CTA images as sample pairs, and dividing them into a training data set and a test data set; Perform enhancement preprocessing on the training set and the test set data to obtain enhanced preprocessed CT images and enhanced preprocessed CTA images; Constructing a deep learning network model based on an adversarial learning framework, wherein the deep learning network model includes a dynamic deformation compensation model, a student mapping model, and a teacher authentication model; The dynamic deformation compensation model is a residual connection encoder and residual connection decoder structure, the residual connection encoder includes a plurality of large-size shallow variable convolution layer blocks and a small-size deep variable convolution layer block, the residual connection decoder and the residual connection encoder blocks are mirror-symmetric and share parameters, and the number of the residual connection encoder blocks is the same as the number of the residual connection decoder blocks; The teacher authentication model is a convolutional neural network, which includes an asymmetric dilated convolution group to form a cross-scale feature pyramid, and each layer output is connected to a dual-branch authentication head; The deep learning network model is trained using the training data set, and then the compound L1 loss function, adversarial loss function, and zero-center penalty loss function of the deep learning network model with a ratio of 3:2:1 are used to perform regularization penalty, and the model parameters of the deep learning network model are adjusted by annealing dynamic cosine variable learning rate scheduling reverse gradient propagation, and then the trained model is verified and tested using the test data set; The trained deep learning network model is used to generate a contrast agent CT liver vascular image for the original CT image to be processed, and a medical vascular contrast agent CTA image corresponding to the original CT image is obtained.

2. The method according to claim 1, characterized in that The training set and test set data are subjected to enhanced preprocessing, wherein the enhanced preprocessing includes -270 to 330 HU value visual range interception, gama enhancement and contrast-limited adaptive histogram equalization.

3. The method according to claim 1, characterized in that The dynamic deformation compensation model comprises a residual connection encoding block and a residual connection decoding block. The residual connection encoding block comprises: an input layer, a downsampling convolution layer, normalization, a self-attention mechanism, and a feature fusion. The residual connection decoding block comprises: an upsampling transposed convolution layer, a skip connection, and a self-attention mechanism. A dynamic weight distribution layer is set between the output end of the self-attention mechanism and the feature fusion module.

4. The method according to claim 1, characterized in that: The 2D encode block and 2D decode block of the student mapping model include: input layer, downsampling convolution layer, downsampling deformable convolution, and 2D decode block includes: convolution layer, variable convolution layer, two-dimensional deconvolution, and the offset of the deformable convolution is constrained in the range of [-Δ, +Δ], where Δ=3×σ, and σ is the standard deviation of the blood vessel diameter.

5. The method according to claim 1, characterized in that The convolutional neural network of the teacher authentication model includes: a convolutional layer, a spatial attention mechanism, and a maximum pooling layer.

6. The method according to claim 1, characterized in that The composite loss function includes an L1 loss function, an adversarial loss function, and a zero-center penalty loss function, wherein the composite loss function is a sum of the L1 loss function, the adversarial loss function, and the zero-center penalty loss function with a weight ratio of 3:2:1; The L1 loss function is as follows: ， Where: N is the number of samples, is the i-th original CT image, is the medical angiography CTA image generated from the i-th original CT image, is the absolute error of the i-th generated medical angiography contrast agent CTA image; The adversarial loss function is as follows: ， ， in, is the judgment result of the teacher authentication model on the original CT image x, It is a medical vascular contrast agent CTA image generated from the original CT image corresponding to the student mapping model; The zero-center penalty loss function is as follows: ， ， in , is the regularization coefficient related to the HU value distribution characteristics of the CT image, is the gradient of the teacher authentication model to the original CT data x, is the data of medical vascular contrast agent CTA images generated by the student mapping model, is the gradient of the teacher authentication model to the data of the generated medical vascular contrast agent CTA image.

7. An electronic device, characterized in that: include: A processor and a storage unit, the storage unit is used to store computer program code, the code includes computer instructions, when the processor executes these instructions, the electronic device executes the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images