GAN neural network creation method, CXR chest image processing and device

Abstract

The present invention discloses a method for creating a GAN neural network, a CXR chest image processing device, and the like. The method comprises the following steps: obtaining a plurality of CXR chest images, obtaining a plurality of DRR chest images before bone removal, and each DRR chest image I d A chest image after bone elimination, a corresponding rib projection in a DRR domain, and a corresponding rib residual map in the DRR domain are correspondingly generated; a GAN neural network is created; and the GAN neural network is trained based on a plurality of CXR chest images, a plurality of DRR chest images, a plurality of chest images, a plurality of rib projections, and a plurality of rib residual maps. The GAN neural network can eliminate ribs from the CXR image.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a method for creating a GAN neural network, and a device for processing CXR chest images. Background Art

[0002] Chest X-ray (CXR) images are widely used in the diagnosis of lung diseases, such as pneumonia, tuberculosis and lung cancer. Among them, the number of deaths from lung cancer ranks first among all cancer deaths. If lung diseases can be detected early, the treatment effect of the disease can be improved and the mortality rate of lung cancer can be reduced. Although chest X-rays are not as good as CT (Computed Tomography) in terms of detection performance, they are still the most important detection method due to their high economy and low radiation dose.

[0003] X-ray chest imaging is based on the fact that different tissues in the lungs differ in density and thickness, resulting in different degrees of absorption during the X-ray penetration process, thus forming images of different intensities on the ray receiving surface. Since the rays are projected onto a two-dimensional plane, different tissues in the direction of the rays will overlap and appear on the image, making it difficult to observe and identify the local lesion area. Overlapping is an inherent problem of chest X-rays and the main reason for many errors in the diagnosis of lung diseases. Studies have shown that the misdiagnosis rate of doctors in chest X-rays is as high as 30%, and 82%-95% of the missed diagnoses are due to the superposition or obstruction of the soft tissue of the lungs by the ribs. The conventional method for obtaining images containing only soft tissue of the lungs is the dual energy difference technique, which uses special equipment to perform two exposure perspectives of different energies on the subject to separate the ribs and soft tissues, and requires a larger radiation dose, which is limited to large hospitals and other institutions.

[0004] Therefore, how to easily remove ribs from CXR images has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the main purpose of the present invention is to provide a method for creating a GAN neural network, and a device for processing CXR chest images.

[0006] To achieve the above object, the technical solution of the present invention is implemented as follows: a method for creating a GAN neural network for CXR chest image processing, comprising the following steps: obtaining a plurality of CXR chest images I x , and obtain several DRR chest images before bone removal I d , and each DRR chest image I d Corresponding chest image Q after bone removal d , the corresponding rib projection B of the DRR domain dand the corresponding rib residual map R in the DRR domain d ; Create a GAN neural network, the generator G of the GAN neural network includes: a first encoder E s , the second encoder E c , the third encoder E R , the first decoder G L , the second decoder G Q , the third decoder G R and the fourth decoder G B ; Among them, the first encoder E s The input chest image I to be processed is processed, and the structural features F after the ribs are removed are obtained. s , the second encoder E c Process the input chest image I to be processed and obtain the contrast feature F C , the third encoder E R Process the input chest image I to be processed and obtain the rib features F R ; First decoder G L For structural features F S Process and obtain the lung segmentation L ′ ; Second decoder G Q For structural features F S and contrast feature F C Process and obtain the initial bone elimination graph Q ′ ; The third decoder G R For the contrast feature F C and rib feature F R Process and obtain the rib residual map R ′ ; The fourth decoder G B Rib feature F R Processing and obtaining rib projection diagram B ′ ; Based on several CXR chest images I x 、Several DRR chest images I d , several chest images Q d 、Several rib projections B d and several rib residual graphs R d , train the GAN neural network.

[0007] As an improvement of the embodiment of the present invention, the "obtaining a plurality of DRR chest images before bone removal" d , and each DRR chest image I d Corresponding chest image Q after bone removal d , the corresponding rib projection B of the DRR domain d and the corresponding rib residual map R in the DRR domain dSpecifically, it includes: obtaining a plurality of CT chest images, and performing the following operations on each CT chest image: performing forward projection processing on the CT chest image, obtaining a DRR chest image before bone removal. d ; Obtain the gray value of the soft tissue in the CT chest image, segment the rib image from the CT chest image based on the pre-trained segmentation neural network, replace the gray value of the rib image with the gray value, and then perform forward projection processing on the rib image to obtain a chest image Q with bone removed but spine retained d ; Projecting a rib projection B from the rib image d ; R d =I d -Q d .

[0008] As an improvement of the embodiment of the present invention, the plurality of CXR chest images I x and several DRR chest images I d It is unpaired.

[0009] As an improvement of the embodiment of the present invention, during the training process, a plurality of DRR chest images I d Input into the generator G and get each DRR chest image I d The corresponding chest image Q′ after bone removal d , the corresponding rib projection B′ of the DRR domain d and the corresponding rib residual map R′ in the DRR domain d ; Several CXR chest images I x Input into the generator G and get each CXR chest image I x The corresponding chest image Q′ after bone removal x ,I x The corresponding rib projection B′ x and I x The corresponding rib residual graph R′ x ; The loss function of the GAN neural network is: Where: adv , f , i , λ inter and λ Δ All are weight values;

[0010] ‖·‖1 is the L1 loss function;

[0011]

[0012] I′ d→x represents the image transferred from the DRR domain to the CXR domain, I′ x→d Represents an image transferred from the CXR domain to the DRR domain, and the discriminator D x It is used to evaluate the domain difference between the image converted to the CXR domain and the CXR image. The converted image retains the structure of the original image domain DRR while having the image contrast of the target domain CXR. d It is used to evaluate the domain difference between the image converted to the DRR domain and the DRR domain image. The converted image retains the structure of the original image domain CXR while having the image contrast of the target domain DRR. The discriminator D B Used to determine whether the input rib projection image is from the CXR domain or the DRR domain;

[0013]

[0014] I′ x =G Q (E C (I x ),E S (I x ))+G R (E C (I x ),E B (I x )),

[0015] I″ x =G Q (E c (I′ d→x ),E s (I′ x→d ))+G R (E c (I′ c→x ),E B (I′ x→d )),

[0016] I′ x For input I x A forward prediction of d For input I d A forward prediction of x and I″ d stands for Cycle Forecast;

[0017]

[0018] is the gradient operator, is the discriminator applied to the gradient map of the bone-removing image, Q′d→x is the bone loss image converted from DRR to CXR, R′ x is the rib residual map predicted from CXR, L′ x and L′ d are the lung segmentations predicted from CXR and DRR images, respectively;

[0019] ,

[0020] M′ x For I x The corresponding rib segmentation is performed by x Apply bilateral filter and threshold segmentation to obtain, ∪ represents the intersection of two input binary images;

[0021] ,

[0022] Δ is the Laplace operator.

[0023] As an improvement of the embodiment of the present invention, λ adv =1,λ f =1,λ i =10, λ inter =500,λ Δ =1.

[0024] The embodiment of the present invention also provides a device for creating a GAN neural network for CXR chest image processing, comprising the following modules: a first data acquisition module, for acquiring a plurality of CXR chest images I x , and obtain several DRR chest images before bone removal I d , and each DRR chest image I d Corresponding chest image Q after bone removal d , the corresponding rib projection B of the DRR domain d and the corresponding rib residual map R in the DRR domain d ; A GAN neural network creation module, used to create a GAN neural network, wherein the generator G of the GAN neural network includes: a first encoder E s , the second encoder E c , the third encoder E R , the first decoder G L , the second decoder G Q , the third decoder G R and the fourth decoder G B ; Among them, the first encoder E s The input chest image I to be processed is processed, and the structural features F after the ribs are removed are obtained. S , the second encoder E cProcess the input chest image I to be processed and obtain the contrast feature F c , the third encoder E R Process the input chest image I to be processed and obtain the rib features F R ; First decoder G L For structural features F R Process and obtain the lung segmentation L ′ ; Second decoder G Q For structural features F S and contrast feature F C Process and obtain the initial bone elimination graph Q ′ ; The third decoder G R For the contrast feature F C and rib feature F R Process and obtain the rib residual map R ′ ; The fourth decoder G B Rib feature F R Processing and obtaining rib projection diagram B ′ ; Training module for training based on several CXR chest images I x 、Several DRR chest images I d , several chest images Q d 、Several rib projections B d and several rib residual graphs R d , train the GAN neural network.

[0025] As an improvement of the embodiment of the present invention, the first data acquisition module is further used to: acquire a plurality of CT chest images, and perform the following operations on each CT chest image: perform forward projection processing on the CT chest image to obtain a DRR chest image before bone elimination. d ; Obtain the gray value of the soft tissue in the CT chest image, segment the rib image from the CT chest image based on the pre-trained segmentation neural network, replace the gray value of the rib image with the gray value, and then perform forward projection processing on the rib image to obtain a chest image Q with bone removed but spine retained d ; Projecting a rib projection B from the rib image d ; R d =I d -Q d .

[0026] As an improvement of the embodiment of the present invention, the plurality of CXR chest images I x and several DRR chest images I d It is unpaired.

[0027] The embodiment of the present invention also provides a method for processing a CXR chest image, comprising the following steps: obtaining a CXR chest image I input Execute the above creation method to create a GAN neural network, control the generator G in the GAN neural network to the CXR chest image I input Process and obtain the rib residual graph R' input , then the CXR chest image I input The corresponding bone elimination diagram = I input -R' input .

[0028] The embodiment of the present invention further provides a device for processing a CXR chest image, comprising the following modules: a second data acquisition module, configured to acquire a CXR chest image I input ; An image processing module, used to execute the above creation method to create a GAN neural network, control the generator G in the GAN neural network to the CXR chest image I input Process and obtain the rib residual graph R' input , then the CXR chest image I input The corresponding bone elimination diagram = I input -R' input .

[0029] The GAN neural network creation method, CXR chest image processing device and apparatus provided in the embodiments of the present invention have the following advantages: The embodiments of the present invention disclose a GAN neural network creation method, CXR chest image processing device and apparatus, the creation method comprises the following steps: obtaining a plurality of CXR chest images, obtaining a plurality of DRR chest images before bone removal, and each DRR chest image I d A chest image after bone elimination, a corresponding rib projection in a DRR domain, and a corresponding rib residual map in the DRR domain are correspondingly generated; a GAN neural network is created; and the GAN neural network is trained based on a plurality of CXR chest images, a plurality of DRR chest images, a plurality of chest images, a plurality of rib projections, and a plurality of rib residual maps. The GAN neural network can eliminate ribs from the CXR image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a process for creating a GAN neural network for CXR chest image processing provided in this embodiment;

[0031] Figure 2 , Figure 3 and Figure 4 A schematic diagram of the structure of the GAN neural network provided in this embodiment;

[0032] Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Fig.11 This is the test result diagram of the GAN neural network. DETAILED DESCRIPTION

[0033] The present invention will be described in detail below in conjunction with the embodiments shown in the accompanying drawings. However, the embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on the embodiments are all within the protection scope of the present invention.

[0034] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0035] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0036] Embodiment 1 of the present invention provides a method for creating a GAN neural network for CXR chest image processing, such as Figure 1 As shown, the following steps are included:

[0037] Step 101: Obtaining a number of CXR chest images I x , and obtain several DRR (Digitally Reconstructed Radiograph) chest images before bone removal I d , and each DRR chest image I d Corresponding chest image Q after bone removal d , the corresponding rib projection B of the DRR domain d and the corresponding rib residual map R in the DRR domain d ;

[0038] Step 102: Create a GAN (Generative Adversarial Networks) neural network, where the generator G of the GAN neural network includes: a first encoder E s , the second encoder E c , the third encoder E R , the first decoder G L , the second decoder G Q , the third decoder G R and the fourth decoder G B ; Among them, the first encoder E s The input chest image I to be processed is processed, and the structural features F after the ribs are removed are obtained. S , the second encoder E c Process the input chest image I to be processed and obtain the contrast feature F C , the third encoder E R Process the input chest image I to be processed and obtain the rib features F R ; First decoder G L For structural features F S Process and obtain the lung segmentation L ′ ; Second decoder G Q For structural features F S and contrast feature F C Process and obtain the initial bone elimination graph Q ′ ; The third decoder G R For the contrast feature F C and rib feature F R Process and obtain the rib residual map R ′ ; The fourth decoder G B Rib feature FR Processing and obtaining rib projection diagram B ′ Here, it can be understood that after the GAN neural network training is completed, the CXR chest image I x (i.e., the chest image I to be processed is the CXR chest image I x ) is input into the GAN neural network, then the CXR chest image I x Corresponding chest image after bone removal = CXR chest image I x - Rib residual plot R ′ In this GAN neural network, by generating the rib projection map B ′ , the GAN neural network can better learn the rib features to predict the rib residual map, and also get a rib projection segmentation map; lung segmentation L ′ It can also help the GAN neural network better focus on the lung area.

[0039] Step 103: Based on a number of CXR chest images I x 、Several DRR chest images I d , several chest images Q d 、Several rib projections B d and several rib residual graphs R d , train the GAN neural network.

[0040] Here, the GAN neural network includes a generator G and a discriminator D, where for each DRR chest image I d For example, the corresponding chest image Q d , Rib projection B d and rib residual plot R d All are correct data, but for CXR chest image I x There is no corresponding correct data, so the GAN neural network can be trained based on the GAN training method.

[0041] In this embodiment, the “obtaining a plurality of DRR chest images before bone removal” d , and each DRR chest image I d Corresponding chest image Q after bone removal d , the corresponding rib projection B of the DRR domain d and the corresponding rib residual map R in the DRR domain d Specifically include:

[0042] Obtain several CT chest images, and perform the following operations on each CT chest image:

[0043] The CT chest image is forward projected to obtain a DRR chest image before bone removal.d Digital Reconstructed Radiography (DRR) is a method of simulating and generating X-ray images at any angle from CT images. X-ray images can be regarded as projections of CT images in a certain direction. In CT images, the attenuation coefficients of various positions of the human body are known. DRR sets up virtual X-ray light sources and detector planes to simulate the result of beams of light of different energy levels passing through the human body and attenuating under the action of the attenuation coefficients of various positions of the human body. Under a beam of a single energy level, making a projection is equivalent to making a Radon transform of the attenuation coefficients of each position, that is, summing the values ​​of the positions passing through a straight line in a certain direction.

[0044] The gray value of the soft tissue in the CT chest image is obtained, and a rib image is segmented from the CT chest image based on a pre-trained segmentation neural network, and the gray value of the rib image is replaced with the gray value. After that, the rib image is forward projected to obtain a chest image Q with bones removed but the spine retained. d ; Projecting a rib projection B from the rib image d ; R d =I d -Q d Here, through DRR technology, we can simulate a more realistic X-ray image. At the same time, we retain the spine when eliminating the ribs. The spine can provide important landmark information when doctors read the film. Moreover, since only the ribs are segmented in the CT image, the bone elimination in the image is limited to the ribs, and the spinal information is completely preserved.

[0045] Here, the chest image Q d The spine is preserved, which can provide position information when the doctor is reading the film. The preservation of the spine is achieved by retaining the spine in the bone-removing image of the DRR auxiliary training, and further by obtaining the DRR image by only segmenting the rib area in the CT.

[0046] In this embodiment, the plurality of CXR chest images I x and several DRR chest images I d Here, in reality, the paired CXR image and the CXR image after bone removal do not exist in reality, so the CXR chest image I can be used. x and DRR chest image I d , perform unsupervised learning.

[0047] In this embodiment, during the training process, several DRR chest images I d Input into the generator G and get each DRR chest image I dThe corresponding chest image Q′ after bone removal d , the corresponding rib projection B′ of the DRR domain d and the corresponding rib residual map R′ in the DRR domain d ; Several CXR chest images I x Input into the generator G and get each CXR chest image I x The corresponding chest image Q′ after bone removal x ,I x The corresponding rib projection B′ x and I x The corresponding rib residual graph R′ x ;

[0048] The loss function of the GAN neural network is:

[0049] in:

[0050] λ adv , f , i , λ inter and λ Δ All are weight values;

[0051]

[0052] ‖·‖1 is the L1 loss function;

[0053]

[0054] I′ d→x represents the image transferred from the DRR domain to the CXR domain, I′ x→d Represents an image transferred from the CXR domain to the DRR domain, and the discriminator D x It is used to evaluate the domain difference between the image converted to the CXR domain and the CXR image. The converted image retains the structure of the original image domain DRR while having the image contrast of the target domain CXR. d It is used to evaluate the domain difference between the image converted to the DRR domain and the DRR domain image. The converted image retains the structure of the original image domain CXR while having the image contrast of the target domain DRR. The discriminator D B Used to determine whether the input rib projection image is from the CXR domain or the DRR domain;

[0055] Due to the difference between the two image domains, it is invalid to directly use the results of supervised learning in the DRR domain for CXR image bone removal; in order to reduce the impact of image domain differences on rib removal, the GAN neural network applies cycle consistency learning and realizes CXR image decoupling through image domain adaptation. In particular, domain-invariant features and domain-dependent features are considered separately in the learning process. Here, it is assumed that rib features and structural features other than ribs are domain-invariant features, while contrast features are domain-dependent features, that is, DRR and CXR images contain the same anatomical structure information, but the contrast of DRR and CXR is different. In this way, by combining the structural features of the current image domain and the contrast features of the target image domain, images of a specific image domain can be generated. In order to apply the bone removal mapping learned in the DRR domain to the CXR domain, the model needs to learn domain-dependent contrast features; adversarial learning is used here to reduce the domain difference between the output prediction and the real image. is the discriminator D R The adversarial loss function is is the adversarial loss function of the generator;

[0056]

[0057] Here, in order to enhance the convertibility between the two image domains, the model learning is constrained by the reconstruction consistency at both the feature and image levels. At the feature level, contrast and structural consistency loss functions are used. Used to ensure that the contrast features extracted from the original image in the target domain and the generated image in the target domain are the same; structural consistency loss Used to prevent structural distortion in contrast-altered images.

[0058]

[0059] I′ x =G Q (E C (I x ),E S (I x ))+G R (E C (I x ),E B (I x )),

[0060] I″ x =G Q (E c (I′ d→x ),E s (I′ x→d ))+G R (E c (I′d→x ),E B (I′ x→d )),

[0061] I′ x For input I x A forward prediction of d For input I d A forward prediction of x and I″ d stands for Cycle Forecast;

[0062]

[0063] is the gradient operator, is the discriminator applied to the gradient map of the bone-removing image, Q′ d→x is the bone loss image converted from DRR to CXR, R′ x is the rib residual map predicted from CXR, L′ x and L′ d The lung segmentations predicted from CXR and DRR images are shown respectively; here, during the image domain adaptation process, the ribs may still remain in the prediction results of bone elimination. A gradient-based constraint is used in the lung region to eliminate the influence of grayscale value differences between different images, while enhancing the smoothness of grayscale changes around the ribs in the bone elimination prediction image. The adversarial loss of gradient information is added to the bone elimination prediction image of the CXR image and its corresponding rib residual image.

[0064] ,

[0065] M′ x For I x The corresponding rib segmentation is performed by x The bilateral filter and threshold segmentation are applied on it, where ∪ represents the intersection of the two input binary images; here, the grayscale information between the ribs should remain unchanged in the input CXR image and its corresponding bone elimination image.

[0066] ,

[0067] Δ is the Laplace operator. Here, in order to prevent the tracheal details overlapping with the ribs from remaining on the rib residual image, a loss function L based on the Laplace operator is used. Δ To eliminate the tracheal overlap in the rib residual map. The Laplacian operator can be seen as a regularization method that can help generate smoother rib residual maps.

[0068] In this embodiment, λ adv =1,λ f =1,λ i=10, λ inter =500,λ Δ =1.

[0069] In the process of the inventor's creation, 2 CT public datasets and 4 CXR public datasets were used. The CT dataset contains 896 CT scan images from LIDC-IDRI and 777 CT scan images from the Tianchi Medical AI Competition held by Alibaba in 2017 and 2019. Among them, the inventor only used CT images with a layer thickness of less than 2.5mm. The CXR dataset contains 11,200 CXR images of size 512×512 from TBX11K, 112,120 CXR images of size 1024×1024 from chest-14, 138 CXR images of size 4020×4892 from Montgomery Country, and 662 CXR images of size approximately 3000×3000 from Shenzhen Hospital.

[0070] For the CT dataset, the inventors annotated ribs on 199 CT scan images in LIDC-IDRI and trained a U-Net model for rib segmentation. The trained model was then applied to the entire CT dataset to obtain rib segmentation for all CT data. Based on rib segmentation, the inventors used an image inpainting algorithm to remove ribs in CT.

[0071] To generate DRR images, the inventors used DeepDRR to evenly project each CT image into 42 projection images within the range of ±10° left and right and up and down. Using a similar method, the inventors generated projection images of CT scans after bone removal. All DRR images were scaled to a size of 320×320 for training using bilinear interpolation. For the CXR dataset, all CXR images were scaled to a size of 320×320 and input into the network. Finally, the inventors adjusted the rib residual image back to its original size for rib removal. The inventors used the training sets of TBX11K, Chest-14, and MontgomerySet for training, and other data for testing.

[0072] The inventors used PyTorch to build the network and trained the network on a GeForce RTX 2080Ti GPU. All networks were trained using the Adam optimizer with a learning rate of 0.00001. In the initial stage, the inventors trained the network for 30,000 iterations, and the batch size during training was set to 1.

[0073] Weber Contrast and Peak Signal Noise Rate (PSNR) are two indicators used to evaluate the quality of rib removal in the inventor's model. For CXR images, Weber Contrast evaluates the performance of rib removal on the boundary by calculating the contrast difference between the rib removal area and the background. The definition is as follows:

[0074]

[0075] In the formula, I r Represents the average gray value of the manually marked rib area, I b Represents the limited background area within 5 pixels around the ribs. A smaller Weber Contrast means a higher similarity between the rib-removed area and the background, and the visual effect of the rib-removed CXR image will be better. Figure 5 shown.

[0076] PSNR is a commonly used image similarity index, and the inventors use it to evaluate the degree of detail preservation in the rib removal prediction results. The similarity calculation here is completed in the manually marked rib-free area. A higher PNSR represents a better prediction result.

[0077] The above indicators are used to compare the method of the present invention with some other methods: U-Net (DRR), U-Net (Cycle), and Li. For U-Net (DRR), the inventor directly uses the U-Net trained in the DRR image with bone removal labels to predict the bone removal map in CXR. For U-Net (Cycle), the CXR image is first converted to the DRR domain through CycleGAN, and then the ribs are eliminated by the U-Net trained on the DRR domain, and then converted back to the CXR domain. The above comparison is based on 192 CXR images with artificially marked ribs and lung areas, of which 99 CXR images were randomly selected from the Shenzhen Hospital dataset, and the other 93 were from the Chest-xray14 dataset.

[0078] Figure 5 and Figure 6The quantitative and visual results of different methods are shown. The GAN neural network has the highest PSNR (Peak Signal to Noise Ratio, an objective standard for evaluating images, which has limitations and is generally used for an engineering project between the maximum signal and the background noise) value and the lowest Weber Contrast (the ratio of the value of the area where the ribs are eliminated to the value of the area without ribs inside the lungs. If the ribs are not completely eliminated, this value should be very large), showing the best effect. The PSNR value of the GAN neural network is much better than that of U-Net (DRR) and U-Net (Cycle). This may be due to the consistency constraint imposed by the GAN neural network in the area between the ribs. Figure 6 As shown, the inventor randomly selected 1 normal and 2 tuberculosis cases from the Shenzhen Hospital dataset. Compared with other methods, the results output by the GAN neural network look cleaner both in general and in detail. In the dark rectangular box of Case 1, U-Net (DRR) and U-Net (Cycle) cannot completely eliminate the ribs, and the details are also smoothed out. Due to the grayscale error of histogram matching, drastic changes will exist in the output results of Li. Among all the methods, the GAN neural network eliminates the most rib components while retaining the richest details, which proves that through the decoupling method, the GAN neural network can obtain the best and most robust rib elimination image. For the retention of details in the rib elimination results, compared with other methods, the GAN neural network proposed by the inventor can make the details clearer and retain large lesions (tuberculosis in the light rectangular box) and very small objects (small blood vessels and trachea, see the dark rectangular box) at the same time. However, due to poor rib prediction, Li has a large afterimage of the clavicle. All the advantages of the inventor's method help to reduce the difficulty of reading the film and the misdiagnosis rate.

[0079] The inventors then developed two downstream applications: lung disease classification and tuberculosis detection, to further evaluate the quality of rib-eliminated CXR prediction maps.

[0080] In order to measure the contribution of rib elimination to lung disease classification, the inventors conducted some experiments on the Chest-xray14 and TBX11K datasets. The inventors applied DenseNet-121 as a classifier network to predict 14 lung diseases in Chest-xray14 and conducted experiments on different input combinations. The inventors tried 5 combinations: (1) DenseNet-121: only the original CXR is used as input; (2) DenseNet-121+Li: only the generated rib-eliminated CXR image is used as input; (3) DenseNet-121+Li(Mix): the combination of the original CXR image of 2 channels and the generated rib-eliminated CXR image of 1 channel is used as input; (4) DenseNet-121+GAN neural network: only the rib-eliminated CXR image generated by the inventor's method GAN neural network is used as input; (5) DenseNet-121+GAN neural network (Mix)x): the combination of the original CXR image of 2 channels and the rib-eliminated CXR image generated by the inventor's method GAN neural network of 1 channel is used as input. The inventors predicted 14 lung diseases on the Chest-xray14 dataset and used the commonly used Area Under Curve (AUC) indicator to compare the effectiveness of the 5 input combinations. The results are shown in Figure 7 and Figure 8 The inventor's method achieved the best results in the application of lung disease prediction. The combination of the original CXR image and the bone elimination map generated by this method brought an average improvement of 0.014 in the AUC index compared with the original CXR image alone. The inventor's CXR bone elimination map alone improved the pneumothorax AUC index by 1% and the fibrosis AUC index by 1.6% compared with the original CXR image alone. This is because these lung diseases cause changes in the appearance of the lung texture, and rib elimination helps to highlight the lung texture.

[0081] The inventors applied Faster R-CNN (FRCNN) as the classifier network class to predict CXR classification into three categories (healthy, unhealthy but not tuberculosis, tuberculosis). Similarly, the inventors made 5 combinations: (1) FRCNN; (2) FRCNN+Li; (3) FRCNN+Li

[19] (Mix); (4) FRCNN+GAN neural network; (5) FRCNN+GAN neural network (Mix). Here, the inventors used the accuracy, AUC, sensitivity, specificity, average precision and average recall of predicting three categories to compare these five input combinations. The comparison is based on the official TBX1K dataset. Figure 7 and Figure 8As shown in the figure, compared with other methods, FRCNN+GAN neural network (Mix) achieved the best results. Compared with direct prediction on the original CXR, this method improved the accuracy by 2.7% and AUC by 1.49%. This confirms that rib elimination can help improve the classification accuracy of CXR images in clinical diagnosis.

[0082] In order to demonstrate the effectiveness of GAN neural network for tuberculosis detection, the inventors applied FRCNN as the detection network to predict the tuberculosis area and classified the lesions in each detection frame into category-agnostic tuberculosis, active tuberculosis, and latent tuberculosis. Similar to the previous classification task, the inventors used 5 combinations of inputs to train the detection network: (1) FRCNN; (2) FRCNN+Li; (3) FRCNN+Li(Mix); (4) FRCNN+GAN neural network; (5) FRCNN+GAN neural network(Mix).

[0083] To evaluate the effectiveness of tuberculosis detection, the inventors used two average precision (AP) metrics for bounding box detection. AP50 represents the AP at an IoU (intersection-over-union) threshold of 0.5. mAP represents the average AP at IoUs with a step size of 0.05 from 0.5 to 0.95. Higher AP50 and mAP represent that the method can obtain more accurate detection results. The comparison is based on the official TBX11K dataset, and the results are as follows: Fig.10 Compared with other types of input, the combination of original CXR images and bone elimination maps can achieve the best detection effect; compared with only using original CXR images as input, it can bring a 5% improvement in AP50. This is probably because the GAN neural network is able to eliminate the clavicle and ribs overlapping the tuberculosis lesions, as shown in Fig.11 As shown in the light-colored rectangular boxes in Case 2 and Case 3.

[0084] The inventors conducted an ablation study on the key components of the GAN neural network, including the basic model (only decoupled learning and domain adaptation), rib elimination adversarial loss Background consistency loss and ribs to eliminate some smoothing loss

[0085] As Fig.11 As shown, when the loss function is eliminated When it is introduced, the Weber contrast decreases by 0.74, which indicates that the intensity difference between the inner area of ​​the ribs and the background is reduced. However, the decrease in PSNR indicates that many details are lost due to When the inter-rib constraint loss function is introduced After that, the improvement of PSNR shows that This helps keep the details between the ribs unchanged after the ribs are removed. Finally, a loss function based on the Laplacian operator is added After that, both indicators in the table are slightly improved, which helps to retain clearer details in the rib area in the rib removal result.

[0086] like Fig.11 As shown in the figure, three CXR images were selected from the Shenzhen Hospital dataset to visually compare the rib removal effects with different loss functions. In the end, the inventors got fewer ribs than the base model, but at the same time retained fewer details. compared to, Helps maintain the strength of the inter-rib area, Improved preservation of detail in the inner rib area.

[0087] Embodiment 2 of the present invention provides a device for creating a GAN neural network for CXR chest image processing, comprising the following modules:

[0088] The first data acquisition module is used to acquire a plurality of CXR chest images I x , and obtain several DRR chest images before bone removal I d , and each DRR chest image I d Corresponding chest image Q after bone removal d , the corresponding rib projection B of the DRR domain d and the corresponding rib residual map R in the DRR domain d ;

[0089] A GAN neural network creation module is used to create a GAN neural network. The generator G of the GAN neural network includes: a first encoder E s , the second encoder E c , the third encoder E R , the first decoder G L , the second decoder G Q , the third decoder G R and the fourth decoder G B ; Among them, the first encoder E s The input chest image I to be processed is processed, and the structural features F after the ribs are removed are obtained. S , the second encoder E c Process the input chest image I to be processed and obtain the contrast feature F C , the third encoder E R Process the input chest image I to be processed and obtain the rib features F R ; First decoder GL For structural features F S Process and obtain the lung segmentation L ′ ; Second decoder G Q For structural features F S and contrast feature F C Process and obtain the initial bone elimination graph Q ′ ; The third decoder G R For the contrast feature F C and rib feature F R Process and obtain the rib residual map R ′ ; The fourth decoder G B Rib feature F R Processing and obtaining rib projection diagram B ′ ;

[0090] Training module for training a chest image based on several CXR images x 、Several DRR chest images I d , several chest images Q d 、Several rib projections B d and several rib residual graphs R d , train the GAN neural network.

[0091] In this embodiment, the first data acquisition module is further used to: acquire a plurality of CT chest images, and perform the following operations on each CT chest image: perform forward projection processing on the CT chest image to obtain a DRR chest image before bone removal. d ; Obtain the gray value of the soft tissue in the CT chest image, segment the rib image from the CT chest image based on the pre-trained segmentation neural network, replace the gray value of the rib image with the gray value, and then perform forward projection processing on the rib image to obtain a chest image Q with bone removed but spine retained d ; Projecting a rib projection B from the rib image d ; R d =I d -Q d Here, the segmentation neural network can be U-Net. The specific method is to mark the ribs in a set of CT data and then train U-Net.

[0092] In this embodiment, the plurality of CXR chest images I x and several DRR chest images I d It is unpaired.

[0093] Embodiment 3 of the present invention provides a method for processing a CXR chest image, comprising the following steps: obtaining a CXR chest image I inputExecute the creation method in Example 1 to create a GAN neural network, control the generator G in the GAN neural network to generate the CXR chest image I input Process and obtain the rib residual graph R' input , then the CXR chest image I input The corresponding bone elimination diagram = I input -R' input .

[0094] Embodiment 3 of the present invention provides a device for processing a CXR chest image, comprising the following modules: a second data acquisition module, configured to acquire a CXR chest image I input ; An image processing module, used to execute the creation method in Example 1 to create a GAN neural network, control the generator G in the GAN neural network to process the CXR chest image I input Process and obtain the rib residual graph R' input , then the CXR chest image I input The corresponding bone elimination diagram = I input -R' input .

[0095] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0096] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for creating a GAN neural network for CXR chest image processing, It is characterized in that The following steps are involved: Acquire several CXR chest images I x , and obtain several DRR chest images before bone removal I d , and each DRR chest image I d Corresponding chest image Q after bone removal d , the corresponding rib projection B of the DRR domain d and the corresponding rib residual map R in the DRR domain d ; Create a GAN neural network, the generator G of the GAN neural network includes: a first encoder E s , the second encoder E c , the third encoder E R , the first decoder G L , the second decoder G Q , the third decoder G R and the fourth decoder G B ; Among them, the first encoder E s The input chest image I to be processed is processed, and the structural features F after the ribs are removed are obtained. S , the second encoder E c Process the input chest image I to be processed and obtain the contrast feature F C , the third encoder E R Process the input chest image I to be processed and obtain the rib features F R ; First decoder G L For structural features F S Processing is performed and the lung segmentation L′ is obtained; the second decoder G Q For structural features F S and contrast feature F C Processing is performed to obtain the initial bone elimination graph Q′; the third decoder G R For the contrast feature F C and rib feature F R Processing is performed to obtain the rib residual image R′; the fourth decoder G B Rib feature F R Processing is performed to obtain the rib projection diagram B′; Based on several CXR chest images I x 、Several DRR chest images I d , several chest images Q d 、Several rib projections B d and several rib residual graphs R d , train the GAN neural network.

2. The creation method according to claim 1, It is characterized in that The method of "obtaining a plurality of DRR chest images before bone removal" d , and each DRR chest image I d Corresponding chest image Q after bone removal d , the corresponding rib projection B of the DRR domain d and the corresponding rib residual map R in the DRR domain d Specifically include: Acquire several CT chest images, and perform the following operations on each CT chest image: perform forward projection processing on the CT chest image to obtain a DRR chest image before bone removal. d ; Obtain the gray value of the soft tissue in the CT chest image, segment the rib image from the CT chest image based on the pre-trained segmentation neural network, replace the gray value of the rib image with the gray value, and then perform forward projection processing on the rib image to obtain a chest image Q with bone removed but spine retained d ; Projecting a rib projection B from the rib image d ; R d =I d -Q d .

3. The creation method according to claim 1, It is characterized in that The plurality of CXR chest images I x and several DRR chest images I d It is unpaired.

4. The creation method according to claim 1, Features: During the training process, several DRR chest images I d Input into the generator G and get each DRR chest image I d The corresponding chest image Q′ after bone removal d , the corresponding rib projection B′ of the DRR domain d and the corresponding rib residual map R′ in the DRR domain d ; Several CXR chest images I x Input into the generator G and get each CXR chest image I x The corresponding chest image Q′ after bone removal x ,I x The corresponding rib projection B′ x and I x The corresponding rib residual graph R′ x ; The loss function of the GAN neural network is: in: λ adv , f , i , λ inter and λ Δ All are weight values; ||·|| 1 YesL 1 Loss function; I d→x represents the image transferred from the DRR domain to the CXR domain, I x→d Represents an image transferred from the CXR domain to the DRR domain, and the discriminator D x It is used to evaluate the domain difference between the image converted to the CXR domain and the CXR image. The converted image retains the structure of the original image domain DRR while having the image contrast of the target domain CXR. d It is used to evaluate the domain difference between the image converted to the DRR domain and the DRR domain image. The converted image retains the structure of the original image domain CXR while having the image contrast of the target domain DRR. The discriminator D B Used to determine whether the input rib projection image is from the CXR domain or the DRR domain; I x =G Q (E C (I x ),E S (I x ))+G R (E C (I x ),E B (I x )), I″ x =G Q (E C (I′ d→x ), E S (I′ x→d ))+G R (E C (I′ d→x ), E B (I′ x→d )),I x For input I x A forward prediction of d For input I d A forward prediction of x and I d stands for Cycle Forecast; is the gradient operator, is the discriminator applied to the gradient map of the bone-removing image, Q′ d→x is the bone loss image converted from DRR to CXR, R′ x is the rib residual map predicted from CXR, L′ x and L′ d are the lung segmentations predicted from CXR and DRR images, respectively; ， M′ x For I x The corresponding rib segmentation is performed by x Apply bilateral filter and threshold segmentation on it, ∪ represents the intersection of two input binary images; ， Δ is the Laplace operator.

5. The creation method according to claim 4, Features: l adv =1,λ f =1,λ i =10, l inter =500, l Δ =1.

6. A device for creating a GAN neural network for CXR chest image processing, It is characterized in that Includes the following modules: The first data acquisition module is used to acquire a plurality of CXR chest images I x , and obtain several DRR chest images before bone removal I d , and each DRR chest image I d Corresponding chest image Q after bone removal d , the corresponding rib projection B of the DRR domain d and the corresponding rib residual map R in the DRR domain d ; A GAN neural network creation module is used to create a GAN neural network. The generator G of the GAN neural network includes: a first encoder E s , the second encoder E c , the third encoder E R , the first decoder G L , the second decoder G Q , the third decoder G R and the fourth decoder G B ; Among them, the first encoder E s The input chest image I to be processed is processed, and the structural features F after the ribs are removed are obtained. S , the second encoder E c Process the input chest image I to be processed and obtain the contrast feature F C , the third encoder E R Process the input chest image I to be processed and obtain the rib features F R ; First decoder G L For structural features F S Processing is performed and the lung segmentation L′ is obtained; the second decoder G Q For structural features F S and contrast feature F C Processing is performed to obtain the initial bone elimination graph Q′; the third decoder G R For the contrast feature F C and rib feature F R Processing is performed to obtain the rib residual image R′; the fourth decoder G B Rib feature F R Processing is performed to obtain the rib projection diagram B′; Training module for training a chest image based on several CXR images x 、Several DRR chest images I d , several chest images Q d 、Several rib projections B d and several rib residual graphs R d , train the GAN neural network.

7. The creation device according to claim 6, It is characterized in that The first data acquisition module is also used for: Obtain several CT chest images, and perform the following operations on each CT chest image: The CT chest image is forward projected to obtain a DRR chest image before bone removal. d ; Obtain the gray value of the soft tissue in the CT chest image, segment the rib image from the CT chest image based on the pre-trained segmentation neural network, replace the gray value of the rib image with the gray value, and then perform forward projection processing on the rib image to obtain a chest image Q with bone removed but spine retained d ; Projecting a rib projection B from the rib image d ; R d =I d -Q d .

8. The creation device according to claim 6, It is characterized in that The plurality of CXR chest images I x and several DRR chest images I d It is unpaired.

9. A method for processing CXR chest images, It is characterized in that The following steps are involved: Get CXR chest image I input ; Execute the creation method of any one of claims 1 to 5 to create a GAN neural network, control the generator G in the GAN neural network to the CXR chest image I input Process and obtain the rib residual map R input , then the CXR chest image I input The corresponding bone elimination diagram = I input -R input .

10. A device for processing CXR chest images, It is characterized in that Includes the following modules: The second data acquisition module is used to acquire CXR chest image I input ; An image processing module, used to execute the creation method of any one of claims 1 to 5 to create a GAN neural network, and control the generator G in the GAN neural network to process the CXR chest image I input Process and obtain the rib residual graph R' input , then the CXR chest image I input The corresponding bone elimination diagram = I input -R' input .

Images