Method for processing radiological images

Abstract

A method for processing a digital radiographic image I(x,y) including at least one radiographic abnormality detected using a convolutional neural network trained to detect radiographic abnormalities in a radiological examination, wherein the radiographic image I(x,y) is characterized by the intensity of each pixel I(x,y) and at least one radiographic abnormality influence map C. k (x,y), which assigns a value (x,y) to each pixel (x,y) of the radiographic image I(x,y), representing the proportion of the pixel's influence on the detection result of the radiographic anomaly k. This method is implemented by a computer and includes the following steps: - Normalizing (3) the radiographic anomaly influence map C k (x,y) is used to give the normalized radiological anomaly effect diagram C. kn (x,y); - Fusion (4) Normalized Radiological Anomaly Effects Diagram C kn (x,y) to give a single fusion influence map C(x,y); - Improve the image I(x,y) using an intensity histogram (5), wherein the contribution of each pixel in the calculation of the intensity histogram is weighted by the fusion influence map C(x,y).

Description

Technical Field

[0001] This invention relates to a method for processing radiographic images, and belongs to the field of medical devices for images obtained using X-ray imaging, and more precisely to the field of image processing and machine learning for diagnosis in assisting radiographic examinations. Background Technology

[0002] To establish a diagnosis based on radiological examinations, radiologists apply various post-processing operations (contrast optimization, windowing, contour enhancement, etc.) to images to highlight certain abnormalities indicating the presence of one or more conditions. Radiological abnormalities correspond to any signs visible on a radiological examination that indicate a patient's condition. For example, these abnormalities may be opacities, coagulations, calcifications, infiltrations, etc.

[0003] These post-processing operations need to be parameterized, either by default or by the user, or automatically and based on measurements performed on the entire image or in a user-defined region of interest (ROI).

[0004] These post-processing operations typically require manual intervention to parameterize them or define the ROI. Therefore, defining parameters or ROIs can be time-consuming and challenging if the anomalies in question are difficult to identify before applying these processing operations. This can increase diagnostic time.

[0005] When these measures are performed on the entire image, the results may not be optimal for visualizing anomalies located in specific regions of the image.

[0006] As an alternative, systems based on machine learning algorithms have recently been used to assist in diagnosis based on radiological examinations, using the acronym CAD for "Computer-Aided Diagnosis." However, most of these are still in the experimental stage, and their results are difficult to interpret. Therefore, it is necessary for radiologists to study the examinations to establish diagnoses.

[0007] Numerous post-processing operations are used to improve the quality of radiographic images, particularly to enhance contrast. Many of these are based on histogram equalization (HE), as disclosed in the following literature: "Computer vision: algorithms and applications", Springer Science & Business Media, by Szeliski, Richard, 2010, inparagraph 3.1.4. In this method, the intensity of pixels is modified to obtain the most homogeneous possible intensity distribution across all pixels of the image. The HE algorithm does not specify any parameterization, nor does it attempt to improve contrast across all regions of the image.

[0008] To overcome this problem, many variations exist, particularly contrast-limited adaptive histogram equalization (CLAHE), as described in the following literature: "Adaptive histogram equalization and its variations", Computer vision, graphics, and image processing 39(3):355–368, by Pizer, Stephen M, E Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. 1987; and "Modified Contrast Limited Adaptive Histogram Equalization Based on Local Contrast Enhancement for Mammogram Images" Mobile Communication and Power Engineering, edited by Vinu V Das and Yogesh Chaba, 296:397-403, Communications in Computer and Information Science, Berlin, Heidelberg: Springer Berlin. Heidelberg, https: / / doi.org / 10.1007 / 978-3-642-35864-7_60, by Mohan, Shelda, and M. Ravishankar, 2013. These variants must be parameterized, and some works propose automatic parameterization methods. In the method "Automatic x-ray image contrast enhancement based on parameter auto-optimization", Journal of Applied Clinical Medical Physics 18(6):218-23, https: / / doi.org / 10.1002 / acm2, 2017, Pizer's solution is coupled with high-pass filters and their parameters are optimized to maximize the entropy of the processed image.The paper "Improved Lung Nodule Visualization on Chest Radiographs Using Digital Filtering and Contrast Enhancement" 5(12):4 by Kwan, Benjamin YM, and Hon Keung Kwan, 2011 proposes using HE after another processing operation or simply applying HE to a user-defined ROI.

[0009] Pizer and Mohan's method proposed HE (histogram equalization) improvements to overcome the aforementioned problems. However, these methods must be parameterized, either utilizing default parameters that may not be suitable for every image, or manually set by the radiologist.

[0010] Kwan's method proposes applying HE after a high-pass filter, and therefore the limitations of HE remain. As an alternative, it proposes calculating HE on the region of interest (ROI), but this requires a user-defined ROI. Therefore, this task requires radiologist intervention and can sometimes prove complex if the radiologist lacks any prior information about the location of the abnormality to be visualized.

[0011] The paper "Automatic x-ray image contrast enhancement based on parameter auto-optimization". Journal of Applied Clinical Medical Physics 18(6):218-23, by Qiu, Jianfeng, H. Harold Li, Tiezhi Zhang, Fangfang Ma, and Deshan Yang, 2017 proposes a variant of the Pizer method by coupling it with a high-pass filter. It also describes an automatic parameterization strategy in which parameters are optimized to maximize the entropy of the processed image.

[0012] Other methods for improving contrast include those described in the following literature: "Contrast Enhancement of Medical X-Ray Image Using Morphological Operators with Optimal Structuring Element", ArXiv:1905.08545[Cs,Eess], http: / / arxiv.org / abs / 1905.08545,2019, by Kushol, Rafsanjany, Md Nishat Raihan, Md Sirajus Salekin, and A.B. Mashikur Rahman. This literature also proposes a strategy for performing automatic parameterization.

[0013] In Kushol and Qiu's method, all pixels in the image have the same effect on the calculation of the variables used to optimize the parameters. The results obtained in this way may not be optimal for certain regions of the image; for example, in the case of HE (Heterochromatic Image), the distribution in some regions differs significantly from the distribution across the entire image.

[0014] Recent work has proposed using neural networks (more specifically, networks that perform image classification) to aid in diagnosis. Examples of this type are described in the following literature: "Abnormality Detection and Localization in Chest X-Rays Using Deep Convolutional Neural Networks", ArXiv:1705.09850[Cs], 2017, http: / / arxiv.org / abs / 1705.09850, by Islam, Mohammad Tariqul, Md Abdul Aowal, Ahmed Tahseen Minhaz, and Khalid Ashraf; and "CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning", ArXiv:1711.05225[Cs,Stat], http: / / arxiv.org / abs / 1711.05225, 2017, by Rajpurkar, Pranav, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, DaisyDing, et al.

[0015] The Islam or Rajpurkar methods do not improve the visibility of abnormalities in radiographic images, but they estimate the presence of radiographic abnormalities associated with pathology. These techniques are still experimental and require the intervention of a radiologist to confirm the diagnosis.

[0016] Numerous methods have been proposed for visual interpretation of results obtained from neural networks used for image classification. These methods generate an influence map that assigns a value to each pixel of the input image, measuring the proportion of the pixel's influence on the classification result. The higher the value, the greater the pixel's influence on the resulting classification. This is the case in: "Grad-cam: Visual explanations from deep networks via gradient-based localization", Proceedings of the IEEE international conference on computervision, 618–626, 2017, by Selvaraju, Ramprasaath R, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra; "Layer-Wise RelevancePropagation for Deep Neural Network Architectures",Information Science andApplications(ICISA)2016,edited by Kuinam J.Kim and Nikolai Joukov,376:913-22,Lecture Notes in Electrical Engineering,Singapore:Springer Singapore.https: / / doi.org / 10.1007 / 978-981-10-0557-2_87,by Binder, Alexander, Sebastian Bach, Gregoire Montavon, Klaus-Robert Müller, and Wojciech Samek; "Learning DeepFeatures for Discriminative Localization", 2016 IEEE Conference on ComputerVision and Pattern Recognition (CVPR), 2921-29, Las Vegas, NV, USA: IEEE. https: / / doi.org / 10.1109 / CVPR.2016.319, by Zhou, Bolei, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba; and "Visualizing and Understanding Convolutional Networks" ArXiv:1311.2901 [Cs], November. http: / / arxiv.org / abs / 1311.2901, 2016, by Zeiler, Matthew D., and Rob Fergus. When these methods are applied to neural networks for radiological image classification (depending on whether the image exhibits radiological anomalies), the resulting influence map corresponds to the radiological anomaly influence map. Summary of the Invention

[0017] One object of the present invention is to overcome the above-mentioned problems, and in particular to improve the post-processing of digital images in order to better highlight detected radiological abnormalities, facilitate the work of radiologists, and reduce diagnostic time.

[0018] Therefore, according to one aspect of the invention, a method is proposed for processing a digitally formatted radiographic image I(x,y), the radiographic image including at least one radiographic abnormality detected using a convolutional neural network trained to detect radiographic abnormalities in radiological examinations. The radiographic image I(x,y) is characterized by the intensity of each pixel I(x,y) and at least one radiographic abnormality influence map C. k (x,y) assigns a value (x,y) to each pixel (x,y) of the radiographic image I, representing the proportion of that pixel's influence on the detection result of the radiographic anomaly k. This method is implemented by a computer and includes the following steps:

[0019] - Normalized Radiological Anomaly Effects Diagram C k (x,y) is used to give the normalized radiological anomaly effect diagram C. kn (x,y);

[0020] -Unified normalized radiological anomaly effect diagram C kn (x,y) to give the single fusion impact diagram C(x,y);

[0021] - The image I(x,y) is improved using an intensity histogram, where the contribution of each pixel in the intensity histogram calculation is weighted by the fusion influence map C(x,y).

[0022] The method of this invention allows for the highlighting of detected abnormalities, thereby facilitating the work of radiologists and reducing diagnostic time.

[0023] The following documents disclose neural networks for identifying images containing such radiological anomalies: "Abnormality Detection and Localization in Chest X-Rays Using Deep Convolutional Neural Networks", ArXiv:1705.09850[Cs], 2017, http: / / arxiv.org / abs / 1705.09850, by Islam, Mohammad Tariqul, Md Abdul Aowal, Ahmed Tahseen Minhaz, and Khalid Ashraf; and "CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning", ArXiv:1711.05225[Cs,Stat], http: / / arxiv.org / abs / 1711.05225, 2017, by Rajpurkar, Pranav, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, TonyDuan, Daisy Ding, et al.

[0024] Training (or optimizing or learning) a neural network allows it to generate a probability p of existence based on the input image for each anomaly k. k When this probability is greater than a given threshold π, for example, π = 0.5, then an anomaly is considered to have been detected.

[0025] In one implementation, each anomaly impact graph C k The normalization of (x,y) uses the following affine transformation:

[0026]

[0027] in:

[0028] C kn (x, y) represents the influence of radiological anomalies in diagram C. k Normalized radiological anomaly effect diagram of (X,y).

[0029] This normalization ensures that all normalized graphs C kn (x, y) are in the same range of values ​​(in this case, between 0 and 1). This process can balance the influence of various graphs during the fusion step (the graph containing values ​​between 0 and 100 has a much larger influence than the graph containing values ​​between 0 and 1).

[0030] In one implementation, C(x,y) is fused using a normalized radiological anomaly influence map C. kn The average value of (x, y) is given by the probability p of each anomaly being expressed using the following relationship. k Weighted:

[0031]

[0032] in:

[0033] |K| represents the number of anomalies detected; and

[0034] p k p represents the probability of each anomaly existing. k It is calculated by a convolutional neural network.

[0035] The advantage of this fusion is that a single graph is used to weight the contribution of each pixel in the image during the calculation of the intensity histogram.

[0036] In one implementation, the processing operation uses intensity histogram equalization, where the histogram is calculated by weighting the contribution of each pixel according to the following relationship:

[0037]

[0038] in:

[0039] H(u) represents the level of the modified histogram of intensity u; and

[0040] Indicator functions:

[0041] I(x,y) represents the intensity of pixel (x,y).

[0042] This processing operation has the advantage of improving image contrast, especially the contrast of regions in the image that show detected anomalies, because the influence of the corresponding pixels in the histogram calculation is increased.

[0043] As a variant, the processing operation can use a variant of the method proposed in the following literature: "Automatic x-ray image contrast enhancement based on parameter auto-optimization". Journal of Applied Clinical Medical Physics 18(6):218-23, by Qiu, Jianfeng, H. Harold Li, Tiezhi Zhang, Fangfang Ma, and Deshan Yang, 2017", where the calculation of the entropy of the processed image is modified using the following relation:

[0044]

[0045] in:

[0046] h(I t ) represents the processed image I t Modified entropy

[0047] U represents all intensity levels of the digital image; and

[0048] p u Replace the probability that a pixel in a digital image has intensity u with the following relationship:

[0049]

[0050] Where H(u) represents the level of the modified histogram of intensity u according to the following relationship:

[0051]

[0052] This processing operation improves the contrast of the image by relatively increasing the contribution of the corresponding pixel to the entropy calculation of the processed image, especially the contrast of image regions that have been detected as anomalous.

[0053] According to another aspect of the invention, a computer program product is also provided, comprising program code instructions recorded on a computer-readable medium for implementing the steps of the above-described method when the program is executed on a computer. Attached Figure Description

[0054] The invention will be better understood by studying some embodiments described by way of completely non-limiting examples and illustrated with the accompanying drawings, wherein:

[0055] [ Figure 1 The illustration schematically depicts a computer-implemented method for processing radiographic images according to one aspect of the present invention. Detailed Implementation

[0056] Figure 1 The illustration depicts a method for processing a digital radiographic image I(x,y) according to one aspect of the invention, the radiographic image including at least one radiographic anomaly detected using a convolutional neural network trained to detect radiographic anomalies in a radiographic examination 1. The radiographic image I(x,y) is characterized by the intensity of each pixel I(x,y) and the influence of at least one radiographic anomaly on the image. k (x,y)2 assigns a value (x,y) to each pixel (x,y) of the radiographic image I(x,y), which represents the proportion of the pixel's influence on the detection result of the radiographic anomaly k.

[0057] This method is implemented by a computer and includes the following steps:

[0058] - Normalized 3-radiological anomaly influence diagram C kn (x,y) is used to give the normalized radiological anomaly effect diagram C. kn (x,y);

[0059] -Fusion 4-normalized radiological anomaly effect diagram C kn (x,y) to give the single fusion impact diagram C(x,y);

[0060] - An improved processing is performed on the image I(x,y) using an intensity histogram5, where the contribution of each pixel in the intensity histogram calculation is weighted by the fusion influence map C(x,y).

[0061] The method according to the invention enables the automatic processing of radiographic images I(x,y), which improves the visibility of areas where abnormalities are detected. Therefore, the invention facilitates symptom diagnosis based on radiographic examinations without intervention from a radiologist or another user, and provides images in which the visibility of any radiographic abnormalities is improved, making them easier for radiologists to study.

[0062] To this end, the existing processing operations have been modified to increase the impact on image regions where anomalies have been detected. Therefore, the processing operations will be more effective on these regions, making it easier to highlight detected anomalies.

[0063] This is mainly achieved by weighting the contribution of each pixel in the image by fusing the corresponding values ​​of the influence map C(x,y) during the calculation of the intensity histogram.

[0064] Detection step 1 can use a convolutional neural network that has been trained to detect radiological abnormalities representing pathological conditions in radiological examinations.

[0065] Different detection methods can be used when compatible radiological anomaly impact map estimations are available.

[0066] The implementation and training / learning of such neural networks are widely described in the literature. This type of network is based on the input image generating a probability p of the presence of each anomaly k. k When this probability is greater than a given threshold π, for example, π = 0.5, an anomaly is considered to have been detected. Let K represent all detected anomalies.

[0067] k∈K if and only if p y >π

[0068] This type of network typically requires the input image to have a precise resolution and a precise number of channels (one for grayscale images and three for RGB images). If the input image has a different resolution, the image can be resized to the desired resolution, and the number of channels can be adjusted.

[0069] For example, (Rajpurkar et al. 2017) used the DenseNet-121 network architecture (Huang, Gao, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. "Densely Connected Convolutional Networks". In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2261-69. Honolulu, HI:IEEE. https: / / doi.org / 10.1109 / CVPR.2017.243), which takes a three-channel image of size 224x224 at the input. In this case, the size of a radiographic image with arbitrary resolution and a single channel is resized to 224x224 and converted to three channels, with the same value repeated for each channel.

[0070] Step 2 of the impact estimation can be achieved using existing techniques, such as those proposed in the following literature: "Grad-cam: Visual explanations from deep networks via gradient-based localization," Proceedings of the IEEE international conference on computervision, 618–626, 2017, by Selvaraju, Ramprasaath R, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. This method is compatible with various architectures and produces a map with a resolution equal to the output of the selected convolutional layer in the detection network architecture. The size of the radiographic anomaly impact map must then be resized to the original resolution of the image, for example, through bilinear interpolation.

[0071] C k (x,y) is used to represent the radiological anomaly effect map corresponding to the detected anomaly k.

[0072] For example, when the influence estimation2 from Selvaraju, Ramprasaath R, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra is applied to the last convolutional layer of the detections from Rajpurkar, Pranav, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, TonyDuan, and Daisy Ding, the output resolution is 7x7. The size of the radiometric anomaly influence map is then resized to match the original image size.

[0073] Any other radiological anomaly effect map estimation that is compatible with the selected detection method can be valid.

[0074] Regarding the influence of each radiological abnormality on Normalized 3, Figure C k One possible implementation of the steps for (x, y) is to apply an affine transformation so that all values ​​are between 0 and 1:

[0075]

[0076] Where C kn (x, y) represents the influence of radiological anomalies in diagram C. kThe normalized graph of (x,y).

[0077] Radiological anomaly influence diagram C k (X,y) can be normalized in different ways.

[0078] Fusion 4 Normalized radiological anomaly impact plot C for each detected anomaly k kn One implementation solution for the (x,y) step is to take the average of the graph weighted by the existence probability of each anomaly k:

[0079]

[0080] in

[0081] |K| represents the number of anomalies detected; and

[0082] p k p represents the probability of each anomaly existing. k It is calculated by a convolutional neural network.

[0083] As a variant, graphs can be merged in different ways.

[0084] The processing steps can use intensity histogram equalization, where the histogram calculation is modified by weighting the contribution of each pixel according to the following relationship:

[0085]

[0086] in:

[0087] H(u) represents the level of the modified histogram of intensity u; and

[0088] Indicator functions:

[0089]

[0090] I(x,y) represents the intensity of pixel (x,y).

[0091] As a variant, the processing operation can use a variant of the method proposed in the following literature: "Automatic x-ray image contrast enhancement based on parameter auto-optimization". Journal of Applied Clinical Medical Physics 18(6):218-23, by Qiu, Jianfeng, H. Harold Li, Tiezhi Zhang, Fangfang Ma, and Deshan Yang, 2017", where the calculation of the entropy of the processed image is modified using the following relation:

[0092]

[0093] in:

[0094] h(I t ) represents the processed image I t Modified entropy

[0095] U represents all intensity levels of the digital image; and

[0096] p u Replace the probability that a pixel in a digital image has intensity u with the following relationship.

[0097]

[0098] Where H(u) represents the level of the histogram of intensity u modified according to the following relationship:

[0099]

[0100] Process 5 can also be applied differently to the above processing methods and other known processing methods.

[0101] The present invention can be implemented in a computer program product comprising computer-executable computer code stored on a computer-readable medium and designed to implement the methods described above.

[0102] This invention can be implemented on a local computer or a distributed network platform.

Claims

1. A method for processing a radiographic image I(x,y) in digital format, the radiographic image including at least one radiographic anomaly k detected using a convolutional neural network trained to detect radiographic anomalies in a radiographic examination, the radiographic image I(x,y) being processed by the intensity of each of its pixels (x,y) and the influence map C of at least one radiographic anomaly. k (x,y) is a feature, and the influence of at least one radiological anomaly is shown in Figure C. k (x,y) corresponds to the radiological anomaly k, and a value representing the proportion of the influence of the pixel on the detection result of the radiological anomaly k is assigned to each pixel (x,y) of the radiological image I. The method is implemented by a computer and includes the following steps: - normalizing (3) the radiological abnormality impact map C k (x,y) to give a normalized radiological abnormality impact map C kn (x,y); - fusing (4) said normalized radiological abnormality influence maps C kn (x,y) to give a single fused influence map C(x,y); - improving processing (5) of said image I(x,y) using an intensity histogram, wherein the contribution of each pixel in the computation of said intensity histogram is weighted by said fusion influence map C(x,y).

2. The method of claim 1, wherein, Each anomaly influences graph C k The normalization (3) of (x, y) uses the following affine transformation: wherein: C kn (x,y) is the normalized radiological abnormality influence map of the radiological abnormality influence map C k (x,y).

3. The method of claim 1 or 2, wherein, The fusion (4) C(x,y) uses the normalized radiological abnormality influence map C kn The probability p of presence of each abnormality is given by the following relation k Weighted average wherein |K| represents the number of detected anomalies; and p k a probability p that each anomaly exists k calculated by the convolutional neural network.

4. The method of claim 1 or 2, wherein, said processing (5) uses intensity histogram equalization, wherein said computation of said histogram is modified by weighting the contribution of each pixel according to the following relation: wherein: H(u) represents the level of the modified histogram of intensity u; and represents an indicator function: I(x,y) represents the intensity of pixel (x,y).

5. A computer program product comprising program code instructions recorded on a computer readable medium for implementing the steps of the method of any one of claims 1 to 4 when said program is executed on a computer.

Images