Methods for processing radiographic images

Abstract

1. A method of processing a radiological image I(x,y) in digital format containing at least one radiological anomaly k, detected using a convolutional neural network trained to detect radiological anomalies in radiological examinations, wherein the radiological image I(x,y) is divided into at least one radiological anomaly influence map C, which assigns a numerical value to each pixel (x,y) of the radiological image I(x,y) according to the intensity of each of its pixels I(x,y) and how much that pixel influenced the detection of the radiological anomaly k. k (x,y), and the method is computer-implemented; - radiological anomaly influence map C k Normalize (x,y) to obtain the normalized radiological anomaly influence map C kn Step (3) to obtain (x,y) and a normalized radiological anomaly influence map C kn (4) fusing (x,y) to obtain a single fused influence map C(x,y); and (5) performing an improvement process on the image I(x,y) using an intensity histogram, where the contribution of each pixel in the calculation of the intensity histogram is weighted by the fused influence map C(x,y).

Description

[Technical Field]

[0001] The present invention relates to a method for processing radiographic images, and belongs to the field of medical devices for utilizing images obtained by X-ray imaging, and more specifically, to the field of image processing and machine learning for supporting the diagnosis of radiological examinations. [Background technology]

[0002] To confirm a diagnosis based on radiographic examinations, radiologists apply various post-processing operations to images (contrast optimization, windowing, contour enhancement, etc.) to highlight specific abnormalities that indicate the presence of one or more pathologies. Radiographic abnormalities correspond to any visible signs in a radiographic examination that suggest pathologies within the patient's body. For example, these abnormalities may include opacities, aggregations, calcifications, infiltrations, etc.

[0003] These post-processing operations are determined by default, require parameterization provided by the user, or are automatic, and are based on measurements performed on the entire image or within a user-specified region of interest (ROI).

[0004] These post-processing operations often require human intervention to parameterize them or specify ROIs. Therefore, specifying parameters or ROIs can be time-consuming, and if the anomalies in the problem are not easily visible until these processes are applied, it may be considered difficult. Consequently, the diagnostic time will be longer.

[0005] If these measurements are performed on the entire image, the results may not be optimal in terms of visualizing anomalies within specific areas of the image.

[0006] As an alternative, in recent years, systems based on machine learning algorithms have been used to assist in diagnosis; this is called CAD, an abbreviation for "computer-aided diagnosis" based on radiological examinations. However, most of these are still in the experimental stage, and the results are difficult to interpret. Therefore, radiologists need to carefully examine the test results to confirm the diagnosis.

[0007] To improve the quality of radiographic images, various post-processing operations exist, particularly to enhance contrast. Many of these are based on histogram equalization (HE), which is disclosed in paragraph 3.1.4 of the literature, “Computer vision: algorithms and applications”, Springer Science & Business Media, by Richard Szeliski, 2010. In this method, the intensity of pixels is modulated so that the intensity distribution is as uniform as possible across all pixels in the image. The HE algorithm does not specify any parameterization, but it does not improve the contrast across the entire image.

[0008] To solve this problem, there are numerous variations, particularly contrast-limited adaptive histogram equalization (CLAHE), as cited in the literature, “Adaptive histogram equalization and its variation”, Computer vision, graphics, and image processing 39(3):355-368, by Pizer, Stephen M, E. Philip Amburn, John D. Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B. Zimmerman, and Karel Zuiderveld, 1987, and “Modified Contrast Limited Adaptive Histogram Equalization Based on Local Contrast Enhancement for Mammogram Images”, Mobile Communication and Power Engineering, Vinu V. Das and Yogesh Chaba, 296:397-403, Communications in Computer and Information Science, Berlin, Heidelberg: Springer Berlin. This is described in Heidelberg, http: / / doi.org / 10.1007 / 978-3-642-35864-7_60, by Mohan, Shelda, and M. Ravishankar, 2013. These variations require parameterization, and some studies have proposed an automated parameterization method.In the method described in “Automatic x-ray image contrast enhancement based on parameter auto-optimization”, Journal of Applied Clinical Medical Physics1 18(6):218-23, https: / / doi.org / 10.1002 / acm2, 2017, Pizer’s solution is combined with a high-pass filter, and their parameters are optimized to maximize the entropy of the processed image. The literature, “Improved Lung Nodule Visualization on Chest Radiographs Using Digital Filtering and Contrast Enhancement” 5(12):4 by Kwan, Benjamin Y M. and Hon Keung Kwan, 2011, suggests using HE after other processing operations or simply applying HE to a user-specified ROI.

[0009] The Pizer and Mohan methods propose improvements to histogram equalization (HE) to overcome the aforementioned problems. However, these methods must be parameterized, which may involve using default parameters that are unsuitable for each image, or it may be done manually by the radiologist.

[0010] Kwan's method proposes applying HE after a high-pass filter, and therefore the limitations of HE still exist. As an alternative, this proposes calculating HE for ROIs, but this requires the user to specify the ROI. Therefore, this task requires the intervention of a radiologist, which may be considered complex if the radiologist does not have empirical information on the location of the internal structures to be visualized.

[0011] The paper, “Automatic x-ray image contrast enhancement based on parameter auto-optimization”, Journal of Applied Clinical Medical Physics 18(6):218-23, by Qiu, Jianfeng, H. Harold Li, Tiezhi Zhang, Fangfang Ma, and Deshan Yang, 2017, proposes a variation of Pizer's method by combining it with a high-pass filter. It also describes an automatic parameterization method in which the parameters are optimized to maximize the entropy of the processed image.

[0012] Other methods for improving contrast also exist, for example, as described in the paper, “Contrast Enhancement of Medical X-ray Image Using Morphological Operators with Optimal Structuring Element”, ArXiv1905.08545[Cs,Eess],http: / / arxiv.org / abs / 1905.08545,2019, by Kushol, Rafsanjany, Md Nishat Raihan, Md Sirajus Salekin, and ABMAshikur Rahman. This paper also proposes an automated parameterization method.

[0013] In the Kushol and Qiu method, every pixel in an image influences the calculation of variables for optimizing the parameters. The results obtained in this way may not be optimal for specific regions of the image, and their distribution will be substantially different from that of the entire image, as is the case with HE.

[0014] Recent research has proposed providing diagnostic support based on neural networks, more specifically, image classification networks. Examples of this type of problem are described in the literature, “Abnormality Detection and Localization in Chest X-Rays Using Deep Convolutional Neural Networks”, ArXiv:1705.09850[Cs], 2017, http: / / arxiv.org / abs / 1705.09850, by Islam, Mohammad Tarqul, Md Abdul Aowal, Ahmed Tahseen Minhaz, and Khalid Asharaf, and in the literature, “CheXNet:Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning”, ArXiv:1711.05225[Cs,Stat], http: / / arxiv.org / abs / 1711.05225, 2017, by Rajpurkar, Pranav, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, et al.

[0015] Islam or Rajukar's methods do not improve the visibility of abnormalities in radiographic images, but rather estimate the presence of radiological abnormalities associated with pathology. Such techniques are still experimental, and the intervention of a radiologist is necessary to confirm the diagnosis.

[0016] Numerous visual explanation methods have been proposed to visually describe the results obtained from neural networks for image classification. These methods generate an influence map, assigning a numerical value to each pixel in the input image to represent the degree of influence that pixel had on the classification result. A higher numerical value indicates a greater impact that pixel had on the resulting classification. This is supported by the document “Grad-cam: Visual explanations from deep networks via gradient-based localization”, Proceedings of the IEEE international conference on computer vision, 618-626, 2017, by Selvaraju, Ramprasaath R. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra, “Layer-Wise Relavance Propagation for Deep Neural Network Architectures”, Information Science and Applications (ICISA) 2016, edited by Kuinam J. Kim and Nikolai Joukov, 376:913-22, Lecture Notes in Electrical Engineering, Singapore: Springer Singapore, https: / / doi.org / 10.1007 / 978-981-10-0557-2_87, by Binder, Alexander, Sebastian Bach,Gregoire Montavon, Klaus-Robert Mueller, and Wojciech Samek, “Learning Deep Features for Discriminative Localization,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2921-29, Las Vegas, NW, USA: IEEE.This is the case described in https: / / doi.org / 10.1109 / CVPR.2016.319, by Zhou, Bolei, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba, and in the literature "Visualizing and Understanding Convolutional Networks", ArXiv:1311.2901 [Cs], November, http: / / arxiv.org / abs / 1311.2901, 2016, by Zeiler, Matthew D., and Rob Fergus. When these methods are applied to neural networks for radiological image classification (depending on whether the image shows radiological abnormalities), the generated influence maps correspond to the influence maps of radiological abnormalities.

Prior Art Documents

Non-Patent Documents

[0017]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Direct Environment 7

Outdoor Tools 8

Outdoor Tools9

[0018] One objective of the present invention is to overcome the above-mentioned problems and, in particular, to improve post-processing operations on digital images to better highlight detected radiological abnormalities, thereby facilitating the work of radiologists and shortening diagnostic time. [Means for solving the problem]

[0019] Accordingly, proposed is a method for processing a digital radiographic image I(x,y) containing at least one radiographic anomaly, detected using a convolutional neural network trained to detect radiographic anomalies in radiographic examinations, according to one aspect of the present invention. The radiographic image I(x,y) is processed by at least one radiographic anomaly influence map C, which assigns a numerical value to each pixel of the radiographic image I(x,y) based on the intensity of each pixel I(x,y) and how much the pixel influenced the detection result of the radiographic anomaly k. k It is characterized by (x,y). This method is implemented by computer, - Radiological Anomaly Influence Map C k Normalize (x,y) to obtain the normalized radiological anomaly influence map C kn The step of giving (x,y), - Normalized radiological anomaly influence map C kn The steps include: fusing (x,y) to obtain a single fused influence map C(x,y), - A step of performing an improvement process on an image I(x,y) using an intensity histogram, wherein the intensity histogram is weighted by an influence map C(x,y) in which the contribution of each pixel is fused, Includes.

[0020] The method of the present invention can highlight detected abnormalities, simplify the work of radiologists, and shorten diagnostic time.

[0021] Literature, “Abnormality Detection and Localization in Chest X-Rays Using Deep Convolutional Neural Networks”, ArXiv:1705.09850[Cs], 2017, http: / / arxiv.org / abs / 1705.09850, by Islam, Mohammad Tarqul, Md Abdul Aowal, Ahmed Tahseen Minhaz, and Khalid Asharaf and References, “CheXNet:Radiologist-Level Pneumonia Detection on Chest Mehta,Tony Duan,Daisy Ding.et al. has disclosed a neural network for identifying images containing such radiological abnormalities.

[0022] Through training (or optimization or learning) of a neural network, the probability of existence p for each anomaly k is determined based on the input image. k If this probability is higher than a certain threshold π, for example π=0.5, then an anomaly is considered to have been detected.

[0023] In one embodiment, each abnormal influence map C k For the normalization of (x, y), the following affine transformation is used:

Equation

[0024] Through such normalization, all of the normalized map C kn (x, y) can be ensured to be within the same numerical range (in this case, 0 to 1). By this procedure, it becomes possible to balance the influence of various maps during the fusion step (a map including values from 0 to 100 will be much more influential than a map including values from 0 to 1).

[0025] According to one embodiment, the fused C(x, y) uses the average of the normalized radiological abnormal influence map C kn (x, y), which is weighted by the probability of existence p k of each abnormality using the following relationship:

Equation

[0026] Such a fusion has the advantage that one map for weighting the contribution of each pixel of the image in the calculation of the intensity histogram is obtained.

[0027] In one embodiment, the processing operation uses intensity histogram equalization, in which case the histogram calculation is modified by weighting the contribution of each pixel according to the following relationship:

number

number

number

[0028] Such processing operations have the advantage of improving the contrast of the image, especially in areas of the image that show detected anomalies, because the influence of the corresponding pixels in the histogram calculation becomes greater.

[0029] As a variation, the processing operation may use a variation of the method proposed in the literature, “Automatic x-ray image contrast enhancement based on parameter auto-optimization”, Journal of Applied Clinical Medical Physics 18(6):218-23, by Qiu, Jianfeng, H. Harold Li, Tiezhi Zhang, Fangfang Ma, and Deshan Yang, 2017, in which the calculation of the entropy of the processed image is modified using the following relationship:

number

number

number

[0030] Such processing operations improve the contrast of the image, particularly that of the areas of the image showing detected anomalies, by relatively increasing the contribution of the corresponding pixels in the calculation of the entropy of the processed image.

[0031] According to another aspect of the present invention, proposed is a computer program product for performing the steps of the above-described method when the program is executed on a computer, the program comprising program code instructions recorded on a computer-readable medium.

[0032] The present invention is described as entirely non-limiting examples and will be better understood by considering several embodiments illustrated by the accompanying drawings below. [Brief explanation of the drawing]

[0033] [Figure 1] A schematic representation of a computer implementation method for processing radiographic images according to one aspect of the present invention is provided. [Modes for carrying out the invention]

[0034] Figure 1 shows a method according to one embodiment of the present invention for processing a digital radiographic image I(x,y) containing at least one radiographic anomaly detected using a convolutional neural network trained to detect radiographic anomalies in radiographic examinations. The radiographic image I(x,y) is processed into at least one radiographic anomaly influence map C, which assigns a numerical value to each pixel (x,y) of the radiographic image I(x,y) indicating how much that pixel influenced the detection of the radiographic anomaly k. k It is characterized by (x,y).

[0035] The method is implemented by computer, - Radiological Anomaly Influence Map C k Normalize (x,y) to obtain the normalized radiological anomaly influence map C kn Step 3 to obtain (x,y), - Normalized radiological anomaly influence map C kn Step 4 involves fusing (x,y) to obtain the fused influence map C(x,y), - Step 5 involves performing an improvement process on image I(x,y) using an intensity histogram, in which the intensity histogram is weighted by an influence map C(x,y) in which the contribution of each pixel is fused during the calculation of the intensity histogram. Includes.

[0036] The method according to the present invention enables the automatic processing of radiographic images I(x,y), which improves the visibility of areas where abnormalities are detected. Therefore, the present invention facilitates pathological diagnosis based on radiological examinations, eliminating the need for intervention by radiologists or other users, and provides images with improved visibility of any radiological abnormalities, thereby making image interpretation easier for radiologists.

[0037] To this end, existing processing operations are modified to increase the impact of areas in the image where anomalies are detected. Therefore, the processing operations become more effective in these areas, making it easier to highlight detected anomalies.

[0038] This is achieved by weighting the contribution of each pixel in the image by the corresponding numerical values ​​of the fused influence map C(x,y) in the calculation of the intensity histogram.

[0039] Detection step 1 may use a convolutional neural network trained to detect radiological abnormalities that represent pathology in radiographic examinations.

[0040] Different detection methods can be used, as long as a suitable radiological anomaly influence map can be estimated.

[0041] The implementation and training of such neural networks are widely described in the literature. This type of network calculates the probability of existence p for each anomaly k based on the input image. k This is calculated, and when this probability is higher than a certain threshold π, for example π = 0.5, it is considered that an anomaly has been detected. If we represent all detected anomalies as K, p k k∈K only when >π This is the result.

[0042] In this type of network, it is often required that the input image has the correct resolution and the correct number of channels (1 for grayscale images, 3 for RGB images). If the input image has a different resolution, the image may be resized to the required resolution and the number of channels may be adjusted.

[0043] For example, (Rajpurkar et al. 2017) uses the DenseNet-121 network architecture (Huang, Gao, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger, 2017, “Densely Connected Convolutional Networks”, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2261-69, Honolulu, HI:IEEE, https: / / doi.org / 10.1109 / CVPR.2017.243), which takes a 3-channel image of size 224x224 as input. In this case, a radiographic image with any resolution and one channel is resized to 224x224, converted to three channels, and the same numerical value is repeated in each channel.

[0044] Step 2 of the influence estimation can be performed according to estimations from prior art, such as the one proposed in the literature, “Grad-cam: Visual explanations from deep networks via gradient-based localization”, Proceedings of the IEEE international conference on computer vision, 618-626, 2017, by Selvaraju, Ramprasaath R. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra, which is adaptable to multiple different architectures and generates a map at the same resolution as the output from the selected convolutional layer of the detection network architecture. The radiological anomaly influence map then must be resized to the original resolution of the image, for example, by bilinear interpolation.

[0045] C k(x,y) is used to show a radiological anomaly influence map corresponding to the detected anomaly k.

[0046] For example, if influence estimate 2 by Selvaraju, Ramprasaath R., Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra is applied to the final convolutional layer of detection by Rajpurkar, Pranav, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, and Daisy Ding, the output resolution is 7x7. Then the radiological anomaly influence map is resized to the size of the original image.

[0047] Any other radiological anomaly influence map estimation 2 that is compatible with the selected detection method may also be valid.

[0048] Each Radiological Abnormality Influence Map C k Regarding step 3, which normalizes (x,y), one possible implementation is to apply an affine transformation so that all the numbers are between 0 and 1:

number

[0049] Radiological Anomaly Influence Map C k (x,y) can be normalized in a different way.

[0050] Normalized radiological anomaly influence map C for each detected anomaly k. kn One solution for performing step 4, which involves fusing (x,y), is to take the average of maps weighted by the probability of each anomaly k:

number

[0051] As a variation, maps can be merged in different ways.

[0052] The processing step may use intensity histogram equalization, in which the histogram calculation is modified by weighting the contribution of each pixel according to the following relationship:

number

number

number

[0053] As a variation, the processing operation may use a modification of the method proposed in the literature, “Automatic x-ray image contrast enhancement based on parameter auto-optimization”, Journal of Applied Clinical Medical Physics 18(6):218-23, by Qiu, Jianfeng, H. Harold Li, Tiezhi Zhang, Fangfang Ma, and Deshan Yang, 2017, in which case the calculation of the entropy of the processed image is modified using the following relationship:

number

number

number

[0054] Process 5 can also be applied in a different way from the processing method described above, or to other known processing methods.

[0055] The present invention can be implemented in a computer program product that includes computer executable computer code stored on a computer-readable medium and designed to perform the method described above.

[0056] This invention can be implemented on a local computer or on a distributed network platform.

Claims

1. A method for detecting at least one radiological anomaly by processing a digital radiological image I(x,y) using a convolutional neural network trained to detect radiological anomalies in radiological examinations, wherein the radiological image I(x,y) is used to process a digital radiological image I(x,y) by processing a radiological image I(x,y) by processing the image, wherein the radiological image I(x,y) is used to process at least one radiological anomaly influence map C, which assigns a numerical value to each pixel (x,y) of the radiological image I(x,y) that represents how much the pixel (x,y) influenced the detection result of the radiological anomaly k. ｋ Characterized by (x, y), the method is implemented by a computer, - The aforementioned radiological abnormality influence map C ｋ (x, y) is normalized to obtain the normalized radiological anomaly influence map C. ｋｎ Step (3) gives (x, y), - The normalized radiological anomaly influence map C ｋｎ Step (4) merges (x, y) to give a single merged influence map C(x, y), - Step (5) of performing an improvement process on the image I(x,y) using an intensity histogram, wherein in the calculation of the intensity histogram, the intensity of each pixel (x,y) is weighted by the fused influence map C(x,y), Methods that include...

2. The normalization (3) of each abnormal influence map Ck(x,y) is the following affine transformation: [Math 1] Use During the ceremony, C ｋｎ (x, y) is the radiological abnormality influence map C mentioned above. ｋ This is a normalized radiological anomaly influence map of (x, y). The method according to claim 1.

3. The aforementioned fusion (4) C(x,y) has a probability of existence p for each anomaly using the following relationship. ｋ A weighted, normalized radiological anomaly influence map C ｋｎ The mean of (x, y) is given: [Math 2] During the ceremony, |K| represents the number of anomalies detected. p ｋ represents the probability p of the existence of each abnormality calculated by the convolutional neural network ｋ . The method according to claim 1 or 2.

4. The process (5) uses intensity histogram equalization, and the calculation of the intensity histogram is modified by weighting the intensity of each pixel (x, y) according to the following relationship: [Math 3] During the ceremony, H(u) represents the modified intensity histogram level for intensity u, [Math 4] is the indicator function [Math 5] This represents, I(x,y) represents the intensity of the pixel (x,y). The method according to any one of claims 1 to 3.

5. A computer program product comprising program code instructions recorded on a computer-readable medium, wherein the computer program product is used to perform the steps of the method described in any one of claims 1 to 4 when the program of the computer program product is executed on a computer.

Images