Calculation and display of local spectral covariance or local spectral covariance deficit to highlight relevant material transitions in spectral CT and MR.

JP2023552333A5Inactive Publication Date: 2024-12-04KONINKLIJKE PHILIPS NV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023532493
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-12-01
Filing Date
2021-11-28
Publication Date
2024-12-04
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Multispectral imaging technologies struggle to aggregate rich information from multiple spectral channels into a single view for efficient reading, as traditional images are often preferred due to established reading habits and expertise.

Method used

An apparatus and method for processing image data using local covariance matrices and superimposing variance and covariance values across monoenergetic images, enabling the identification of material transitions and enhancing spectral information visualization.

Benefits of technology

Enables the visualization of material transitions within multispectral images without disturbing the reference image, allowing for efficient reading and identification of subtle changes using color or gray overlays, reducing computational burden, and maintaining familiar viewing experiences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

The present invention relates to multispectral imaging. To improve the identification of relevant multispectral material transitions (especially those caused by injected contrast agents), an apparatus is proposed that uses local maxima of the variance and / or covariance of the intensities of a multi-channel image to locate the material transitions. Compared to gradient vectors, local variances are not directed and are less susceptible to noise. An alternative apparatus is proposed that uses local covariance defects of the intensities of a multi-channel image to locate the material transitions. The proposed alternative approach is independent of spatial drift across the image volume.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present invention generally relates to multispectral imaging, and more particularly to two alternative apparatuses and methods for processing image data of interest, including at least two single-energy images acquired from different spectral channels, as well as a medical imaging system and a computer program element. [Background technology]

[0002] Multispectral imaging is becoming increasingly common in clinical practice due to rapid advances in computer technology and the growing literature demonstrating significant advantages over conventional single-energy imaging. Spectral CT generates several spectral channels ranging from low to high energy levels, which are governed to varying degrees by, for example, the Compton effect and the photoelectric effect. These spectral channels are further translated into various representations: pure Compton images, pure photoelectric images, and pure (pseudo / virtual) single-energy images. A typical dual-layer or dual-energy CT scanner generates two independent spectral channels, while photon-counting CT scanners, MR scanners, or ultrasound (US) probes generate many N independent spectral channels.

[0003] However, the rich multispectral information per voxel provided by spectral images is difficult to consolidate into a single view for efficient reading. Furthermore, conventional images are often preferred for observation due to existing reading habits and accumulated expertise. [Overview of the project] [Problems that the invention aims to solve]

[0004] It is necessary to consolidate multispectral information into a single image. [Means for solving the problem]

[0005] The object of the present invention is solved by the subject matter of the independent claims, and further embodiments are incorporated in the dependent claims. It should be noted that the aspects of the present invention described below are further applicable to two alternative devices and methods, a medical imaging system, and a computer program element.

[0006] According to a first aspect of the present invention, there is provided an apparatus for processing image data of an object of interest including at least two single-energy images obtained from different spectral channels. The apparatus includes an input module, a calculation module, a rendering module, and an output module. The input module is configured to receive the image data of the object of interest. The calculation module is configured to determine a local covariance matrix at a plurality of image positions within at least two single-energy images. Each local covariance matrix is a matrix of local variances and local covariances between image intensities at one of the plurality of image positions within at least two single-energy images. The rendering module is configured to overlay the values of local variances and / or local covariances at the plurality of image positions with one or more of at least two single-energy images. The output module is configured to provide the result of the overlay.

[0007] In one example, the image data is obtained by a dual or multi-energy CT scanner. For example, in dual-energy CT, additional attenuation measurements are obtained using a second X-ray spectrum (i.e., a second "energy"), thereby enabling the differentiation of a number of materials. Alternatively, this enables the quantification of the mass densities of two or three materials in a mixture with a known elemental composition.

[0008] In another example, the image data is obtained by multi-parametric magnetic resonance imaging (MRI) including the following sequences, namely, T1-weighted image, T2-weighted image, diffusion-weighted image (DWI), and dynamic contrast-enhanced imaging (DCEI).

[0009] In another example, the image data is acquired by a set of ultrasonic (US) modes such as A / B / C / M mode, or a set of harmonic modes.

[0010] Local covariance matrices at a plurality of image positions, such as pixel positions or voxel positions in the first and second single-energy images, are determined. Each local covariance matrix is a matrix of local variances and local covariances between image intensities at one of the plurality of image positions in the first and second single-energy images. For each image position, the local variances and covariances are calculated using an image patch of adjacent pixels or voxels. An "image patch" refers to a patch or group of pixels or voxels having a specific size, shape, and location corresponding to the image. For example, the image patch has a predetermined pixel width / height (e.g., 7×7, 9×9, 11×11, etc.), and the location of each image patch is defined based on one or more central pixels or voxels.

[0011] The local variances and covariances are positive definite numbers and reach their maximum values at the locations of material transitions. Local maxima of the variances and / or covariances of the intensities of the multi-channel image are used to identify the positions of material transitions. These material transitions are caused by a contrast agent and indicate blood perfusion.

[0012] The covariance matrix values characterize specific material transition types such as adipose tissue, muscle tissue, air, contrast-agent-tagged materials, etc. For example, FIGS. 2A-2C demonstrate that the spectral local variance in the "photograph" image and the "scattered" image is sensitive to the folds of the colon submerged in contrast-tagged fecal residue, while the spectral local covariance particularly means the material transition from the tag to air.

[0013] The values of the local variances and / or local covariances at the plurality of image positions are overlaid with the first single-energy image and / or the second single-energy image to generate an overlaid display of the first single-energy image and / or the second single-energy image. The overlaid display of the first single-energy image and / or the second single-energy image is visualized on a display.

[0014] Local variances and / or covariances are discretely selected only for specific transition types (e.g., transitions between air and tissue), and the trace of the matrix is ​​used as a weight for representation. Alternatively, local variances and / or covariances are continuously weighted due to application-specific relevances.

[0015] In one example, the local variance and / or covariance values ​​are superimposed on a standard slice image using color coding.

[0016] In another example, material transition cues are communicated using gray values, such as a gradual black-to-white color transition, ensuring a familiar viewing experience similar to conventional images, as well as possible deployment to commonly available black-and-white monitors. Overlay presence / absence and weighting are interactively controlled or toggled by the operator (hereinafter referred to as "user").

[0017] Further visualization options include three-dimensional (3D) rendering of the magnitude of local variance or regression values ​​into an interactively rotatable virtual display.

[0018] In other words, related material transitions are superimposed by color or grayscale values, thereby conveying additional spectral information. In this way, the user does not need to read two or even all of a set of different image representations, but can evaluate only one image type, i.e., one spectral band. Certain material transitions that are not discernible in conventional images are discernible in spectral image channels. Furthermore, since only transition lines (rather than entire areas) are marked, the reference image is not disturbed by superimposition that occludes areas, and standard reading expertise does not need to be changed.

[0019] According to one embodiment of the present invention, the computing module is configured to compute each local variance matrix by a Gaussian convolution operation.

[0020] In other words, local variances and covariances are computed by a Gaussian convolution, i.e., a smoothing operation. Gaussian filtering is computed for various kernel widths corresponding to the scale space, thereby computed the entire vector of covariance matrices for various values ​​of the covariance scale space tensor.

[0021] According to one embodiment of the present invention, the computing module is further configured to determine the number of eigenvalues ​​for at least one of the local covariance matrices in order to determine the amount of material in which a material transition occurs.

[0022] In other words, the computation module is further configured to determine the number of eigenvalues ​​for at least one of the local covariance matrices, and based on the number of eigenvalues, classify at least one of the local covariance matrices into material transitions among N materials, where N is 2 or greater.

[0023] For example, a single eigenvalue of the local variance matrix represents a transition between two materials (e.g., between air and soft tissue).

[0024] In one example, two eigenvalues ​​of the local variance matrix represent three material transitions (e.g., between air, soft tissue, and contrast-tagged material).

[0025] According to one embodiment of the present invention, the apparatus further includes a classifier module configured to apply a pre-trained classifier to classify each of the local covariance matrices into material transition types.

[0026] Examples of classifiers include, but are not limited to, linear, maximum likelihood, random forest, support vector machine, or artificial neural networks, and are used to classify covariances at any given location into material transition types.

[0027] The classification results are one or more type-specific scalar values ​​output from each local covariance matrix or covariance scale space tensor.

[0028] According to one embodiment of the present invention, the pre-trained classifier includes at least one of an artificial neural network, a support vector machine, a maximum likelihood model, or a random forest.

[0029] According to one embodiment of the present invention, the multiple image locations include locations of material transitions.

[0030] The location of material transitions is identified based on edge detection. Edge detection is the name of a set of mathematical methods aimed at identifying points in a digital image where the image brightness changes abruptly or, more formally, has a discontinuity. Points where the image brightness changes abruptly are generally organized into a set of curved segments called edges. Various techniques such as Prewitt edge detection, Laplacian edge detection, LOG edge detection, and Canny edge detection are used for edge detection.

[0031] In this way, the local covariance matrix is ​​calculated only at the locations of material transitions in order to identify the relevant multispectral material transitions. As a result, the computational burden is reduced.

[0032] According to one embodiment of the present invention, the apparatus further includes a display device configured to display the result of the superposition.

[0033] In one example, the image data includes at least one of the following: image data acquired by spectral computed tomography (CT) or image data acquired by spectral magnetic resonance imaging (MRI).

[0034] According to a second aspect of the present invention, an apparatus is provided for processing image data of interest, comprising at least two single-energy images acquired from different spectral channels. The apparatus includes an input module, a computation module, and an output module. The input module is configured to receive image data of interest. The computation module is configured to determine local covariance matrices at a plurality of image locations within the at least two single-energy images and to determine local covariance defects based on a comparison between the product of local variances across all combinations of spectral channels and the product of local covariances. Each local covariance matrix is ​​a matrix of local variances and local covariances between image intensities at one of the plurality of image locations within the at least two single-energy images. The output module is configured to provide the local covariance defects as machine learning features for superimposing them with one or more of the at least two single-energy images by a graphical cue and / or for performing material classification.

[0035] In other words, alternative methods for processing image data are proposed. Unlike the methods expressed in the first aspect of the present invention for determining local variance and / or local covariance, the alternative methods in the second aspect of the present invention perform voxel- or pixel-level calculations of inter-channel covariance defects. Multispectral covariance defects are mapped to individual spectral channels or to color overlays on so-called conventional slice images. Alternatively or additionally, inter-channel covariance defects are used as machine learning features for performing material classification.

[0036] The proposed method is independent of spatial drift across image volumes. As will be discussed below, it further enables high-speed computation using standard filters suitable for massively parallel processing, for example, on a GPU. This alternative method does not require eigenvalue decomposition or singular value decomposition.

[0037] According to one embodiment of the present invention, the apparatus further includes a rendering module configured to superimpose local covariance loss values ​​at multiple image locations with one or more of at least two single-energy images.

[0038] Multispectral covariance defects are communicated to the reader via graphical cues, such as slice-level color overlay, maximum intensity projection (MIP), direct volume rendering (DVR), and 3D texture mapping.

[0039] By superimposing local covariance defects with one or more single-energy images, it is possible to identify and navigate spectrally prominent locations within a multispectral image volume, such as those generated by photon counting CT. This allows the user to evaluate only one image type, i.e., one spectral band, saving time compared to reading all channel image volumes separately. Furthermore, when various covariance defects are superimposed with different hues and opacities according to their magnitude, this enables an unobstructed view of smooth areas where dispersion disappears, while flagging different material transitions. Thus, although differences between different spectral channels are subtle and may be further obscured by the different overall dynamic scaling and offsets of the multispectral channels, the proposed superimposed image still allows the user to identify different material transitions within the superimposed image. In particular, the user can identify subtle changes in injected contrast agent that indicate differences in local blood perfusion.

[0040] According to one embodiment of the present invention, the apparatus further includes a display device configured to display the result of the superposition.

[0041] According to one embodiment of the present invention, the apparatus further includes a classifier module configured to apply a pre-trained classifier to perform material classification at one or more of a plurality of image locations based on local covariance defects.

[0042] Examples of pre-trained classifiers include, but are not limited to, linear discriminant analysis-based classifiers (LDAs), support vector machines (SVMs), random forest classifiers, and convolutional neural networks (CNNs).

[0043] The accuracy of the classification is improved by adding local covariance defects. In two spectral channels, material classification at the voxel or pixel level relies on two single energy values ​​for each voxel / pixel and the value of the local covariance defect, thereby yielding three features per voxel / pixel. In spectral multiband images, local covariance defects are a computationally efficient generalization of gradient magnitude and correlation.

[0044] According to one embodiment of the present invention, the computation module is configured to determine local covariance loss in image space or projection space.

[0045] According to a third aspect of the present invention, a medical imaging system is provided. The medical imaging system includes a scanner and an apparatus according to the first aspect and related examples or the second aspect and related examples. The scanner is configured to scan an object of interest to acquire image data of the object of interest. The apparatus is configured to process the image data of the object of interest.

[0046] A fourth aspect of the present invention provides a computer implementation for processing image data of interest, which includes at least two single-energy images acquired from different spectral channels. The computer implementation is as follows: The steps include receiving image data of the subject of interest, A step of determining local covariance matrices at multiple image locations within at least two single-energy images, wherein each local covariance matrix is ​​a matrix of local variances and local covariances between image intensities at one of the multiple image locations within at least two single-energy images. A step of superimposing the local variance and / or local covariance values ​​at multiple image locations with one or more of at least two single-energy images, The process includes the step of providing the result of superposition.

[0047] In one example, the step of determining the local covariance matrix further involves the step of calculating each local covariance matrix by a Gaussian convolution operation.

[0048] In one example, the step of determining the local covariance matrix further includes the step of determining the number of eigenvalues ​​for at least one of the local covariance matrices in order to determine the amount of material in which material transitions occur.

[0049] In one example, this method further includes the step of applying a pre-trained classifier to classify each of the local covariance matrices into material transition types.

[0050] According to a fifth aspect of the present invention, a computer implementation for processing image data of interest, comprising at least two single-energy images obtained from different spectral channels of different energies, wherein the computer implementation is The steps include receiving image data of the subject of interest, A step of determining local covariance matrices at multiple image locations within at least two single-energy images, wherein each local covariance matrix is ​​a matrix of local variances and local covariances between image intensities at one of the multiple image locations within at least two single-energy images. A step of determining local multispectral covariance defects based on a comparison between the product of local variances and the product of local covariances across all combinations of spectral channels, The process includes the step of providing local covariance defects, preferably, so that they are communicated by a graphical cue and / or as machine learning features.

[0051] In one example, the computer implementation method further includes the step of superimposing the values ​​of local covariance loss at multiple image locations with one or more of at least two single-energy images.

[0052] In one example, the computer implementation method further includes a step of displaying the superposition results.

[0053] In one example, the computer implementation method further includes the step of applying a pre-trained classifier to perform material classification at one or more of a plurality of image locations based on local covariance defects.

[0054] In one example, the computer implementation method further includes the step of determining local covariance defects in image space or projection space.

[0055] According to another aspect of the present invention, a computer program element is provided which, during execution, is configured to perform steps of the method relating to a third aspect and related examples.

[0056] In one example, a computer-readable medium containing computer program elements is provided.

[0057] For the sake of advantage, any benefit provided by any of the above-described embodiments is equally applicable to all the other embodiments, and vice versa.

[0058] As used herein, the term “learning” in the context of machine learning refers to the identification and training of an appropriate algorithm for performing a task of interest. The term “learning” includes, but is not limited to, federative learning, classification learning, clustering, and numerical prediction.

[0059] As used herein, the term “machine learning” refers to the field of computer science that studies the design of computer programs that can derive patterns, regularities, or rules from past experience to develop appropriate responses to future data or to describe data in some meaningful way.

[0060] As used herein, the term “data-driven model” in the context of machine learning refers to a suitable algorithm that is learned based on appropriate training data.

[0061] As used herein, the term “module” means, is part of, or includes application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped) and / or memory (shared, dedicated, or grouped) that run one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the functions described.

[0062] It should be recognized that all combinations of the aforementioned concepts and additional concepts discussed in more detail below (provided that such concepts are not contradictory) are considered to be part of the subject matter of the invention disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter of the invention disclosed herein.

[0063] These and other aspects of the present invention will become apparent and will be clarified by reference to the embodiments described below.

[0064] In drawings, similar reference letters generally refer to the same part throughout different drawings. Furthermore, drawings are not necessarily to scale; instead, the focus is generally on illustrating the principles of the invention. [Brief explanation of the drawing]

[0065] [Figure 1] This is a flowchart illustrating the computer implementation method. [Figure 2A] This figure shows an exemplary "photograph" image with superimposed local variances. [Figure 2B] This figure shows an exemplary "scattering" image where the magnitudes of local dispersion are superimposed. [Figure 2C] This figure shows an exemplary "scattering" image with the local covariance magnitudes superimposed. [Figure 3A] This figure shows an exemplary composite image of a "photographic" image and a "scattered" image with local dispersion magnitude. [Figure 3B] This figure shows a further exemplary composite image obtained by reducing the spectral local covariance from the exemplary composite image in Figure 3A. [Figure 4] This is a flowchart illustrating an alternative computer implementation method. [Figure 5A] This figure shows an exemplary covariance-defect image volume, where multispectral covariance defects are superimposed by 3D text mapping. [Figure 5B] This figure shows an exemplary covariance-defect image volume, where multispectral covariance defects are represented using rotational maximum intensity projection (MIP). [Figure 6] This is a schematic diagram showing an example of the apparatus. [Figure 7] This diagram schematically shows examples of alternative devices. [Figure 8] This is a schematic diagram illustrating an example of a medical imaging system. [Modes for carrying out the invention]

[0066] The present application's approach is described below in relation to abdominal spectral CT scanning of contrast-tagged fecal residue in the colon. The following detailed description will be illustrated using an application to CT virtual colonoscopy, but those skilled in the art will recognize that the methods, apparatus, and medical imaging systems described above and below are adaptable to other areas of interest, e.g., vascular walls, and other imaging modalities, e.g., MRI. Accordingly, the examples described below are provided without loss of generality to the claimed invention and without limitation to the claimed invention.

[0067] Figure 1 shows a flowchart of a computerized implementation 200 according to some embodiments of the present disclosure. A computerized implementation 200 is proposed for processing image data of interest, which includes at least two single-energy images, such as a first single-energy image acquired at a first energy and a second single-energy image acquired at a second energy different from the first energy.

[0068] Computer implementation method 200 is implemented in a set of logic instructions stored in a non-temporary machine or computer-readable storage medium such as random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc., as a device, module, or related component; in configurable logic such as programmable logic arrays (PLAs), field-programmable gate arrays (FPGAs), composite programmable logic circuits (CPLDs), etc.; in fixed-function hardware logic using circuit technologies such as application-specific integrated circuits (ASICs), complementary metal-oxide-semiconductor (CMOS), or transistor-transistor logic (TTL) technology, etc., or in any combination thereof. For example, computer program code for performing the operations shown in method 200 is written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java®, Smalltalk®, C++, Python, etc., and conventional procedural programming languages ​​such as the C programming language or similar programming languages.

[0069] In step 210, i.e., step a), the image data of interest is received.

[0070] Medical images are either two-dimensional images containing image pixel data or three-dimensional images containing image voxel data.

[0071] In one example, the object of interest is the colon. In another example, the object of interest is the lumen of the vascular system.

[0072] In one example, image data is acquired by spectral CT. For instance, the basic idea of ​​dual-energy imaging in CT is to acquire two datasets at low and high energy levels and use the paired datasets to estimate additional information about the patient. The physical basis of dual-energy imaging involves two main mechanisms of X-ray interaction with matter in the clinically relevant diagnostic energy range of 30 keV to 140 keV, the two interactions being photoelectric absorption and Compton scattering, each having its own functional dependence on X-ray energy. Photoelectric absorption is a rapidly decreasing energy function for producing a "photographic" image, while Compton scattering is a slower function of energy for producing a "scattered" image.

[0073] In another example, image data is acquired by multiparametric MRI.

[0074] In step 220, i.e., step b), a local covariance matrix is ​​determined for multiple image locations within the first and second single-energy images. Each local covariance matrix is ​​a matrix of local variances and local covariances between image intensities at one of the multiple image locations within the first and second single-energy images. For the local variance at each pixel or voxel location, statistics of small patches of adjacent pixels or voxels, also called image patches, are used. Each image patch represents a single pixel or group of pixels in a two-dimensional medical image, or a single voxel or group of voxels in a three-dimensional medical image. For example, an image patch may have a given pixel width / height (e.g., 7x7, 8x8, 9x9, etc.), and the location of each image patch may be defined based on one or more central pixels or voxels.

[0075] In one example, for two spectral channels, the local covariance matrix has a 2x2 matrix of covariances, three of which are independent.

[0076] In one example, for the case of N (N>2) spectral channels, the local covariance matrix has an N×N matrix of covariances.

[0077] Optionally, the local variance and covariance are calculated by a Gaussian convolution operation, var ii (x)=G σ *(I i (x)·I i (x))-(G σ *(I i (x)·G*(I i (x)) cov ik (x)=G σ *(I i (x)·I k (x))-(G σ *(I i (x)·G*(I k (x)) Here, G σ is the Gaussian convolution, σ is the kernel width of the Gaussian filter, I i (x) is the image intensity at a specific image position x in spectral band i, and I k (x) is the image intensity at a specific image position x in spectral band k.

[0078] The Gaussian filter G σ is calculated for various kernel widths σ corresponding to the scale space, whereby the entire vector of the covariance matrix is calculated for various values of σ to obtain the covariance scale space tensor.

[0079] The local variance and covariance are positive definite numbers and reach their maximum values at the locations of material transitions. The softness of the filter response is affected by the Gaussian filter width σ.

[0080] Compared to the gradient vector, the local variance is not directional and is less susceptible to noise.

[0081] Optionally, this method further includes the step of determining the number of eigenvalues ​​for at least one of the local covariance matrices in order to determine the amount of material in which a material transition occurs.

[0082] For example, in dual-energy CT, the two spectral bands ("photographic" image and "scattering" image) yield a 2x2 local covariance matrix. A single eigenvalue in the local covariance matrix represents two material transitions, while two eigenvalues ​​represent three material transitions.

[0083] In step 230, i.e., step c), the values ​​of local variance and / or local covariance at multiple image locations are superimposed on the first single-energy image and / or the second single-energy image.

[0084] In one example, the values ​​of local variance and / or local covariance are superimposed on a standard slice image using color coding.

[0085] Another visualization option is a 3D rendering of the magnitude of local variance or regression values ​​into an interactively rotatable virtual display.

[0086] As a further option, the transition queue can be communicated using gray values ​​(gradual black-to-white color transitions, rather than color overlays), ensuring a familiar viewing experience similar to traditional images, as well as the possibility of deployment on commonly available black-and-white monitors. Overlay (on / off) and its weight are interactively controlled / toggled by the user.

[0087] For example, Figures 2A and 2C show an example of two spectral bands ("photographic" image and "scattered" image) from an abdominal spectral CT scan of contrast-tagged fecal residue in the colon.

[0088] In particular, Figure 2A shows an exemplary "photographic" image in which the magnitude of local dispersion, shown as a brighter contour, is superimposed. This image demonstrates that the spectral local dispersion in the "photographic" image responds sensitively to the folds of the colon submerged in contrast-tagged fecal residue.

[0089] Figure 2B shows an exemplary "scattering" image with superimposed local dispersion magnitudes, indicated by brighter contours. This image demonstrates that spectral local dispersion in the "scattering" image is also sensitive to colonic folds submerged in contrast-tagged fecal residue.

[0090] Figure 2C shows an exemplary “scattering” image with the magnitudes of local covariance superimposed, indicated by brighter contours. This image demonstrates that spectral local covariance, in particular, signifies material transitions from the tag to the air.

[0091] Since the spectral local dispersion in both the "photographic" and "scattered" images is sensitive to the colonic folds submerged in contrast-tagged stool residue, these two images are combined to highlight the colonic folds submerged in contrast-tagged stool residue. An exemplary combined image is shown in Figure 3A.

[0092] Additionally, the contours of the colonic folds submerged in contrast-tagged fecal residue are further enhanced from the composite image in Figure 3A, particularly by subtracting the spectral local covariance, which represents the material transition from the tag to the air. An exemplary subtraction result is shown in Figure 3B.

[0093] Returning to Figure 1, in step 240, i.e., step d), the superposition result is provided, for example, to a display (e.g., an internal screen, a connected monitor or projector) or to file storage (e.g., a hard drive or solid-state drive).

[0094] Unless otherwise explicitly stated, it should also be understood that in any method claimed herein that includes two or more steps or actions, the order of the steps or actions of the method is not necessarily limited to the order in which the steps or actions of the method are enumerated.

[0095] The proposed method allows users to evaluate only one image type, i.e., one spectral band, without having to read two or even all of a set of different image representations. Additionally, relevant material transitions are superimposed with color or gray values, thereby conveying additional spectral information. Certain material transitions that are not discernible in conventional images are discernible in spectral image channels. Furthermore, since only transition lines (rather than entire regions) are marked, the reference image is not disturbed by superimpositions that occlude regions, and standard reading expertise does not need to be altered.

[0096] Optionally, a pre-trained classifier is applied to classify each of the local covariance matrices into a material transition type.

[0097] In one example, the classifier is a pre-trained machine learning model. The machine learning model is pre-trained on historical patient data that can be retrieved from image repositories in the same hospital or other hospitals. Preferably, a supervised learning method is used in which the historical images are pre-labeled by an experienced clinician. The labeling provides data including local covariance matrices at multiple image locations of a large number of images and corresponding classified material transition types (e.g., transitions between air-tissue, tissue-contrast-tagged material, or others).

[0098] Training a classifier involves the following steps, namely receiving training data and applying the classifier to the training data, in one or more iterations. As a result of this application, a pre-trained classifier is obtained, which is then used for deployment. In deployment, new image data is applied to the pre-trained classifier to obtain the classification result for this new data.

[0099] For example, a neural network model, also known as an artificial neural network (ANN), is used as an example of a classifier. However, other machine learning techniques such as support vector machines, maximum likelihood, and random forests may be used instead of neural networks.

[0100] A trained classifier attempts to approximate the correlation between the local covariance matrix and the material transition type. This approximation is achieved through a learning or training process in which parameters that themselves form a high-dimensional space are tuned using optimization techniques based on the training data.

[0101] More specifically, the classifier is implemented as an artificial neural network ("ANN"). The ANN can operate in two modes: a "training mode / phase" and an "expansion mode / phase". In training mode, an initial model of the ANN is trained on a set of training data to create a trained ANN model. In expansion mode, the pre-trained ANN model is fed new, untrained data so that it can operate in normal use. Training mode is either a one-time operation or it can be continued with repeated training phases to improve performance. Everything described so far regarding the two modes is applicable to any type of machine learning algorithm and is not limited to ANNs.

[0102] An ANN consists of a set of interconnected nodes organized into layers. The ANN includes an output layer and an input layer. The input layer is a matrix whose size (rows and columns) matches the size of the training input local covariance matrix. The output layer is a vector or matrix whose size matches the size chosen for the material transition type.

[0103] The ANN preferably has a deep learning architecture, that is, it has at least one, preferably two or more, hidden layers between the output layer and the input layer.

[0104] Each node is associated with a numerical value called a "weight," which represents how it responds to inputs from previous nodes in the preceding layer.

[0105] The set of weights defines the configuration of the ANN. During the training phase, the initial configuration is adjusted based on the training data using training algorithms such as forward-backward ("FB") propagation, other optimization methods, or other gradient descent methods. The gradient is taken with respect to the parameters of the objective function.

[0106] The training mode is preferably supervised, i.e., based on annotated training data. The annotated training data includes pairs or training data items. For each pair, one item is training input data, and the other item is target training data that is known a priori to be correctly associated with the training input data item. This association is defined by annotations, which are preferably provided by human experts. The training pairs include a local covariance matrix as training input and are associated with material transition types as output.

[0107] In training mode, preferably, a large number of such pairs are applied to the input layer and propagated through the ANN until the output appears in the output layer. Initially, the output generally differs from the target. During optimization, the initial configuration is readjusted for all pairs to achieve a good match between the input training data and their respective targets. The match is measured via a similarity measure formulated with respect to the objective function or cost function. The objective is to tune the parameters to produce low cost, i.e., a good match.

[0108] The training dataset is applied to the initially configured ANN and then processed according to a learning algorithm such as the FB propagation algorithm, as previously described. At the end of the training phase, the then pre-trained ANN is used in the unfolding phase to compute material transition types for new data, i.e., newly acquired images that were not present in the training data.

[0109] The classification results are one or more type-specific scalar values ​​output from each local covariance matrix or covariance scale space tensor.

[0110] In one example, local variances are discretely selected only for specific transition types, and the matrix trace is used as a weight for representation.

[0111] Instead of a "hard" classification, local variances and / or covariances are continuously weighted, for example, by regression rather than classification, due to application-specific relevances.

[0112] Figure 4 shows a flowchart of an alternative embodiment of computer implementation 400 for processing image data of interest, which includes at least two single-energy images acquired from different spectral channels at different energies.

[0113] In step 410, the image data of interest is received.

[0114] In step 420, local covariance matrices are determined for multiple image locations within the first and second single-energy images. Each local covariance matrix is ​​a matrix of local variances and local covariances between image intensities at one of the multiple image locations within the first and second single-energy images. The local covariance matrices are determined according to the method described above with respect to step 220 in Figure 1.

[0115] In step 430, the local multispectral covariance defect is determined based on a comparison between the product of local variances across all combinations of spectral channels and the product of local covariances.

[0116] As discussed above, local variance and covariance are calculated using a spatially symmetric Gaussian convolution filter or a binomial convolution filter (or multiplication in Fourier space).

[0117] For the two spectral channels i and k, the local variance and covariance are calculated by the following Gaussian convolution operation: var ii (x=G σ *(I i (x)·I i (x)-(G σ *(I i (x)·G*(I i (x)) cov ik (x=G σ *(I i (x)·I k (x)-(G σ *(I i (x)·G*(I k (x)) Here is the G σ is a Gaussian convolution, where σ is the kernel width of the Gaussian filter, and I i (x) is the image intensity of spectral band i at a specific image position x, where I k (x) is the image intensity of the spectral band k at a specific image position x.

[0118] The correlation loss between the two spectral channels is Correlation loss = 1 - (cov ik (x)·cov ik (x) / (var ii (x)·var kk (x)) It is defined as follows.

[0119] However, as the denominator approaches zero, the variance disappears locally, making the correlation deficit numerically unstable.

[0120] Therefore, this disclosure proposes considering local covariance defects that are positive semidefinite. A local covariance defect at a specific image position x is: missing covariance=cov ik (x)·cov ik (x)-var ii (x)·var kk It is defined as (x)≧0.

[0121] Optionally, the sign of the cross-covariance may be multiplied to obtain the following signed covariance deficit. Signed missing covariance=sign(cov ik (x) · Covariance deficit

[0122] Covariance deficits are positive semi-definite and vanish smoothly without singularities at the point where local variances vanish.

[0123] For N (N>2) spectral channels, the following quadratic matrix represents the covariance. C={[I i (x)-m i ]·[I k (x)-m k ] T} Considering this, here, I i (x) is the image intensity of a specific image position x in the spectral band i from N spectral channels, where I k (x) is the image intensity of a specific image position x in the spectral band k from N spectral channels, and m i This is the local average value in spectral band i, and mk This is the local mean value in spectral band k.

[0124] This results in N(N-1) / 2 independent covariance defects being added to the multispectral covariance defect, each responding regardless of which spectral channel provides the added information.

[0125] Covariance deficits are,

number

number

[0126] As an alternative formulation, the Kronecker product

number

number

[0127] In some embodiments, addition is efficiently performed by directly interleaving with Gaussian convolution operations, for example, using superposition addition techniques, to take advantage of their inherent parallelism and small memory footprint. Thus, the computational complexity is O(N). 2 M) runtime and O(M) memory, where M is the number of voxels in the image.

[0128] In the example above, the calculation of multispectral covariance defect is performed in image space. Alternatively, the calculation of multispectral covariance defect is performed in projection space before reconstruction of the image volume. In this case, local variance and covariance are calculated using the fitted anisotropic smoothing filter shape. Finally, the projected covariance defect is reconstructed in image space.

[0129] In step 440, the local covariance defect is preferably provided as a machine learning feature for superimposing onto one or more of at least two single-energy images and / or for performing material classification.

[0130] One option involves overlaying local covariance loss values ​​at multiple image locations with one or more single-energy images from at least two sources. For example, multispectral covariance loss can be overlaid using graphical cues, such as slice-level color overlay, maximum intensity projection (MIP), direct volume rendering (DVR), or 3D texture mapping.

[0131] Of particular interest is the mapping of multispectral covariance defects to individual spectral channels or to color overlays on so-called conventional slice images. Through construction, covariance defects are highest at material transition locations. Therefore, when various covariance defects are overlaid with different hues and opacities according to their size, this allows for an unobstructed view of smooth regions where dispersion disappears, while flagging different material transitions.

[0132] Figures 5A and 5B show two exemplary covariance-deficient image volumes using a 9×9×9 binomial smoothing kernel.

[0133] In the example shown in Figure 5A, multispectral covariance defects are superimposed by 3D text mapping. The dark gray areas in Figure 5A correspond to the blue areas representing the disappearance of covariance defects. The light gray areas in Figure 5A correspond to the yellow areas representing significant covariance defects corresponding to the added multispectral information.

[0134] In the example shown in Figure 5B, the covariance deficit is represented using the rotational maximum intensity projection (MIP).

[0135] For specific applications, it is beneficial to evaluate or present covariance deficits only in selected regions of interest (e.g., myocardium). For example, augmentation may be performed using organ morphology models.

[0136] Another option is to apply a pre-trained classifier (e.g., SVM, CNN, etc.) to perform material classification at one or more of multiple image locations based on local covariance defects. The local covariance defects can be supplemented with spectral information to improve classification accuracy. For example, the classification relies on a single energy value for each pixel / voxel (e.g., two single energy values ​​in a 2-channel spectral CT) and the value of the local covariance defect, resulting in numerous features for each pixel. Of course, one or more other features, such as local mean values ​​and local covariance matrices, may be added to each pixel / voxel.

[0137] Figure 6 schematically shows an example of an apparatus 100 for processing image data of interest, which includes a first single-energy image acquired at a first energy and a second single-energy image acquired at a second energy different from the first energy.

[0138] The apparatus 100 includes an input module 110, a computing module 120, a rendering module 130, and an output module 140. Each module is part of or includes an application-specific integrated circuit (ASIC), electronic circuitry, a processor (shared, dedicated, or grouped) and / or memory (shared, dedicated, or grouped) that runs one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the functions described.

[0139] Apparatus 100 is any computing device such as a mobile device, laptop and desktop computer, wearable computing device, and other computing devices suitable for processing image data.

[0140] The input module 110 is configured to receive image data of interest. Medical images are either two-dimensional images containing image pixel data or three-dimensional images containing image voxel data. Examples of imaging modalities include, but are not limited to, spectral CT or multiparametric MRI.

[0141] The computation module 120 is configured to determine local covariance matrices at multiple image locations within first and second single-energy images. These multiple image locations include material transition locations. Each local covariance matrix is ​​a matrix of local variances and local covariances between image intensities at one of the multiple image locations within the first and second single-energy images. For each pixel or voxel location, statistics of small patches of adjacent pixels or voxels, i.e., image patches, are used for the calculation.

[0142] Optionally, the computation module 120 is configured to compute each local variance matrix by Gaussian convolution.

[0143] Optionally, the computation module 120 is further configured to determine the number of eigenvalues ​​for at least one of the local covariance matrices in order to determine the amount of material at which material transitions occur.

[0144] For example, a single eigenvalue of the local dispersion matrix could represent a transition between two materials, such as air and soft tissue.

[0145] In one example, two eigenvalues ​​of the local dispersion matrix might represent three material transitions, for instance, between air, soft tissue, and contrast-tagged material.

[0146] The rendering module 130 is configured to superimpose local variance and / or local covariance values ​​(e.g., color values ​​or gray values) at multiple image locations onto a first single-energy image and / or a second single-energy image. Exemplary superposition results are shown in Figures 2A-2C and 3A-3C.

[0147] The output module 140 is configured to provide the superposition results to, for example, a display (e.g., a built-in screen, a connected monitor, or a projector) or to file storage (e.g., a hard drive or a solid-state drive).

[0148] Optionally, the device 100 further includes a classifier module configured to apply a pre-trained classifier such as an artificial neural network, support vector machine, maximum likelihood, or random forest to classify the local covariance matrix into material transition types.

[0149] Optionally, the apparatus includes a display device (not shown) configured to display the superposition results.

[0150] Figure 7 schematically shows an alternative embodiment of the device 500, which includes an input module 510, a calculation module 520, and an output module 530.

[0151] Input module 510 is substantially functionally equivalent to input module 110 shown in Figure 6.

[0152] The computation module 520 is configured to determine local covariance matrices at multiple image locations within the first and second single-energy images. Each local covariance matrix is ​​a matrix of local variances and local covariances between image intensities at one of multiple image locations within at least two single-energy images. The local covariance matrices are determined according to the method described with respect to step 220 in Figure 1.

[0153] The computation module 520 is further configured to determine local covariance defects based on a comparison between the product of local variances and the product of local covariances across all combinations of spectral channels. The local covariance defects are determined according to the method described with respect to step 430 in Figure 4. The computation module is configured to determine local covariance defects in image space or projection space.

[0154] The output module 530 is configured to provide local covariance defects as machine learning features for superimposing them, preferably, with one or more of at least two single-energy images and / or for performing material classification.

[0155] In some examples, the apparatus 500 further includes a rendering module (not shown) configured to superimpose local covariance loss values ​​at multiple image locations onto one or more of at least two single-energy images. Exemplary superposition results are shown in Figures 5A and 5B. The apparatus 500 further includes a display (not shown) for displaying the superposition results.

[0156] In some examples, the device 500 further includes a classifier module (not shown) configured to apply a pre-trained classifier to perform material classification at one or more of a plurality of image locations based on local covariance defects.

[0157] Figure 8 schematically shows a medical imaging system 300 according to several embodiments of the present disclosure.

[0158] The medical imaging system 300 includes a scanner 310 configured to scan an object of interest to acquire image data of the object of interest. The scanner 310 is a spectral CT scanner or an MRI scanner.

[0159] The medical imaging system 300 further includes an apparatus 100 for processing image data of interest, as described in the example with respect to Figure 6 or Figure 7.

[0160] All definitions defined and used herein should be understood to take precedence over dictionary definitions, definitions in documents incorporated by reference, and / or the ordinary meanings of the defined terms.

[0161] In this text and in the claims, singular elements should be understood to mean “at least one” unless explicitly stated otherwise.

[0162] In this specification and in the claims, the phrase "and / or" should be understood to mean "either or both" of the elements thus combined, that is, elements that are sometimes conjunctive and sometimes disjunctive. The numerous elements enumerated by "and / or" should similarly be interpreted as "one or more" of the elements thus combined. Other elements besides those specifically identified by the "and / or" phrase may exist as options, whether or not they relate to the specifically identified elements.

[0163] In this specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when distinguishing items in a list, “or” or “and / or” should be interpreted as inclusive, that is, including at least one of several elements or lists of elements, but also including two or more, and optionally including additional items not listed in the list. Only terms that are explicitly indicated as not being inclusive, such as “only one of,” or “exactly one of,” or “consisting of” as used in the claims, refer to including exactly one element of several elements or lists of elements. In general, the term “or” as used herein should be interpreted as indicating an exclusive choice (i.e., “one or the other, but not both”) only when preceded by terms of exclusivity such as “either,” “one of,” “only one of,” or “exactly one of.”

[0164] When used herein and in the claims, the phrase “at least one” with respect to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list of elements, but not necessarily including at least one of all elements specifically enumerated in the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than those specifically identified in the list of elements to which the phrase “at least one” refers, whether or not they are related to the specifically identified elements.

[0165] In the claims and the above-mentioned specification, all transitional phrases such as “having,” “including,” “carrying,” “possessing,” “containing,” “related to,” “holding,” and “composed of” should be understood to be non-restrictive, meaning they include but do not restrict. Only the transitional phrases “consisting of” and “essentially consisting of” are considered restrictive or semi-restrictive transitional phrases, respectively.

[0166] In another exemplary embodiment of the present invention, a computer program or computer program element is provided, characterized by being configured to perform a step of the method according to one of the preceding embodiments on a suitable system.

[0167] Therefore, the computer program elements are stored in a computer unit, which is also part of one embodiment of the present invention. This computing unit is configured to perform or induce the execution of the steps of the method described above. Furthermore, this computing unit is configured to operate the components of the apparatus described above. The computing unit is configured to operate automatically and / or to execute user instructions. The computer program is loaded into the working memory of a data processor. Thus, the data processor is equipped to perform the method of the present invention.

[0168] This exemplary embodiment of the present invention covers both computer programs that use the present invention from the outset and computer programs that, through updates, transform an existing program into one that uses the present invention.

[0169] Furthermore, computer program elements can provide all the necessary steps to satisfy the procedures of the exemplary embodiment of the method described above.

[0170] According to a further exemplary embodiment of the present invention, a computer-readable medium such as a CD-ROM is presented, the computer-readable medium having computer program elements stored therein, the computer program elements being as described in the previous section.

[0171] Computer programs may be stored and / or distributed on suitable media such as optical storage media or solid media supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0172] However, the program may also be presented via a network such as the World Wide Web and downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the present invention, a medium is provided that allows a computer program element to be downloaded, and the computer program element is configured to perform a method according to one of the aforementioned embodiments of the present invention.

[0173] While several embodiments of the invention are described and illustrated herein, those skilled in the art will readily conceive of various other means and / or structures for performing the functions described herein and / or obtaining the results and / or one or more advantages, and each of such modifications and / or variations will be considered within the scope of the embodiments of the invention described herein. More generally, those skilled in the art will readily recognize that all parameters, dimensions, materials and configurations described herein are intended to be illustrative, and that actual parameters, dimensions, materials and / or configurations will depend on one or more specific applications in which the teachings of the invention are used. Those skilled in the art will recognize many equivalents of the particular embodiments of the invention described herein, or at best, can verify them using routine experimentation. Therefore, it should be understood that the embodiments described herein are presented merely as examples, and that embodiments of the invention may be practiced in ways different from those specifically described and claimed, within the scope of the appended claims and their equivalents. Embodiments of the invention of this disclosure relate to the individual features, systems, articles, materials, kits and / or methods described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present invention as long as such features, systems, articles, materials, kits, and / or methods are not inconsistent with each other.

Claims

1. 1. An apparatus for processing image data of an object of interest comprising at least two monoenergetic images acquired from different spectral channels, the apparatus comprising: an input module for receiving the image data of the object of interest; a computation module for determining local covariance matrices at a plurality of image locations in the at least two mono-energy images, each local covariance matrix being a matrix of local variances and local covariances between image intensities at one of the plurality of image locations in the at least two mono-energy images; a rendering module for overlaying local variance and / or local covariance values ​​at the plurality of image locations with one or more of the at least two mono-energy images; An output module to provide the result of the overlay; 13. An apparatus comprising:

2. The apparatus of claim 1 , wherein the computation module computes each local variance matrix by a Gaussian convolution operation.

3. The apparatus of claim 1 or 2, wherein the computation module is further configured to determine a number of eigenvalues ​​for at least one of the local covariance matrices to determine an amount of material at which a material transition occurs.

4. The apparatus of claim 1 or 2, further comprising a classifier module for applying a pre-trained classifier to classify each of the local covariance matrices into a material transition type.

5. The apparatus of claim 1 or 2, wherein the plurality of image locations includes a location of a material transition.

6. The apparatus of claim 1 or 2, further comprising a display device for displaying the result of the overlay.

7. 1. An apparatus for processing image data of an object of interest comprising at least two monoenergetic images acquired from different spectral channels, the apparatus comprising: an input module for receiving the image data of the object of interest; a computation module for determining local covariance matrices at a plurality of image locations in the at least two mono-energy images and determining a local covariance deficit based on a comparison between a product of local variances and a product of local covariances across all spectral channel combinations, each local covariance matrix being a matrix of local variances and local covariances between image intensities at one of the plurality of image locations in the at least two mono-energy images; an output module for providing the local covariance deficit, preferably as a machine learning feature for registration with one or more of the at least two mono-energy images and / or for performing material classification; 13. An apparatus comprising:

8. The apparatus of claim 7 , further comprising a rendering module for overlaying values ​​of local covariance deficit at the plurality of image locations with one or more of the at least two mono-energy images.

9. The apparatus of claim 8 further comprising a display device for displaying the result of the overlay.

10. 10. The apparatus of claim 7, further comprising a classifier module for applying a pre-trained classifier to perform material classification at one or more of the plurality of image locations based on the local covariance deficit.

11. The apparatus of claim 7 , wherein the computation module determines the local covariance deficit in image space or projection space.

12. a scanner for scanning an object of interest to obtain image data of the object of interest; An apparatus according to any one of claims 1 or 2 or claims 7 to 9 for processing the image data of the object of interest; 2. A medical imaging system comprising:

13. 1. A computer-implemented method for processing image data of an object of interest comprising at least two monoenergetic images acquired from different spectral channels, the computer-implemented method comprising: receiving the image data of the object of interest; determining local covariance matrices at a plurality of image locations in the at least two mono-energy images, each local covariance matrix being a matrix of local variances and local covariances between image intensities at one of the plurality of image locations in the at least two mono-energy images; registering local variance and / or local covariance values ​​at said plurality of image locations with one or more of said at least two mono-energy images; providing a result of the superposition; 13. A computer-implemented method comprising:

14. 1. A computer-implemented method for processing image data of an object of interest comprising at least two monoenergetic images acquired from different spectral channels, the computer-implemented method comprising: receiving the image data of the object of interest; determining local covariance matrices at a plurality of image locations in the at least two mono-energy images, each local covariance matrix being a matrix of local variances and local covariances between image intensities at one of the plurality of image locations in the at least two mono-energy images; determining a local multispectral covariance deficit based on a comparison between a product of local variances and a product of local covariances over all spectral channel combinations; providing said local covariance deficit, preferably as a machine learning feature for registration with one or more of said at least two mono-energy images and / or for performing material classification; 13. A computer-implemented method comprising:

15. A computer program comprising instructions which, when executed by a processing unit, cause the processing unit to perform the steps of the method according to claim 13 or 14.