Method and apparatus for motion detection based on deep learning in nuclear imaging systems

Abstract

A method and apparatus for deep learning based motion detection in nuclear imaging systems. Systems and methods for detecting object movement within a medical image based on a trained deep learning process are disclosed. In some examples, an image processing system receives positron emission tomography (PET) measurement data and common modal measurement data from an image scanner. The image processing system generates a PET image and a common-mode image based on the PET measurement data and the common-mode measurement data, respectively. Further, the image processing system inputs the PET image and the common modal image to a first trained neural network and generates a first feature of the PET measurement data and a second feature of the common modal measurement data. The image processing system inputs the first feature and the second feature to a second trained neural network and generates displacement data characterizing a displacement between the first feature and the second feature. Based on the displacement data, the image processing system generates display data for display.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to medical diagnostic systems and, more particularly, to detecting object motion in nuclear images for diagnostic and reporting purposes. Background Art

[0002] Nuclear imaging systems can capture images using various techniques. For example, some nuclear imaging systems use positron emission tomography (PET) or single photon emission computed tomography (SPECT) to capture anatomical images. PET is a nuclear medicine imaging technique that produces tomographic images representing the distribution of positron-emitting isotopes in the body, while SPECT relies on the detection of gamma rays to produce tomographic images representing the distribution of radioactive tracer molecules in the body. Some nuclear imaging systems use co-modalities, such as computed tomography (CT) or magnetic resonance imaging (MRI). CT is an imaging technique that uses x-rays to produce anatomical images. Magnetic resonance imaging (MRI) is an imaging technique that uses magnetic fields and radio waves to generate anatomical and functional images. Some nuclear imaging systems combine images from PET and CT scanners during the image fusion process to produce images showing information from both PET scans and CT scans (e.g., PET / CT systems). Similarly, some nuclear imaging systems combine images from PET scanners and MRI scanners to produce images showing information from both PET scans and MRI scans.

[0003] Typically, these nuclear imaging systems capture measurement data and use mathematical algorithms to process the captured measurement data to reconstruct medical images. For example, the reconstruction can be based on a model based on an analytical or iterative algorithm, or more recently, a deep learning algorithm. In at least some instances, traditional image reconstruction (e.g., PET image reconstruction) and clinical interpretation assume, and may depend on, spatial consistency between the various modal scans. For example, the reconstruction and / or clinical interpretation of PET and CT scans can assume that the PET and CT scans are spatially consistent. However, during image capture, the subject (e.g., the patient) may move. For example, the subject may intentionally or unintentionally move their head, arms, or legs. As another example, the subject may move due to breathing. These movements may result in misalignment of the PET image and the CT image. For example, the tissue captured in the PET image may not be aligned with the tissue captured in the CT image. As a result, in addition to other potential problems, the movement may cause errors during the reconstruction process and may further hinder the clinical interpretation of the reconstructed image. As such, there is an opportunity to address the shortcomings in nuclear imaging systems. Summary of the Invention

[0004] Systems and methods are disclosed for detecting object motion within medical images based on a trained deep learning process.

[0005] In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: receiving positron emission tomography (PET) measurement data and common modality measurement data from an image scanning system. The operations also include generating a PET image based on the PET measurement data and generating a common modality image based on the common modality measurement data. Furthermore, the operations include inputting the PET image and the common modality image into a first trained neural network, and generating a first feature of the PET image and a second feature of the common modality image based on inputting the PET image and the common modality image into the first trained neural network. The operations also include inputting the first feature and the second feature into a second trained neural network, and generating displacement data representing a displacement between the first feature and the second feature based on inputting the first output data into the second trained neural network. The operations also include generating display data based on the displacement data, and transmitting the display data for display.

[0006] In some embodiments, a system includes a memory device storing instructions and at least one processor communicatively coupled to the memory device. The at least one processor is configured to execute instructions to receive positron emission tomography (PET) measurement data and co-modality measurement data from an image scanning system. The at least one processor is further configured to execute instructions to generate a PET image based on the PET measurement data and a co-modality image based on the co-modality measurement data. Furthermore, the at least one processor is configured to execute instructions to input the PET image and the co-modality image into a first trained neural network and, based on inputting the PET image and the co-modality image into the first trained neural network, generate a first feature of the PET image and a second feature of the co-modality image. The at least one processor is further configured to execute instructions to input the first feature and the second feature into a second trained neural network and, based on inputting the first output data into the second trained neural network, generate displacement data representing a displacement between the first feature and the second feature. The at least one processor is further configured to execute instructions to generate display data based on the displacement data and transmit the display data for display.

[0007] In some embodiments, a computer-implemented method includes receiving positron emission tomography (PET) measurement data. The method also includes receiving common-modality measurement data. Furthermore, the method includes inputting the PET measurement data and the common-modality measurement data into a neural network, and generating output data characterizing a first feature of the PET measurement data and a second feature of the common-modality measurement data based on inputting the PET measurement data and the common-modality measurement data into the neural network. The method also includes determining, based on the output data, that the neural network is trained. Based on the determination, the method also includes storing parameters characterizing the neural network in a data repository.

[0008] In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: receiving positron emission tomography (PET) measurement data. The operations also include receiving common-modality measurement data. Furthermore, the operations include inputting the PET measurement data and the common-modality measurement data into a neural network, and generating output data representing a first feature of the PET measurement data and a second feature of the common-modality measurement data based on inputting the PET measurement data and the common-modality measurement data into the neural network. The operations also include determining, based on the output data, that the neural network is trained. Based on the determination, the operations also include storing parameters representing the neural network in a data repository.

[0009] In some embodiments, a system includes a memory device storing instructions and at least one processor communicatively coupled to the memory device. The at least one processor is configured to execute instructions to receive positron emission tomography (PET) measurement data. The at least one processor is further configured to execute instructions to receive common-modality measurement data. Furthermore, the at least one processor is configured to execute instructions to input the PET measurement data and the common-modality measurement data into a neural network, and based on inputting the PET measurement data and the common-modality data into the neural network, generate output data representing a first feature of the PET measurement data and a second feature of the common-modality measurement data. The at least one processor is further configured to execute instructions to determine, based on the output data, that the neural network is trained. Based on the determination, the at least one processor is further configured to execute instructions to store parameters representing the neural network in a data repository.

[0010] In some embodiments, a computer-implemented method includes receiving a first feature generated from positron emission tomography (PET) measurement data and a second feature generated from common modality measurement data. The method also includes inputting the first feature and the second feature into a neural network, and based on inputting the first feature and the second feature into the neural network, generating output data representing a displacement between the first feature and the second feature. Furthermore, the method includes determining, based on the output data, that the neural network is trained. Based on the determination, the method also includes storing parameters representing the neural network in a data repository.

[0011] In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: receiving a first feature generated from positron emission tomography (PET) measurement data and a second feature generated from common modality measurement data. The operations also include inputting the first and second features into a neural network, and generating output data representing a displacement between the first and second features based on the inputting the first and second features into the neural network. Furthermore, the operations include determining, based on the output data, that the neural network is trained. Based on the determination, the operations also include storing parameters representing the neural network in a data repository.

[0012] In some embodiments, a system includes a memory device storing instructions and at least one processor communicatively coupled to the memory device. The at least one processor is configured to execute instructions to receive a first feature generated from positron emission tomography (PET) measurement data and a second feature generated from common modality measurement data. The at least one processor is further configured to execute instructions to input the first feature and the second feature into a neural network, and based on inputting the first feature and the second feature into the neural network, generate output data representing a displacement between the first feature and the second feature. Furthermore, the at least one processor is configured to execute instructions to determine, based on the output data, that the neural network is trained. Based on the determination, the at least one processor is further configured to execute instructions to store parameters representing the neural network in a data repository.

[0013] In some embodiments, a computer-implemented method includes receiving positron emission tomography (PET) measurement data and common modality measurement data from an image scanning system. The method also includes generating a PET image based on the PET measurement data and generating a common modality image based on the common modality measurement data. In addition, the method includes inputting the PET image and the common modality image into a first trained neural network, and generating a first feature of the PET image and a second feature of the common modality image based on inputting the PET image and the common modality image into the first trained neural network. The method also includes inputting the first feature and the second feature into a second trained neural network, and generating displacement data representing a displacement between the first feature and the second feature based on inputting the first output data into the second trained neural network. The method also includes generating display data based on the displacement data and transmitting the display data for display. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] As will become apparent hereinafter from elements of the various figures, the figures are provided for illustrative purposes and are not necessarily drawn to scale.

[0015] Figure 1 A nuclear imaging system is illustrated in accordance with some embodiments.

[0016] Figure 2 Illustrated is a block diagram of an example computing device that can perform one or more of the functions described herein in accordance with some embodiments.

[0017] Figure 3A and Figure 3B Illustrated is a diagram of a Figure 1 Example display data generated by the Nuclear Imaging System.

[0018] Figure 4A An image reconstructed from misaligned positron emission tomography (PET) and computed tomography (CT) image data is illustrated.

[0019] Figure 4B Illustrated is a representation according to some embodiments Figure 4A The displacement vector values ​​of the image.

[0020] Figure 4C The diagram illustrates a method based on some embodiments of the present invention. Figure 4B The heat map generated by the displacement vector values.

[0021] Figure 5 Illustrated is the training of a neural network in a nuclear imaging system, according to some embodiments.

[0022] Figure 6 is a flowchart of an example method of training a neural network, according to some embodiments.

[0023] Figure 7 is a flow chart of an example method for detecting object motion using a trained neural network, according to some embodiments. DETAILED DESCRIPTION

[0024] This description of the exemplary embodiments is intended to be read in conjunction with the accompanying drawings, which are to be considered a part of the entire written description.

[0025] Exemplary embodiments are described with respect to the claimed systems and with respect to the claimed methods. Furthermore, exemplary embodiments are described with respect to methods and systems for image reconstruction, and with respect to methods and systems for training functions for image reconstruction. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. For example, a claim for providing a system may be modified with features described or claimed in the context of a method, and vice versa. Furthermore, functional features of the described or claimed method are embodied by an object unit of the providing system. Similarly, a claim for a method and system for training an image reconstruction function may be modified with features described or claimed in the context of a method and system for image reconstruction, and vice versa.

[0026] Various embodiments of the present disclosure may employ machine learning methods or processes to provide clinical information from nuclear imaging systems. For example, embodiments may employ machine learning methods or processes to reconstruct images based on captured measurement data and provide the reconstructed images for clinical diagnosis. In some embodiments, the machine learning methods or processes are trained to improve image reconstruction and clinical interpretation of those reconstructed images.

[0027] Hybrid imaging systems, such as PET / CT imaging systems, can provide two independent modalities. For example, a PET / CT imaging system can capture a PET scan of a subject as well as a CT scan of the subject. If the subject moves during or between these image captures, the PET and CT images may be misaligned (e.g., tissue located at one location in the PET image may be located at another location in the CT image). For example, Figure 4A The diagram shows a display of an unregistered PET / CT image 400 (e.g., the PET image is misaligned with the CT image). The PET / CT image 400 includes coronal and sagittal views of the scanned subject and is an orthogonal two-dimensional (2D) slice from the same 3D volume. As shown, there is a visible misalignment between the PET and CT images due to respiration. For example, there is a mismatch at the boundary between the lungs and the liver. In fact, some liver activity on the PET image appears to be located within the lung space on the CT image.

[0028] Embodiments described herein can detect such intermodal movement (e.g., determine such intermodal movement and generate an estimate of such intermodal movement), and in some examples can display an indication of intermodal movement. For example, an embodiment can include a deep learning motion estimation framework that includes at least two trained artificial intelligence or machine learning models, such as neural networks (e.g., convolutional neural networks (CNNs)). A first trained neural network can process images (e.g., normalized images) from each modality to generate features corresponding to each modality. A second trained neural network processes the features generated from the first trained neural network and generates displacement data characterizing relative motion displacement between the features generated from each modality. For example, the displacement data can include a 3-dimensional (3D) displacement value for each of a plurality of pixel positions.

[0029] During the clinical interpretation of the images, the displacement data can be utilized in various ways to assist medical professionals (e.g., physicians). For example, a graphical display, such as a heat map, can be generated based on the displacement data. The heat map can indicate the displacement magnitude (e.g., Euclidean magnitude) of each of a plurality of voxels. The heat map can be displayed to the medical professional to assist in the clinical interpretation of the fused image reconstructed from the images of each modality. In some examples, an alarm (e.g., a warning icon for display) can be generated based on the displacement data. For example, an alarm can be generated when the displacement magnitude exceeds a corresponding threshold. The alarm can alert the medical professional that relatively large movement has been detected between the image scans of the various modalities.

[0030] As described herein, a first neural network can be trained based on training data comprising measurement data for each modality. For example, the first neural network can be trained based on training data comprising PET measurement data and corresponding co-modal measurement data (e.g., CT measurement data corresponding to a PET scan of the same subject). In some examples, the PET measurement data and / or the co-modal measurement data are labeled to identify various features (e.g., anatomical regions, tissues, etc.). During training, the first neural network generates output data that characterizes first features of the PET measurement data and second features of the co-modal measurement data. For example, the first neural network can perform various linear (e.g., convolution) and nonlinear operations and, as a result, output a first feature that comprises a set of learned spatial features. The first feature and the second feature can comprise common features detected in each of the PET measurement data and the co-modal measurement data.

[0031] A determination as to whether the first neural network is sufficiently trained can be made based on the output data. For example, a metric value can be calculated based on the output data and the expected feature data characterizing the detection of the expected feature. For example, the metric value can be a metric value calculated from a loss function, such as a calculated precision value, a calculated recall value, a calculated AUC value, any receiver operating characteristic (ROC) curve or precision-recall (PR) curve value, or any other suitable metric value. A determination as to whether the first neural network is trained can be made based on the metric value. For example, when the calculated metric value exceeds (e.g., is below, is above) a corresponding threshold, the first neural network can be considered trained.

[0032] A second neural network can be trained based on training data comprising a first feature of PET measurement data and a corresponding second feature of common modality measurement data (e.g., CT measurement data). The training data can be labeled to indicate a displacement between corresponding pixels of the first feature and the second feature. During training, the second neural network generates output data representing the displacement between corresponding pixels of the first feature and the second feature. A determination can be made based on the output data as to whether the second neural network is sufficiently trained. For example, the displacement provided by the output data (i.e., the displacement value) can be compared to an expected displacement to determine whether the second neural network is sufficiently trained.

[0033] In some examples, a metric value is calculated based on the output data and the expected displacement value, and a determination is made as to whether the second neural network is trained based on the metric value. For example, the metric value can be a metric value calculated from a loss function, such as a calculated precision value, a calculated recall value, a calculated AUC value, any ROC curve or PR curve value, or any other suitable metric value. When the calculated metric value exceeds (e.g., is lower than, higher than) a corresponding threshold value, the second neural network can be considered to be trained.

[0034] In some examples, to generate training data for the second neural network, one or more of the first features of the PET measurement data and / or the second features of the common modality measurement data can be adjusted to introduce a displacement between corresponding features (e.g., tissue). Alternatively, the PET measurement data and / or the common modality measurement data can be adjusted to introduce the displacement, and then the first and second features can be generated based on inputting the adjusted PET measurement data and / or the common modality measurement data into the trained first neural network. The training data can be labeled to indicate the displacement between corresponding pixels of the first and second features. For example, the training data can include a 3D vector that includes a displacement offset of at least some pixels of the first and second features in three dimensions (e.g., in the x, y, and z directions of an x, y, z coordinate system).

[0035] Among other advantages, embodiments can provide medical professionals with automatic quality control features to assist in the clinical interpretation of images. Additionally, embodiments can provide an indication that the captured PET and CT images are spatially inconsistent, thereby allowing the patient to be rescanned before leaving the imaging room if necessary (e.g., such as when the PET and CT images are misaligned by at least a threshold amount, as described herein).

[0036] Figure 1An exemplary nuclear imaging system 100 is illustrated. As shown, the nuclear imaging system 100 includes an image scanning system 102, an image processing system 104, a data repository 160, and, in some examples, a monitor 162. The image scanning system 102 can be, for example, a PET / CT scanner that can capture PET images and CT images. For example, the image scanning system 102 can capture a CT image of anything (e.g., a person) within the scanner's field of view (FOV) of the CT and generate CT measurement data 133 based on the CT scan. The image scanning system 102 can also capture a PET image of anything (e.g., a person) within the scanner's FOV of the PET and generate PET measurement data 111 (e.g., sinogram data) based on the captured PET image. The PET measurement data 111 can represent anything imaged in the scanner's FOV that contains a positron-emitting isotope. In at least some examples, the CT measurement data 133 and the PET measurement data 111 correspond to scans of the same object (e.g., a patient). The image scanning system 102 may transmit the CT measurement data 133 and the PET measurement data 111 to the image processing system 104 .

[0037] The image processing system 104 includes a CT image reconstruction engine 139, a PET image reconstruction engine 113, a feature extraction engine 142, a relative motion displacement engine 144, and a display generation engine 146. In some examples, all or part of the image processing system 104 is implemented in hardware, such as in one or more field programmable gate arrays (FPGAs), one or more application specific integrated circuits (ASICs), one or more state machines, one or more computing devices, digital circuits, or any other suitable circuits. In some examples, part or all of the image processing system 104 can be implemented in software as executable instructions, such that when the executable instructions are executed by one or more processors, the one or more processors perform the corresponding functions as described herein. For example, the instructions can be stored in a non-transitory computer-readable storage medium.

[0038] For example, Figure 2 Illustrated is a computing device 200 that may be employed by the image processing system 104. The computing device 200 may implement, for example, one or more of the functionalities of the image processing system 104 described herein.

[0039] The computing device 200 may include one or more processors 201, a working memory 202, one or more input / output devices 203, an instruction memory 207, a transceiver 204, one or more communication ports 209, and a display 206, all operatively coupled to one or more data buses 208. The data buses 208 allow communication between the various devices. The data buses 208 may include wired or wireless communication channels.

[0040] The processor 201 may include one or more different processors, each having one or more cores. Each different processor may have the same or different architectures. The processor 201 may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), application specific integrated circuits (ASICs), digital signal processors (DSPs), and the like.

[0041] The processor 201 may be configured to perform a specific function or operation by executing code stored in the instruction memory 207, thereby embodying the function or operation. For example, the processor 201 may be configured to perform one or more of any functions, methods, or operations disclosed herein.

[0042] The instruction memory 207 may store instructions that can be accessed (e.g., read) and executed by the processor 201. For example, the instruction memory 207 may be a non-transitory computer-readable storage medium such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a removable disk, a CD-ROM, any non-volatile memory, or any other suitable memory. For example, the instruction memory 207 may store instructions that, when executed by the one or more processors 201, cause the one or more processors 201 to perform one or more of the functions of the CT image reconstruction engine 139, the PET image reconstruction engine 113, the feature extraction engine 142, the relative motion displacement engine 144, and the display generation engine 146.

[0043] The processor 201 can store data to and read data from the working memory 202. For example, the processor 201 can store a working instruction set (such as instructions loaded from the instruction memory 207) to the working memory 202. The processor 201 can also use the working memory 202 to store dynamic data created during operation of the computing device 200. The working memory 202 can be a random access memory (RAM), such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), or any other suitable memory.

[0044] Input / output device 203 may include any suitable device that allows data to be input or output. For example, input / output device 203 may include one or more of a keyboard, touchpad, mouse, stylus, touch screen, physical buttons, speakers, microphone, or any other suitable input or output device.

[0045] Communication port(s) 207 may include, for example, a serial port, such as a universal asynchronous receiver / transmitter (UART) connection, a universal serial bus (USB) connection, or any other suitable communication port or connection. In some examples, communication port(s) 207 allow for programming of executable instructions in instruction memory 207. In some examples, communication port(s) 207 allow for transfer (e.g., uploading or downloading) of data, such as CT measurement data 133 and PET measurement data 111.

[0046] Display 206 may display user interface 205. User interface 205 may enable a user to interact with computing device 200. For example, user interface 205 may be a user interface of an application that allows viewing of final image volume 191. In some examples, a user may interact with user interface 205 by engaging input / output device 203. In some examples, display 206 may be a touch screen on which user interface 205 is displayed.

[0047] The transceiver 204 allows for communication with a network, such as a Wi-Fi network, an Ethernet network, a cellular network, or any other suitable communication network. For example, if operating in a cellular network, the transceiver 204 is configured to allow for communication with the cellular network. The processor(s) 201 are operable to receive data from the network or send data to the network via the transceiver 204.

[0048] Return Reference Figure 1 , the CT image reconstruction engine 139 receives the CT measurement data 133 (e.g., CT raw data) and processes the CT measurement data 133 to generate a reconstructed CT image 137. The CT image reconstruction engine 139 may generate the reconstructed CT image 137 based on the corresponding CT measurement data 133 using any suitable method known in the art. For example, the CT image reconstruction engine 139 may apply a back-projection-based algorithm or an iterative method to the CT measurement data 133 to generate the reconstructed CT image 137. In addition, the PET image reconstruction engine 113 receives the PET measurement data 111 and processes the PET measurement data 111 to generate a reconstructed PET image 115. For example, the PET image reconstruction engine 113 may apply an iterative MLEM-based algorithm to the PET measurement data 111 to generate the reconstructed PET image 115. The PET image reconstruction engine 113 may generate the reconstructed PET image 115 based on the corresponding PET measurement data 111 using any suitable method known in the art.

[0049] Additionally, feature extraction engine 142 receives CT image 137 and PET image 115 and applies a first trained neural network (e.g., a trained CNN) to CT image 137 and PET image 115 to generate joint feature data 143. For example, the first trained neural network is configured to extract features from CT image 137 and PET image 115. As such, joint feature data 143 may include CT features (e.g., a CT feature map) that characterize features of each CT image 137 and may also include PET features (e.g., a PET feature map) that characterize features of each PET image 115.

[0050] As described herein, a first trained neural network can be trained to detect features based on labeled CT images and labeled PET images (e.g., ground truth data) during a training period and further validated using unlabeled CT images and PET images during a validation period.

[0051] In some examples, feature extraction engine 142 includes two neural networks, one of which is trained to generate CT features based on CT image 137 and the other of which is trained to generate PET features based on PET image 115. In these examples, feature extraction engine 142 may input CT image 137 and PET image 115 into corresponding neural networks and may combine the outputs of the neural networks to generate joint feature data 143.

[0052] In addition, the relative motion displacement engine 144 receives the joint feature data 143 from the feature extraction engine 142 and applies a second trained neural network (e.g., a trained CNN) to the joint feature data 143 to generate displacement data 145, which characterizes the displacement between the CT features and the PET features received within the joint feature data 143. For example, the joint feature data 143 may include 3D vectors for each corresponding pixel of the CT features and the PET features, where each 3D vector includes a displacement value (e.g., an offset value) in each of three directions (e.g., the x, y, and z directions of an x, y, z coordinate system). For example, the displacement value may identify a number of pixel locations.

[0053] In some examples, displacement data 145 represents a displacement from a CT feature to a PET feature (e.g., a PET pixel is offset from a corresponding CT pixel in each of three directions by a displacement value of a corresponding 3D vector). In other examples, displacement data 145 represents a displacement from a PET feature to a CT feature (e.g., a CT pixel is offset from a corresponding PET pixel in each of three directions by a displacement value of a corresponding 3D vector).

[0054] As described herein, a second trained neural network can be trained to detect pixel displacement based on labeled CT features and labeled PET features (e.g., ground truth data) during a training period and further validated with unlabeled CT images and PET images during a validation period.

[0055] In addition, the relative motion displacement engine 144 can store the displacement data 145 in the data repository 160. In some examples, the relative motion displacement engine 144 transmits the displacement data 145 to the monitor 162 for display. For example, the monitor 162 can display the corresponding displacement value of the displacement data 145. In some examples, the displacement data 145 is transmitted over a network (e.g., via the transceiver 204) to a remote computing device, such as a laptop, smartphone, tablet, or any other suitable computing device. For example, Figure 4B Graph 420 of exemplary displacement values ​​is illustrated.

[0056] As shown, display generation engine 146 can receive displacement data 145 from relative motion displacement engine 144, or in some examples, from data repository 160. Based on displacement data 145, display generation engine 146 can generate display data 147 for display, such as for display on monitor 162. Display data 147 can include warning messages (e.g., icons), heat maps, images (e.g., alone or overlaid with one or more of corresponding CT images 137 and PET images 115), or any other suitable display data based on displacement data 145. In some examples, display data 147 is transmitted over a network (e.g., via transceiver 204) to a remote computing device, such as a laptop, smartphone, tablet, or any other suitable computing device.

[0057] For example, based on the displacement data 145, the display generation engine 146 can generate a heat map that represents 3D displacement magnitudes for each of the plurality of portions of the displacement data 145, such as for each corresponding voxel of the displacement data 145. The display generation engine 146 can generate the 3D displacement magnitude for each voxel by calculating the displacement magnitude for each pixel of the voxel based on the corresponding displacement value identified within the displacement data 145 and summing the displacement magnitudes for the voxel. The display generation engine 146 can calculate the displacement magnitude for each pixel by determining the magnitude of the 3D displacement vector for the pixel. For example, the display generation engine 146 can calculate the displacement magnitude for the pixel based on the Euclidean magnitude of the following vector formula:

[0058]

[0059] in:

[0060] d xis the "x" coordinate displacement value;

[0061] d y is the "y" coordinate displacement value; and

[0062] d z is the "z" coordinate displacement value.

[0063] The display generation engine 146 may package the calculated 3D displacement values ​​into display data 147 and transmit the display data 147 to the monitor 162 for display. Figure 4C Illustrated is an exemplary heat map 440 that may be generated based on the displacement data 145 as described herein.

[0064] In some examples, display generation engine 146 can generate display data 147 as an alert to a medical professional. For example, display data 147 can indicate a warning to the medical professional. For example, display generation engine 146 can determine whether displacement data 145 indicates object movement (e.g., unacceptable object movement) based on comparing displacement data 145 to one or more thresholds. If display generation engine 146 determines that displacement data 145 indicates object movement, display generation engine 146 can generate display data 147 to indicate a warning or alert.

[0065] As an example, Figure 3A Illustrated is an alert window 302 displayed by monitor 162 and generated based on exemplary display data 147 received from display generation engine 146. Alert window 302 indicates that too much motion has been detected and that the subject should be rescanned (eg, before leaving the imaging room).

[0066] To determine whether the displacement data 145 indicates too much movement, the display generation engine 146 may compare the calculated 3D displacement magnitude values ​​within the display data 147 to a threshold value and may determine unacceptable object movement when any of the calculated 3D displacement magnitude values ​​exceeds the threshold value. As another example, the display generation engine 146 may sum the calculated 3D displacement magnitude values ​​within the display data 147 to determine a total displacement magnitude value and may compare the total displacement magnitude value to a threshold value. When the total displacement magnitude value exceeds the threshold value, the display generation engine 146 may determine unacceptable object movement.

[0067] In yet other examples, the display generation engine 146 may receive selection data representing an image region, such as any of the CT image 137 or the PET image 115, from a display, such as the monitor 162. For example, Figure 3BThe diagram illustrates a display of an unregistered PET / CT image 350 and a selection box 360. The medical professional can adjust the size of the selection box 360 (e.g., using the input / output device 203) to select an area of ​​the PET / CT image 350. The display generation engine 146 can receive selection data representing the selection box 360 from the monitor 162 and can determine whether one or more displacement magnitudes within the display data 147 corresponding to pixels within the selection box 360 exceed corresponding thresholds. When the one or more displacement magnitudes exceed their corresponding thresholds, the display generation engine 146 can determine that the object movement is unacceptable.

[0068] Figure 5 A training engine 502 is shown, which can train any of the neural networks described herein, such as for Figure 1 The training engine 502 may be implemented by the image processing system 104 or the computing device 200.

[0069] As shown, the training engine 502 is communicatively coupled with the first CNN 504, the second CNN 526, and the data repository 560. The first CNN 504 can be implemented by the feature extraction engine 142, and the second CNN 526 can be implemented by the relative motion displacement engine 526. In some examples, the first CNN 504, the second CNN 526, and the training engine 502 are implemented by one or more processors 201 executing corresponding instructions.

[0070] Data repository 560 includes PET data 501 and CT data 503. Each of PET data 501 and CT data 503 can be labeled with features (e.g., identifying anatomical regions, tissues, etc.). PET data 501 can represent a PET image, and CT data 503 can represent a corresponding CT image. For example, each PET image can have a corresponding CT image (e.g., the CT image is a scan of the same object as the corresponding PET image). Data repository 560 also includes PET features 551 and CT features 553. PET features 551 represent features of a PET image (e.g., the PET image of PET data 501), and CT features represent features of a CT image (e.g., the CT image of CT data 503). In addition, data repository 560 includes expected feature data 541 representing image features of PET data 501 and CT data 503, and expected displacement data 543 representing displacement values ​​between PET features 551 and CT features 553.

[0071] In some examples, training engine 502 obtains PET data 501 and applies adjustments to PET data 501 to generate warped PET data 511. For example, training engine 502 may adjust the position of one or more pixels of PET data 501 and may store the adjusted PET data as warped PET data 511. To adjust the position of the pixels, training engine 502 may copy the pixel values ​​(e.g., for each of the three dimensions) and write the pixel values ​​to the shifted pixel positions. In this way, warped PET data 511 may represent a PET image with the shifted (e.g., moved) pixels, thereby generating a PET image that is at least partially shifted from a corresponding CT image. Similarly, training engine 502 may adjust the position of one or more pixels of CT data 503 and may store the adjusted CT data as warped CT data 513. Warped CT data 513 may represent a CT image with the shifted (e.g., moved) pixels, thereby generating a CT image that is at least partially shifted from a corresponding PET image.

[0072] The training engine 502 can further update the expected feature data 541 and the expected displacement data 543 for the generated images of the warped PET data 511 and the warped CT data 513. For example, the training engine 502 can determine the new position of the feature based on the displacement used to generate any of the warped PET data 511 and the warped CT data 513. In addition, and based on the new position of the feature, the training engine 502 can store values ​​representing the feature and displacement in the expected feature data 541 and the expected displacement data 543, respectively, for any generated warped PET data 511 and warped CT data 513.

[0073] The training engine 502 can obtain the PET data 501 and the CT data 503 from the data repository 560 and can generate first training data 533. The first training data 533 can include, for example, labeled 3D PET image vectors and corresponding labeled 3D CT image vectors. In some examples, the training engine 502 generates at least a portion of the first training data 533 based on one or more of warping the PET data 511 and the CT data 503, the PET data 501 and the warping CT data 513, or the warping the PET data 511 and the warping CT data 513. The training engine 502 inputs the first training data 533 to the first CNN 504 and receives corresponding output data 534 from the first CNN 504. The output data 534 can represent detected features of each of the input PET image and the input CT image.

[0074] The training engine 502 may receive output data 534 from the first CNN 504 and may determine whether the first CNN 504 is trained based on the output data 534. For example, the training engine 502 may calculate a metric value based on the output data 534 and the expected feature data 541 characterizing the expected feature detection. For example, the metric value may be a metric value calculated from a loss function, such as a calculated precision value, a calculated recall value, a calculated AUC value, any ROC curve or PR curve value, or any other suitable metric value. When the calculated metric value exceeds (e.g., is higher than, is lower than) a corresponding threshold, the training engine 502 may determine that the first CNN 504 is trained.

[0075] The training engine 502 may train the first CNN 504 using the first training data 533 during the training period, and may further verify the first CNN 504 using the additional unlabeled first training data 533 during the verification period. To determine whether the first CNN 504 is verified, the training engine 502 may calculate a metric value based on the output data 534 generated during the verification period. When the metric value exceeds a corresponding threshold, the training engine 502 may determine that the first CNN 504 is trained.

[0076] When training the first CNN 504, the training engine 502 can obtain first CNN parameters 522 from the trained first CNN 504. The first CNN parameters 522 may include, for example, hyperparameters, weights, coefficients, and / or any other values ​​required to build the trained first CNN 504. The training engine 502 can store the first CNN parameters 522 in a data repository 560. For inference, the feature extraction engine 142 can obtain the first CNN parameters 522 from the data repository 560 and can build (e.g., execute) the trained first CNN 504 based on the first CNN parameters 522.

[0077] Furthermore, the training engine 502 may obtain the PET features 551 and the CT features 553 from the data repository 560 and may generate second training data 535. The second training data 535 may include, for example, labeled PET features and corresponding labeled CT features. In some instances, the PET features 551 include labeled features generated from one or both of the PET data 501 and the warped PET data 511. In some instances, the CT features 553 include labeled features generated from one or both of the CT data 503 and the warped CT data 513. The training engine 502 inputs the second training data 535 into the second CNN 526 and receives corresponding output data 536 from the second CNN 526. The output data 536 may represent displacement values ​​between the PET features 551 and the corresponding CT features 553.

[0078] The training engine 502 may receive output data 536 from the second CNN 526 and may determine whether the second CNN 526 is trained based on the output data 536. For example, the training engine 502 may calculate a metric value based on the output data 536 and the expected displacement data 543 representing the expected displacement. For example, the metric value may be a metric value calculated from a loss function, such as a calculated precision value, a calculated recall value, a calculated AUC value, any ROC curve or PR curve value, or any other suitable metric value. When the calculated metric value exceeds (e.g., is higher than, is lower than) a corresponding threshold, the training engine 502 may determine that the second CNN 526 is trained.

[0079] The training engine 502 may train the second CNN 526 using the second training data 535 during the training period, and may further verify the second CNN 526 using the additional unlabeled second training data 535 during the verification period. To determine whether the second CNN 526 is verified, the training engine 502 may calculate a metric value based on the output data 536 generated during the verification period. When the metric value exceeds a corresponding threshold, the training engine 502 may determine that the second CNN 526 is trained.

[0080] When training the second CNN 526, the training engine 502 can obtain second CNN parameters 524 from the trained second CNN 526. The second CNN parameters 524 can include, for example, hyperparameters, weights, coefficients, and / or any other values ​​required to build the trained second CNN 526. The training engine 502 can store the second CNN parameters 524 in the data repository 560. For inference, the relative motion displacement engine 144 can obtain the second CNN parameters 524 from the data repository 560 and can build (e.g., execute) the trained second CNN 526 based on the second CNN parameters 524.

[0081] Figure 6 is to train a neural network (such as Figure 1 Flowchart of an example method 600 for a neural network (described herein). The method may be performed by one or more computing devices (such as image processing system 104) executing corresponding instructions.

[0082] Beginning at block 602, a PET image and a corresponding CT image are received. For example, the image processing system 104 can generate a PET image 115 and a CT image 137, respectively, based on the PET measurement data 111 and the CT measurement data 133 received from the image scanning system 102. The CT image and the PET image can be labeled. For example, the labeling can identify features of each of the CT image and the PET image. At block 604, the PET image and the CT image can be input to a first neural network. Based on inputting the PET image and the CT image to the first neural network, output data is generated. The output data (e.g., joint feature data 143) characterizes a first feature of the PET image and a second feature of the CT image.

[0083] Proceeding to block 606, a determination is made as to whether the first neural network is trained based on the output data. For example, as described herein, the image processing system 104 can calculate a metric value based on the output data and the expected feature data characterizing the detection of the expected feature. For example, the metric value can be a metric value calculated from a loss function, such as a calculated precision value, a calculated recall value, a calculated AUC value, any ROC curve or PR curve value, or any other suitable metric value. When the calculated metric value exceeds (e.g., is above, is below) a corresponding threshold value, the image processing system 104 can determine that the first neural network is trained.

[0084] In some examples, the first trained neural network is trained during a training period and further validated using unlabeled CT images and unlabeled PET images during a validation period. A metric value can be calculated based on the output data generated during the validation period, and the first neural network can be considered trained when the metric value exceeds a corresponding threshold. If the first neural network is not sufficiently trained, the method proceeds to return to block 602 to continue training the first neural network. Otherwise, if the first neural network is sufficiently trained, the method proceeds to block 608.

[0085] At block 608, the first feature and the second feature are input to a second neural network. Based on the input of the first feature and the second feature to the second neural network, output data is generated. The output data (e.g., displacement data 145) represents a displacement value between the first feature of the PET image and the second feature of the CT image.

[0086] Proceeding to block 610, a determination is made as to whether the second neural network is trained based on the output data. For example, as described herein, the image processing system 104 can calculate a metric value based on the output data and expected displacement data representing an expected displacement value between the first feature and the second feature. For example, the metric value can be a metric value calculated from a loss function, such as a calculated precision value, a calculated recall value, a calculated AUC value, any ROC curve or PR curve value, or any other suitable metric value. When the calculated metric value exceeds (e.g., is above, is below) a corresponding threshold, the image processing system 104 can determine that the second neural network is trained.

[0087] In some examples, the second trained neural network is trained during a training period and further validated with unlabeled displacement data during a validation period. Metric values ​​can be calculated based on the output data generated during the validation period, and the second neural network can be considered trained when the metric values ​​exceed corresponding thresholds. If the second neural network is not sufficiently trained, the method proceeds to return to block 608 to continue training the second neural network, or alternatively, to return to block 602 to continue training the first neural network. Otherwise, if the second neural network is sufficiently trained, the method proceeds to block 612.

[0088] At block 612, parameters representing the trained first neural network and the trained second neural network are stored in a data repository (e.g., data repository 160). These parameters may include, for example, hyperparameters, weights, coefficients, and any other relevant values. The image processing system 104 may establish each of the trained first neural network and the trained second neural network based on the stored parameters.

[0089] Figure 7 is the use of trained neural networks (such as Figure 1 Flowchart of an example method 700 for detecting object motion using a trained neural network as described above. The method may be performed by one or more computing devices (such as image processing system 104) executing corresponding instructions.

[0090] Beginning at block 702, PET measurement data and CT measurement data are received from an image scanning system. At block 704, a PET image is generated based on the PET measurement data, and a CT image is generated based on the CT measurement data. For example, as described herein, image processing system 104 may receive PET measurement data 111 and CT measurement data 133 from image scanning system 102 and may generate PET image 115 and CT image 137 based on PET measurement data 111 and CT measurement data 133, respectively.

[0091] At block 706, the PET image and the CT image are input to a first trained neural network. Based on inputting the PET image and the CT image to the first trained neural network, a first feature of the PET image and a second feature of the CT image are generated. For example, as described herein, the image processing system 104 may input the CT image 137 and the PET image 115 to the first trained neural network, and in response, the first trained neural network may output joint feature data 143 representing the CT feature and the PET feature, respectively.

[0092] Furthermore, at block 708, the first feature and the second feature are input to a second trained neural network. Based on the input of the first feature and the second feature to the second trained neural network, displacement data is generated. The displacement data represents a displacement value between the first feature and the second feature. For example, as described herein, the image processing system 104 may input joint feature data 143 to the second trained neural network. Based on the input of joint feature data 143 to the second trained neural network, the second trained neural network may output displacement data 145 representing a displacement value between the CT feature and the PET feature.

[0093] Proceeding to block 710, display data is generated based on the displacement data. For example, as described herein, the display data can be a warning message (e.g., alert window 302), a heat map (e.g., heat map 440), or an image (e.g., image 420). At block 712, the display data is transmitted for display (e.g., on monitor 162). In some examples, the display data is transmitted over a network (e.g., via transceiver 204) to a remote computing device, such as a laptop, smartphone, tablet, or any other suitable computing device.

[0094] The embodiments described herein may employ a deep learning motion estimation framework that includes two trained CNNs. The first trained CNN processes a PET / SPECT image and a co-modality (e.g., CT) image to extract features (e.g., common features) from each of the PET / SPECT image and the co-modality image. The second trained CNN uses the two sets of generated features to detect relative motion displacement between the two sets of features. The magnitude of the relative motion displacement can be determined and used to display information related to the detected inter-modality movement, such as a heat map.

[0095] The following is a list of non-limiting illustrative embodiments disclosed herein:

[0096] Illustrative embodiment 1: A computer-implemented method comprising:

[0097] receiving positron emission tomography (PET) measurement data and common modality measurement data from an image scanning system;

[0098] generating a PET image based on the PET measurement data and generating a common modality image based on the common modality measurement data;

[0099] Inputting the PET image and the co-modality image to a first trained neural network, and generating a first feature of the PET image and a second feature of the co-modality image based on inputting the PET image and the co-modality image to the first trained neural network;

[0100] inputting the first feature and the second feature into a second trained neural network, and generating displacement data representing a displacement between the first feature and the second feature based on inputting the first output data into the second trained neural network; and

[0101] Display data is generated based on the displacement data and transmitted for display.

[0102] Illustrative embodiment 2: The computer-implemented method of illustrative embodiment 1, wherein the common modality measurement data is computed tomography (CT) measurement data and the common modality image is a CT image.

[0103] Illustrative embodiment 3: The computer-implemented method of any one of illustrative embodiments 1-2, wherein the first trained neural network is a convolutional neural network (CNN).

[0104] Illustrative Embodiment 4: The computer-implemented method of any one of Illustrative Embodiments 1-3, wherein the second trained neural network is a convolutional neural network (CNN).

[0105] Illustrative Embodiment 5: The computer-implemented method of any of Illustrative Embodiments 1-4, wherein the first feature of the PET image and the second feature of the common modality image comprise a common feature.

[0106] Illustrative Embodiment 6: The computer-implemented method of any one of Illustrative Embodiments 1-5, wherein the displacement data comprises at least one displacement value for each of a plurality of pixels of the PET image and the common modality image.

[0107] Illustrative embodiment 7: The computer-implemented method of illustrative embodiment 6, wherein the at least one displacement value for each of the plurality of pixels comprises a first displacement value in a first direction, a second displacement value in a second direction, and a third displacement value in a third direction.

[0108] Illustrative embodiment 8: The computer-implemented method of illustrative embodiment 7, comprising:

[0109] determining a magnitude for each of the plurality of pixels based on the first displacement value, the second displacement value, and the third displacement value; and

[0110] Generate display data based on the magnitude.

[0111] Illustrative embodiment 9: The computer-implemented method of illustrative embodiment 8, wherein a data representation heat map is displayed.

[0112] Illustrative Embodiment 10: The computer-implemented method of any one of Illustrative Embodiments 1-9, wherein the displacement data comprises displacement values ​​identifying pixel offsets between the PET image and the common modality image.

[0113] Illustrative Embodiment 11: The computer-implemented method of any one of Illustrative Embodiments 1-10, wherein the PET measurement data and the common modality measurement data are based on corresponding scans of the same subject.

[0114] Illustrative Embodiment 12: The computer-implemented method of any one of Illustrative Embodiments 1-11, comprising training a first trained neural network, the training comprising:

[0115] Inputting the labeled PET image and the labeled CT image into the neural network, and generating output data representing the PET features and the CT features based on inputting the labeled PET image and the labeled CT image into the neural network; and

[0116] It is determined based on the output data that the neural network is trained.

[0117] Illustrative embodiment 13: The computer-implemented method of illustrative embodiment 12, comprising:

[0118] determining at least one metric value based on the output data; and

[0119] The neural network is determined to be trained based on the at least one metric.

[0120] Illustrative Embodiment 14: The computer-implemented method of any of Illustrative Embodiments 12-13, comprising storing parameters characterizing the first trained neural network in a data repository.

[0121] Illustrative Embodiment 15: The computer-implemented method of any of Illustrative Embodiments 1-14, comprising training a second trained neural network, the training comprising:

[0122] inputting the labeled PET features and the labeled CT features into a neural network, and generating output data representing a displacement value between the labeled PET features and the labeled CT features based on inputting the labeled PET features and the labeled CT features into the neural network; and

[0123] It is determined based on the output data that the neural network is trained.

[0124] Illustrative embodiment 16: The computer-implemented method of illustrative embodiment 15, comprising:

[0125] determining at least one metric value based on the output data; and

[0126] The neural network is determined to be trained based on the at least one metric.

[0127] Illustrative Embodiment 17: The computer-implemented method of any of Illustrative Embodiments 15-16, comprising storing parameters characterizing the first trained neural network in a data repository.

[0128] Illustrative Embodiment 18: A non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

[0129] receiving positron emission tomography (PET) measurement data and common modality measurement data from an image scanning system;

[0130] generating a PET image based on the PET measurement data and generating a common modality image based on the common modality measurement data;

[0131] Inputting the PET image and the co-modality image to a first trained neural network, and generating a first feature of the PET image and a second feature of the co-modality image based on inputting the PET image and the co-modality image to the first trained neural network;

[0132] inputting the first feature and the second feature into a second trained neural network, and generating displacement data representing a displacement between the first feature and the second feature based on inputting the first output data into the second trained neural network; and

[0133] Display data is generated based on the displacement data, and the display data is transmitted for display.

[0134] Illustrative Embodiment 19: The non-transitory computer readable medium of Illustrative Embodiment 18, wherein the common modality measurement data is computed tomography (CT) measurement data and the common modality image is a CT image.

[0135] Illustrative Embodiment 20: The non-transitory computer-readable medium of any one of Illustrative Embodiments 18-19, wherein the first trained neural network is a convolutional neural network (CNN).

[0136] Illustrative Embodiment 21: The non-transitory computer readable medium of any one of Illustrative Embodiments 18-20, wherein the second trained neural network is a convolutional neural network (CNN).

[0137] Illustrative Embodiment 22: The non-transitory computer readable medium of any one of Illustrative Embodiments 18-21, wherein the first feature of the PET image and the second feature of the common modality image comprise a common feature.

[0138] Illustrative Embodiment 23: The non-transitory computer readable medium of any one of Illustrative Embodiments 8-22, wherein the displacement data comprises at least one displacement value for each of a plurality of pixels of the PET image and the common modality image.

[0139] Illustrative Embodiment 24: The non-transitory computer-readable medium of Illustrative Embodiment 23, wherein the at least one displacement value for each of the plurality of pixels comprises a first displacement value in a first direction, a second displacement value in a second direction, and a third displacement value in a third direction.

[0140] Illustrative embodiment 25: The non-transitory computer readable medium of illustrative embodiment 24, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations comprising:

[0141] determining a magnitude for each of the plurality of pixels based on the first displacement value, the second displacement value, and the third displacement value; and

[0142] Generate display data based on the magnitude.

[0143] Illustrative Embodiment 26: The non-transitory computer readable medium of Illustrative Embodiment 25, wherein a data representation heat map is displayed.

[0144] Illustrative Embodiment 27: The non-transitory computer readable medium of any one of Illustrative Embodiments 18-26, wherein the displacement data comprises a displacement value identifying a pixel offset between the PET image and the common modality image.

[0145] Illustrative Embodiment 28: The non-transitory computer readable medium of any one of Illustrative Embodiments 18-27, wherein the PET measurement data and the common modality measurement data are based on corresponding scans of the same subject.

[0146] Illustrative Embodiment 29: The non-transitory computer-readable medium of any one of Illustrative Embodiments 18-28, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations comprising training a first trained neural network, the training comprising:

[0147] Inputting the labeled PET image and the labeled CT image into the neural network, and generating output data representing the PET features and the CT features based on inputting the labeled PET image and the labeled CT image into the neural network; and

[0148] It is determined based on the output data that the neural network is trained.

[0149] Illustrative embodiment 30: The non-transitory computer readable medium of illustrative embodiment 29, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations comprising:

[0150] determining at least one metric value based on the output data; and

[0151] The neural network is determined to be trained based on the at least one metric.

[0152] Illustrative Embodiment 31: The non-transitory computer-readable medium of any one of Illustrative Embodiments 29-30, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations comprising storing parameters characterizing a first trained neural network in a data repository.

[0153] Illustrative Embodiment 32: The non-transitory computer-readable medium of any one of Illustrative Embodiments 18-31, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations comprising training a second trained neural network, the training comprising:

[0154] inputting the labeled PET features and the labeled CT features into a neural network, and generating output data representing a displacement value between the labeled PET features and the labeled CT features based on inputting the labeled PET features and the labeled CT features into the neural network; and

[0155] It is determined based on the output data that the neural network is trained.

[0156] Illustrative embodiment 33: The non-transitory computer readable medium of illustrative embodiment 32, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations comprising:

[0157] determining at least one metric value based on the output data; and

[0158] The neural network is determined to be trained based on the at least one metric.

[0159] Illustrative embodiment 34: The non-transitory computer-readable medium of any one of illustrative embodiments 32-33, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform operations comprising storing parameters characterizing a first trained neural network in a data repository.

[0160] Illustrative Embodiment 35: A system comprising:

[0161] a memory device storing instructions; and

[0162] at least one processor communicatively coupled to the memory device, the at least one processor configured to execute instructions to:

[0163] receiving positron emission tomography (PET) measurement data and common modality measurement data from an image scanning system;

[0164] generating a PET image based on the PET measurement data and generating a common modality image based on the common modality measurement data;

[0165] Inputting the PET image and the co-modality image to a first trained neural network, and generating a first feature of the PET image and a second feature of the co-modality image based on inputting the PET image and the co-modality image to the first trained neural network;

[0166] inputting the first feature and the second feature into a second trained neural network, and

[0167] generating displacement data representing a displacement between the first feature and the second feature based on inputting the first output data into a second trained neural network; and

[0168] Display data is generated based on the displacement data, and the display data is transmitted for display.

[0169] Illustrative Embodiment 36: The system of Illustrative Embodiment 35, wherein the common modality measurement data is computed tomography (CT) measurement data and the common modality image is a CT image.

[0170] Illustrative Embodiment 37: The system of any one of Illustrative Embodiments 35-36, wherein the first trained neural network is a convolutional neural network (CNN).

[0171] Illustrative Embodiment 38: The system of any one of Illustrative Embodiments 35-37, wherein the second trained neural network is a convolutional neural network (CNN).

[0172] Illustrative Embodiment 39: The system of any of Illustrative Embodiments 35-38, wherein the first feature of the PET image and the second feature of the common modality image comprise a common feature.

[0173] Illustrative Embodiment 40: The system of any of Illustrative Embodiments 35-39, wherein the displacement data comprises at least one displacement value for each of a plurality of pixels of the PET image and the common modality image.

[0174] Illustrative Embodiment 41: The system of Illustrative Embodiment 40, wherein the at least one displacement value for each of the plurality of pixels comprises a first displacement value in a first direction, a second displacement value in a second direction, and a third displacement value in a third direction.

[0175] Illustrative Embodiment 42: The system of Illustrative Embodiment 41, wherein the at least one processor is configured to execute instructions to:

[0176] determining a magnitude for each of the plurality of pixels based on the first displacement value, the second displacement value, and the third displacement value; and

[0177] Generate display data based on the magnitude.

[0178] Illustrative Embodiment 43: The system of Illustrative Embodiment 42, wherein a data representation heat map is displayed.

[0179] Illustrative Embodiment 44: The system of any of Illustrative Embodiments 35-43, wherein the displacement data comprises displacement values ​​identifying pixel offsets between the PET image and the common modality image.

[0180] Illustrative Embodiment 45: The system of any of Illustrative Embodiments 35-44, wherein the PET measurement data and the common modality measurement data are based on corresponding scans of the same subject.

[0181] Illustrative Embodiment 46: The system of any of Illustrative Embodiments 35-45, wherein to train the first trained neural network, the at least one processor is configured to execute instructions to:

[0182] Inputting the labeled PET image and the labeled CT image into the neural network, and generating output data representing the PET features and the CT features based on inputting the labeled PET image and the labeled CT image into the neural network; and

[0183] It is determined based on the output data that the neural network is trained.

[0184] Illustrative Embodiment 47: The system of Illustrative Embodiment 46, wherein the at least one processor is configured to execute instructions to:

[0185] determining at least one metric value based on the output data; and

[0186] The neural network is determined to be trained based on the at least one metric.

[0187] Illustrative Embodiment 48: The system of any of Illustrative Embodiments 46-47, wherein the at least one processor is configured to execute instructions to store parameters characterizing the first trained neural network in a data repository.

[0188] Illustrative Embodiment 49: The system of any of Illustrative Embodiments 35-48, wherein to train the second trained neural network, the at least one processor is configured to execute instructions to:

[0189] inputting the labeled PET features and the labeled CT features into a neural network, and generating output data representing a displacement value between the labeled PET features and the labeled CT features based on inputting the labeled PET features and the labeled CT features into the neural network; and

[0190] It is determined based on the output data that the neural network is trained.

[0191] Illustrative embodiment 50: The system of illustrative embodiment 49, wherein the at least one processor is configured to execute instructions to:

[0192] determining at least one metric value based on the output data; and

[0193] The neural network is determined to be trained based on the at least one metric.

[0194] Illustrative Embodiment 51: The system of any of Illustrative Embodiments 49-50, wherein the at least one processor is configured to execute instructions to store parameters characterizing the first trained neural network in a data repository.

[0195] Illustrative Embodiment 52: A system comprising:

[0196] means for receiving positron emission tomography (PET) measurement data and common modality measurement data from an image scanning system;

[0197] means for generating a PET image based on the PET measurement data and generating a common modality image based on the common modality measurement data;

[0198] means for inputting the PET image and the common modality image into a first trained neural network, and generating a first feature of the PET image and a second feature of the common modality image based on inputting the PET image and the common modality image into the first trained neural network;

[0199] means for inputting the first feature and the second feature into a second trained neural network, and generating displacement data representing a displacement between the first feature and the second feature based on inputting the first output data into the second trained neural network; and

[0200] Means for generating display data based on the displacement data and transmitting the display data for display.

[0201] Illustrative Embodiment 53: The system of Illustrative Embodiment 52, wherein the common modality measurement data is computed tomography (CT) measurement data and the common modality image is a CT image.

[0202] Illustrative Embodiment 54: The system of any one of Illustrative Embodiments 52-53, wherein the first trained neural network is a convolutional neural network (CNN).

[0203] Illustrative Embodiment 55: The system of any one of Illustrative Embodiments 52-54, wherein the second trained neural network is a convolutional neural network (CNN).

[0204] Illustrative Embodiment 56: The system of any of Illustrative Embodiments 52-55, wherein the first feature of the PET image and the second feature of the common modality image comprise a common feature.

[0205] Illustrative Embodiment 57: The system of any of Illustrative Embodiments 52-56, wherein the displacement data comprises at least one displacement value for each of a plurality of pixels of the PET image and the common modality image.

[0206] Illustrative Embodiment 58: The system of Illustrative Embodiment 57, wherein the at least one displacement value for each of the plurality of pixels comprises a first displacement value in a first direction, a second displacement value in a second direction, and a third displacement value in a third direction.

[0207] Illustrative embodiment 59: The system of illustrative embodiment 58, comprising:

[0208] means for determining a magnitude for each of the plurality of pixels based on the first displacement value, the second displacement value, and the third displacement value; and

[0209] Means for generating display data based on the quantity value.

[0210] Illustrative Embodiment 60: The system of Illustrative Embodiment 59, wherein a data representation heat map is displayed.

[0211] Illustrative Embodiment 61: The system of any of Illustrative Embodiments 52-60, wherein the displacement data comprises displacement values ​​identifying pixel offsets between the PET image and the common modality image.

[0212] Illustrative Embodiment 62: The system of any of Illustrative Embodiments 52-61, wherein the PET measurement data and the common modality measurement data are based on corresponding scans of the same subject.

[0213] Illustrative Embodiment 63: The system of any of Illustrative Embodiments 52-62, comprising means for training a first trained neural network, the means for training comprising:

[0214] means for inputting the labeled PET image and the labeled CT image into the neural network, and generating output data characterizing the PET features and the CT features based on inputting the labeled PET image and the labeled CT image into the neural network; and

[0215] Means for determining that the neural network is trained based on the output data.

[0216] Illustrative Embodiment 64: The system of Illustrative Embodiment 63, comprising:

[0217] means for determining at least one metric value based on the output data; and

[0218] Means for determining that the neural network is trained based on at least one metric.

[0219] Illustrative Embodiment 65: The system of any of Illustrative Embodiments 63-64, comprising means for storing parameters characterizing the first trained neural network in a data repository.

[0220] Illustrative Embodiment 66: The system of any of Illustrative Embodiments 52-65, comprising means for training a second trained neural network, the means for training comprising:

[0221] means for inputting the labeled PET features and the labeled CT features into the neural network, and generating output data representing a displacement value between the labeled PET features and the labeled CT features based on inputting the labeled PET features and the labeled CT features into the neural network; and

[0222] Means for determining that the neural network is trained based on the output data.

[0223] Illustrative Embodiment 67: The system of Illustrative Embodiment 66, comprising:

[0224] means for determining at least one metric value based on the output data; and

[0225] Means for determining that the neural network is trained based on at least one metric.

[0226] Illustrative Embodiment 68: The system of any of Illustrative Embodiments 66-67, comprising means for storing parameters characterizing the first trained neural network in a data repository.

[0227] The apparatus and processes are not limited to the specific embodiments described herein. In addition, components of each apparatus and each process can be practiced independently and separately from other components and processes described herein.

[0228] The previous description of the embodiments is provided to enable any person skilled in the art to practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without resorting to inventive concept. The present disclosure is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A computer-implemented method comprising: receiving positron emission tomography (PET) measurement data and common modality measurement data from an image scanning system; generating a PET image based on the PET measurement data and generating a common modality image based on the common modality measurement data; Inputting the PET image and the co-modality image to a first trained neural network, and generating a first feature of the PET image and a second feature of the co-modality image based on inputting the PET image and the co-modality image to the first trained neural network; inputting the first feature and the second feature into a second trained neural network, and generating displacement data representing a displacement between the first feature and the second feature based on inputting the first output data into the second trained neural network; and Display data is generated based on the displacement data, and the display data is transmitted for display.

2. The computer-implemented method of claim 1, wherein the common-modality measurement data is computed tomography (CT) measurement data and the common-modality image is a CT image.

3. The computer-implemented method of claim 1 , wherein the first trained neural network is a convolutional neural network (CNN).

4. The computer-implemented method of claim 1 , wherein the second trained neural network is a convolutional neural network (CNN). 5 . The computer-implemented method of claim 1 , wherein the first feature of the PET image and the second feature of the common modality image comprise a common feature. 6 . The computer-implemented method of claim 1 , wherein the displacement data comprises at least one displacement value for each of a plurality of pixels of the PET image and the common modality image. 7 . The computer-implemented method of claim 6 , wherein the at least one displacement value for each of the plurality of pixels comprises a first displacement value in a first direction, a second displacement value in a second direction, and a third displacement value in a third direction.

8. The computer-implemented method of claim 7, comprising: determining a magnitude for each of the plurality of pixels based on the first displacement value, the second displacement value, and the third displacement value; and Generate display data based on the magnitude. The computer-implemented method of claim 8 , wherein the display data represents a heat map.

10. The computer-implemented method of claim 1, wherein the displacement data comprises displacement values ​​identifying pixel offsets between the PET image and the common modality image.

11. The computer-implemented method of claim 1 , wherein the PET measurement data and the common modality measurement data are based on corresponding scans of the same subject.

12. The computer-implemented method of claim 1 , comprising training a first trained neural network, the training comprising: Inputting the labeled PET image and the labeled CT image into the neural network, and generating output data representing PET features and CT features based on inputting the labeled PET image and the labeled CT image into the neural network; and It is determined based on the output data that the neural network is trained.

13. The computer-implemented method of claim 12, comprising: determining at least one metric value based on the output data; and The neural network is determined to be trained based on the at least one metric.

14. The computer-implemented method of claim 12, comprising storing parameters characterizing the first trained neural network in a data repository.

15. The computer-implemented method of claim 1 , comprising training a second trained neural network, the training comprising: inputting the labeled PET features and the labeled CT features into a neural network, and generating output data representing a displacement value between the labeled PET features and the labeled CT features based on inputting the labeled PET features and the labeled CT features into the neural network; and It is determined based on the output data that the neural network is trained.

16. The computer-implemented method of claim 15, comprising: determining at least one metric value based on the output data; and The neural network is determined to be trained based on the at least one metric.

17. The computer-implemented method of claim 15, comprising storing parameters characterizing the first trained neural network in a data repository.

18. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: receiving positron emission tomography (PET) measurement data and common modality measurement data from an image scanning system; generating a PET image based on the PET measurement data and generating a common modality image based on the common modality measurement data; Inputting the PET image and the co-modality image to a first trained neural network, and generating a first feature of the PET image and a second feature of the co-modality image based on inputting the PET image and the co-modality image to the first trained neural network; inputting the first feature and the second feature into a second trained neural network, and generating displacement data representing a displacement between the first feature and the second feature based on inputting the first output data into the second trained neural network; and Display data is generated based on the displacement data, and the display data is transmitted for display.

19. The non-transitory computer-readable medium of claim 18, wherein the common-modality measurement data is computed tomography (CT) measurement data and the common-modality image is a CT image.

20. A system comprising: a memory device for storing instructions; and at least one processor communicatively coupled to the memory device, the at least one processor configured to execute instructions to: receiving positron emission tomography (PET) measurement data and common modality measurement data from an image scanning system; generating a PET image based on the PET measurement data and generating a common modality image based on the common modality measurement data; Inputting the PET image and the co-modality image to a first trained neural network, and generating a first feature of the PET image and a second feature of the co-modality image based on inputting the PET image and the co-modality image to the first trained neural network; inputting the first feature and the second feature into a second trained neural network, and generating displacement data representing a displacement between the first feature and the second feature based on inputting the first output data into the second trained neural network; as well as Display data is generated based on the displacement data, and the display data is transmitted for display.

Images