Method and apparatus for performing image enhancement using a neural network in a medical imaging system

Abstract

This invention provides a method for performing neural network-based image enhancement in a medical imaging system. [Solution] A method comprising: acquiring an image of an imaging subject having multiple patches, wherein adjacent patches have overlapping regions; performing patch-by-patch image enhancement on the acquired image using a trained neural network; and synthesizing a neural network-enhanced image based on the output from the trained neural network and a first blending function, wherein the synthesis step performs weighted summation on each voxel in the overlapping region based on the first blending function to reduce checkerboard artifacts in the neural network-enhanced image.

Description

[Technical Field]

[0001] This application relates to medical imaging systems, including positron emission tomography (PET) scanners and computed tomography (CT) scanners. Specifically, this application relates to image processing performed by artificial neural networks (ANNs) in medical diagnostic imaging systems. [Background technology]

[0002] In recent years, the potential applications of artificial neural networks (ANNs) in medical imaging systems have been widely explored. Compared to other data formats such as text and numbers, training and inference with image-based networks requires far more storage capacity and computational power.

[0003] To address the high computational demands of ANN (Artificial Neural Network)-based image processing, several strategies are typically applied. (1) Utilize multi-GPU (graphics processing unit) computing, (2) Before inputting the image into the network, reduce the dimensionality of the image by downsampling, and (3) Cut the image into small volumes (patches) before inputting it into the network. [Overview of the project] [Problems that the invention aims to solve]

[0004] However, implementing multi-GPU computing requires significant investment in hardware and infrastructure. As the scale of the problem increases, these investments must also increase accordingly. This can lead to potential financial burdens or limit applicable network architectures and designs.

[0005] Downsampling images is an effective technique to reduce computational demands, but it often results in information loss, which is highly undesirable in medical applications. The degree of information loss depends on the available computing power and the specific network architecture or design applied.

[0006] When computational resources are limited, cropping the image before inputting it into the network allows for the use of more complex network architectures and designs without losing image information. As a result, this method is a preferred solution in ANN-based image processing. However, this technique can introduce checkerboard artifacts when the network-processed patches are recombined into the original image volume during inference.

[0007] Therefore, it is desirable to improve ANN-based image processing to mitigate these limitations. [Means for solving the problem]

[0008] Embodiments of this application relate to a method for performing neural network-based image enhancement in a medical imaging system. The method includes acquiring an image of an imaging subject comprising multiple patches, wherein adjacent patches have overlapping regions; performing patch-by-patch image enhancement on the acquired image using a trained neural network; and synthesizing a neural network-enhanced image based on the output from the trained neural network and a first blending function. The synthesis step performs weighted summation on each voxel in the overlapping region based on the first blending function to reduce checkerboard artifacts in the neural network-enhanced image.

[0009] Another embodiment of this application relates to a method for performing neural network-based image enhancement in a medical imaging system. The method includes receiving a neural network-enhanced image of a first image of an imaging subject, which is generated based on image enhancement by a trained neural network, wherein the first image comprises a plurality of patches, adjacent patches having overlapping regions, and the trained neural network performs image enhancement on a patch-by-patch basis; acquiring another image of the imaging subject different from the neural network-enhanced image; and generating a second image of the imaging subject using the neural network-enhanced image and the other image based on a blending function, thereby reducing checkerboard artifacts in the second image.

[0010] Further embodiments of this application relate to an apparatus for performing neural network-based image enhancement in a medical imaging system, the apparatus comprising a processing circuit configured to acquire an image of an imaging subject comprising a plurality of patches, wherein adjacent patches have overlapping regions; perform patch-by-patch image enhancement on the acquired image using a trained neural network; and synthesize a neural network-enhanced image based on the output from the trained neural network and a first blending function. The processing circuit is further configured to perform weighted summation on each voxel in the overlapping region based on the first blending function during synthesis to reduce checkerboard artifacts in the neural network-enhanced image.

[0011] It should be noted that this summary section of the invention does not specify all embodiments and / or incrementally novel aspects of the invention claimed in this application. Rather, the summary of the invention provides only a preliminary description of different embodiments and corresponding points of novelty. For further details and / or possible perspectives of the invention and embodiments, readers are directed to the sections on embodiments for carrying out the invention of this disclosure and the corresponding figures, which are further described below. [Brief explanation of the drawing]

[0012] This application will be better understood in light of the description provided in a non-limiting manner with the accompanying drawings. [Figure 1A] This shows an example of a checkerboard artifact observed in an image processed patch-by-patch by a neural network. [Figure 1B] This shows an example of a checkerboard artifact observed in an image processed patch-by-patch by a neural network. [Figure 2] A flowchart illustrating an exemplary method for mitigating checkerboard artifacts in images processed by a neural network, according to embodiments of this application, is shown. [Figure 3] An exemplary data flow of a method for mitigating checkerboard artifacts in images processed by a neural network is shown according to embodiments of this application. [Figure 4] An exemplary data flow of a method for mitigating checkerboard artifacts in images processed by a neural network is shown according to embodiments of this application. [Figure 5A] A perspective view of a positron emission tomography (PET) scanner that can be used in the techniques described herein is shown. [Figure 5B] A schematic diagram of a PET scanner that can be used with the technology described herein is shown. [Figure 6]The structure of ResNet, the weighting method of the weighted loss function, the sample coronal slices of the test subjects, and the box plots of the liver NSTD in the subjects in ResNet, ResNet_weighted, and ResNet_weighted_SUVclip before and after applying 4mm Gaussian smoothing are shown. [Figure 7] The network structure of AiCE, the sample coronal slices of one test subject, the box plots of the liver NSTD in the test subjects, and the correlation plots of the lesion SUVmax in each AiCE image and the lesion SUVmax in the 4mm Gaussian-filtered input image before and after applying 4mm Gaussian smoothing in AiCE-sharp, AiCE-standard, and AiCE-smooth are shown.

Embodiments for Carrying Out the Invention

[0013] Embodiments or examples for implementing various aspects of the present application may be described in the following sections. Specific examples of components and arrangements are described below to simplify the present application. Of course, these are merely examples and are not intended to be limiting.

[0014] The order of the descriptions of the different steps described in this specification is presented for clarity. Generally, these steps can be executed in any suitable order. Further, although each of the different features, techniques, configurations, etc. in this specification may be discussed at different places in the present application, it is intended that each concept can be executed independently of each other or in combination with each other. Therefore, the present application can be embodied and considered in many different ways.

[0015] Furthermore, as used in this specification, words such as "a," "an," etc. generally have the meaning of "one or more" unless otherwise specified.

[0016] As mentioned earlier, cutting an image into small patches for neural network processing can introduce checkerboard artifacts when the processed patches are reassembled into the original image volume during the inference phase.

[0017] One method to remove these checkerboard artifacts is to perform inference directly on the original, uncropped image. However, this method requires a GPU card with enough memory to handle the entire image size, which is often impractical or even impossible. Alternatively, it is possible to perform inference on the original image using the CPU (Central Processing Unit), but this significantly increases processing time.

[0018] Another technique to reduce checkerboard artifacts is to introduce overlap between adjacent patches during patch preparation. When the patches are combined into the original image volume, overlapping voxels are typically averaged, and the greater the overlap, the weaker the artifact. This technique is called the "overlap-and-average" method. However, in addition to increasing the total inference time, residual artifacts may still be observed, especially in uniform areas of the image.

[0019] Figures 1A and 1B show an example of checkerboard artifacts in images processed by a neural network using the overlap-and-average method. Figure 1A is an imaging tool useful in oncology. 18The image shows a coronal slice of a sample obtained from an F-FDG PET scan. Figure 1B shows the same image after processing with a denoising neural network in a patch-by-patch manner, followed by an overlap-and-average procedure to reduce checkerboard artifacts. The original image volume is 336 × 336 × 256 voxels. The patch size is 32 × 32 × 32 voxels, with 16 voxel overlap between adjacent patches in all directions. Both images use the same display scale (

[0005] ). As shown in Figure 1B, residual checkerboard artifacts are still visible, particularly in the liver region, despite the overlap-and-average processing.

[0020] This disclosure provides a method for synthesizing a network-processed image from patches without introducing checkerboard artifacts at the patch boundaries. This method uses weighted summation on overlapping voxels in adjacent patches based on weights provided by a user-defined blending function. The blending function is designed to ensure smooth intensity transitions across patch boundaries, effectively eliminating checkerboard artifacts in the synthesized image.

[0021] Figure 2 shows a flowchart of an exemplary procedure 200 for using a neural network to perform image enhancement in a medical imaging system according to an embodiment of the present application. This process begins at step S210, where a subject image is acquired. For example, a subject image can be acquired from an imaging subject by a medical imaging system. The acquired image is divided into multiple patches, with overlapping regions placed between adjacent patches.

[0022] In step S220, the acquired images are processed patch by patch using a trained neural network for image enhancement. This neural network can apply any desired enhancement to the input patch, including but not limited to improving image quality, reducing noise, improving smoothness, and adding sharper details.

[0023] A neural network can process all input patches or selectively process a subset that meets specific criteria. For example, a neural network can process only patches with obvious radioactive tracer activity exceeding a predefined threshold. Therefore, a neural network can be trained using only patches with obvious radioactive tracer activity exceeding a predefined threshold. This selective processing can reduce computation time in both the training and inference phases.

[0024] In step S230, the output from the neural network can be used to synthesize a neural network-enhanced image based on a first blending function. Specifically, a weighted sum is applied to each voxel in the overlapping region to reduce checkerboard artifacts in the synthesized image.

[0025] Weighted summation based on the first blending function effectively reduces most checkerboard artifacts, but residual artifacts may still appear in widespread, uniformly low-intensity areas (e.g., the air background surrounding the imaging subject). To further eliminate these residual artifacts, additional processing to refine the affected areas can optionally be applied.

[0026] In step S240, related but different images of the imaging subject are acquired. These images can originate from various sources, as long as they relate to the same imaging subject and are different from the neural network-enhanced image synthesized in step S230. In step S250, the acquired images are blended with the neural network-enhanced image based on a second blending function. By combining the voxels of these two images, checkerboard artifacts can be further minimized across the entire image.

[0027] As shown in Figure 1A 18 Using an example of an F-FDG PET image, Figure 3 illustrates the process of performing weighted summation on overlapping voxels between adjacent patches based on a first blending function. All images in Figure 3 use the same display scale (

[0005] ). For simplification, this illustration is shown in two dimensions (2D).

[0028] As shown in Figure 3, the patches of the noise input image in Figure 1A are processed by the denoising network 310 to generate denoised patches. These denoised patches are multiplied by weights determined by the first blending function, resulting in denoised weighted patches. Subsequently, a denoised weighted image can be synthesized from the denoised weighted patches.

[0029] To avoid intensity changes in the synthesized image, it is desirable that the sum of the weights applied to each pixel equals 1. Alternatively, if the sum of weights across the entire image volume is not always 1, a normalization map can be maintained to track the total weights at each location. The synthesized image can then be normalized by dividing it by this normalization map, correcting for any potential intensity variations caused by the weighting process.

[0030] As mentioned above, the weights used in weighted summation are determined by the first blending function. In Figure 3, a Gaussian kernel function with a standard deviation equal to 1 / 4 of the patch size (sigma = 8) is used as an exemplary blending function. However, those skilled in the art will understand that the blending function can take various forms, as long as it ensures smooth intensity transitions across the overlapping regions between image patches.

[0031] Figure 3 shows the processing and resynthesis of all patches in the input image, but as mentioned above, the denoising network 310 can also be configured to selectively process patches that meet predefined criteria. This method can reduce computation time in the training and inference phases.

[0032] Figure 3 further compares the denoised, weighted, and normalized image using the method described in this application with the network-processed image shown in Figure 1B. As illustrated, the method of this application significantly reduces checkerboard artifacts, particularly in the liver region of the network-weighted image.

[0033] If checkerboard artifacts remain visible in widespread, uniform, and low-intensity areas, additional blending can be applied between corresponding voxels from related but different images. This blending process is performed on the entire image, rather than on individual patches.

[0034] Figure 4 illustrates the data flow in the process of performing weighted summation between corresponding voxels from two related but different images. To make residual checkerboard artifacts in the air background more visible, all images in Figure 4 are displayed using a low and narrow display scale of [0 0.02]. This illustration is shown in 2D for simplification.

[0035] Figure 4 shows the blending of the following two images. • Image A: Network-enhanced image shown in Figure 3, and • Image B: Smoothed image of the noise input image shown in Figure 1A, generated by a 3mm full width at half maximum (FWHM) Gaussian filter.

[0036] The weights assigned to the voxels of the two images are determined by a second blending function. This blending function can be designed to ensure smooth transitions across the intensity range observed in the region where residual checkerboard artifacts need to be removed.

[0037] In some embodiments of this application, the second blending function has an active intensity window bounded by upper and / or lower intensity limits. For example, in the example shown in Figure 4, a sigmoid blending function is used with a lower limit of 0.012 and an upper limit of 0.1 in Standardized Uptake Value (SUV) units. This active range covers the intensity distribution of the air background region.

[0038] For example, the active intensity window of the second blending function can also guide the neural network in deciding which patches to process. For instance, voxels in the original image with intensity values ​​outside the defined intensity window may remain untouched by the neural network. In the blending process shown in Figure 4, these skipped voxels are replaced by corresponding voxels from the 3mm Gaussian smoothed image. Allowing the neural network to skip such voxels during the training and inference phases reduces both training and inference times, leading to improved computational efficiency.

[0039] It should be noted that the aforementioned sigmoid blending function is merely one example. The second blending function can take various forms. Furthermore, those skilled in the art can design multiple blending functions according to the local characteristics of the image volume and apply them to different image regions. This method allows for the incorporation of multiple blending functions according to local image characteristics, thus enabling more natural and effective artifact removal across the entire image volume.

[0040] As shown in Figure 4, the weight maps used for image A and image B are inversely related to each other. For example, if the weight of voxel j from image A is w aj If so, the weight of the corresponding voxel j' from image B is (1-w aj This results in the total weight of each voxel in the image volume always being maintained at 1, ensuring that the original intensity of the voxel remains unaffected.

[0041] Similar to the method described in Figure 3, a normalization map can be used when the sum of weights across the entire image volume is not necessarily equal to 1. The normalization map tracks the total weights in each voxel and allows for normalization to compensate for potential intensity variations.

[0042] The embodiments described in this application provide an intensity blending strategy for removing checkerboard artifacts in images synthesized from network-processed patches, while simultaneously significantly reducing inference time. By incorporating a user-designed blending function to achieve smooth intensity transitions, checkerboard artifacts can be effectively removed across the entire image volume.

[0043] Figures 5A and 5B illustrate embodiments of a medical imaging system including a PET scanner that can implement the method described in this application. It is important to note that this image preprocessing method is applicable to other imaging modalities, including but not limited to X-ray / CT imaging.

[0044] In Figure 5A, the PET scanner includes multiple gamma-ray detectors (GRDs) (e.g., GRD1, GRD2-GRDN), each configured as a rectangular detector module. Each GRD may include a two-dimensional array of individual detector crystals that absorb gamma rays and emit scintillation photons. These scintillation photons can be detected by a two-dimensional array of photodetectors or photosensors, such as photomultiplier tubes (PMTs) or silicon photomultiplier tubes (SiPMs). Optical waveguides can be placed between the array of detector crystals and the photodetectors.

[0045] Each photodetector (e.g., PMT or SiPM) can generate an analog signal indicating when a scintillation event occurs, as well as the energy of the gamma ray that generates the detection event. Furthermore, photons emitted from one detector crystal can be detected by two or more photodetectors, and based on the analog signals generated by each photodetector, the detector crystal corresponding to the detection event can be determined, for example, using Anger logic and crystal decoding.

[0046] Figure 5B shows an example of a PET scanner configuration, where the subject OBJ to be imaged is placed on a stage 516, and the GRD modules GRD1 through GRDN are arranged circumferentially around the subject OBJ and stage 516. The GRDs can be fixedly connected to annular components 520 which are fixedly connected to a gantry 540. The gantry 540 houses many of the components of the PET scanner. The gantry 540 of the PET scanner also includes an opening through which the subject OBJ and stage 516 can pass, allowing the GRDs to detect gamma rays emitted in the opposite direction from the subject OBJ due to annihilation events, and timing and energy information can be used to determine the coincidence of gamma ray pairs.

[0047] Figure 5B also shows the circuitry and hardware for acquiring, storing, processing, and distributing gamma-ray detection data. This circuitry and hardware includes a processor 570, a network controller 574, memory 578, and a Data Acquisition System (DAS) 576. The PET scanner also includes data channels that route detection measurement results from the GRD to the DAS 576, processor 570, memory 578, and network controller 574. The Data Acquisition System 576 can control the acquisition, digitization, and routing of detection data from the detector. In one embodiment, the DAS 576 controls the movement of the bed 516. The processor 570 performs functions including image reconstruction from detection data, pre-reconstruction processing of detection data, and post-reconstruction processing of image data, as described herein.

[0048] The processor 570 can be configured to perform various steps and variations of the method described herein. The processor 570 may include a CPU that can be implemented as individual logic gates, such as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other Complex Programmable Logic Device (CPLD). The FPGA or CPLD implementation may be coded in VHDL, Verilog, or other hardware description language, and the code may be stored directly in the electronic memory within the FPGA or CPLD, or in separate electronic memory. Furthermore, the memory may be non-volatile, such as ROM, EPROM, EEPROM, or flash memory. The memory may also be volatile, such as static RAM or dynamic RAM, and a processor, such as a microcontroller or microprocessor, may be provided to manage the electronic memory and manage the interaction between the FPGA or CPLD and the memory.

[0049] Alternatively, the CPU within processor 570 may execute a computer program containing a set of computer-readable instructions that perform various steps of the described method, and the program may be stored in the non-transient electronic memory and / or hard disk drive, CD, DVD, flash drive, or any other known storage medium as described above. Furthermore, the computer-readable instructions may be provided in utility applications, background daemons, or components of an operating system, or a combination thereof, and may be executed in cooperation with processors such as Intel Xeon or AMD Opteron processors, and Microsoft Vista, UNIX, Solaris, Linux, Apple, MAC-OS, and other operating systems known to those skilled in the art. Furthermore, the CPU may be implemented as multiple processors that work concurrently and cooperatively to execute instructions.

[0050] The memory 578 may be a hard disk drive, CD-ROM drive, DVD drive, flash drive, RAM, ROM, or other electronic storage device known in the art.

[0051] Network controllers 574, such as the Intel Ethernet PRO network interface card manufactured by Intel Corporation in the United States, can interface between various parts of the PET scanner. Furthermore, network controller 574 can also interface with an external network. As can be understood, the external network can be a public network such as the Internet, a private network such as a LAN or WAN network, or any combination thereof, and may include a PSTN or ISDN subnetwork. The external network can be wired, such as an Ethernet network, or wireless, such as a cellular network including EDGE, 3G, and 4G wireless cellular systems. The wireless network can be WiFi, Bluetooth, or other known wireless communication formats.

[0052] To aid in understanding the various embodiments, various techniques have been described as multiple separate operations. The order of the description should not be interpreted as meaning that these operations are necessarily order-dependent. In fact, these operations do not need to be performed in the order presented. The described operations may be performed in a different order than in the embodiments described. In additional embodiments, various additional operations may be performed, and / or the described operations may be omitted.

[0053] In light of the above teachings, numerous modifications and variations of the embodiments presented herein are possible. Therefore, it should be understood that, within the scope of the claims, this application may be implemented in ways other than those specifically described herein. The present invention is not limited to the embodiments described herein. In particular, features of the illustrated embodiments can be combined with each other in variations not shown.

[0054] Furthermore, embodiments of this disclosure may be described in the following appendix.

[0055] (1) A method for performing neural network-based image enhancement in a medical imaging system, comprising: acquiring an image of an imaging subject comprising a plurality of patches, wherein adjacent patches have overlapping regions; performing patch-by-patch image enhancement on the acquired image using a trained neural network; and synthesizing a neural network-enhanced image based on the output from the trained neural network and a first blending function, wherein the synthesis step performs weighted summation on each voxel in the overlapping region based on the first blending function to reduce checkerboard artifacts in the neural network-enhanced image.

[0056] (2) The method according to (1), wherein the step of performing image enhancement using a trained neural network further includes classifying a plurality of patches into a first group of patches and a second group of patches based on a second blending function, performing image enhancement on each patch of the first group of patches in a patchwise manner using the trained neural network to derive enhanced patches of the first group of patches, and leaving the patches of the second group unchanged by the trained neural network, and the synthesis step further includes synthesizing the enhanced patches of the first group of patches and the unmodified patches of the second group to produce a neural network-enhanced image.

[0057] (3) The method according to (2), further comprising: the classification step of determining an intensity window based on a second blending function; and for each of a plurality of patches, classifying the patch into a first group of patches if the patch has an intensity range within the determined intensity window; and classifying the patch into a second group of patches if the patch has an intensity range outside the determined intensity window.

[0058] (4) The method of (3), wherein the trained neural network is trained using a set of input patches and a corresponding set of target patches, and each input patch in the set of input patches has an intensity range within a determined intensity window.

[0059] (5) The method according to (2), wherein the first blending function is predefined to maintain intensity transitions below a predefined threshold across overlapping regions related to the first group of patches, thereby minimizing checkerboard artifacts in the neural network-enhanced image.

[0060] (6) The first blending function is a Gaussian kernel function having a standard deviation corresponding to the ratio of the patch sizes of the multiple patches, as described in (5).

[0061] (7) The method according to (1), wherein the synthesis step further includes keeping the sum of the weights equal to 1 during the weighted sum for each voxel in the overlapping region.

[0062] (8) The method of (1), wherein the synthesis step further includes, if during weighted summation the sum of weights for voxels in overlapping regions is not always 1, recording the weights used for the voxels using a normalization map and canceling out the intensity changes in the neural network-enhanced image caused by the weights used, using a normalization map.

[0063] (9) The method according to (3), further comprising: obtaining another image of the imaging subject different from the neural network-enhanced image; and generating a final image of the imaging subject using the neural network-enhanced image and the other image based on a second blending function, such that checkerboard artifacts in the final image are reduced.

[0064] (10) Another image is obtained by performing Gaussian filtering on the acquired image, as described in (9).

[0065] (11) The method according to (9), further comprising the step of generating a final image using a neural network-enhanced image and another image, the step of generating a first weight map for the neural network-enhanced image and a second weight map for the acquired other image, which are inversely related to each other, based on a second blending function, and synthesizing the neural network-enhanced image and the acquired other image by performing a weighted sum on the generated first and second weight maps to reduce checkerboard artifacts in the final image.

[0066] (12) The method according to (9), wherein a second blending function is predefined to maintain intensity transitions below a predefined threshold across overlapping regions related to the second group of patches, thereby minimizing checkerboard artifacts in the final image.

[0067] (13) The method according to (3), further comprising the determination step of determining an intensity window based on the active intensity range of a second blending function, thereby classifying among multiple patches patches related to the imaging subject into a first group of patches and patches related to the air background region into a second group of patches.

[0068] (14) A method for performing neural network-based image enhancement in a medical imaging system, comprising: receiving a neural network-enhanced image of a first image of an imaging subject, which is generated based on image enhancement by a trained neural network, wherein the first image comprises a plurality of patches, adjacent patches have overlapping regions, and the trained neural network performs image enhancement on a patch-by-patch basis; acquiring another image of the imaging subject different from the neural network-enhanced image; and generating a second image of the imaging subject using the neural network-enhanced image and the other image based on a blending function, and reducing checkerboard artifacts in the second image.

[0069] (15) Another image is obtained by performing Gaussian filtering on the first image, as described in (14).

[0070] (16) A medical imaging system comprising a device for performing neural network-based image enhancement, the device comprising a processing circuit configured to acquire an image of an imaging subject comprising a plurality of patches, wherein adjacent patches have overlapping regions, perform patch-by-patch image enhancement on the acquired image using a trained neural network, and synthesize a neural network-enhanced image based on the output from the trained neural network and a first blending function, wherein the processing circuit is further configured to perform weighted summation on each voxel in the overlapping region based on the first blending function during synthesis, thereby reducing checkerboard artifacts in the neural network-enhanced image.

[0071] (17) The apparatus according to (16), further configured such that the processing circuit classifies a plurality of patches into a first group of patches and a second group of patches based on a second blending function, performs patch-by-patch image enhancement on each patch of the first group of patches using a trained neural network to derive enhanced patches of the first group of patches, leaves the patches of the second group unchanged by the trained neural network, and synthesizes the enhanced patches of the first group of patches and the unchanged patches of the second group to generate a neural network-enhanced image.

[0072] (18) The apparatus according to (17), further configured such that the processing circuit determines an intensity window based on a second blending function, and for each of a plurality of patches, classifies the patch into a first group of patches if the patch has an intensity range within the determined intensity window, and classifies the patch into a second group of patches if the patch has an intensity range outside the determined intensity window.

[0073] (19) The apparatus according to (16), further configured to acquire another image of the imaging subject, different from the neural network-enhanced image, and to generate a final image of the imaging subject using the neural network-enhanced image and the other image based on a second blending function, thereby reducing checkerboard artifacts throughout the final image.

[0074] (20) Another image is obtained by performing Gaussian filtering on the acquired image, using the apparatus described in (19).

[0075] In light of the above teachings, numerous modifications and variations of the embodiments presented herein are possible. Therefore, it should be understood that, within the scope of the claims, this disclosure may be implemented in ways other than those specifically described herein.

[0076] The aspects of this disclosure can be further described as follows.

[0077] 18 F-FDG: A deep convolutional neural network based on physiological information for PET image noise reduction.

[0078] Introduction: The diagnostic accuracy of positron emission tomography (PET) images is often affected by noise levels. In practice, regularization reconstruction and post-reconstruction smoothing have been applied to reduce noise, but require additional parameter adjustments to achieve a good signal-to-noise trade-off in various applications. In recent years, deep learning-based algorithms for PET image denoising have been widely studied due to their computational efficiency and the potential to handle more complex scenarios than conventional methods if properly trained. In this study, we propose two training strategies to improve network performance by increasing attention to clinically relevant information. Both strategies are applicable to clinical research. 18 It is developed based on general knowledge about the F-FDG distribution.

[0079] Methods: In this study, a residual network (ResNet) was used for the image noise reduction task (Figure 6(a)). The training dataset consisted of 16 cases acquired using a clinical PET-CT system (Cartesion Prime; Canon Medical Systems) in list mode with long-term acquisition of 10 minutes per bed position. 18 F-FDG examination was included. List-mode data from each 10-minute session were randomly downsampled to generate six noise inputs corresponding to acquisitions at 30, 45, 60, 90, 120, and 180 seconds. All data were reconstructed using four iterations of list-mode OSEM with 12 subsets. Image intensity was converted to standardized acquisition (SUV) units. Images reconstructed from full-count data were post-processed and smoothed using a 4mm Gaussian filter to serve as training targets. Patch-based training was employed to accommodate available computational resources. We,18 A training strategy was developed to compensate for the lack of high-intensity features in F-FDG images. This strategy incorporates a weighted loss function (Fig. 6(b)) that assigns higher weights to patches with SUV values exceeding 5, which are often associated with cancerous tissues. In the urinary system, the 18 To prevent the learning of the network from being affected by the high intensity of excreted 18 F, SUV clipping was also implemented, which sets voxel SUV values exceeding 20 to 20. This SUV clipping also unified the intensity range of the training images. The network was trained with and without incorporating the weighted loss function and SUV clipping. The impact of these on network performance was evaluated using 7 separate clinical whole-body

[0080] Results: Sample coronal slices from test subjects are shown in Figure 6(c) for a standard ResNet, a ResNet trained with a weighted loss function (ResNet_weighted), and a ResNet trained with a weighted loss function and SUV clipping (ResNet_weighted_SUVclip), along with the corresponding noise input before and after smoothing with a 4mm Gaussian filter (GF4). Magnified images of the mediastinal region and profiles traversing two small lesions within that region are also provided to facilitate visual evaluation. In the standard ResNet image, one of the lesions is almost completely blurred. In the image from the ResNet trained with a weighted loss function, this lesion is better preserved. All ResNet images show significantly less visual noise than the input images and the 4mm Gaussian smoothed images. This is further demonstrated in the box plots of liver NSTDs across all test images (Figure 6(d)). In ResNet box plots, the minimum variability in liver NSTD between images is obtained when SUV clipping is applied.

[0081] Discussion: In order to improve network performance in clinical research 18 Based on general knowledge about the distribution of F-FDG, we designed two training strategies. 18 In applications of F-FDG PET image denoising, the proposed weighted loss function and the incorporation of SUV clipping help preserve small lesions and reduce performance variability, respectively. Current applications are 18 While the focus is on F-FDG PET image denoising, these strategies should be applicable to other applications with appropriate modifications.

[0082] Figure 6(a) Structure of ResNet; Figure 6(b) Weighting scheme of the weighted loss function; Figure 6(c) Coronal slices of test subjects' samples before and after 4mm Gaussian smoothing in ResNet, ResNet_weighted, and ResNet_weighted_SUVclip; Figure 6(d) Box plot of liver NSTD in subjects.

[0083] The aspects of this disclosure can be further described as follows.

[0084] 18 Performance evaluation of deep convolutional neural networks based on physiological information for F-FDG PET image noise reduction.

[0085] Introduction: Noise reduction is one of the most promising applications of deep learning algorithms in medical imaging. Recently, we have found that in clinical research... 18 We developed a physiologically-based deep convolutional neural network that incorporates general knowledge about the distribution of F-FDG into its training process. This network, called the Advanced Intelligent Clear-IQ Engine (AiCE), demonstrates the potential for significant noise reduction without compromising feature rendering and quantification. To provide insights into the clinical utility of AiCE, this study involved 50 patients. 18 This report presents qualitative and quantitative evaluations of F-FDG PET images.

[0086] Methods: The AiCE network architecture is shown in Figure 7(a). The training dataset consisted of 16 cases acquired using a Canon Medical Cartesion Prime PET / CT system in list mode, with a protocol of 10 minutes per bed position. 18The dataset consisted of F-FDG PET scans. Low-count data, serving as training noise input, were generated by randomly downsampling each 10-minute list-mode data to six count levels corresponding to 30, 45, 60, 90, 120, and 180-second acquisitions. Each data point was then reconstructed using four iterations of Ordered-Subsets Expectation Maximization (OSEM) with 12 subsets and converted to standardized acquired values ​​(SUVs). To accommodate a wide range of physician preferences for image noise levels, AiCE offers three levels of denoising, referred to in this study as AiCE-sharp, AiCE-standard, and AiCE-smooth. This was achieved by using reconstructed images of full-count data post-smoothed with 2mm, 3mm, and 4mm Gaussian filters as targets for the three levels, respectively. A weighted loss function was incorporated to enhance the network's attention to SUVs potentially associated with cancerous tissue. To unify the intensity range of the training images and prevent network learning from being influenced by high intensity values ​​in the urinary tract, SUV clipping was applied, which replaces SUVs above a specific threshold with that threshold. The AiCE network was trained on whole-body images from 50 patients acquired using a protocol of 2 minutes per bed position. 18 The study was tested using F-FDG PET scans. In addition to visual assessment, the normalized standard deviation (NSTD) of the volume of interest (VOI) of the liver and the maximum SUV (SUVmax) of 42 lesions visualized by experienced physicians were also measured. For each AiCE denoising level, linear regression analysis was performed using the corresponding SUVmax in the noise input, post-processed and smoothed with a 4 mm Gaussian filter, as a reference.

[0087] Results: Sample coronal slices from one test subject at each level of AiCE denoising are shown in Figure 7(b), along with the noise input before and after smoothing with a 4mm Gaussian filter. Fine lesion structures (indicated by red arrows) are preserved in the AiCE images, with a slight smoothness in the AiCE-smooth image compared to other images. All AiCE images show significantly less visual noise than the input images and the 4mm Gaussian filtered images. The lowest visual noise levels were observed in the AiCE-smooth image, followed by the AiCE-standard image and the AiCE-sharp image. This observation is consistent with the box plots of liver NSTD values ​​in the test subject shown in Figure 7(c). The lesion SUVmax in each AiCE image was plotted against the corresponding value in the 4mm Gaussian filtered input image, and linear regression analysis was performed (Figure 7(d)). The lesion SUVmax in AiCE images is highly correlated with the SUVmax in 4mm Gaussian filtered images, regardless of the AiCE denoising level. The regression line also shows that applying AiCE-sharp / AiCE-smooth leads to slightly higher / lower lesion SUVmax than in 4mm Gaussian filtered images. AiCE-standard provides images with lesion SUVmax equivalent to that of 4mm Gaussian filtered images.

[0088] Discussion: We, 18We developed a physiologically-based deep convolutional neural network for denoising F-FDG PET images. Three levels of denoising are provided by training the network with high-count images post-processed and smoothed using different Gaussian filters. The results show that all three levels can achieve significant noise reduction while preserving fine features and lesion SUVmax. Quantitative analysis focuses on lesion SUVmax, as it is often used as a reference in lesion segmentation using thresholding, despite being relatively sensitive to the application of denoising filters. Although not shown, the mean SUV of lesions is consistent between the Gaussian post-processed and smoothed images and the images provided by the proposed network.

[0089] Figure 7(a) Network structure of AiCE; Figure 7(b) Coronal slices of a sample from one test subject before and after 4mm Gaussian smoothing, in AiCE-sharp, AiCE-standard, and AiCE-smooth; Figure 7(c) Box plot of liver NSTD in the test subject; Figure 7(d) Correlation plot between lesion SUVmax in each AiCE image and lesion SUVmax in the 4mm Gaussian filtered input image.

Claims

1. A method for performing image enhancement based on a neural network in a medical imaging system, Acquiring an image of an imaging subject that includes multiple patches, with adjacent patches having overlapping regions, Using a trained neural network, image enhancement is performed on the acquired images in a patch-by-patch manner, The process includes synthesizing a neural network-enhanced image based on the output from the trained neural network and a first blending function, The synthesis step performs weighted summation on each voxel in the overlapping region based on the first blending function to reduce checkerboard artifacts in the neural network-enhanced image. method.

2. The step of performing image enhancement using the aforementioned trained neural network is: Based on a second blending function, the plurality of patches are classified into a first group of patches and a second group of patches, Using the trained neural network, image enhancement is performed on each patch of the first group of patches in a patch-by-patch manner, and the enhanced patches of the first group are derived. The method further includes leaving the second group of patches unchanged by the trained neural network, The synthesis step further includes synthesizing the enhanced first group of patches and the unmodified second group of patches to generate the neural network enhanced image. The method according to claim 1.

3. The aforementioned classification step is, Determining the intensity window based on the second blending function described above, For each of the aforementioned multiple patches, If the patch has an intensity range within the determined intensity window, the patch is classified as a patch of the first group. If the patch has an intensity range outside the determined intensity window, the patch is classified as a patch of the second group. The method according to claim 2, further comprising:

4. The method according to claim 3, wherein the trained neural network is trained using a set of input patches and a corresponding set of target patches, each input patch in the set of input patches having an intensity range within the determined intensity window.

5. The method according to claim 2, wherein the first blending function is predefined to maintain intensity transitions below a predefined threshold across the overlapping region relating to the first group of patches, thereby minimizing the checkerboard artifact in the neural network-enhanced image.

6. The method according to claim 5, wherein the first blending function is a Gaussian kernel function having a standard deviation corresponding to the ratio of the patch sizes of the plurality of patches.

7. The method according to claim 1, wherein the synthesis step further includes keeping the sum of the weights equal to 1 during the weighted sum for each voxel in the overlapping region.

8. The aforementioned synthesis step is During the weighted summation, if the sum of the weights for the voxels in the overlapping region is not always 1, a normalization map is used to record the weights used for the voxels. Using the normalization map, the intensity changes that occurred in the neural network-enhanced image due to the weights used are canceled out. The method according to claim 1, further comprising:

9. To acquire another image of the imaging subject, which is different from the neural network-enhanced image, Based on the second blending function, the neural network-enhanced image and the other image are used to generate a final image of the imaging subject, and the checkerboard artifact in the final image is reduced. The method according to claim 3, further comprising:

10. The method according to claim 9, wherein the aforementioned other image is obtained by performing Gaussian filtering on the acquired image.

11. The step of generating the final image using the neural network-enhanced image and the other image is: Based on the second blending function, generate a first weight map for the neural network-enhanced image and a second weight map for the acquired image, which are inversely related to each other. The neural network-enhanced image and the acquired image are synthesized by performing a weighted summation based on the generated first and second weight maps, thereby reducing checkerboard artifacts in the final image. The method according to claim 9, further comprising:

12. The method according to claim 9, wherein the second blending function is predefined to maintain intensity transitions below a predefined threshold across the overlapping region relating to the second group of patches, thereby minimizing the checkerboard artifact in the final image.

13. The aforementioned decision step is, The intensity window is determined based on the active intensity range of the second blending function, thereby classifying the patches related to the imaging subject into the first group of patches and the patches related to the air background region into the second group of patches. The method according to claim 3, further comprising:

14. A method for performing image enhancement based on a neural network in a medical imaging system, The method involves receiving a neural network-enhanced image generated based on image enhancement by a trained neural network for a first image of an imaging subject, wherein the first image comprises a plurality of patches, adjacent patches have overlapping regions, and the trained neural network performs the image enhancement in a patch-by-patch manner. To acquire another image of the imaging subject, which is different from the neural network-enhanced image, Based on a blending function, a second image of the imaging subject is generated using the neural network-enhanced image and the other image, and checkerboard artifacts in the second image are reduced. Methods that include...

15. The method according to claim 14, wherein the other image is obtained by performing Gaussian filtering on the first image.

16. A device for performing image enhancement based on a neural network in a medical imaging system, An image of an imaging subject is obtained that includes multiple patches, with adjacent patches having overlapping regions. Using a trained neural network, image enhancement is performed on the acquired images in a patch-by-patch manner. The system includes a processing circuit configured to synthesize a neural network-enhanced image based on the output from the trained neural network and a first blending function, The processing circuit is further configured to perform weighted summation on each voxel within the overlapping region based on the first blending function during synthesis, thereby reducing checkerboard artifacts in the neural network-enhanced image. Device.

17. The aforementioned processing circuit is Based on the second blending function, the plurality of patches are classified into a first group of patches and a second group of patches. Using the trained neural network, image enhancement is performed on each patch of the first group of patches in a patch-by-patch manner, and the enhanced patches of the first group are derived. The second group of patches is left untouched by the trained neural network. The enhanced first group of patches and the unmodified second group of patches are combined to generate the neural network enhanced image. The apparatus according to claim 16, further configured as follows.

18. The aforementioned processing circuit is The intensity window is determined based on the second blending function described above. For each of the aforementioned multiple patches, If the patch has an intensity range within the determined intensity window, the patch is classified as a patch of the first group. If the patch has an intensity range outside the determined intensity window, the patch is classified as a patch of the second group. The apparatus according to claim 17, further configured as follows.

19. The aforementioned processing circuit is A different image of the imaging subject, distinct from the neural network-enhanced image, is obtained. Based on the second blending function, the neural network-enhanced image and the other image are used to generate a final image of the imaging subject, and the checkerboard artifact is reduced throughout the final image. The apparatus according to claim 16, further configured as follows.

20. The apparatus according to claim 19, wherein the aforementioned other image is obtained by performing Gaussian filtering on the acquired image.

Images