Image harmonization for deep learning model optimization

Abstract

A method for optimizing deep learning model performance using image harmonization as a preprocessing step includes decomposing, by a system operatively coupled to a processor, an input image into sub-images. The method also includes harmonizing the sub-images with corresponding reference sub-images based on two or more different statistical values computed for the sub-images and the corresponding reference sub-images, respectively, of at least one reference image, resulting in a transformation of the sub-images into modified sub-image images. The modified sub-images can be combined into a harmonized image that has a more similar appearance relative to the input image and the at least one reference image. The harmonized images and / or modified sub-images generated using these techniques can be used as ground truth training samples for training one or more deep learning models to transform input images having appearance variations into harmonized images.

Description

Technical Field

[0001] This application relates to image harmonization techniques for deep learning model optimization. Background Technology

[0002] Advances in artificial intelligence (AI) and machine learning (ML) technologies, such as deep neural networks (DNNs), have led to the development of AI / ML models that have demonstrated excellent performance in medical image processing and analysis tasks such as diagnosis, organ segmentation, anomaly detection, and image reconstruction. Most typically, these models are trained on images from a specific source domain. When applied to images that differ in appearance from those in the source domain due to various factors (e.g., image capture protocol, dosage, exposure settings, photon-receiving materials, field of view (FOV), demographics, contrast versus non-contrast, etc.), model performance degradation is commonly observed. Tuning these models to accurately perform on images from other domains is difficult and expensive. Therefore, efficient and effective techniques are needed to maintain or improve model performance on images with different appearance variations relative to the original training images. Summary of the Invention

[0003] The following summary is presented to provide a basic understanding of one or more embodiments of the invention. This summary is not intended to identify key or essential elements, nor is it intended to depict any scope of the different embodiments or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to the more detailed description that follows. In one or more embodiments described herein, systems, computer-implemented methods, apparatuses, and / or computer program products providing image harmonization techniques for deep learning model optimization are described.

[0004] According to one embodiment, a system is provided that includes a memory and a processor, the memory storing computer-executable components, the processor executing the computer-executable components stored in the memory. These computer-executable components may include: an image decomposition component that decomposes an input image into sub-images; and a harmonization component that harmonizes the sub-images with the corresponding reference sub-images based on two or more distinct characteristics calculated respectively for the sub-images and corresponding reference sub-images of at least one reference image, thereby resulting in a transformation of the sub-images into a modified sub-image image. In some specific embodiments, these computer-executable components may further include: a reconstruction component that combines the modified sub-images into a harmonized image, the harmonized image having a more similar appearance to the at least one reference image relative to the input image.

[0005] In various specific embodiments, these sub-images include band images, and these reference sub-images include corresponding reference band images. In these specific embodiments, the two or more distinct characteristics may include statistical values ​​selected from the group consisting of: the mean of these band images and these corresponding reference band images, the standard deviation of these band images, the percentiles of these band images, and histograms of these band images.

[0006] These computer-executable components further include: a reference image selection component that selects the at least one reference image from candidate reference images based on the degree of similarity between a first feature vector of the input image and a second feature vector of each reference image. In some embodiments, the first feature vector and the second feature vector are each based on at least one of two or more different statistical values. For example, in embodiments where the sub-images include band images and the reference sub-images include corresponding reference band images, the feature vectors may be based on one or more statistical values ​​such as, but not limited to, the mean of the band images, the standard deviation of the band images, the percentiles of the band images, and the histograms of the band images. Additionally or alternatively, the first feature vector and the second feature vector may be based on output features generated by applying a deep learning neural network model to the input image and the at least one reference image, respectively, wherein the deep learning neural network model is trained on a corpus of images associated with the input image and the at least one reference image.

[0007] In some embodiments, these computer-executable components may further include: a vectorization component that generates feature vectors of candidate reference images; and a reference image set generation component that selects a subset of the candidate reference images based on the differences between these feature vectors. In these embodiments, the reference image selection component may select the at least one reference image from the subset. For example, in one embodiment, these computer-executable components may further include: a clustering component that clusters the candidate reference images into different groups based on the differences between these feature vectors, and the reference image set generation component may select a candidate reference image from each of these different groups to include in the subset.

[0008] In one or more additional embodiments, these computer-executable components may further include: a training component that uses the harmonized images as benchmark training samples to facilitate the training of an image harmonization model to transform the input image into a harmonized image that has a more similar appearance to the at least one reference image relative to the new input image. The image harmonization model may include one or more machine learning models, such as deep learning neural network models. For example, the image harmonization model may include multiple sub-image harmonization models that transform the newly decomposed sub-images from the new input image into new modified sub-images, respectively. In these embodiments, the reconstruction component may combine these new sub-images to generate the harmonized image.

[0009] In another embodiment, a system is provided that includes a memory storing computer-executable components and a processor executing the computer-executable components stored in the memory. These computer-executable components may include: a decomposition component that decomposes training images into sub-images; and a harmonization component that harmonizes these sub-images with corresponding reference sub-images of reference images, resulting in a transformation of these sub-images into modified sub-images. The training images may include images representing images from different domains with different appearance variations. The computer-executable components further include: a reconstruction component that combines groups of these modified sub-images associated with corresponding training images in these training images to generate baseline harmonized images of these corresponding training images. The computer-executable components further include: a training component that trains a full-image harmonization model (i.e., an image harmonization model) to transform these training images into these baseline harmonized images. The computer-executable components may also include: a model application component that applies these (trained) full-image harmonization models to a new input image to transform the new input image into a new harmonized image that has a more similar appearance to these reference images relative to the new input image.

[0010] Additionally or alternatively, another system is provided, comprising a memory storing computer-executable components, the processor executing such computer-executable components stored in the memory. These computer-executable components may include: a decomposition component that decomposes a training image into sub-images; and a harmonization component that harmonizes these sub-images with corresponding reference sub-images of a reference image, resulting in a transformation of these sub-images into modified sub-images. In this case, the harmonized images may be images using band harmonization transformation or images from the same imaging setup. These computer-executable components further include: a training component that trains a sub-image harmonization model to transform these sub-images into these modified sub-images. These training images may include images representing images from different domains with different appearance variations. These computer-executable components may also include: a model application component that applies the sub-image harmonization model to transform the newly decomposed sub-images from the input image into new modified sub-images; and a reconstruction component that combines these new modified sub-images to generate a harmonized image of the input image that has a more similar appearance to the reference images relative to the input image.

[0011] In some implementations, the elements described in the system disclosed herein may be embodied in different forms, such as computer-implemented methods, computer program products, or another form. Attached Figure Description

[0012] Figure 1 A block diagram of an exemplary non-limiting system for facilitating image harmonization for deep learning model optimization, based on one or more embodiments of the disclosed subject matter, is shown.

[0013] Figure 2 The decomposition of images into different band images is shown according to one or more embodiments of the disclosed subject matter.

[0014] Figure 3 A flowchart of an exemplary image harmonization process based on one or more embodiments of the disclosed subject is presented.

[0015] Figure 4 Chest X-ray images before and after image harmonization are presented according to one or more embodiments of the disclosed subject matter.

[0016] Figure 5 Magnetic resonance (MR) scan images before and after image harmonization are presented according to one or more embodiments of the disclosed subject matter.

[0017] Figure 6X-ray images of different anatomical sites before and after image harmonization are presented according to one or more embodiments of the disclosed subject matter.

[0018] Figure 7 An exemplary high-level flowchart of a computer-implemented process for image harmonization according to one or more embodiments of the disclosed subject matter is shown.

[0019] Figure 8 A block diagram of another exemplary non-limiting system for facilitating image harmonization for deep learning model optimization, based on one or more embodiments of the disclosed subject matter, is shown.

[0020] Figure 9 An exemplary high-level flowchart of another computer implementation of a process for image harmonization according to one or more embodiments of the disclosed subject is shown.

[0021] Figure 10 An exemplary high-level flowchart of another computer implementation of a process for image harmonization according to one or more embodiments of the disclosed subject is shown.

[0022] Figure 11 A block diagram of another exemplary non-limiting system for facilitating image harmonization for deep learning model optimization, based on one or more embodiments of the disclosed subject matter, is shown.

[0023] Figure 12 The training and use of a full-image harmonization model based on one or more embodiments of the disclosed subject are shown.

[0024] Figure 13A Training of a sub-image harmonization model for image harmonization based on one or more embodiments of the disclosed subject is shown.

[0025] Figure 13B The application of a sub-image harmonization model for image harmonization according to one or more embodiments of the disclosed subject is shown.

[0026] Figure 14 An exemplary high-level flowchart of a computer-implemented process for image harmonization using a full image harmonization model, according to one or more embodiments of the disclosed subject matter, is shown.

[0027] Figure 15 An exemplary high-level flowchart is shown of a computer implementation of a process for developing a full-image harmonization model according to one or more embodiments of the disclosed subject matter.

[0028] Figure 16An exemplary high-level flowchart of a computer-implemented process for image harmonization using a sub-image harmonization model, according to one or more embodiments of the disclosed subject matter, is shown.

[0029] Figure 17 A block diagram is shown of an exemplary non-limiting operating environment in which one or more embodiments described herein may be facilitated. Detailed Implementation

[0030] The following specific embodiments are merely exemplary and are not intended to limit the implementation and / or application or use of the embodiments. Furthermore, they are not intended to be construed as being bound by any express or implied information presented in the foregoing "Summary of the Invention" or "Detailed Description" sections.

[0031] The disclosed subject matter relates to systems, computer-implemented methods, apparatuses, and / or computer program products that provide various image harmonization techniques to facilitate the optimization of the performance of AI / ML-based image processing models on images that differ in appearance from training images. In the context of the disclosed subject matter, image harmonization refers to adapting a given image to appear more similar to one or more reference images, which represent training images used to train a particular image processing model. For example, the one or more reference images may include one or more of the training image and / or one or more images corresponding to the training image. The disclosed image harmonization techniques are independent of the trained model and serve as a preprocessing step to correct model predictions for images from different domains relative to the training image. In this respect, the disclosed image harmonization techniques can be applied to facilitate the optimization of the performance of various types of image processing models configured to perform multiple inference tasks.

[0032] In one or more non-limiting embodiments, the image processing model may include a deep learning-based medical image processing model configured to perform various medically relevant inference tasks on medical images, such as image-based diagnosis, organ segmentation, anomaly detection, image reconstruction, etc. For example, various deep learning-based AI models have been proposed for interpreting and prioritizing chest X-ray / radiography (CXR) images, the most commonly used modality for screening and diagnosing various lung diseases such as pneumonia, lung cancer, tuberculosis, and lung scarring. However, often, these modulated products are only authorized for use with images captured using a specific chest X-ray system, as the appearance of CXR images can vary between different X-ray systems due to dose usage, exposure settings, photon-receiving materials, and various other factors. These modulated AI models can be very sensitive to image variations caused by these imaging factors, as is generally the case with deep learning networks. Typically, performance degradation can be observed on image datasets that differ from the training dataset due to even minor variations in appearance.

[0033] Therefore, in the context of medical image processing model optimization, the disclosed image harmonization technique can be used to adapt medical images that are visually different from the model's training images to be more visually similar to the training images before they are input into the model. For example, variational medical images may include medical images of the same type as the training images that are visually different from the training images (e.g., CXR images) due to differences in the capture protocol / system used (e.g., which may differ between medical centers) or other imaging factors. The disclosed image harmonization technique can also be applied to harmonize image datasets for use in AI / ML image analysis / processing models configured to perform various inference tasks on non-medical images.

[0034] According to one or more embodiments, the disclosed image harmonization technique involves decomposing an original image to be harmonized with a training dataset of a specific model into sub-images, and harmonizing these sub-images with corresponding reference sub-images decomposed from one or more reference images. Specifically, the harmonization of sub-images with corresponding reference sub-images can be achieved by adapting or modifying the sub-images to appear more similar to the corresponding reference sub-images, thereby producing modified sub-images of the original image. In various embodiments, the harmonization process may involve altering one or more features of each sub-image to make them more similar to corresponding features of the corresponding reference sub-image. The modified sub-images can then be recombinated to generate a reconstructed harmonized image, which is a modified version of the original image that has a more similar visual appearance to the original image and one or more reference images.

[0035] In various implementations, decomposition involves using a novel decomposition process to generate subband images from a low-pass signal, decomposing an original image and one or more reference images into different band images with different frequency ranges. After the original image and one or more reference images have been decomposed into band images, each band of the original image can be harmonized with one or more corresponding reference band images by making one or more statistical measures of each band image more similar to the statistical measures of one or more corresponding reference band images. For example, various statistical measures can be calculated or generated for each band image based on its band image intensity values, including but not limited to: the mean of its band image intensity values, the standard deviation of its band image intensity values, the percentiles of its band image intensity values, and a histogram of its band image intensity values. In this regard, harmonizing the band images of the original image with the corresponding reference band images may include (e.g., relative to a defined degree of variation) modifying the intensity values ​​of the band images so that the statistical measures of the band image intensity values ​​are more similar to the statistical measures of the corresponding reference band images.

[0036] In another embodiment, decomposing the original input image and one or more reference images into sub-images may include segmenting the image into two or more distinct regions. For example, when applied to medical images, a pre-trained segmentation model can be used to segment the medical image into distinct regions of interest (e.g., organs). In this regard, each segmented region can be considered a sub-image and will have different features (e.g., relative to appearance features and / or spatial features). According to this embodiment, the corresponding segmented region of the original image can be harmonized with the corresponding segmented region of one or more reference images by making one or more features of the original image fragment more similar to corresponding features of the corresponding reference image fragment. The modified fragment of the original image can then be reconstructed to generate a harmonized image.

[0037] In various additional implementations, the harmonized images and / or modified sub-images generated using the techniques described above can also be used as benchmark truth training samples for training one or more ML / AI models to transform input images with appearance changes into harmonized images (e.g., harmonized with the training dataset of the image processing model) before being input into the image processing model. Such ML / AI models are referred to herein as image harmonization models. For example, one or more image harmonization models may include one or more deep learning models, such as autoencoders, generative adversarial autoencoders, generative adversarial networks (GANs), etc. In the cases of these implementations, once trained, one or more image harmonization models can be used to transform new input images with different feature changes into harmonized images that have a more similar appearance to the original source training images.

[0038] For example, in some implementations, the techniques described above for generating harmonic images can be applied to a training image set of an image harmonicization model with different appearance variations to generate a harmonic image for each training image. The image harmonicization model can then be trained to transform the training images into harmonic images. Additional training images without a baseline harmonic version can also be added to the training dataset. Once the image harmonicization model is trained, it can be applied to transform new input images with feature variations relative to the original model training images into harmonic images.

[0039] Additionally or alternatively, multiple sets of sub-image / modified sub-image pairs generated for each training image in the training images can be used to train individual sub-image harmonization models to map sub-images to corresponding modified sub-images. For example, suppose the decomposition process decomposes each input image into a first-pass bandgap image, a second-pass bandgap image, a third-pass bandgap image, and so on, up to a predetermined number (i). According to this example, a first sub-image harmonization model can be trained to transform the first-pass bandgap image into a first modified sub-image, a second sub-image harmonization model can be trained to transform the second-pass bandgap image into a second modified sub-image, and so on. Once trained, the sub-image harmonization models can be applied to new first-pass, second-pass, third-pass, etc., sub-images decomposed from a new input image to generate modified sub-images of the new input image. The modified sub-images can then be recombined to generate the harmonized image.

[0040] The disclosed techniques for developing and applying one or more image harmonization models provide an efficient end-to-end process for image harmonization because the mechanism for generating benchmark truth training data is automated. A major limitation in ML / AI model development in this regard is the availability of benchmark truth training data, which most often requires manual annotation—a tedious and costly effort. With the disclosed techniques, benchmark truth training data can be generated without any manual intervention, providing unlimited, low-cost, and efficient benchmark truth training data generation. Furthermore, using a trained image harmonization model to transform an input image into a harmonized image reduces the overall processing time of image harmonization compared to the aforementioned decomposition, sub-image harmonization, and reconstruction processes.

[0041] The disclosed subject matter further provides tools for enhancing the accuracy and effectiveness of the disclosed image harmonization techniques by optimizing the selection of reference images. In this regard, in some embodiments, the same single pre-selected reference image can be used to transform all variational images into harmonized images for input into a specific image processing model. In other embodiments, multiple (e.g., two or more) reference images can be used. According to these embodiments, a weighting scheme for reference sub-images, determined based on the degree of similarity between the input image and the corresponding reference image, can be used to harmonize the sub-images decomposed from the input image with their corresponding reference sub-images among the multiple reference images.

[0042] In various implementations, the degree of similarity can be determined based on a comparison of feature vectors generated for the input image and the reference image, respectively. In some implementations, the feature vector can be generated based on one or more statistical measures among those used to harmonize the sub-image with the corresponding reference sub-image. For example, in an implementation where the image is decomposed into band images, the feature vector can be generated based on one or more statistical data of the corresponding band images of the image (e.g., the mean of the band image intensity values ​​of each band image, the standard deviation of the band image intensity values ​​of each band image, the percentile of the band image intensity values ​​of each band image, etc.). Additionally or alternatively, the feature vector can be computed from the output features of one or more layers (e.g., fully connected layers) of a pre-trained network trained on a corpus of similar images (e.g., using the ImageNet dataset, etc.). The feature vector can also represent other image-based features (e.g., dose usage, exposure settings, photon receiving material, FOV, demographics, contrast versus non-contrast, etc.) and / or non-image-based features (e.g., patient-specific features such as patient demographics, patient history, etc.) included in metadata associated with the corresponding image.

[0043] In some implementations, multiple reference images may include a selected subset of the original model training images, which provides a representative subset of variational images of different types included in the training dataset. For example, in one or more implementations, feature vectors may be generated for all (or a smaller subset) of the training images. The training images may then be clustered into different groups of related images based on the similarity and differences between the feature vectors of the training images (e.g., using a suitable clustering algorithm, such as k-means clustering). One or more representative reference images may then be selected from each (or in some implementations, one or more) group to generate a reference image set for harmonization using a weighted scheme based on the degree of similarity between the input image and the corresponding reference images included in the set.

[0044] Alternatively or additionally, a single reference image can be selected from a set of reference images or the entire training image set for each input image to be harmonized, based on feature vector similarity. In this regard, the feature vector of the input image can be compared with the feature vectors of all candidate reference images to select the single candidate reference image that best matches the feature vector of the input image with the highest degree of similarity. This process can be performed for each input image, allowing for the customization of the reference image used to harmonize each input image.

[0045] The term "image processing model" is used herein to refer to an AI / ML model configured to perform image processing or analysis tasks on images. Image processing or analysis tasks can vary. In various implementations, image processing or analysis tasks may include (but are not limited to): segmentation tasks, image reconstruction tasks, object recognition tasks, motion detection tasks, video tracking tasks, optical flow tasks, etc. The image processing models described herein may include two-dimensional (2D) image processing models and three-dimensional (3D) image processing models. Image processing models may employ various types of AI / ML algorithms, including (but not limited to): deep learning models, neural network models, deep neural network models (DNN), convolutional neural network models (CNN), etc.

[0046] As used herein, a “medical imaging processing model” refers to an image processing model customized to perform image processing / analysis tasks on one or more medical images. For example, medical imaging processing / analysis tasks may include (but are not limited to): organ segmentation, anomaly detection, anatomical feature representation, medical image reconstruction, diagnosis, etc. The types of medical images processed / analyzed by the medical image processing model may include images captured using various types of imaging modalities. For example, medical images may include (but are not limited to): radiotherapy (RT) images, X-ray images, CXR images, digital radiography (DX) X-ray images, X-ray angiography (XA) images, panoramic X-ray (PX) images, computed tomography (CT) images, mammography (MG) images (including tomography synthesis devices), magnetic resonance imaging (MRI) images, ultrasound (US) images, color stream Doppler (CD) images, positron emission tomography (PET) images, single-photon emission computed tomography (SPECT) images, nuclear medicine (NM) images, etc. Medical images may include two-dimensional (2D) images as well as three-dimensional (3D) images.

[0047] The terms “source domain model,” “source model,” “source image processing model,” and “source domain image processing model” are used interchangeably herein to refer to an image processing model trained on images from a specific domain (referred to herein as the source domain). Images included in the source domain are referred to herein as “source domain images” or “source images.” When applied to image harmonicization, one or more reference images are or correspond to source domain images. In this respect, images that differ in appearance from source domain images are considered herein to be images from different but similar domains relative to the source domain. These images may include images of the same “type” as source domain images but with some appearance variations relative to the source domain images. For example, appearance variations may be attributable to a different imaging factor, including but not limited to: image capture protocol, image capture modality, capture voltage, sequence intensity, dose usage, exposure settings, photon receiving material, FOV, demographics, contrast versus non-contrast, and other factors affecting image quality (IQ) or appearance. For example, source domain images and variational images may include the same type of medical images, but captured from different acquisition sources used at different medical centers. In another example, the source domain image and variational image may include medical images of the same anatomical body part (e.g., the same organ) but differ in capture modality (CT images with or without contrast). In another example, the source domain image and variational image may differ with respect to different capture voltages. In yet another example, the source domain image and variational image may include MR images with different sequence intensities.

[0048] One or more embodiments will now be described with reference to the accompanying drawings, wherein the same reference numerals are always used to denote the same elements. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a more thorough understanding of one or more embodiments. However, it will be apparent, in various cases, that one or more embodiments may be practiced without these specific details.

[0049] Now turn to the attached image. Figure 1 A block diagram of an exemplary non-limiting system 100 for facilitating image harmonization for deep learning model optimization, according to one or more embodiments of the disclosed subject matter, is shown. Embodiments of the system described herein may include one or more machine-executable components embodied within one or more machines (e.g., embodied in one or more computer-readable storage media associated with one or more machines). When executed by one or more machines (e.g., processors, computers, computing devices, virtual machines, etc.), such components may enable the one or more machines to perform the operations.

[0050] For example, system 100 includes computing device 104, which may include various computer / machine executable components, including decomposition component 106, image harmonization component 108, and reconstruction component 110. Computing device 104 may also include or be operatively coupled to at least one memory 118 and at least one processor 116. In various embodiments, at least one memory 118 may store executable instructions (e.g., decomposition component 106, image harmonization component 108, reconstruction component 110, and additional components described herein) that, when executed by at least one processor 116, facilitate the performance of operations defined by the executable instructions. Computing device 104 also includes a reference image data source 114 capable of storing one or more reference images. In other specific embodiments, one or more reference images may be stored in memory 118 or in another suitable data structure accessible to computing device 104. Computing device 104 may also include a device bus 112 coupled to various components of computing device 104. Examples of the processor 116 and memory 118, as well as other suitable computer-based or computing elements, may be found in [reference]. Figure 17 The processing unit 1716 and system memory 1714 were found and can be combined to achieve integration. Figure 1 It may be used in accordance with one or more of the other figures shown and described herein and the systems or components thereof.

[0051] In some implementations, system 100 may be deployed using any type of component, machine, device, facility, apparatus, and / or instrument, including a processor and / or capable of communicating effectively and / or operatively with wired and / or wireless networks. All such implementations are contemplated. For example, computing device 104 may be or correspond to server equipment, general-purpose computer, special-purpose computer, tablet computing device, handheld device, server-type computing machine and / or database, laptop computer, notebook computer, desktop computer, cellular phone, smartphone, consumer device and / or instrument, industrial and / or commercial equipment, digital assistant, multimedia internet-enabled telephone, multimedia player, and / or another type of device.

[0052] It should be understood that the embodiments of this subject matter depicted in the various accompanying figures are for illustrative purposes only, and therefore the architecture of such embodiments is not limited to the systems, devices, and / or components described herein. In some embodiments, one or more components of system 100 and other systems described herein may be executed individually or in parallel by different computing devices (e.g., including virtual machines) according to a distributed computing system architecture. System 100 may also include the operating environment 1700 referenced herein and Figure 17 The various additional computer- and / or computing-based elements described herein. In several embodiments, such computer- and / or computing-based elements may be combined with the implementation of what is disclosed herein.Figure 1 It may be used in combination with one or more of the operations implemented by the systems, devices, components and / or computers shown or described in other accompanying drawings.

[0053] The decomposition component 106, image harmonization component 108, and reconstruction component 110 can jointly perform an image harmonization process that can be used to transform the input image 102 into a harmonized image 118. The harmonized image 118 generated according to this image harmonization process is a modified version of the input image 102 that has a more similar appearance to one or more reference images (e.g., included in reference image data source 114) relative to the input image 102. In various embodiments, the one or more reference images can be or correspond to training images from a specific source domain used to train a specific AI / ML image processing model. Therefore, the image harmonization process provided by the decomposition component 106, image harmonization component 108, and reconstruction component 110 can be used to transform the input image 102, which has or may have some appearance changes relative to the training images, into a harmonized image 118 that corrects or normalizes those appearance changes, making the harmonized image 118 look more similar to the training images. Therefore, the image harmonization process provided by the decomposition component 106, the image harmonization component 108, and the reconstruction component 110 can be used as a preprocessing step to transform images with appearance changes relative to the training images of the image processing model into harmonized images before they are input into the image processing model. Thus, the image processing model can be applied to images from various domains other than the source domain without performance degradation or with minimal performance degradation.

[0054] According to one or more embodiments, the image harmonization process involves: decomposing an input image 102 into sub-images by a decomposition component 106; harmonizing the sub-images with corresponding reference sub-images decomposed from one or more reference images by a harmonization component 108 to generate modified sub-images; and reconstructing the modified sub-images by a reconstruction component 110 to generate a harmonized image 118. In some embodiments, a single reference image may be used for harmonization. For example, a single reference image may be used as a single representative image representing a training image dataset. In some embodiments, a single reference image may be pre-selected. In other embodiments, a single reference image may be selected from candidate reference images based on its degree of similarity to the input image. In other embodiments, multiple reference images (e.g., two or more) may be used for harmonization. References below... Figure 8 This describes additional features and functions related to reference image selection and the use of multiple reference images. For ease of explanation, we will first refer to an implementation using a single reference image. Figures 1-7 Describe the features and functions of system 100.

[0055] In this regard, the decomposition unit 106 may use one or more decomposition processes to decompose the input image 102 into two or more sub-images. In some embodiments, a reference image provided in the reference image data source 114 may be pre-decomposed into a corresponding reference sub-image generated using the same decomposition process used to decompose the input image 102. In other specific embodiments, the decomposition unit 106 may decompose both the input image 102 and the reference image (or multiple reference images) at runtime.

[0056] In various embodiments, the decomposition unit 106 may decompose the input image 102 using a novel band decomposition process, which includes decomposing the input image 102 into different band images (also referred to as subband images). In these embodiments, the same band decomposition process is applied to a reference image to decompose the reference image into corresponding reference band images. As used herein, the term "band image" (or subband image) is used to refer to a frequency-domain filtered version of an image that contains a subset of the spectral components of the original image. In this respect, each band image generated from the original image has a different frequency band or range and different energy frequency values ​​or intensity values ​​within its frequency band / range.

[0057] For example, Figure 2 The decomposition of an input image 202 into different bandgap images according to one or more embodiments of the disclosed subject matter is illustrated. In various embodiments, the input image 202 may be or correspond to the input image 102. In this example, the input image 202 includes a CXR image; however, it should be understood that the input image may include other types of medical and non-medical images. In the illustrated embodiment, the input image 202 has been decomposed into four different bandgap images, identified as bandgap image 204, bandgap image 206, bandgap image 208, and bandgap image 210, respectively. Each of the different bandgap images is a filtered version of the input image 202 with different frequency intensities removed.

[0058] Given Figure 2 refer to Figure 1In various embodiments, the decomposition unit 106 may use a decomposition process to decompose an input image (e.g., input image 102, input image 202, etc.) into different band images (e.g., band image 204, band image 206, band image 208, and band image 210), which involves generating new band images from low-pass signal bands to high-pass bands. In this respect, the decomposition unit 106 may decompose the original image into different band images with different frequency bands, wherein each band image is filtered from a previously generated low-pass signal (e.g., from the low-pass band to a higher pass band). A last low-pass signal band may also be included as the last band covering all frequency bands. In this respect, as mathematically expressed, the decomposition unit 106 may decompose the original input image I(x) into different band images I with different band frequencies according to the following equations 1-3. i The number (i) of different bandgap images can vary, where i = 1, 2, ..., B, such that the last bandgap image is labeled B.

[0059] L i(x) =L i-1 (x)*G(x;σ i ),

[0060] Where G is the Gaussian kernel and

[0061] σ i The number can be randomly selected to increase.

[0062] Equation 1.

[0063] I i(x) =L i-1 (x)-L i (x) At this point, i = 1, ..., B-1,

[0064] Where L 0(x) =I(x)

[0065] Equation 2.

[0066] I B(x) =L B-1 (x) At this time, i = B

[0067] Equation 3.

[0068] According to equations 1-3, the decomposition component 106 can first perform a Gaussian calculation (convolution) on the original input image I(x) according to equation 1 to obtain the image L. i(x) This image is a smoother version of the original input image. In this respect, L i(x) It is a smoothed image based on the previous image and the defined Gaussian kernel. For the first calculation (i.e., the first pass), the previous smoothed image is the input image. Next, according to Equation 2, the decomposition component 106 can be decomposed in L...(i-1)(x) With the smoothed image L from the previous step i(x) Perform subtraction between them to obtain the bandgap image I i The resulting band structure belongs to the high-frequency range. As mentioned above, starting from this first pass where i = 1, l 0(x) =I(x), this first subtraction will be the first smoothed image L i(x) Subtracting from the original input I(x) yields the first bandgap image I1. As reflected in Equation 3, the decomposition unit 106 can perform the operations of Equations 1 and 2 multiple times, wherein for each iteration, a lower-frequency smoothed version of the input image generated using Equation 1 is subtracted to generate the next bandgap image. In this respect, each new bandgap image is generated from a low-pass bandgap image, wherein the generated new bandgap image has a higher frequency band relative to the low-pass band from which it was generated (i.e., from which it was subtracted).

[0069] For example, refer to Figure 2 Four different band images correspond to the images generated by each pass of equations 1-3, where four passes are performed. In this respect, band image 204 may be or correspond to a first pass subband image (I1) including the first subband energy frequency of the input image 202, band image 206 may be or correspond to a second pass subband image (I2) including the second subband energy frequency of the input image 202, band image 208 may be or correspond to a third pass subband image (I3) including the third subband energy frequency of the input image 202, and band image 210 may be or correspond to a fourth pass subband image (I4) including the fourth subband energy frequency of the input image. Each subsequent band image generated by each pass has a higher frequency band range relative to the previous image, except for the last one (i.e., band image 210), which belongs to the lowest frequency band from which the previous image was generated.

[0070] The same decomposition process described above can also be used to decompose the reference image into different bandgap images. In this regard, the reference image can be decomposed into bandgap images having the same frequency band / range as the input image. Therefore, each of the different reference bandgap images can be paired with the corresponding bandgap image of the input image. As described above, in some specific embodiments, the decomposition unit 106 can decompose the reference image to generate reference bandgap images in the same manner as described for the input image (e.g., using Equations 1-3).

[0071] In various implementations, after the input image (e.g., input image 102, input image 202, etc.) and the reference image have been decomposed into band images, the harmonization component 108 harmonizes the band images with the corresponding reference band images by making one or more statistical measures of each band image of the input image 102 the same as or similar to the statistical measures of the corresponding reference band image. For example, various statistical measures can be calculated or generated for each band image based on its band frequency / intensity, including but not limited to: the mean of its band image intensity values, the standard deviation of its band image intensity values, the percentile of its band image intensity values, and a histogram of its band image intensity values. In this regard, the harmonization component 108 can harmonize the band images with the corresponding reference band images by modifying the band frequency / intensity of each band image of the input image 102 (e.g., relative to a defined degree of variation) so that one or more statistical measures of the band frequency / intensity are the same as or more similar to the statistical measures of the corresponding reference band image. Therefore, the image harmonization component 108 can transform each band image of the input image 102 into a modified band image that is more similar to the corresponding reference band image.

[0072] For example, Figure 3 A flowchart of an exemplary image harmonization process 300 according to one or more embodiments of the disclosed subject is presented. Figure 3 The image harmonization process 300 shown provides an exemplary embodiment of the disclosed image harmonization technique applied to CXR images and harmonizing with energy bands. However, the image harmonization process 300 can be applied to various other types of images using other decomposition techniques for generating sub-images.

[0073] Given Figure 1 refer to Figure 3 According to the image harmonization process 300, at 301, the decomposition component 106 may decompose the input image 302 into sub-images 304. In one or more exemplary embodiments, the input image 302 may be or correspond to a CXR image to be harmonized with a training image dataset used to train a particular image processing model. In this regard, the input image 302 may include CXR images from different domains relative to the training image dataset. For example, the input image 302 may have been captured using a different X-ray system / protocol relative to the training images. Regardless of the reason, the input image 302 may have at least some appearance variations relative to the training images.

[0074] In the illustrated embodiment, the different sub-images are respectively identified as sub-image 3041, sub-image 3042, sub-image 3043, and sub-image 3044. For example, when applied to band image decomposition, each sub-image can be a different band image generated from each pass by equations 1-3. In this respect, sub-image 3041 may correspond to a first band image having a first frequency range, sub-image 3042 may correspond to a second band image having a second frequency range, sub-image 3043 may correspond to a third band image having a third frequency range, and sub-image 3044 may correspond to a fourth band image having a fourth energy frequency range.

[0075] Similarly, at 303, the decomposition unit 106 can decompose the reference image 306 into reference sub-images 308 using the same decomposition process used for the input image 302. In the illustrated embodiment, the different reference sub-images are respectively identified as reference sub-image 3081, reference sub-image 3082, reference sub-image 3083, and reference sub-image 3084. For example, when applied to bandgap image decomposition, each reference sub-image in reference sub-image 308 can also be a different bandgap image generated from each pass by equations 1-3. In this respect, reference sub-image 3081 may correspond to a first reference bandgap image having a first frequency range, reference sub-image 3082 may correspond to a second reference bandgap image having a second frequency range, reference sub-image 3083 may correspond to a third reference bandgap image having a third frequency range, and reference sub-image 3084 may correspond to a fourth reference bandgap image having a fourth frequency range.

[0076] At position 105, the image harmonization component 108 harmonizes sub-image 304 with the corresponding reference sub-image 308 to generate a modified sub-image 310. In this respect, the image harmonization component 108 harmonizes sub-image 3041 with reference sub-image 3081 to generate a modified sub-image 3101, harmonizes sub-image 3042 with reference sub-image 3082 to generate a modified sub-image 3102, harmonizes sub-image 3043 with reference sub-image 3083 to generate a modified sub-image 3103, and harmonizes sub-image 3042 with reference sub-image 3082 to generate a modified sub-image 3102.

[0077] In various embodiments where the sub-images are band images, the image harmonization component 108 may calculate, generate, or otherwise determine their statistical measurements based on their respective energy frequencies / intensities within the frequency range / band of each sub-image included in the sub-images and each reference sub-image in the reference sub-images. For example, the statistical measurements of each of the sub-images 304 and 308 may include, but are not limited to, the mean of their band image intensity values, the standard deviation of their band image intensity values, the percentiles of their band image intensity values, and a histogram of their band image intensity values. The image harmonization component 108 may further harmonize the sub-images with their corresponding reference sub-images (e.g., having the same frequency range / band) by modifying the intensity / frequency of each sub-image so that its statistical measure is the same as or similar to the statistical measure of the corresponding reference sub-image. For example, the harmonization component 108 can adjust the intensity / frequency adjustment of sub-image 3041 to achieve one or more of the following: 1.) its mean intensity / frequency is the same as or closer to the mean intensity / frequency of reference sub-image 3081 (relative to a defined degree of variation); 2.) its standard deviation of intensity / frequency is the same as or closer to the standard deviation of intensity / frequency of reference sub-image 3081 (relative to a defined degree of variation); 3.) its percentile of intensity / frequency is the same as or closer to the percentile of intensity / frequency of reference sub-image 3081 (relative to a defined degree of variation); and 4.) its histogram of intensity / frequency is the same as or closer to the histogram of intensity / frequency of reference sub-image 3081 (relative to a defined degree of variation). The image harmonization component 106 can similarly apply this same harmonization process to harmonize sub-image 3042 with reference sub-image 3082, sub-image 3043 with reference sub-image 3083, and sub-image 3044 with reference sub-image 3084. It should be understood that the disclosed harmonization techniques are not limited to these statistical measures, and other statistical measures may be used.

[0078] In various implementations, two or more statistical measures may be combined to increase the output of image harmonization. For example, in some implementations, the image harmonization component 106 may employ both the mean and standard deviation of the bandgap image intensity / frequency. In some specific embodiments of these implementations, the harmonization component 106 may harmonize each bandgap image with its corresponding reference bandgap image according to the following equation 4, where I (harm_i)(x) This represents a harmonized (or modified) sub-image (e.g., a modified sub-image in modified sub-image 310), I (input_i)(x) I represents a sub-image of input image 302 (e.g., a sub-image in sub-image 304). (ref_i)(x) This indicates a reference sub-image (e.g., a reference sub-image in reference sub-image 308), and "std" is an abbreviation for standard deviation.

[0079]

[0080] Again, given Figure 1 Referring to process 300, once the modified sub-image 310 has been generated at 307, reconstruction component 110 can combine the modified sub-images to reconstruct the harmonized image 312. The resulting harmonized image 312 has a greater degree of similarity to the reference image in appearance relative to the input image 302. For example, in the illustrated embodiment, the harmonized image 312 looks more like the reference image 306 in terms of intensity compared to the input image 302.

[0081] Figure 4 Exemplary CXR images before and after image harmonization are presented according to one or more embodiments of the disclosed subject matter. Specifically, the harmonized input image 406 is generated from input image 402 using reference image 404 and the band decomposition / harmonization process of Equations 1-4. (As in...) Figure 4 As can be seen, the harmonized input image 406 is more similar in appearance to the reference image 404 than the input image 402.

[0082] Experiments were conducted to evaluate the effectiveness of the image harmonization process applied to a CXR image processing model configured to handle CXR images from a specific source domain (such as...). Figure 4 Those shown perform pneumoperitoneum diagnosis, segmentation, and endotracheal tube location estimation. Based on these experiments, model performance was evaluated when applied to unharmonized CRX images from various other domains (e.g., input image 402) and the same CRX images harmonized according to the band decomposition / harmonization process according to Equations 1-4 (e.g., harmonized image 404). Results from different datasets consistently demonstrate that this image harmonization process increases the effectiveness of the deep learning pipeline in terms of accuracy, area under the receiver operating characteristic curve, DICE coefficient, and other application-related error metrics.

[0083] Figure 5 Exemplary MR scan images before and after image harmonization, according to one or more embodiments of the disclosed subject matter, are presented. Specifically, the harmonized input image 504 is generated from input image 502 using reference image 506 and the band decomposition / harmonization process of Equations 1-4. Figure 5 As can be seen, the harmonized input image 504 is more similar in appearance to the reference image 506 than the input image 502.

[0084] Experiments were conducted to evaluate the effectiveness of the image harmonization process applied to an MR image processing model configured to handle MR images from a specific source domain (such as...). Figure 5The models (those shown) perform organ and lesion segmentation. Based on these experiments, model performance was evaluated when applied to unharmonized MR images from various other domains (e.g., input image 502) and the same MR images harmonized according to the band decomposition / harmonization process according to Equations 1-4 (e.g., harmonized image 504). Results from different datasets consistently demonstrate that this image harmonization process increases the effectiveness of the image processing model. For example, in a test dataset covering 73 organs, the success rate of the organ segmentation model increased from 86.3% to 89% after harmonization. Additionally, the MR brain lesion detection / segmentation model was tested on a dataset of 35 patients with multiple MR sequences (T1, T1 ce, T2, and Flair). The model performance increased from a DICE score of 0.655 + / - 0.232 before harmonization to 0.709 + / - 0.19 after harmonization.

[0085] Figure 6 X-ray images of different anatomical sites before and after image harmonization, according to one or more embodiments of the disclosed subject matter, are presented. The disclosed image harmonization technique can be applied to correct various appearance variations in images, as well as to correct errors and remove artifacts. For example, Figure 6 The results of the disclosed bandgap image harmonization technique, applied to X-ray images to correct window levels and window widths from different X-ray images, are shown.

[0086] Refer again Figure 1 In some implementations, the decomposition component 106 may employ other decomposition techniques to decompose the input image 102 (and the reference image) into sub-images. For example, in one specific implementation, the decomposition component 106 may decompose the input image 102 and the reference image by segmenting them into two or more distinct regions. For instance, when applied to medical images, a pre-trained segmentation model may be used to segment the medical image into distinct regions of interest (e.g., organs). In this regard, each segmented region may be considered a sub-image and will have different features (e.g., relative to appearance features and / or spatial features).

[0087] In these embodiments, the harmonization component 108 harmonizes a sub-image with a corresponding reference sub-image by making one or more features of each sub-image identical or more similar to corresponding features of its corresponding reference sub-image (e.g., relative to a defined degree of variation). For example, the harmonization component 108 may modify one or more visual features of a sub-image (e.g., pixel shading, pixel tone, saturation, etc.) to be more similar to corresponding visual features of the reference sub-image. In another example, the harmonization component 108 may modify one or more spatial features of a sub-image (e.g., pixel location, sub-image thickness, sub-image size / shape, etc.) to be more similar to corresponding visual features of the reference sub-image. In some specific embodiments, the harmonization component 108 may also calculate or otherwise generate statistical measures of the sub-image based on its spatial and / or visual features, and modify the sub-image such that its statistical measures are identical or similar to those of the corresponding sub-image. For example, when applied to a segmented sub-image, the harmonization component 108 can harmonize the corresponding segmented region of the input image 102 with the corresponding segmented region of the reference image by making one or more visual and / or spatial features of the original image segment more similar to the corresponding visual and / or spatial features of the corresponding reference image segment. Then, the reconstruction component 110 can reconstruct the modified sub-image to generate the harmonized image 118.

[0088] Figure 7 An exemplary high-level flowchart of a computer-implemented process 700 for image harmonization according to one or more embodiments of the disclosed subject matter is shown. For brevity, repeated descriptions of similar elements employed in the respective embodiments are omitted.

[0089] At 702, method 700 includes: decomposing an input image (e.g., input image 302) into sub-images (e.g., sub-image 304, using decomposition component 106) by a system (e.g., system 100) operatively coupled to a processor. In various embodiments, the sub-images may include band images with different energy frequencies generated according to equations 1-3. In other specific embodiments, the sub-images may include different segmented regions of the input image. Various other decomposition mechanisms and resulting sub-images may also be used. At 704, method 700 includes: (e.g., using image harmonization component 108) harmonizing the sub-image with a corresponding reference sub-image (e.g., reference sub-image 308) of at least one reference image (e.g., reference image 306) based on two or more different statistical values ​​calculated respectively for the sub-image and the corresponding reference sub-image, thereby resulting in a transformation of the sub-image to a modified sub-image image (e.g., modified sub-image 310). For example, in an implementation where the sub-images are band images, harmonization component 106 can modify the intensity of each sub-image such that one or more statistical measures of intensity (e.g., the mean of intensity, the standard deviation of intensity, the percentile of intensity, a histogram of intensity, etc.) are the same as or more similar (e.g., relative to a defined degree of variation) to the corresponding statistical measures of the corresponding reference band image. At 706, method 700 includes (e.g., using reconstruction component 110) combining the modified sub-images into a harmonized image (e.g., harmonized image 312) that has a more similar appearance to at least one reference image relative to the input image.

[0090] Figure 8 A block diagram of another exemplary non-limiting system 800 for facilitating preprocessed image harmonization for deep learning model optimization, according to one or more embodiments of the disclosed subject matter, is shown. System 800 includes the same or similar components as system 100, wherein a reference image module 806 is added and the image harmonization component 108 is divided into a single image harmonization component 802 and a multi-reference harmonization component 804. For brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0091] As described above, in some embodiments, the same single pre-selected reference image can be used to transform all input images (e.g., input image 102) into harmonized images for input into a particular image processing model. In other embodiments, multiple (e.g., two or more) reference images can be used, or a specific reference image can be selected from candidate reference images based on the maximum similarity between a specific reference image and a specific input image for harmonization with the specific input image. In specific embodiments using multiple references, a weighted scheme for reference sub-images, determined based on the degree of similarity between the input image and the corresponding reference image, can be used to harmonize the sub-images decomposed from input image 102 with corresponding reference sub-images among the multiple reference images. The reference image module 806 provides various components to facilitate this purpose, including a vectorization component 808, a clustering component 810, a reference set generation component 812, a similarity scoring component 814, and a reference image selection component 816.

[0092] In some implementations, vectorization unit 808 may generate feature vectors for input image 102 and candidate reference images included in reference image data source 114. The feature vectors can be used to determine the degree of similarity between input image 102 and the corresponding candidate reference image. In some specific implementations, the feature vectors may be generated based on one or more statistical measures among those used to harmonize a sub-image with its corresponding reference sub-image. For example, in an implementation where the image is decomposed into band images, vectorization unit 808 may generate feature vectors based on one or more statistical data of the corresponding band images of the image (e.g., the mean of band image intensity values ​​for each band image, the standard deviation of band image intensity values ​​for each band image, the percentile of band image intensity values ​​for each band image, etc.). In some implementations, vectorization unit 808 may also employ other image-based features (e.g., dose usage, exposure settings, photon receiving material, FOV, demographics, contrast versus non-contrast, etc.) and / or non-image-based features (e.g., patient-specific features such as patient demographics, patient history, etc.) included in metadata associated with the corresponding image to generate feature vectors.

[0093] Additionally or alternatively, the vectorization component 808 may generate feature vectors based on the outputs of one or more fully connected layers of one or more pre-trained deep neural network models trained on a corpus of similar images (e.g., using the ImageNet dataset, etc.). For example, the pre-trained deep neural network may include classical neural network models, convolutional neural network models, recurrent neural network models, autoencoder networks, etc. The task of the pre-trained model can vary, as long as it is configured to consistently identify and extract relevant and discriminative features from the input image (as the outputs of one or more fully connected layers). For example, in one specific implementation, the pre-trained model may include a deep autoencoder model, rather than consisting of encoder and decoder neural networks. Such autoencoder models typically operate by mapping input data to a latent representation in a feature space, which is then decoded by a decoder network to reconstruct the input data from the reduced feature representation in the latent space. When applied to the disclosed topic, the autoencoder model may be trained on a corpus of images similar to the input image (e.g., of the same type, such as CXR images) but from various domains or otherwise having appearance variations. The trained encoder network of an autoencoder can be applied to an input image and a reference image to extract a reduced feature set for the corresponding image, which can then be used to generate a feature vector for the corresponding image.

[0094] As described above, the one or more reference images used for harmonization can be or correspond to training images used to train a specific image processing model. In some embodiments, the vectorization unit 808 can generate feature vectors for each training image or a randomly selected reduced subset of training images. Therefore, in this context, these training images can be considered as candidate reference images. In some embodiments, the similarity scoring unit 814 can compare the feature vector of the input image 102 with all (or a reduced subset) feature vectors of the candidate reference images and determine the degree of similarity between the input image feature vector and the feature vector of the corresponding candidate reference image. For example, the similarity scoring unit 814 can generate a similarity score for each input image / candidate reference image pair based on the degree of similarity between their feature vectors.

[0095] In some specific implementations of these schemes, the reference image selection component 816 may select a single candidate reference image for harmonization with the input image 102 based on the highest similarity score (i.e., most similar or best match) of the input image / candidate reference image pair. In these specific implementations, the image harmonization component 108 may employ a single reference harmonization component 802 to harmonize the reference image with the input image using the techniques described above.

[0096] In another implementation, the reference set generation component 816 may select the top N highest-rated (where N is a qualified integer greater than one) candidate reference images to include in the reference image set to be used for harmonizing with the input image 102 according to the multi-reference harmonization scheme. In another implementation using multiple reference images, to provide a more diverse representation of the training dataset population, the reference set generation component 812 may select a subset of candidate reference images that have different degrees of similarity to the input image (e.g., different similarity scores) to include in the reference image set. For example, the reference set generation component 812 may select the highest-rated candidate reference image (e.g., the one with the closest similarity to the input image), the lowest-rated candidate reference image, and intermediate-rated candidate reference images.

[0097] Alternatively or additionally, instead of comparing the feature vectors of the input image with the feature vectors of all training images (i.e., candidate reference images), the reference set generation component 812 may generate a fixed set of reference images that provide diverse representations of the images included in the training dataset. This fixed set of reference images may then be applied to each input image 102. According to these embodiments, the clustering component 810 may (e.g., using a suitable clustering algorithm, such as k-means clustering, etc.) cluster the candidate reference images into different groups of related images based on the similarity and differences between the feature vectors of the candidate reference images. The number of clusters may vary based on the desired number of reference images to be included in the reference image set. The reference set generation component 812 may further select one or more representative reference images from each cluster to be included in the fixed reference image set. In this respect, the reference set generation component 812 may select several representative reference images with as many vectors as possible covering the groups in the training dataset.

[0098] In some specific implementations of these schemes, after generating a fixed set of reference images, each time a new input image is received for harmonization, the vectorization unit 808 generates a feature vector of the input image. The similarity scoring unit 814 can further generate a similarity score for each input image / reference image included in the set based on its corresponding feature vector. The reference image selection unit 816 can further select one reference image from the set for harmonization with the input image using the single-reference harmonization unit 802.

[0099] In various alternative implementations that select multiple images (e.g., images included in a set of reference images), the multi-reference harmonization component 804 can harmonize the input image 102 using each of the reference images in the set and a weighted scheme based on the degree of similarity between the input image feature vector and each feature vector of the reference images included in the set. For example, suppose the set of reference images has 10 reference images, each with a different feature vector. At runtime, the similarity scoring component 814 can generate a similarity score for each of the 10 reference images, which reflects the degree of similarity between the corresponding feature vector of the reference image and the feature vector of the input image. The multi-reference harmonization component 804 can further use a weighted scheme of reference sub-images determined based on the similarity scores to harmonize each sub-image of the input image with its corresponding reference sub-image. For example, each of the 10 reference images will be decomposed into sub-images, resulting in multiple sets of 10 sub-images corresponding to the same filtering pass (e.g., 10 first reference sub-images, 10 second pass reference sub-images, etc.). Then, the multi-reference harmonization component 804 can use a weighting scheme for reference sub-images, giving greater weight to sub-images associated with higher (e.g., more similar) similarity scores, to harmonize the first-pass sub-images of the input image with each of the 10 corresponding reference sub-images. The multi-reference harmonization component 804 can repeat this harmonization process for each sub-image and the corresponding group of reference sub-images to generate modified sub-images.

[0100] In a specific implementation that extends Equation 4 to multi-reference image harmonicization, Equation 4 can be transformed into Equation 5, where sm(I(x),Ir r (x) is the standardized similarity score between the input image and the r-th reference image in the set. The similarity score can be the absolute correlation between the feature vectors of the input image and the feature vectors of the r-th reference image.

[0101]

[0102] Alternatively, according to Equation 5, among multiple references, the multi-reference harmonization component 804 can apply a similarity index of 1.0 to the most similar vector and apply 0 to all other vectors, thereby resulting in the input image being harmonized with only the closest matching reference image.

[0103] Figure 9 An exemplary high-level flowchart of another computer implementation of a process 900 for image harmonization according to one or more embodiments of the disclosed subject matter is shown. For the sake of brevity, repeated descriptions of similar elements employed in the respective embodiments are omitted.

[0104] At 902, method 900 includes: decomposing an input image (e.g., input image 102, input image 302, etc.) into sub-images (e.g., sub-image 304, using decomposition unit 106) by a system (e.g., system 800) operatively coupled to a processor. At 904, method 900 includes: selecting a reference image (e.g., reference image 306) from a set of reference images by a system (e.g., using reference image selection unit 816) based on the degree of similarity between a first feature vector of the input image and a second feature vector of the reference image. At 906, method 900 includes: (e.g., using single-reference harmonization unit 802) harmonizing a sub-image with a corresponding reference sub-image (e.g., reference sub-image 308) of the reference image, thereby resulting in a transformation of the sub-image to a modified sub-image image (e.g., modified sub-image 310). At 908, method 900 includes (e.g., using reconstruction component 110) combining modified sub-images into a harmonized image (e.g., harmonized image 312) that has a more similar appearance to the input image and the reference image.

[0105] Figure 10 An exemplary high-level flowchart of another computer implementation of a process 1000 for image harmonization according to one or more embodiments of the disclosed subject matter is shown. For the sake of brevity, repeated descriptions of similar elements employed in the respective embodiments are omitted.

[0106] At 1002, method 1000 includes: decomposing an input image (e.g., input image 102, input image 302, etc.) into sub-images (e.g., sub-image 304, using decomposition unit 106) by a system (e.g., system 800) operatively coupled to a processor. At 1004, method 900 includes: harmonizing a sub-image with a reference sub-image of a corresponding reference image by a system (e.g., using multi-reference harmonization unit 804), resulting in a transformation of the sub-image to a modified sub-image, wherein the harmonization includes using a weighting scheme for the reference sub-image determined based on the degree of similarity between the input image and the corresponding reference image. At 1008, method 1000 includes: (e.g., using reconstruction unit 110) combining the modified sub-images into a harmonized image that has a more similar appearance to the input image and the corresponding reference image.

[0107] Figure 11A block diagram of another exemplary non-limiting system 1100 for facilitating preprocessed image harmonization for deep learning model optimization, according to one or more embodiments of the disclosed subject matter, is shown. System 1100 includes the same or similar components as system 800, with the addition of a training component 1102, harmonized training images 1104, one or more harmonization models, and a harmonization model application component 1108. For brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0108] System 1100 provides an extension of deep learning to the aforementioned image harmonization techniques. Specifically, System 1100 uses harmonized images and / or modified sub-images generated according to the aforementioned harmonization techniques as benchmark truth training samples for training one or more ML / AI models to transform input images with appearance changes into harmonized images (e.g., harmonized with the training dataset of the image processing model) before being input into the image processing model. Such ML / AI models are referred to herein as image harmonization models 1106. For example, one or more image harmonization models 1106 may include one or more deep learning models, such as autoencoders, generative adversarial autoencoders, generative adversarial networks (GANs), etc. In these embodiments, once trained, one or more image harmonization models 1102 can be applied by harmonization model application component 1108 to new input images (not shown) to transform new input images with different feature changes into harmonized images with a more similar appearance to the original source training images.

[0109] In this regard, system 1100 may include a plurality of harmonicized training images 1104, which may comprise a diverse set of variational images having feature variations relative to the original source domain images used to train / develop a particular image processing model. For example, harmonicized training images 1104 may include multiple images of the same type as the source domain images (e.g., all CXR images) but from one or more different domains relative to the source images. At least some of these diverse variational harmonicized images 1104 may be processed using the image harmonicization techniques described above to generate harmonicized images of the corresponding harmonicized training images. For example, all (or some) (or at least some) of the harmonicized training images 1104 may be decomposed into sub-images by decomposition component 106 according to the techniques described above, harmonicized with corresponding reference sub-images by image harmonicization component 108, and reconstructed into harmonicized images (such as harmonicized image 118) by reconstruction component 110. These harmonized training images / harmonized image pairs and / or sub-images / modified sub-image pairs may also be stored together with the harmonized training images 1104, and the training component 1102 may use them as baseline truth training samples for training one or the harmonized model 1106. In some specific implementations, the harmonized training images 1104 may also include at least some images that do not have baseline truth samples (e.g., have not been processed to generate baseline truth harmonized images).

[0110] The disclosed technique 1106 for developing and applying one or more image harmonization models provides an efficient end-to-end process for image harmonization because the mechanism for generating benchmark truth training data is automated. In this regard, a major limitation in ML / AI model development is the availability of benchmark truth training data, which most often requires manual annotation—a tedious and costly effort. With the disclosed technique, benchmark truth training data can be generated without any manual intervention, thus providing unlimited, low-cost, and efficient benchmark truth training data generation. Furthermore, using a trained image harmonization model to transform an input image into a harmonized image reduces the overall processing time of image harmonization compared to the aforementioned decomposition, sub-image harmonization, and reconstruction processes.

[0111] Image harmonization techniques based on harmonization models include two different approaches: the whole-image approach and the sub-image approach. (See full-image approach reference.) Figure 12 For a more detailed description, please refer to the sub-image method reference. Figure 13A and Figure 13B To describe in more detail.

[0112] in this regard, Figure 12The training and use of an image full-image harmonization model 1204 according to the full-image method are illustrated. Specifically, process 1200 illustrates model training, and process 1201 illustrates the application of the trained model to a new input image (e.g., input image 1210) to generate a harmonized image 1212 generated by the model generation (MG). Referring to process 1200, according to the full-image method, training images 1202 may include multiple variational images (e.g., images with appearance variations relative to source domain images used to train a particular image processing model). Thus, at least some of the training images 1202 have a baseline true harmonized image generated using the image harmonization technique described above. In this respect, each baseline true harmonized image 1206 is a harmonized image generated for a corresponding training image 1202 using decomposition component 106, image harmonization component 108, and reconstruction component 110. According to this full-image method, sub-images generated for each training image in training images 1202 during the image harmonization process and corresponding modified sub-images may be discarded.

[0113] The type of the full image harmonization model 1204 can vary, and therefore the training process adopted by the training unit 1102 to train the full image harmonization model 1204 can also vary. At a high level, the training process involves training the full image harmonization model 1204 to transform a training image 1202 having a reference harmonized image into its reference harmonized image. More specifically, the training process involves applying the full image harmonization model 1204 to the training image 1202 to generate a model-generated harmonized image 1206. The reference harmonized training image 1206 is further compared with the model-generated harmonized image 1208, and then the parameters of the full image harmonization model 1204 are tuned based on the difference between the reference harmonized image 1206 and the model-generated harmonized image 1208. For example, in one specific implementation, the full image harmonization model 1204 may include a generative model followed by a discriminator, and the training may involve unsupervised and / or semi-supervised training.

[0114] In the illustrated implementation, an asterisk (e.g., harmonicized model 204') is used to indicate that the harmonicization model is undergoing training. The asterisk is removed in process 1201 to indicate that model training is complete. In this regard, after the full-image harmonicization model 1204 has been trained, the harmonicization model application component 1108 can apply the harmonicization model to a new input image (e.g., input image 1210) to transform the new input image into a harmonicized image 1212 generated by the model that has a greater degree of similarity in appearance to the source domain image relative to the input image 1210.

[0115] Figure 13A The following diagram illustrates the training of a sub-image harmonization model 1304 based on a sub-image model-based harmonization method.1-i The exemplary process 1300. According to the sub-image method, a separate sub-image harmonization model can be trained for each set of related sub-images to transform them into modified sub-images. In this embodiment, multiple sets of sub-images and corresponding modified sub-images generated for each training image in training image 1202 during the image harmonization process are used as benchmark truth training samples.

[0116] For example, as shown in process 1300, each training image in training image 1202 can be decomposed into different sub-images (e.g., different band images) according to the decomposition technique described in reference decomposition component 106. For example, a single training image can be decomposed into a first training sub-image, a second training sub-image, and so on, up to the i-th sub-image (where i = 1, 2, ..., B). In the illustrated embodiment, sub-images of the same type can be grouped together as separate groups (e.g., grouping first-pass sub-images together, grouping second-pass sub-images together, etc.). For example, in the illustrated embodiment, sub-images in different groups are identified as first training sub-image 13011, second training sub-image 13022, and so on, up to the i-th group. The image harmonization technique of image harmonization component 108 can be used to process at least some sub-images in each training iteration to generate modified sub-images of the corresponding sub-images. For example, in the illustrated embodiment, each group of sub-images has at least some base truth (GT) modified sub-images (e.g., GT first modified sub-image 13081, GT second modified sub-image 13082, etc.). These ground truth (GT) modified sub-images can be used as baseline truths for training individual sub-image harmonization models 1304 to transform a specific group of sub-images into model-generated modified sub-images 1306. For example, the first GT modified sub-image 13081 can be used as baseline truth to train the first sub-image harmonization model 13041 to transform the first training sub-image 13021 into the first MG modified sub-image 13061. The second GT modified sub-image 13082 can be used as baseline truth to train the second sub-image harmonization model 13042 to transform the second training sub-image 13022 into the second MG modified sub-image 13062. This process can be repeated for each group of sub-images until the i-th group.

[0117] The type of the sub-image / full-image harmonization model 1204 can vary, and therefore the training process adopted by the training unit 1102 to train the sub-image harmonization model 1304 can also vary. At a high level, the training process may involve sub-harmonization model 1304... i Applied to its training sub-image 1302 i 1306 harmonized and modified sub-images generated by the generative model i Further, the baseline image was modified by training sub-image 1308. iThe modified sub-image 1308 can be compared with the model output, and then the modified sub-image 1306 generated by the model can be compared with the baseline truth. i The difference between them is used to tune the sub-image harmonization model 1304 i Parameters. For example, in one specific implementation, the sub-image harmonization model 1304 may include a generative real model followed by a discriminator, and training may involve unsupervised and / or semi-supervised training.

[0118] Now for reference Figure 13B Process 1301 illustrates the application of a (trained) sub-image harmonization model 1304 for preprocessed image harmonization according to one or more embodiments of the disclosed subject matter. Figure 13A In the illustrated implementation, an asterisk (e.g., sub-image harmonization model 1304') is used to indicate that the sub-image harmonization model is undergoing training. The asterisk is removed in process 1301 to indicate that model training is complete. In this regard, the sub-image harmonization model 1304 has already been trained. 1-i Subsequently, the decomposition unit 106 can decompose the new input image 1208 into sub-images, which are identified as sub-images 1310 in process 1301. 1-i The harmonization model application component 1108 can further apply the corresponding sub-image harmonization model 1304 to its appropriate sub-image 1310. 1-i To the corresponding sub-image 1310 1-i Transformed into the modified sub-image 1312 generated by the model 1-i At 1304, the reconstructed component 110 is then combined with the model to generate the modified sub-image 1312. 1-i The model generates a harmonized image 1316 that is more similar in appearance to the source domain image than the input image 1210.

[0119] Figure 14 An exemplary high-level flowchart of a computer-implemented process 1400 for image harmonization using a full image harmonization model, according to one or more embodiments of the disclosed subject matter, is shown. For brevity, repeated descriptions of similar elements employed in the respective embodiments are omitted.

[0120] At 1402, method 1400 includes: generating a baseline truth harmonic image (e.g., baseline truth harmonic image 1206) of a corresponding training image (e.g., training image 1202) with different appearance variations by a system (e.g., system 1100) operatively coupled to a processor (e.g., using a single reference harmonicization component 802 or a multi-reference harmonicization component 804), wherein the generation includes employing at least one reference image, and wherein the baseline truth harmonic image has a more similar appearance to the corresponding training image and the at least one reference image. Additional details regarding an exemplary process for generating the baseline truth harmonic image are described below with reference to procedures 1500 and operations 1502-1506. At 1404, method 1400 further includes: training a full image harmonicization model (e.g., full image harmonicization model 1204) by the system (e.g., using training component 1102) to transform the corresponding training image into the baseline truth harmonic image. At 1406, method 1400 further includes: applying a full-image harmonization model to a new input image (e.g., input image 1210) by a system (e.g., using harmonization model application component 1108) to transform the new input image into a new harmonized image (e.g., the model-generated harmonized image 1210) that has a more similar appearance to at least one reference image relative to the new input image.

[0121] Figure 15 An exemplary high-level flowchart of a computer implementation of a process 1500 for developing a full image harmonization model according to one or more embodiments of the disclosed subject matter is shown. For the sake of brevity, repeated descriptions of similar elements used in the respective embodiments are omitted.

[0122] At 1502, method 1500 includes: a system (e.g., system 1100) operatively coupled to a processor (e.g., using decomposition component 106) decomposing a training image (e.g., training image 1202) with different appearance variations into sub-images (e.g., training sub-image 1302). 1-i At 1504, method 1500 includes: harmonizing a subimage with a corresponding reference subimage of a reference image using the techniques described above for reference image harmonization component 108 by a system (e.g., using a single reference harmonization component 802 or a multi-reference harmonization component 804), thereby resulting in a modified subimage (e.g., a modified subimage 1308 of the reference image). 1-i The transformation of ) . At 1506, method 1500 further includes: combining a group of modified sub-images associated with the corresponding training image in the training images by a system (e.g., using reconstruction component 110) to generate a baseline truth harmonic image of the corresponding training image (e.g., baseline truth harmonic image 1206). For example, as referenced Figure 3 and Figure 11Each training image is decomposed into a set of sub-images (e.g., sub-image 304), and then the set of sub-images is transformed into modified sub-images (e.g., modified sub-image 310) using the image harmonization technique described in the corresponding reference sub-image (e.g., reference sub-image 308) and the reference image harmonization component 108. The modified sub-images (e.g., modified sub-image 310) are then combined by the reconstruction component 110 to generate a baseline true harmonized image of the training images (e.g., harmonized image 312). At 1508, method 1500 further includes training a full image harmonization model (e.g., full image harmonization model 1204) by the system (e.g., using training component 1102) to transform the training images into baseline true harmonized images, such as the reference... Figure 12 As described in process 1200.

[0123] Figure 16 An exemplary high-level flowchart of a computer-implemented process 1600 for image harmonization using a sub-image harmonization model, according to one or more embodiments of the disclosed subject matter, is shown. For brevity, repeated descriptions of similar elements employed in the respective embodiments are omitted.

[0124] At 1602, method 6100 includes: a system (e.g., system 1100) operatively coupled to a processor generates a corresponding sub-image (e.g., training sub-image 1302) decomposed from a training image (e.g., training image 1202). 1-i The baseline truth sub-image (e.g., the baseline truth modified sub-image 1308) 1-i The generation of the modified sub-image may involve employing at least one reference image. For example, the generation of the modified sub-image may be performed by the decomposition unit 106 and the image harmonization unit 108 according to operations 1502-1504 of process 1500 (e.g., by harmonizing the sub-image with a corresponding reference sub-image, resulting in a transformation of the sub-image to a reference truth sub-image). At 1604, method 1600 further includes training a sub-image harmonization model (e.g., sub-image harmonization model 1304) by a system (e.g., using training unit 1102). 1-i This transforms the sub-image into a reference truth sub-image, such as the reference image. Figure 13A As described in process 1300.

[0125] At 1608, method 1600 further includes: applying a sub-image machine learning model by the system (e.g., using harmonization model application component 1108) to decompose a new sub-image (e.g., sub-image 1310) from the input image (e.g., input image 1210). 1-i Transformed into a new modified sub-image (e.g., the modified sub-image 1312 generated by the model). 1-i (See reference) Figure 13BAs described in process 1301. At 1610, method 1600 includes: combining new modified sub-images by a system (e.g., using reconstruction component 110) to generate a harmonized image (model-generated harmonized image 1316) of the input image, which has a more similar appearance to at least one reference image relative to the input image.

[0126] Exemplary Operating Environment

[0127] One or more embodiments may be systems, methods, and / or computer program products at any possible level of technical detail in integration. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0128] Computer-readable storage media can be tangible devices that hold and store instructions for use by instruction execution devices. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punch cards or raised structures in grooves on which instructions are recorded), and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0129] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them for use in a computer-readable storage medium stored within the suitable computing / processing device.

[0130] Computer-readable program instructions used to implement the operations of this invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit system, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute the computer-readable program instructions to personalize the electronic circuitry in order to implement aspects of the invention by utilizing the state information of the computer-readable program instructions.

[0131] This document describes aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium having the instructions stored therein includes an article of writing comprising instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus or other device implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible specific implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may not occur in the order shown in the figures. For example, depending on the functions involved, two blocks shown consecutively may be executed substantially simultaneously, or sometimes the blocks may be executed in reverse order. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or implements a combination of dedicated hardware and computer instructions.

[0135] Combination Figure 17 The systems and processes described below may be embodied in hardware, such as a single integrated circuit (IC) chip, multiple ICs, application-specific integrated circuits (ASICs), etc. Furthermore, the order in which some or all program blocks appear in each program should not be considered restrictive. Rather, it should be understood that some program blocks can be executed in various orders, and not all of the orders described herein are explicitly shown.

[0136] refer to Figure 17 An exemplary environment 1700 for implementing various aspects of the claimed subject matter includes a computer 1702. The computer 1702 includes a processing unit 1704, system memory 1706, a codec 1735, and a system bus 1708. The system bus 1708 couples system components, including but not limited to system memory 1706, to the processing unit 1704. The processing unit 1704 can be any of a variety of available processors. Dual microprocessors and other multiprocessor architectures may also be used as the processing unit 1704.

[0137] The system bus 1708 can be any of several types of bus architectures, including memory bus or memory controller, peripheral bus or external bus, or local bus using any of the available bus architectures, including but not limited to Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), PCMCIA Bus, FireWire (IEEE 1394), and Small Computer System Interface (SCSI).

[0138] In various embodiments, system memory 1706 includes volatile memory 1710 and non-volatile memory 1712, which may employ one or more of the disclosed memory architectures. A basic input / output system (BIOS) containing basic routines such as transferring information between elements within computer 1702 during startup is stored in non-volatile memory 1712. Additionally, according to the invention, codec 1735 may include at least one of an encoder or a decoder, wherein the encoder or decoder may be composed of hardware, software, or a combination of hardware and software. Although codec 1735 is depicted as a separate component, codec 1735 may be contained within non-volatile memory 1712. By way of illustration and not limitation, non-volatile memory 1712 may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, 3D flash memory, or resistive memory, such as resistive random access memory (RRAM). In at least some embodiments, the non-volatile memory 1712 may employ one or more of the disclosed memory devices. Furthermore, the non-volatile memory 1712 may be computer memory (e.g., physically integrated with computer 1702 or its motherboard) or removable memory. Examples of suitable removable memory that can be used to implement the disclosed embodiments may include a Secure Digital (SD) card, a Compact Flash (CF) card, a Universal Serial Bus (USB) Memory Stick, etc. The volatile memory 1710 includes random access memory (RAM) that acts as external cache memory, and may also employ one or more of the disclosed memory devices in various embodiments. By way of illustration and not limitation, RAM can be provided in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (ESDRAM), etc.

[0139] Computer 1702 may also include removable / non-removable, volatile / non-volatile computer storage media. Figure 17Disk storage device 1714 is illustrated, for example. Disk storage device 1714 includes, but is not limited to, devices such as disk drives, solid-state drives (SSDs), flash memory cards, or memory sticks. Additionally, disk storage device 1714 may include storage media, alone or in combination with other storage media, including, but not limited to, optical disc drives, such as optical disc ROM devices (CD-ROM), CD recordable drives (CD-R drives), CD rewritable drives (CD-RW drives), or digital versatile optical disc ROM drives (DVD-ROM). To facilitate connection of disk storage device 1714 to system bus 1708, removable or non-removable interfaces, such as interface 1716, are typically used. It should be understood that disk storage device 1714 may store user-related information. Such information may be stored on a server or provided to applications running on a server or user device. In one embodiment, the type of information stored on disk storage device 1714 or transmitted to a server or application may be notified to the user (e.g., via output device 1736). Users may be given the option to opt in or opt out of the collection or sharing of such information via a server or application (e.g., via input from input device 1728).

[0140] It should be understood that Figure 17 Software is described that acts as an intermediary between the user and the basic computer resources described in the suitable operating environment 1700. This software includes an operating system 1718. The operating system 1718, which may be stored on a disk storage device 1714, is used to control and allocate the resources of the computer 1702. Application program 1720 utilizes the operating system 1718 to manage resources through program module 1724, and program data 1726, such as power-on / power-off transaction tables, stored in system memory 1706 or on disk storage device 1714. It should be understood that the claimed subject matter can be implemented using various operating systems or combinations of operating systems.

[0141] Users input commands or information into computer 1702 via input device 1728. Input device 1728 includes, but is not limited to, pointing devices such as mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, gamepad, satellite dish, scanner, TV tuner card, digital camera, digital camcorder, webcam, etc. These and other input devices are connected to processing unit 1704 via interface port 1730 through system bus 1708. Interface port 1730 includes, for example, serial port, parallel port, game port, and Universal Serial Bus (USB). Output device 1736 uses some of the same type of ports as input device 1728. Thus, for example, a USB port can be used to provide input to computer 1702 and output information from computer 1702 to output device 1736. Output adapter 1734 is provided to indicate some output devices 1736 such as monitors, speakers, and printers, as well as other output devices 1736 that require special adapters. By way of illustration and not limitation, output adapter 1734 includes a video card and a sound card that provide a means of connection between output device 1736 and system bus 1708. It should be noted that other devices or systems of devices provide both input and output capabilities, such as remote computer 1738.

[0142] Computer 1702 can operate in a networked environment using a logical connection to one or more remote computers, such as remote computer 1738. Remote computer 1738 can be a personal computer, server, router, network PC, workstation, microprocessor-based device, peer-to-peer device, smartphone, tablet, or other network node, and typically includes many of the elements described with respect to computer 1702. For simplicity, only memory storage device 1740 is shown for remote computer 1738. Remote computer 1738 is logically connected to computer 1702 via network interface 1742 and subsequently connected via communication connection 1744. Network interface 1742 encompasses wired or wireless communication networks, such as local area networks (LANs) and wide area networks (WANs), as well as cellular networks. LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ring, etc. WAN technologies include, but are not limited to, point-to-point links, such as Integrated Services Digital Network (ISDN) and its variants, circuit-switched networks, packet-switched networks, and Digital Subscriber Line (DSL).

[0143] Communication connection 1744 refers to the hardware / software used to connect network interface 1742 to bus 1708. Although communication connection 1744 is shown as being inside computer 1702 for clarity of illustration, the communication connection may also be external to computer 1702. For illustrative purposes only, the hardware / software required to connect to network interface 1742 includes internal and external technologies such as modems, including conventional telephone-grade modems, cable modems and DSL modems, ISDN adapters, and wired and wireless Ethernet cards, hubs and routers.

[0144] Although the subject matter has been described above in the general context of computer executable instructions for a computer program product running on one or more computers, those skilled in the art will recognize that the present disclosure may also be implemented in combination with other program modules. Generally, program modules include routines, programs, components, data structures, etc., that perform specific tasks and / or implement specific abstract data types. Furthermore, those skilled in the art will understand that the computer implementation of the methods of the present invention can be practiced using other computer system configurations, including single-processor or multi-processor computer systems, small computing devices, mainframe computers, and computers, handheld computing devices (e.g., PDAs, telephones), microprocessor-based or programmable consumer or industrial electronic devices, etc. The illustrated aspects can also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked via a communication network. However, some (if not all) aspects of the present disclosure can be practiced on a standalone computer. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0145] As used herein, the terms “component,” “system,” “platform,” “interface,” etc., may refer to and / or include computer-related entities or entities associated with an operator having one or more specific functions. Entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a program running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, an application running on a server and a server may both be components. One or more components may reside within a program and / or an execution thread, and components may be located on a single computer and / or distributed across two or more computers. In another example, a corresponding component may execute on various computer-readable media on which various data structures are stored. Components may communicate via local and / or remote programs, such as based on signals having one or more data packets (e.g., data from one component that interacts with another component in a local system, a distributed system, and / or a network (such as the Internet with other systems). As another example, a component can be a device having specific functions provided by mechanical parts operated by an electrical or electronic circuitry system, which is operated by a software or firmware application executed by a processor. In this case, the processor can be internal or external to the device and can execute at least a portion of the software or firmware application. As yet another example, a component can be a device that provides specific functions through electronic components rather than mechanical parts, wherein the electronic components can include a processor or other means for executing software or firmware that at least partially gives the electronic components functionality. In one aspect, a component can be emulated, for example, via a virtual machine within a cloud computing system.

[0146] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X adopts A or B" is intended to mean any natural inclusive substitution. That is, if X adopts A; X adopts B; or X adopts both A and B, then "X adopts A or B" is satisfied in any of the foregoing cases. Additionally, unless otherwise specified or clear from the context to be directed to the singular form, the articles "a" and "an" used in this specification and accompanying drawings should generally be construed as meaning "one or more." As used herein, the terms "example" and / or "exemplary" are used to mean serving as an example, instance, or illustration, and are intended to be non-limiting. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. Furthermore, any aspect or design described herein as "example" and / or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs, nor does it exclude equivalent exemplary structures and techniques known to those skilled in the art.

[0147] As used herein, the term "processor" can refer substantially to any computing processing unit or device, including but not limited to a single-core processor; a single processor with software multithreading capabilities; a multi-core processor; a multi-core processor with software multithreading capabilities; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. Additionally, a processor can refer to an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Furthermore, a processor can utilize nanoscale architectures (such as, but not limited to, molecular and quantum dot-based transistors, switches, and gates) to optimize space usage or enhance the performance of user devices. A processor can also be implemented as a combination of computing processing units. In this disclosure, terms such as "storage," "storage device," "data storage," "data storage apparatus," "database," and substantially any other information storage component associated with the operation and function of a component are used to refer to a "memory component," an entity embodied in "memory," or a component that includes memory. It should be understood that the memory and / or memory components described herein may be volatile or non-volatile memory, or may include both volatile and non-volatile memory. By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM that can act as external cache memory. By way of illustration and not limitation, RAM can be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). Furthermore, the memory components disclosed in the systems or computer-implemented methods described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0148] The foregoing description includes only examples of systems and computer-implemented methods. Of course, it is impossible to describe every conceivable combination of components or computer-implemented methods for the purposes of describing this disclosure, but those skilled in the art will recognize that many other combinations and substitutions are possible. Furthermore, regarding the extent to which the terms “comprising,” “having,” “possessing,” etc., are used in the detailed description, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term “comprising,” as interpreted when “comprising” is used as a transitional word in the claims. Various embodiments have been described for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, the practical application or improvement of technology superior to that found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A system for image harmonization optimized by a deep learning model, the system comprising: Memory, which stores computer-executable components; and A processor that executes computer-executable components stored in the memory, wherein the computer-executable components include: An image decomposition component, which decomposes an input image into sub-images; and A harmonization component, which harmonizes the sub-image with the corresponding reference sub-image based on two or more different statistical values ​​calculated respectively for the sub-image and a corresponding reference sub-image of at least one reference image, thereby resulting in a transformation of the sub-image to a modified sub-image; A clustering component that clusters the candidate reference images into different groups based on the differences between feature vectors between the candidate reference images; A reference image set generation component selects a candidate reference image from each of the different groups as a subset of the candidate reference images, and a reference image selection component selects the at least one reference image from the subset.

2. The system of claim 1, wherein the computer-executable component further comprises: A reconstruction component that combines the modified sub-images into a harmonized image, the harmonized image having a more similar appearance to the input image and the at least one reference image.

3. The system according to claim 1, wherein the sub-image includes an energy band image, and the corresponding reference sub-image includes a corresponding reference energy band image.

4. The system of claim 3, wherein the two or more different statistical values ​​are selected from the group consisting of: the mean of the band image intensity values ​​of the band image and the corresponding reference band image, the standard deviation of the band image intensity values, the percentiles of the band image intensity values, and a histogram of the band image intensity values.

5. The system of claim 1, wherein the image decomposition component generates the sub-image from the low-pass signal.

6. The system of claim 1, wherein the computer-executable component further comprises: A reference image selection component selects the at least one reference image from candidate reference images based on the degree of similarity between a first feature vector of the input image and a second feature vector of the at least one reference image.

7. The system of claim 6, wherein the first feature vector and the second feature vector are each based on at least one of the two or more different statistical values.

8. The system of claim 7, wherein the sub-image includes an energy band image, and the corresponding reference sub-image includes a corresponding reference energy band image, and wherein the two or more distinct statistical values ​​are selected from the group consisting of: the mean of the energy band image intensity values ​​of the energy band image and the corresponding reference energy band image, the standard deviation of the energy band image intensity values, the percentiles of the energy band image intensity values, and a histogram of the energy band image intensity values.

9. The system of claim 6, wherein the first feature vector and the second feature vector are respectively based on output features generated by applying a deep learning neural network model to the input image and the at least one reference image, wherein the deep learning neural network model is trained on a corpus of images associated with the input image and the at least one reference image.

10. The system of claim 6, wherein the computer-executable component further comprises: A vectorization component generates feature vectors for the candidate reference image.

11. The system of claim 2, wherein the computer-executable component further comprises: A training component uses the harmonized image and an additional harmonized image generated in the same manner as the harmonized image as baseline truth training samples to facilitate the training of a full-image harmonization model to transform an input image into a harmonized image that has a more similar appearance to the at least one reference image relative to the input image.

12. The system of claim 11, wherein the full image harmonization model comprises one or more deep learning neural network models.

13. The system of claim 11, wherein the computer-executable component further comprises: The model application component applies the full image harmonization model to transform a new input image into a new harmonized image.

14. The system of claim 1, wherein the computer-executable component further comprises: The training component uses the modified sub-image as a baseline truth training sample to facilitate the training of a sub-image harmonization model to transform the sub-image into the modified sub-image.

15. The system of claim 14, wherein the sub-image harmonization model comprises a deep learning neural network model.

16. The system of claim 14, wherein the computer-executable component further comprises: The model application component applies the sub-image harmonization model to transform a new sub-image decomposed from a new input image into a new modified sub-image from the harmonized image; and A reconstruction component combines the new modified sub-images to generate a new harmonic image that has a more similar appearance to the at least one reference image relative to the new input image.

17. A system for image harmonization optimized by a deep learning model, the system comprising: Memory, which stores computer-executable components; and A processor that executes computer-executable components stored in the memory, wherein the computer-executable components include: An image decomposition component, which decomposes an input image into sub-images; and A harmonization component harmonizes the sub-image with a reference sub-image of a corresponding reference image, thereby resulting in a transformation of the sub-image to a modified sub-image, wherein the harmonization component uses a weighting scheme for the reference sub-image determined based on the degree of similarity between the input image and the corresponding reference image to harmonize the sub-image with the reference sub-image. A clustering component that clusters the candidate reference images into different groups based on the differences between feature vectors between the candidate reference images; A reference image set generation component selects a candidate reference image from each of the different groups as a subset of the candidate reference images, and a reference image selection component selects at least one reference image from the subset.

18. The system of claim 17, wherein the computer-executable component further comprises: A reconstruction component that combines the modified sub-images into a harmonized image, the harmonized image having a more similar appearance to the input image and the corresponding reference image.

19. The system of claim 17, wherein the harmonization component harmonizes the sub-image with the reference sub-image based on one or more statistical values ​​calculated for the sub-image and the reference sub-image, respectively.

20. The system of claim 19, wherein the sub-image includes an energy band image, and the reference sub-image includes a corresponding reference energy band image, and wherein the one or more statistical values ​​are selected from the group consisting of: the mean of the energy band image intensity values ​​of the energy band image and the corresponding reference energy band image, the standard deviation of the energy band image intensity values, the percentiles of the energy band image intensity values, and a histogram of the energy band image intensity values.

21. The system of claim 20, wherein the image decomposition component generates the bandgap image from a low-pass signal.

22. The system of claim 20, wherein the degree of similarity is based on a similarity measure between a first feature vector of the input image and a second feature vector of the corresponding reference image.

23. The system of claim 22, wherein the first feature vector and the second feature vector are respectively based on statistical values ​​calculated according to corresponding band image intensity values ​​associated with the band image and the corresponding reference band image.

24. The system of claim 22, wherein the first feature vector and the second feature vector are respectively based on output features generated by applying a deep learning neural network model to the input image and the corresponding reference image, wherein the deep learning neural network model is trained on a corpus of images associated with the input image and the corresponding reference image.

25. The system of claim 17, wherein the computer-executable component further comprises: A vectorization component that generates feature vectors for candidate reference images.

26. The system of claim 17, further comprising: A training component uses the modified sub-image as a baseline truth training sample to facilitate the training of one or more image harmonization models to transform the input image into a harmonized image that has a more similar appearance to the corresponding reference image relative to the input image.

27. The system of claim 26, wherein the one or more image harmonization models comprise one or more deep learning neural network models.

28. A method for image harmonization optimized by a deep learning model, the method comprising: A system operatively coupled to a processor decomposes an input image into sub-images; as well as The system harmonizes the sub-image with a reference sub-image of a corresponding reference image, thereby resulting in a transformation of the sub-image into a modified sub-image, wherein the harmonization includes employing a weighting scheme for the reference sub-image based on the degree of similarity between the input image and the corresponding reference image; Specifically, at least one reference image is selected from a subset of candidate reference images, the candidate reference images are clustered into different groups based on the differences between the feature vectors of the candidate reference images, and a candidate reference image is selected from each of the different groups as a subset of the candidate reference images.

29. The method according to claim 28, further comprising: The system combines the modified sub-images into a harmonized image, which has a more similar appearance to the input image and the corresponding reference image.

30. The method of claim 28, wherein the sub-image comprises an energy band image, and the reference sub-image comprises a reference energy band image, and wherein the harmonization comprises harmonizing the energy band image with the reference energy band image based on two or more characteristics selected from the group consisting of: the mean of the energy band image intensity values ​​of the energy band image and the reference energy band image, the standard deviation of the energy band image intensity values, the percentiles of the energy band image intensity values, and a histogram of the energy band image intensity values.

31. The method of claim 28, wherein the degree of similarity is based on a similarity measure between a first feature vector of the input image and a second feature vector of the corresponding reference image.

32. A method for image harmonization optimized by a deep learning model, the method comprising: A system operatively coupled to a processor generates a baseline truth harmonic image of a corresponding training image with different appearance variations, wherein the generation includes using at least one reference image, and wherein the baseline truth harmonic image has a more similar appearance to the corresponding training image and the at least one reference image. The system trains a full-image harmonization model to transform the corresponding training images into the baseline true harmonized images; as well as The system applies the full image harmonization model to a new input image to transform the new input image into a new harmonized image that has a more similar appearance to the at least one reference image relative to the new input image; The at least one reference image is selected from a subset of candidate reference images, and the candidate reference images are clustered into different groups based on the differences between the feature vectors of the candidate reference images. A candidate reference image is selected from each of the different groups as a subset of the candidate reference images.

33. The method of claim 32, wherein the generation comprises: The system decomposes the corresponding training images into sub-images; The system harmonizes the sub-image with a corresponding reference sub-image of the at least one reference image, thereby causing a transformation of the sub-image into a modified sub-image; as well as The system combines the modified sub-images associated with the corresponding training images in the training images to generate the baseline truth harmonic image of the corresponding training images.

34. The method of claim 33, wherein the sub-image comprises an energy band image, and the corresponding reference sub-image comprises a reference energy band image, and wherein the harmonization comprises harmonizing the energy band image with the reference energy band image based on two or more characteristics selected from the group consisting of: the mean of the energy band image intensity values ​​of the energy band image and the reference energy band image, the standard deviation of the energy band image intensity values, the percentiles of the energy band image intensity values, and a histogram of the energy band image intensity values.

35. A method for image harmonization optimized by a deep learning model, the method comprising: A system operatively coupled to a processor generates a baseline truth sub-image of the corresponding sub-image decomposed from a training image, wherein the generation includes employing at least one reference image; The system trains a sub-image harmonization model to transform the corresponding sub-image into the baseline truth sub-image; The system applies a sub-image machine learning model to transform the newly decomposed sub-image from the input image into a new modified sub-image; as well as The system combines the new modified sub-images to generate a harmonic image of the input image, the harmonic image having a more similar appearance to the at least one reference image relative to the input image; The at least one reference image is selected from a subset of candidate reference images, and the candidate reference images are clustered into different groups based on the differences between the feature vectors of the candidate reference images. A candidate reference image is selected from each of the different groups as a subset of the candidate reference images.

36. The method of claim 35, wherein the generation comprises: The system decomposes the training image into the corresponding sub-images; as well as The system harmonizes the sub-image with a corresponding reference sub-image of the at least one reference image, thereby causing the sub-image to transform into the reference truth sub-image.

37. The method of claim 36, wherein the sub-image comprises an energy band image, and the corresponding reference sub-image comprises a reference energy band image, and wherein the harmonization comprises harmonizing the energy band image with the reference energy band image based on two or more characteristics selected from the group consisting of: the mean of the energy band image intensity values ​​of the energy band image and the reference energy band image, the standard deviation of the energy band image intensity values, the percentiles of the energy band image intensity values, and a histogram of the energy band image intensity values.

Images