An image preprocessing method for medical image enhancement

Abstract

The application relates to the field of medical image processing, in particular to an image preprocessing method for medical image enhancement, which sequentially performs image registration through low-resolution semantic coarse registration, high-resolution semantic fine registration, local similarity feature fine registration and multi-stage fusion fine-tuning, guarantees the accuracy and efficiency of image registration, guarantees global semantic alignment through semantic-based registration, guarantees the consistency of local features through image-based registration, aligns the plain scan image and the enhanced image through image registration under the premise of guaranteeing clear image details, improves the reliability of the enhanced image after registration, and improves the effect of image preprocessing.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an image preprocessing method for medical image enhancement. Background Technology

[0002] Currently, plain scan images are commonly used in medical image-assisted diagnosis and analysis. However, although plain scan images are safer and faster, the distinction between lesions and normal tissues in plain scan images is relatively limited, making it difficult for plain scan images to provide effective assistance and analysis in some scenarios.

[0003] To address the aforementioned issues, existing methods typically involve patients receiving intravenous or oral contrast agents followed by the acquisition of enhanced images. Enhanced images can improve the differentiation between different tissues, better identify lesions, and provide biochemical information and assess the functional performance of specific tissues.

[0004] However, the use of contrast agents increases the patient's treatment time and cost, and contrast agents are nephrotoxic and can easily cause adverse reactions in patients. The patient's normal organs will also be irradiated multiple times.

[0005] In existing technologies, synthesizing enhanced images from plain scan images using image generation methods can avoid the use of contrast agents. However, image generation models require strictly paired plain scan images and enhanced images as training sample pairs during training. Since the plain scan images and enhanced images are taken at different times, and slight movements such as patient breathing and internal organs inevitably cause misalignment between the plain scan images and enhanced images, this seriously hinders the training performance of the image generation model. Existing image preprocessing methods are difficult to align plain scan images and enhanced images through image registration while ensuring image clarity, resulting in poor image preprocessing effects.

[0006] Therefore, how to improve the effect of image preprocessing and thus provide higher quality flat scan images and enhanced images has become an urgent problem to be solved. Summary of the Invention

[0007] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows: An image preprocessing method for medical image enhancement, the image preprocessing method for medical image enhancement comprising: S11, the first semantic segmentation image corresponding to the initial flat scan image and the second semantic segmentation image corresponding to the initial enhanced image are downsampled respectively to obtain the first intermediate segmentation image corresponding to the initial flat scan image and the second intermediate segmentation image corresponding to the initial enhanced image. S12, perform image registration on the first intermediate segmented image and the second intermediate segmented image, determine the first spatial transformation parameters, and map the initial enhanced image to the first intermediate registered image according to the first spatial transformation parameters; S13, perform image registration on the first semantic segmentation image and the third semantic segmentation image obtained by semantic segmentation of the first semantic segmentation image and the first intermediate registration image, determine the second spatial transformation parameters, and map the first intermediate registration image to the second intermediate registration image according to the second spatial transformation parameters; S14, perform image registration on the initial flat scan image and the second intermediate registration image, determine the third spatial transformation parameters, and map the second intermediate registration image to the third intermediate registration image according to the third spatial transformation parameters; S15, perform image registration on the initial flat scan image and the third intermediate registration image, determine the fourth spatial transformation parameters, and map the third intermediate registration image to the target enhancement image corresponding to the initial flat scan image according to the fourth spatial transformation parameters.

[0008] Compared with the prior art, the present invention has significant advantages. Through the above technical solution, the image preprocessing method for medical image enhancement provided by the present invention achieves considerable technical progress and practicality, and has broad industrial application value. It has at least the following advantages: This invention provides an image preprocessing method for medical image enhancement. The method performs image registration sequentially through several steps: low-resolution semantic coarse registration, high-resolution semantic fine registration, local similarity feature fine registration, and multi-stage fusion fine-tuning. This ensures the accuracy and efficiency of image registration. Semantic-based registration guarantees global semantic alignment, while image-based registration ensures the consistency of local feature details. Under the premise of ensuring clear image details, the method aligns the flat scan image and the enhanced image through image registration, thereby improving the reliability of the enhanced image after registration and thus improving the effect of image preprocessing. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic flowchart of an image preprocessing method for medical image enhancement provided in Embodiment 1 of the present invention; Figure 2This is a flowchart illustrating a deep learning-based medical image enhancement method provided in Embodiment 2 of the present invention. Figure 3 This is a training architecture for the image generator and discriminator in a deep learning-based medical image enhancement method provided in Embodiment 2 of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] This embodiment provides an image preprocessing method for medical image enhancement. See [link to previous section]. Figure 1 The present invention provides a schematic flowchart of an image preprocessing method for medical image enhancement according to Embodiment 1. The image preprocessing method for medical image enhancement includes: S11, the first semantic segmentation image corresponding to the initial flat scan image and the second semantic segmentation image corresponding to the initial enhanced image are downsampled respectively to obtain the first intermediate segmentation image corresponding to the initial flat scan image and the second intermediate segmentation image corresponding to the initial enhanced image. S12, perform image registration on the first intermediate segmented image and the second intermediate segmented image, determine the first spatial transformation parameters, and map the initial enhanced image to the first intermediate registered image according to the first spatial transformation parameters; S13, perform image registration on the first semantic segmentation image and the third semantic segmentation image obtained by semantic segmentation of the first semantic segmentation image and the first intermediate registration image, determine the second spatial transformation parameters, and map the first intermediate registration image to the second intermediate registration image according to the second spatial transformation parameters; S14, perform image registration on the initial flat scan image and the second intermediate registration image, determine the third spatial transformation parameters, and map the second intermediate registration image to the third intermediate registration image according to the third spatial transformation parameters; S15, perform image registration on the initial flat scan image and the third intermediate registration image, determine the fourth spatial transformation parameters, and map the third intermediate registration image to the target enhancement image corresponding to the initial flat scan image according to the fourth spatial transformation parameters.

[0013] The initial plain scan image refers to the scan image acquired without the injection of contrast agent, while the initial enhanced image refers to the scan image acquired with the injection of contrast agent. The initial plain scan image and the initial enhanced image are acquired from the same object and the acquisition time interval is short to ensure that the initial plain scan image and the initial enhanced image have a corresponding relationship.

[0014] The first semantic segmentation image is obtained by inputting the initial plain scan image into the trained semantic segmentation model, and the second semantic segmentation image is obtained by inputting the initial augmented image into the trained semantic segmentation model. The semantic segmentation model can adopt the U-Net architecture. The training dataset and training loss function of the semantic segmentation model used for medical image segmentation are existing technologies and will not be described in detail here.

[0015] The implementer sets the downsampling step size to perform downsampling processing on the first semantic segmentation image and the second semantic segmentation image to obtain the first intermediate segmentation image and the second intermediate segmentation image. The downsampling step size is an integer greater than or equal to 2. The purpose of downsampling processing is to reduce the amount of computation required for subsequent image registration in order to avoid exceeding the computational load of the computing device.

[0016] In one specific embodiment, the image preprocessing method for medical image enhancement further includes: According to a preset first scale, local relative contrast is calculated for the initial flat scan image and the initial enhanced image respectively to obtain a first local contrast image corresponding to the initial flat scan image and a second local contrast image corresponding to the initial enhanced image; Based on the preset first contrast difference threshold, first contrast difference normalization threshold and first contrast normalization threshold, a first region of interest image is extracted from the first local contrast image, and a second region of interest image is extracted from the second local contrast image. Accordingly, the step of image registration of the first intermediate segmented image and the second intermediate segmented image to determine the first spatial transformation parameters includes: A global segmentation similarity metric is calculated based on a first temporary segmentation image obtained by mapping the first intermediate segmentation image and the second intermediate segmentation image with the initialized first initial transformation parameters. Based on the downsampling result of the first region of interest image, the first region of interest is extracted from the first intermediate segmented image; based on the downsampling result of the second region of interest image, the second region of interest is extracted from the first temporary segmented image. Based on the first region of interest and the second region of interest, a region segmentation similarity metric is calculated. The first spatial transformation parameters are determined based on the global segmentation similarity metric and the region segmentation similarity metric.

[0017] The first scale is used to determine the neighborhood range when calculating the local relative contrast. For example, if the first scale is set to 5, then when calculating the local relative contrast, the calculation is performed in a 5×5 neighborhood centered on the pixel. The standard deviation and mean of the pixel values ​​of all pixels within the neighborhood are calculated, and the ratio of the standard deviation and the mean is used as the local relative contrast value corresponding to the center pixel. It should be noted that, in order to avoid the mean being zero within the neighborhood, the implementer can add a bias parameter to the mean. The bias parameter is a small constant.

[0018] For any pixel, calculate the difference between the local relative contrast value of the pixel and the average local relative contrast value of all pixels in the neighborhood determined by the first scale, and then compare it with the first contrast difference threshold to obtain the first comparison result. For example, the first contrast difference threshold is set to 150 / 700.

[0019] The ratio of the difference between the local relative contrast value of the pixel and the average local relative contrast value of all pixels in the neighborhood determined by the first scale to the average local relative contrast value of all pixels in the neighborhood is calculated to obtain a second comparison result. This result is then compared with a first contrast difference normalization threshold. For example, the first contrast difference normalization threshold is set to 0.91.

[0020] Calculate the ratio of the local relative contrast value of the pixel to the maximum local relative contrast value of all pixels, and then compare it with the first contrast normalization threshold to obtain the third comparison result. For example, the first contrast normalization threshold is set to 0.97.

[0021] If the first comparison result, the second comparison result, and the third comparison result are all greater than 1, the mask pixel value of the pixel is determined to be 1; otherwise, the mask pixel value of the pixel is determined to be 0. By traversing all pixels in the initial flat scan image, the corresponding first region of interest image can be obtained. By traversing all pixels in the initial enhanced image, the corresponding second region of interest image can be obtained. It can be seen that both the first region of interest image and the second region of interest image are binary images.

[0022] Specifically, since the first intermediate segmented image and the second intermediate segmented image have been downsampled, their image sizes are different from those of the initial flat scan image and the initial enhanced image. Therefore, the first region of interest image and the second region of interest image also need to be downsampled, and the downsampling step size set above is used for processing.

[0023] The first region of interest (ROI) refers to the set of pixels in the first intermediate segmented image whose masking pixel value is 1 in the downsampling result of the first ROI image. The second region of interest (ROI) refers to the set of pixels in the first temporary segmented image whose masking pixel value is 1 in the downsampling result of the second ROI image.

[0024] By performing a similarity measurement on the first intermediate segmented image and the first temporary segmented image, the global segmentation similarity measurement value can be obtained. As an example, the similarity measurement method can be the minimum mean square error. Similarly, by performing a similarity measurement on the first region of interest and the second region of interest, the local segmentation similarity measurement value can be obtained.

[0025] It should be noted that this embodiment uses a first scale, a first contrast difference threshold, a first contrast difference normalization threshold, and a first contrast normalization threshold to extract high-contrast portions in the global image as regions of interest. However, in actual medical image processing scenarios, the pixel values ​​corresponding to blood vessels and other parts are not high-contrast portions in the global image, but rather relatively high-contrast portions in a local area. Therefore, in one embodiment, the implementer can use a second scale, a second contrast difference threshold, and a second contrast difference normalization threshold to extract local high-contrast portions as regions of interest. Accordingly, the second scale should be smaller than the first scale to enhance the focus on locality during extraction. As an example, the second scale is set to 3, the second contrast difference threshold is set to 60 / 700, and the second contrast difference normalization threshold is set to 0.7.

[0026] Similarly, in one embodiment, the region of interest extraction of organs such as the spleen, liver, and pancreas can be performed using a second scale and a second contrast normalization threshold. For example, the second contrast normalization threshold is set to 0.98.

[0027] As can be seen, by setting different neighborhood sizes and thresholds, multiple groups of regions of interest corresponding to different information can be extracted. Each group of regions of interest can be calculated using the above calculation method to obtain the region segmentation similarity coefficient. Accordingly, the first spatial transformation parameter can be determined by the global segmentation similarity coefficient and multiple region segmentation similarity coefficients.

[0028] It should be noted that the initial flat scan image and the initial enhanced image mentioned above are both described as two-dimensional images. In the three-dimensional image registration scenario, the corresponding three-dimensional image is determined by the image aggregation method of multiple two-dimensional images under different preset axes, and then the three-dimensional image is registered. The preset axes may include the first preset axis X, the second preset axis Y, and the third preset axis Z. For any three-dimensional pixel, the pixel value of the three-dimensional pixel is determined according to the pixel value of the corresponding two-dimensional pixel in the two-dimensional image under each preset axis.

[0029] When performing image aggregation on the region of interest, for any three-dimensional pixel, the maximum value of the pixel value of the corresponding two-dimensional pixel in the two-dimensional image under each preset axis is taken as the pixel value of the three-dimensional pixel.

[0030] In one specific implementation, determining the first spatial transformation parameter based on the global segmentation similarity metric and the region segmentation similarity metric includes: The first objective function is determined based on the global segmentation similarity metric and the region segmentation similarity metric. The first initial transformation parameters are adjusted using a first preset search algorithm to minimize the first objective function, and the adjusted first initial transformation parameters corresponding to the minimized first objective function are used as the first spatial transformation parameters.

[0031] In this embodiment, the sum of the global segmentation similarity metric and the regional segmentation similarity metric is used as the first objective function. In one embodiment, the sum of the global segmentation similarity coefficient and multiple regional segmentation similarity coefficients is used as the first objective function. It should be noted that the implementer can add other sub-functions as needed when determining the first spatial transformation parameters to improve the registration effect or meet the implementer's needs for registration effects in different dimensions. For example, the added sub-function can be the joint standard deviation of the first intermediate segmented image and the first temporary segmented image, etc.

[0032] The first initial transformation parameters are in matrix form. The first initial transformation parameters can map the second intermediate segmented image to perform rigid transformation, affine transformation, or nonlinear transformation of the second intermediate segmented image. Depending on the transformation method determined by the implementer, the first initial transformation parameters contain different parameter types. For example, when the transformation method is rigid, the first initial transformation parameters contain a rotation matrix and a translation vector. When the transformation method is affine, the first initial transformation parameters contain a rotation matrix, a translation vector, a scaling parameter, and a shearing parameter. When the transformation method is nonlinear, the first initial transformation parameters contain a displacement field parameter.

[0033] As an example, the first preset search algorithm is set to a coupled convex discrete optimization algorithm.

[0034] It should be noted that, in this embodiment, all types of objective functions can be expressed as T. ’ =argmin T (S(F,T(M))+a×R(T)), that is, finding T such that S(F,T(M))+a×R(T) is minimized, and taking the T corresponding to the minimum S(F,T(M))+a×R(T) as T. ’Let T be the transformation parameter, F be the reference image, M be the moving image, T(M) be the result of mapping the moving image to the transformation parameter, S(F,T(M)) be the evaluation value between F and T(M), and a be the regularization parameter, which serves as the weight coefficient of R(T) to balance similarity and smoothness. A larger a value results in a smoother transformation, while a smaller value tends to more accurately align image grayscale. R(T) is the regularization term, used to constrain the smoothness or physical rationality of the transformation parameter T. For example, in the first objective function, F is the first intermediate segmented image, M is the second intermediate segmented image, and T(M) is the first temporary segmented image. Evaluation values ​​can include mean squared error, cross-correlation, mutual information, etc., while regularization terms can include smoothness constraints, elastic potential, etc. It is known that different regularization parameters, different evaluation value settings, and different search algorithms will all affect the determined T. ’ Because the initial transformation parameters differ, before adjusting them using a preset search algorithm, it is necessary to first determine the regularization parameters, evaluation values, and search algorithm using a small dataset. Multiple sets of parameter settings are then determined based on different values ​​of these parameters. For any given set of parameters, T is searched under that set of parameters. ’ According to F and T ’ (M) Determine the similarity index, which may include global segmentation similarity coefficient, local segmentation similarity coefficient, image similarity coefficient, structural similarity, etc. Determine the preset search algorithm, preset regularization parameters and preset evaluation value from a set of setting parameters corresponding to the optimal similarity index. In practical applications, there is no need to redetermine the preset search algorithm, preset regularization parameters and preset evaluation value under this type of objective function. Only the initial transformation parameters need to be optimized.

[0035] The calculation of global segmentation similarity coefficient and local segmentation similarity coefficient in this embodiment is explained. The pixel values ​​in the first intermediate segmentation image and the second intermediate segmentation image belong to the semantic segmentation category identifier value set. The semantic segmentation category identifier value set includes several semantic segmentation category identifier values. As an example, when using the existing semantic segmentation model for semantic segmentation, the semantic segmentation category identifier value set can contain up to 118 semantic segmentation category identifier values.

[0036] For each semantic segmentation category, a first set is formed by pixels in the first intermediate segmentation image whose pixel value is the semantic segmentation category identifier value corresponding to that semantic segmentation category, and a second set is formed by pixels in the first temporary segmentation image whose pixel value is the semantic segmentation category identifier value corresponding to that semantic segmentation category. The similarity coefficient corresponding to that semantic segmentation category can be obtained by comparing twice the intersection of the first set and the second set with the total number of pixels in the first set and the second set. By traversing all semantic segmentation categories and performing statistical processing on the similarity coefficients corresponding to all semantic segmentation categories, the global segmentation similarity coefficient between the first intermediate segmentation image and the first temporary segmentation image can be obtained. As an example, the statistical processing can use the mean, maximum, median, minimum, etc.

[0037] Since both the first and second regions of interest are represented by sets, the same calculation method can be used to determine the region segmentation similarity coefficient between the first and second regions of interest.

[0038] In one specific implementation, the step of performing image registration on the third semantic segmentation image obtained by semantic segmentation of the first semantic segmentation image and the first intermediate registration image, and determining the second spatial transformation parameters, includes: The first semantic segmentation image is divided into M first segmentation sub-images according to the first preset size, and the third semantic segmentation image is divided into M second segmentation sub-images according to the first preset size, where M is a positive integer; The similarity metric of the m-th sub-image is calculated based on the second temporary segmentation image obtained by mapping the m-th first intermediate segmentation image and the m-th second intermediate segmentation image with the initialized second initial transformation parameters, where m is an integer in the range [1, M]. The second objective function is determined based on the similarity metrics of the M sub-images; The second preset search algorithm is used to adjust the second initial transformation parameters so as to minimize the second objective function. The adjusted second initial transformation parameters corresponding to the minimum second objective function are then used as the second spatial transformation parameters.

[0039] To avoid excessive computational cost for single image registration, the semantic segmentation image is divided into M sub-images during semantic fine registration. Image registration is performed on each sub-image, and the registration results of each sub-image are then stitched together to obtain the semantic fine registration result.

[0040] Specifically, the second preset search algorithm can adopt a multi-level relevance balance optimization algorithm.

[0041] In one specific implementation, the step of image registration of the initial flat scan image and the second intermediate registered image to determine the third spatial transformation parameters includes: The initial flat scan image is divided into M initial flat scan sub-images according to the first preset size, and the first temporary registration image obtained by mapping the second intermediate registration image with the initialized third initial transformation parameters is divided into M intermediate registration sub-images according to the first preset size, where M is a positive integer; Feature extraction is performed on the M initial flat scan sub-images to obtain the first image feature tensor corresponding to the initial flat scan image; Feature extraction is performed on the M intermediate registration sub-images to obtain the second image feature tensor corresponding to the first temporary registration image; Based on the first image feature tensor and the second image feature tensor, a feature similarity metric is determined, and based on the feature similarity metric, a third objective function is determined. The third preset search algorithm is used to adjust the third initial transformation parameters so as to minimize the third objective function. The adjusted third initial transformation parameters corresponding to the minimum third objective function are then used as the third spatial transformation parameters.

[0042] Among them, the third preset search algorithm can adopt a multi-level correlation balance optimization algorithm, and the feature similarity metric can be determined based on the minimum mean square error of the first image feature tensor and the second image feature tensor.

[0043] In one specific implementation, the step of extracting features from the M initial flat-scan sub-images to obtain the first image feature tensor corresponding to the initial flat-scan image includes: For any pixel in the m-th initial flat-scan sub-image, the neighborhood corresponding to the pixel is determined according to the second preset size; Based on the self-similar context and nearest neighbor distance mutual information of the neighborhood corresponding to the pixel, determine the local similarity evaluation vector corresponding to the pixel; Based on the local similarity evaluation vectors corresponding to each pixel in the m-th initial flat scan sub-image, the first local similarity feature tensor corresponding to the m-th initial flat scan sub-image is obtained. The first image feature tensor corresponding to the initial flat scan image is determined based on the first local similarity feature tensor corresponding to each initial flat scan sub-image.

[0044] The second preset size is used to determine the range of the local similarity evaluation of the corresponding pixel. For example, the second preset size is set to 7.

[0045] Specifically, for any given pixel, several pixel pairs are sampled in the neighborhood of that pixel. The gray-level difference is calculated for each pixel pair. The gray-level differences of each pixel pair form a self-similar context, which is in vector form. Based on the self-similar context corresponding to that pixel, the self-similar context of the other pixel in the image to which that pixel belongs is determined as the nearest neighbor context of that pixel. Mutual information is calculated based on the self-similar context corresponding to that pixel and the nearest neighbor context to obtain the nearest neighbor distance mutual information. The local similarity evaluation vector corresponding to that pixel is obtained by concatenating the self-similar context corresponding to that pixel and the nearest neighbor distance mutual information.

[0046] In one specific implementation, the step of extracting features from the M intermediate registration sub-images to obtain the second image feature tensor corresponding to the first temporary registration image includes: For any pixel in the m-th intermediate registration sub-image, the neighborhood corresponding to the pixel is determined according to the second preset size; Based on the self-similar context and nearest neighbor distance mutual information of the neighborhood corresponding to the pixel, determine the local similarity evaluation vector corresponding to the pixel; Based on the local similarity evaluation vectors corresponding to each pixel in the m-th intermediate registration sub-image, the second local similarity feature tensor corresponding to the m-th intermediate registration sub-image is obtained. The second image feature tensor corresponding to the first temporary registration image is determined based on the second local similarity feature tensor corresponding to each intermediate registration sub-image.

[0047] The fourth preset search algorithm used for image registration between the initial flat scan image and the third intermediate registration image can be an instance optimization algorithm based on Adam.

[0048] It should be noted that the above registration algorithms can all be directly applied to GPU chips, thus significantly improving the registration speed compared to traditional registration algorithms such as the deedsBCV algorithm.

[0049] In one specific embodiment, the image preprocessing method for medical image enhancement further includes: The initial flat scan image is thresholded to obtain a first threshold segmentation image, and the target enhancement image corresponding to the initial flat scan image is thresholded to obtain a second threshold segmentation image. The initial flat scan image is subjected to local variance calculation to obtain a first local variance image. The first local variance image is then thresholded with a preset variance threshold to obtain a third threshold segmentation image. The local variance of the target enhancement image corresponding to the initial flat scan image is calculated to obtain a second local variance image. The second local variance image is then thresholded using the preset variance threshold to obtain a fourth threshold segmentation image. The union of the first threshold segmented image and the third threshold segmented image is used as the first foreground segmented image, and the union of the second threshold segmented image and the fourth threshold segmented image is used as the second foreground segmented image. Determine the foreground similarity coefficient based on the first foreground segmentation image and the second foreground segmentation image; If the foreground similarity coefficient is greater than a preset foreground similarity coefficient threshold, then the initial flat scan image and the target enhancement image corresponding to the initial flat scan image are used as a sample pair, and the sample pair is used for training the image enhancement model.

[0050] The thresholding of the initial flat scan image and the target enhancement image can be performed using the Otsu thresholding method to obtain the first threshold segmentation image and the second threshold segmentation image.

[0051] Specifically, by eliminating images with large misalignments through foreground similarity coefficients, higher-quality initial flat scan images and target enhancement images with corresponding relationships are provided as sample pairs, providing more reliable training samples for the image enhancement model, thereby improving the training effect of the image enhancement model.

[0052] It should be noted that when performing image aggregation on the foreground segmentation image, for any three-dimensional pixel, the minimum pixel value of the corresponding two-dimensional pixel in the two-dimensional image under each preset axis is taken as the pixel value of the three-dimensional pixel.

[0053] In this first embodiment, image registration is performed sequentially through low-resolution semantic coarse registration, high-resolution semantic fine registration, local similarity feature fine registration, and multi-stage fusion fine-tuning. This ensures the accuracy and efficiency of image registration. Semantic-based registration ensures global semantic alignment, while image-based registration ensures the consistency of local feature details. Under the premise of ensuring clear image details, image registration aligns the flat scan image and the enhanced image, thereby improving the reliability of the enhanced image after registration, which in turn improves the effect of image preprocessing.

[0054] This second embodiment provides a deep learning-based medical image enhancement method, see [link to documentation]. Figure 2 The present invention provides a flowchart of a deep learning-based medical image enhancement method, which includes: S21, acquire a number of real two-dimensional flat scan images, wherein the real two-dimensional flat scan images correspond to preset axis labels; S22, For any real two-dimensional flat scan image, input the real two-dimensional flat scan image into the trained image generator corresponding to the preset axis label of the real two-dimensional flat scan image to obtain the prediction difference image corresponding to the real two-dimensional flat scan image, and determine the prediction enhancement image corresponding to the real two-dimensional flat scan image based on the real two-dimensional flat scan image and the prediction difference image corresponding to the real two-dimensional flat scan image. S23, perform image aggregation processing on the predicted enhanced images corresponding to all real two-dimensional flat scan images to obtain the initial three-dimensional enhanced image; S24, the initial 3D enhanced image is input into the trained 3D image generator for fine-tuning to obtain the predicted 3D enhanced image.

[0055] Among them, the real two-dimensional flat scan image refers to the flat scan image that needs to be enhanced by deep learning. The preset axis labels include the first preset axis label, the second preset axis label, and the third preset axis label. The first preset axis label corresponds to the axial position, the second preset axis label corresponds to the coronal position, and the third preset axis label corresponds to the sagittal position. The image generators corresponding to different preset axis labels are different, and the image generators corresponding to different preset axis labels need to be trained separately.

[0056] Image aggregation processing obtains an initial 3D enhanced image by aggregating multiple predicted enhanced images under different preset axes. For any 3D pixel, the pixel value of the 3D pixel is determined according to the pixel value of the corresponding 2D pixel in the 2D predicted enhanced images under each preset axis. For example, the average pixel value of the 3D pixel in the 2D predicted enhanced images under each preset axis can be used as the pixel value of the 3D pixel.

[0057] Specifically, the 3D image generator is trained on the basis of the trained image generator corresponding to each preset axis label. The training of the 3D image generator requires the addition of a 3D image discriminator. The 3D image generator can adopt a 3D Unet network structure, including two convolution modules, multiple downsampling modules, multiple upsampling modules and a first 3D output module. The output module uses a 1×1×1 3D convolution kernel for convolution processing.

[0058] The 3D image discriminator includes multiple 3D convolutional modules and a second 3D output module. The first 3D convolutional module receives the input 3D image, performs 3D convolution processing with a second 3D convolution stride, and then maps the result through an activation function, sending the mapping result to the next 3D convolutional module. Except for the first and last 3D convolutional modules, each of the other 3D convolutional modules includes a 3D convolutional layer, a 3D normalization layer, and an activation function. The last 3D convolutional module performs 3D convolution processing with a first 3D convolution stride to obtain the output of the 3D image discriminator. For example, the first 3D convolution stride is 1, the second 3D convolution stride is 2, and the activation function is implemented using LeakyReLU.

[0059] In one specific implementation, the real two-dimensional flat scan image also corresponds to a flat scan domain identifier; For any real two-dimensional flat scan image, the real two-dimensional flat scan image is input into a trained image generator corresponding to a preset axis label of the real two-dimensional flat scan image to obtain a predicted difference image corresponding to the real two-dimensional flat scan image. Based on the real two-dimensional flat scan image and the predicted difference image corresponding to the real two-dimensional flat scan image, a predicted enhancement image corresponding to the real two-dimensional flat scan image is determined, including: For any real two-dimensional flat scan image, the real two-dimensional flat scan image and the flat scan domain identifier corresponding to the real two-dimensional flat scan image are input into the trained image generator corresponding to the preset axis identifier of the real two-dimensional flat scan image to obtain the predicted difference image corresponding to the real two-dimensional flat scan image. The predicted difference image corresponding to the real two-dimensional flat scan image is added together to obtain the predicted enhanced image corresponding to the real two-dimensional flat scan image.

[0060] The input of the image generator introduces a domain identifier, which facilitates improved training performance by sharing the generator during subsequent image generator training.

[0061] In one specific implementation, the training process of the image generator includes: Obtain several sample pairs corresponding to the preset axis identifier of the image generator. The sample pairs include sample flat scan images and sample enhanced images that have a corresponding relationship. The sample flat scan images correspond to the flat scan domain identifier, and the sample enhanced images correspond to the enhancement domain identifier. The sample flat scan image and the flat scan region identifier are input into the image generator to obtain the first sample difference image; The sample plain scan image and the first sample difference image are added together to obtain a virtual enhanced image, wherein the virtual enhanced image corresponds to the enhancement domain identifier; The virtual enhanced image and the enhanced domain identifier are input into the preset discriminator corresponding to the image generator to obtain the first discrimination result; Based on the first discrimination result and the preset first category, a first discrimination sub-loss is determined; The enhanced image of the sample and the enhanced region identifier are input into the image generator to obtain the second sample difference image; Subtracting the enhanced sample image from the second sample difference image yields a virtual flat scan image, wherein the virtual flat scan image corresponds to the flat scan domain identifier; The virtual flat scan image and the flat scan domain identifier are input into the preset discriminator corresponding to the image generator to obtain the second discrimination result; Based on the second discrimination result and the preset first category, determine the second discrimination sub-loss; The discrimination loss is determined based on the first discriminant loss and the second discriminant loss; Based on the discrimination loss, a first training loss is determined, the weights of the discriminator corresponding to the image generator are frozen, and the image generator is trained based on the first training loss.

[0062] The sample plain scan image and the sample enhanced image can be obtained by the image preprocessing method provided in Example 1.

[0063] The first category is assumed to be the true category. Based on the first discrimination result, the first category, and the adversarial loss function, the first discriminant sub-loss is calculated. For example, the adversarial loss function adopts the vanilla GAN loss.

[0064] Similarly, the second discriminant sub-loss is calculated based on the second discrimination result, the preset first category, and the adversarial loss function.

[0065] Specifically, the sum of the first discriminant loss and the second discriminant loss is used as the discrimination loss.

[0066] During the training of the image generator and its corresponding discriminator, an adversarial learning training method is adopted, which alternately freezes the weights of the image generator or the discriminator, and trains the discriminator or the image generator according to the training loss.

[0067] Two image generators with shared weights are used during training to improve training effectiveness and efficiency.

[0068] In one embodiment, a sample difference image is determined based on the difference between the sample augmented image and the sample unscanned image. A first structural similarity loss and a first mean loss are calculated based on the first sample difference image and the sample difference image. As an example, the first structural similarity loss adopts the SSIM loss function, and the first mean loss adopts the GTLose loss function. The first structural similarity loss and the first mean loss are introduced into the first training loss.

[0069] In one embodiment, a sample difference image is determined based on the difference between the sample augmented image and the sample unscanned image. A second structural similarity loss and a second mean loss are calculated based on the second sample difference image and the sample difference image. The second structural similarity loss and the second mean loss are introduced into the first training loss.

[0070] In one embodiment, a first perceptual loss and a first precise feature distribution matching loss are calculated based on the virtual augmented image and the sample augmented image. For example, the first perceptual loss adopts the Perceptual Loss function, the first mean loss adopts the EFDM loss function, and the first perceptual loss and the first precise feature distribution matching loss are introduced into the first training loss.

[0071] In one embodiment, a second perceptual loss and a second precise feature distribution matching loss are calculated based on the virtual flat scan image and the sample flat scan image, and the second perceptual loss and the second precise feature distribution matching loss are introduced into the first training loss.

[0072] As an example, the Adam optimizer is used during training, with optimizer parameter β1 set to 0.5, optimizer parameter β2 set to 0.999, and the initial training learning rate set to 1×10⁻⁶. -3 The training rate is reduced by half every 50 training rounds, with two batches used for training and a training cycle of 400 rounds.

[0073] In one specific implementation, the training process of the discriminator corresponding to the image generator includes: Based on the first discrimination result and the preset second category, a third discrimination sub-loss is determined; The enhanced image of the sample and the enhanced domain identifier are input into the discriminator corresponding to the image generator to obtain a third discrimination result; Based on the third discrimination result and the preset first category, a fourth discrimination sub-loss is determined; Based on the second discrimination result and the preset second category, determine the fifth discrimination sub-loss; The sample flat scan image and the flat scan domain identifier are input into the discriminator corresponding to the image generator to obtain the fourth discrimination result; Based on the fourth discrimination result and the preset first category, the sixth discrimination sub-loss is determined; The discriminator training loss is determined based on the third discriminant loss, the fourth discriminant loss, the fifth discriminant loss, and the sixth discriminant loss; The weights of the image generator are frozen, and the discriminator corresponding to the image generator is trained based on the discriminator training loss.

[0074] The third, fourth, fifth, and sixth discriminant losses are all calculated using the adversarial loss function, and the sum of the third, fourth, fifth, and sixth discriminant losses is used as the discriminant training loss.

[0075] Specifically, during training, two discriminators with shared weights but different domain labels are used to improve training performance and efficiency. See [link to documentation]. Figure 3 This is the training architecture for the image generator and discriminator in a deep learning-based medical image enhancement method provided in Embodiment 2 of the present invention.

[0076] In one specific implementation, the image generator includes a first input layer, a first encoder layer, a backbone network layer, a first decoder layer, and a first output layer.

[0077] The first input layer performs convolution processing on the received input image and sends the convolution processing result to the first first encoder in the first encoder layer. The first encoder layer includes multiple first encoders. Each first encoder performs convolution processing on the received input. Except for the last first encoder, each first encoder sends the convolution processing result to the next first encoder. The last first encoder sends the convolution processing result to the backbone network layer. Each first encoder also sends the convolution processing result to the corresponding first decoder. The first decoder layer includes multiple first decoders. The number of first encoders is the same as the number of first decoders, and each first encoder has a unique corresponding first decoder.

[0078] The backbone network layer receives the output of the last first encoder and sends the processing result to the first first decoder. Each first decoder upsamples the received input. Except for the last first decoder, each first decoder sends the upsampled result to the next first decoder. The last first decoder sends the upsampled result to the first output layer. The first output layer convolves the received upsampled result using a 1×1 kernel and inputs the convolution result into an activation function to obtain the output of the image generator. As an example, the backbone network layer adopts the ViT network architecture, and the activation function in the first output layer adopts the tanh activation function.

[0079] In one specific implementation, the image generator further includes a second output layer; Accordingly, the step of inputting the sample flat scan image and the flat scan region identifier into the image generator to obtain the first sample difference image includes: The sample flat scan image and the flat scan region identifier are input into the image generator to obtain the first sample difference image output by the first output layer and the predicted region of interest image output by the second output layer. Based on the predicted region of interest image, the first sample difference image, the sample enhancement image, and the first sample region of interest image corresponding to the sample flat scan image, determine the region of interest loss; Accordingly, determining the discrimination loss based on the first discriminant sub-loss and the second discriminant sub-loss includes: The discrimination loss is determined based on the first discriminant loss, the second discriminant loss, and the region of interest loss.

[0080] The second output layer also includes convolution processing with 1×1 size convolution kernels and activation functions. For example, the activation function in the second output layer is the Sigmoid activation function.

[0081] Specifically, the first sample region of interest image can be obtained by extracting the sample flat scan image using the region of interest extraction method provided in Example 1. Based on the predicted region of interest image and the first sample region of interest image, the first region of interest sub-loss is determined. After adding the sample flat scan image and the first sample difference image, the second region of interest sub-loss is calculated in the region of interest determined by the first sample region of interest image and the sample enhancement image. Based on the first region of interest sub-loss and the second region of interest sub-loss, the region of interest loss is determined. As an example, the region of interest sub-loss can be calculated using the mean loss function, and the mean loss function adopts the GTLLoss loss function.

[0082] In one embodiment, based on the first sample region of interest image, the brightness is enhanced within the region of interest corresponding to the first sample region of interest image in the sample flat scan image with a preset probability, so as to enhance the model's learning ability for the enhanced region. For example, the brightness enhancement is randomly increased by 50-100 HU value.

[0083] In one embodiment, based on the foreground image corresponding to the sample flat scan image and the foreground image corresponding to the sample augmented image, foreground image augmentation is performed on the sample augmented image, replacing the background region in the sample augmented image with the background region in the sample flat scan image. This constrains the model's learning of the foreground region and reduces the impact of image misalignment on training performance. It should be noted that, based on the foreground image corresponding to the sample flat scan image and the foreground image corresponding to the sample augmented image, the two foreground images are divided into several flat scan foreground image blocks and several augmented foreground image blocks according to a preset block size. The number of flat scan foreground image blocks and augmented foreground image blocks is the same, and each flat scan foreground image block has a unique corresponding augmented foreground image block. For any augmented foreground image block, if the number of pixels belonging to foreground pixels in the augmented foreground image block is greater than a preset pixel count threshold, and the similarity between the augmented foreground image block and the corresponding flat scan foreground image block is greater than a preset similarity threshold, then the augmented foreground image block is taken as the foreground region of the sample augmented image, and the background region of the sample augmented image is determined based on the foreground region of the sample augmented image.

[0084] In one specific implementation, the training process of the image generator further includes: The virtual flat scan image and the flat scan domain identifier are input into the image generator to obtain the third sample difference image; The virtual flat scan image and the third sample difference image are added together to obtain a virtual enhanced loop image; The first recurrent sub-loss is determined based on the sample augmented image and the virtual augmented recurrent image; The virtual enhanced image and the enhanced domain identifier are input into the image generator to obtain the fourth sample difference image; Subtract the virtual enhanced image from the fourth sample difference image to obtain the virtual flat scan loop image; The second cyclic sub-loss is determined based on the sample flat scan image and the virtual flat scan cyclic image; The cyclic loss is determined based on the first cyclic sub-loss and the second cyclic sub-loss; Accordingly, determining the discrimination loss based on the first discriminant loss, the second discriminant loss, and the region of interest loss includes: The discrimination loss is determined based on the first discriminant loss, the second discriminant loss, the region of interest loss, and the cyclic loss.

[0085] In this embodiment, the training architecture is used to enhance the training effect through cyclic verification.

[0086] As an example, the absolute value of the difference between the sample augmented image and the virtual augmented loop image is used as the first loop sub-loss, the absolute value of the difference between the sample flat scan image and the virtual flat scan loop image is used as the second loop sub-loss, and the sum of the first loop sub-loss and the second loop sub-loss is used as the loop loss.

[0087] In one specific implementation, the discriminator includes a second input layer, a feature extraction layer, and a third output layer, wherein the feature extraction layer includes several residual modules, and the third output layer outputs the discrimination results corresponding to the flat scan domain identifier and the enhanced domain identifier, respectively.

[0088] The second input layer performs convolution processing on the received input and sends the convolution processing result to the first residual module. Except for the last residual module, each residual module sends its output to the next residual module. The last residual module sends its output to the third output layer. The third output layer includes a convolution layer and a dimension adjustment layer. For example, the dimension adjustment layer includes Flatten and View operations.

[0089] It should be noted that the training methods for the 3D image generator and the 3D image discriminator are similar to those for the 2D image generator and its corresponding discriminator, and will not be elaborated further here.

[0090] In this second embodiment, an image generator performs image enhancement processing on each real two-dimensional flat scan image. The resulting multiple two-dimensional predicted enhanced images are aggregated to obtain an initial three-dimensional enhanced image. Image enhancement is performed using two-dimensional images, which ensures the reliability of the image enhancement process. The image aggregation method provides three-dimensional structural information, which is then fine-tuned by the three-dimensional image generator. This avoids the poor model effect caused by the three-dimensional image generator directly performing three-dimensional image enhancement, thereby further improving the reliability of three-dimensional image enhancement.

[0091] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of this invention is defined by the appended claims.

Claims

1. An image preprocessing method for medical image enhancement, characterized in that, The image preprocessing method for medical image enhancement includes: S11, the first semantic segmentation image corresponding to the initial flat scan image and the second semantic segmentation image corresponding to the initial enhanced image are downsampled respectively to obtain the first intermediate segmentation image corresponding to the initial flat scan image and the second intermediate segmentation image corresponding to the initial enhanced image. S12, perform image registration on the first intermediate segmented image and the second intermediate segmented image, determine the first spatial transformation parameters, and map the initial enhanced image to the first intermediate registered image according to the first spatial transformation parameters; S13, perform image registration on the first semantic segmentation image and the third semantic segmentation image obtained by semantic segmentation of the first semantic segmentation image and the first intermediate registration image, determine the second spatial transformation parameters, and map the first intermediate registration image to the second intermediate registration image according to the second spatial transformation parameters; S14, perform image registration on the initial flat scan image and the second intermediate registration image, determine the third spatial transformation parameters, and map the second intermediate registration image to the third intermediate registration image according to the third spatial transformation parameters; S15, perform image registration on the initial flat scan image and the third intermediate registration image, determine the fourth spatial transformation parameters, and map the third intermediate registration image to the target enhancement image corresponding to the initial flat scan image according to the fourth spatial transformation parameters.

2. The image preprocessing method for medical image enhancement according to claim 1, characterized in that, The image preprocessing method for medical image enhancement further includes: According to a preset first scale, local relative contrast is calculated for the initial flat scan image and the initial enhanced image respectively to obtain a first local contrast image corresponding to the initial flat scan image and a second local contrast image corresponding to the initial enhanced image; Based on the preset first contrast difference threshold, first contrast difference normalization threshold and first contrast normalization threshold, a first region of interest image is extracted from the first local contrast image, and a second region of interest image is extracted from the second local contrast image. Accordingly, the step of image registration of the first intermediate segmented image and the second intermediate segmented image to determine the first spatial transformation parameters includes: A global segmentation similarity metric is calculated based on a first temporary segmentation image obtained by mapping the first intermediate segmentation image and the second intermediate segmentation image with the initialized first initial transformation parameters. Based on the downsampling result of the first region of interest image, the first region of interest is extracted from the first intermediate segmented image; based on the downsampling result of the second region of interest image, the second region of interest is extracted from the first temporary segmented image. Based on the first region of interest and the second region of interest, a region segmentation similarity metric is calculated. The first spatial transformation parameters are determined based on the global segmentation similarity metric and the region segmentation similarity metric.

3. The image preprocessing method for medical image enhancement according to claim 2, characterized in that, Determining the first spatial transformation parameters based on the global segmentation similarity metric and the region segmentation similarity metric includes: The first objective function is determined based on the global segmentation similarity metric and the region segmentation similarity metric. The first initial transformation parameters are adjusted using a first preset search algorithm to minimize the first objective function, and the adjusted first initial transformation parameters corresponding to the minimized first objective function are used as the first spatial transformation parameters.

4. The image preprocessing method for medical image enhancement according to claim 1, characterized in that, The step of performing image registration on the third semantic segmented image obtained by semantic segmentation of the first semantic segmented image and the first intermediate registered image to determine the second spatial transformation parameters includes: The first semantic segmentation image is divided into M first segmentation sub-images according to the first preset size, and the third semantic segmentation image is divided into M second segmentation sub-images according to the first preset size, where M is a positive integer; The similarity metric of the m-th sub-image is calculated based on the second temporary segmentation image obtained by mapping the m-th first intermediate segmentation image and the m-th second intermediate segmentation image with the initialized second initial transformation parameters, where m is an integer in the range [1, M]. The second objective function is determined based on the similarity metrics of the M sub-images; The second preset search algorithm is used to adjust the second initial transformation parameters so as to minimize the second objective function. The adjusted second initial transformation parameters corresponding to the minimum second objective function are then used as the second spatial transformation parameters.

5. The image preprocessing method for medical image enhancement according to claim 1, characterized in that, The step of image registration of the initial flat scan image and the second intermediate registered image to determine the third spatial transformation parameters includes: The initial flat scan image is divided into M initial flat scan sub-images according to the first preset size, and the first temporary registration image obtained by mapping the second intermediate registration image with the initialized third initial transformation parameters is divided into M intermediate registration sub-images according to the first preset size, where M is a positive integer; Feature extraction is performed on the M initial flat scan sub-images to obtain the first image feature tensor corresponding to the initial flat scan image; Feature extraction is performed on the M intermediate registration sub-images to obtain the second image feature tensor corresponding to the first temporary registration image; Based on the first image feature tensor and the second image feature tensor, a feature similarity metric is determined, and based on the feature similarity metric, a third objective function is determined. The third preset search algorithm is used to adjust the third initial transformation parameters so as to minimize the third objective function. The adjusted third initial transformation parameters corresponding to the minimum third objective function are then used as the third spatial transformation parameters.

6. The image preprocessing method for medical image enhancement according to claim 5, wherein the step of extracting features from the M initial plain scan sub-images to obtain the first image feature tensor corresponding to the initial plain scan image includes: For any pixel in the m-th initial flat-scan sub-image, the neighborhood corresponding to the pixel is determined according to the second preset size; Based on the self-similar context and nearest neighbor distance mutual information of the neighborhood corresponding to the pixel, determine the local similarity evaluation vector corresponding to the pixel; Based on the local similarity evaluation vectors corresponding to each pixel in the m-th initial flat scan sub-image, the first local similarity feature tensor corresponding to the m-th initial flat scan sub-image is obtained. The first image feature tensor corresponding to the initial flat scan image is determined based on the first local similarity feature tensor corresponding to each initial flat scan sub-image.

7. The image preprocessing method for medical image enhancement according to claim 5, characterized in that, The step of extracting features from the M intermediate registration sub-images to obtain the second image feature tensor corresponding to the first temporary registration image includes: For any pixel in the m-th intermediate registration sub-image, the neighborhood corresponding to the pixel is determined according to the second preset size; Based on the self-similar context and nearest neighbor distance mutual information of the neighborhood corresponding to the pixel, determine the local similarity evaluation vector corresponding to the pixel; Based on the local similarity evaluation vectors corresponding to each pixel in the m-th intermediate registration sub-image, the second local similarity feature tensor corresponding to the m-th intermediate registration sub-image is obtained. The second image feature tensor corresponding to the first temporary registration image is determined based on the second local similarity feature tensor corresponding to each intermediate registration sub-image.

8. The image preprocessing method for medical image enhancement according to claim 1, characterized in that, The image preprocessing method for medical image enhancement further includes: The initial flat scan image is thresholded to obtain a first threshold segmentation image, and the target enhancement image corresponding to the initial flat scan image is thresholded to obtain a second threshold segmentation image. The initial flat scan image is subjected to local variance calculation to obtain a first local variance image. The first local variance image is then thresholded with a preset variance threshold to obtain a third threshold segmentation image. The local variance of the target enhancement image corresponding to the initial flat scan image is calculated to obtain a second local variance image. The second local variance image is then thresholded using the preset variance threshold to obtain a fourth threshold segmentation image. The union of the first threshold segmented image and the third threshold segmented image is used as the first foreground segmented image, and the union of the second threshold segmented image and the fourth threshold segmented image is used as the second foreground segmented image. Determine the foreground similarity coefficient based on the first foreground segmentation image and the second foreground segmentation image; If the foreground similarity coefficient is greater than a preset foreground similarity coefficient threshold, then the initial flat scan image and the target enhancement image corresponding to the initial flat scan image are used as a sample pair, and the sample pair is used for training the image enhancement model.

Images