A long-term medical image interpolation method based on hausdorff decomposition and flow model

Abstract

The application discloses a long-term medical image interpolation method based on Householder decomposition and a flow model and relates to the field of medical image repair. First, the long-term medical images of different periods are registered by using a regional differential homeomorphism measurement mapping. Second, the medical images are subjected to feature extraction by a convolutional encoder, the image pixel value input is extracted, and then missing image features are predicted by using Householder decomposition and a flow-based model. Finally, the features are reconstructed by using a decoder to obtain the missing images. The application can not only effectively interpolate the missing medical images, but also effectively restore the detail information of the lesions in the original images, and reduces the prediction deviation caused by data loss, thereby improving the efficiency of auxiliary diagnosis from an objective aspect.

Description

Technical Field

[0001] This invention relates to the field of medical image restoration, specifically a long-term medical image interpolation method based on Haushold decomposition and flow model. Background Technology

[0002] For early-stage suspected lung tumors, regular follow-up and appropriate treatment plans can significantly improve patient survival rates and reduce the risk of developing cancer. However, obtaining complete longitudinal data is often ideal. In clinical practice, due to subjective factors such as patient discomfort (recovery, migration, physical discomfort, etc.), patients may be unable to attend follow-up appointments at the scheduled time; some may even abandon subsequent observation, resulting in missing longitudinal imaging data. Missing data undermines the integrity of longitudinal studies, leading to varying degrees of bias in the results, reducing accuracy and precision, and affecting the overall quality of the longitudinal study.

[0003] Therefore, a method is needed to supplement missing image data, thereby improving the accuracy and precision of the results. Summary of the Invention

[0004] To address the problem of missing longitudinal imaging data in the treatment of lung tumors, which can disrupt the integrity and quality of longitudinal studies, this invention provides a long-term medical image interpolation method based on Haushold decomposition and flow models.

[0005] This invention is achieved through the following technical solution: a long-term medical image interpolation method based on Haushold decomposition and flow model, comprising the following steps:

[0006] S1: Acquire images of the same patient at different pathological stages. Because images deform as the pathology progresses, the long-term medical images at multiple stages are first mapped using regional differential homeomorphism metric mapping I = {i0, i1, i2, ..., i n Registration is performed to obtain images of the same anatomical point. For the reference image i0 and the floating image i1, in order to make i0 and i1 similar and to achieve spatial and temporal continuity and smoothness of the entire transformation, the region differential homeomorphism metric mapping uses the continuously differentiable time-dependent velocity field v(t) of the ordinary differential equation to construct the differential homeomorphism transformation group, and then generates the velocity space V through the Hilbert space. The optimal solution of energy E in the velocity space V is solved by optimization methods including the variational method, and finally the registration between the reference image i0 and the floating image i1 is achieved. The registered image is used as the image library for the next step of the operation to prepare for the interpolation of missing images. The registration process of the region differential homeomorphism metric mapping is as follows:

[0007] S1-1: Let the transformation region Ω be a bounded open set, and v(t) be a time-dependent velocity field, where V is the Hilbert space formed by continuously differentiable vector fields on Ω; when constructing a differential homeomorphism using the velocity vector field, by the principle of identity transformation, the velocity vector field v(t) satisfies the ordinary differential equation: y is the transformed coordinate;

[0008] S1-2: When an element of a differential homeomorphic transformation group is applied to image i1, a deformed object satisfying the differential homeomorphic non-rigid transformation is obtained; at this time, the matching problem based on the differential homeomorphic transformation is to solve the corresponding energy functional minimization problem, and the calculation formula is as follows:

[0009]

[0010] In formula (1), the first term on the right-hand side of the equals sign is the regularization term, and the second term is the similarity, where the velocity field v(t) determines how image i1 flows to i0; and the goal of the region differential homeomorphism metric mapping is to find v * To minimize the objective function above; by minimizing the objective function, the floating image is transformed into the registered image i1+Δi;

[0011] S2: A feature map encoder is used to extract features from the registered image. The feature map encoder mainly consists of two convolutional layers and two pooling layers. The convolutional layer is composed of several convolutional units, and the parameters of each convolutional unit are optimized through backpropagation. It is mainly used to extract local features of the original CT image. The first convolutional layer uses 32 3x3 convolutional kernels, the activation function is ReLU, the stride is 2, and the padding method is same. The second convolutional layer uses 64 3x3 convolutional kernels, the activation function is ReLU, the stride is 2, and the padding method is same. The pooling layer is used to reduce the feature dimension, compress the size of the data and parameters, reduce overfitting, and improve the fault tolerance of the model. The pooling layer uses a 2x2 max pooling kernel. Then, the image feature map extracted by the encoder is input into the streaming model to predict the missing image. After multiple convolutions, the size of the resulting feature map will become smaller. This process can be regarded as the compression and abstraction of image information, similar to converting the image into some higher-level feature representation.

[0012] S3: A flow model is used to predict missing image features π(z) using the extracted image features p(x). To improve the convergence and computational speed of the model, Haushold decomposition is introduced into the original flow model. The specific calculation process is as follows:

[0013] S3-1 Affine coupling; constructing a flexible and easily solvable transformation function by stacking simple bijections; in each single-step bijection, using an easily invertible function to update the input vector; therefore, using affine coupling to update the input vector; given a D-dimensional input x and a partition interval d<D, the calculation formula for the output z of the corresponding affine coupling layer is as follows:

[0014] z 1:d =h(x 1:d ) (2)

[0015]

[0016] f(·) and k(·) denote the scaling transformation and shift transformation respectively, ⊙ denotes the dot product between vectors, denotes element-wise addition, and H(·) denotes Householder decomposition;

[0017] S3-2: Layer normalization; for samples of different lengths, layer normalization is used to regularize the model; layer normalization normalizes the scale and bias of all neuron nodes of the samples in each layer, instead of normalizing by channels; let H be the number of hidden layer nodes in one layer, and l be the number of layers of the multilayer perceptron, the scaling parameter μ and bias parameter σ for calculating layer normalization are:

[0018]

[0019]

[0020] It is reflected by formulas (4) and (5) that the calculation of the statistics is independent of the number of samples and only depends on the number of hidden layer nodes; therefore, as long as the number of hidden layer nodes is sufficiently large, the statistics of layer normalization can be guaranteed;

[0021]

[0022] In formula (6), is the normalized feature of each layer, a l is the original feature of each layer, μ l is the scaling parameter of each layer, σ l is the bias parameter of each layer; after normalization, the scaling parameter and the bias parameter are regarded as conventional trainable parameters independent of the data, and ε is set to prevent the denominator from being zero;

[0023] S3-3: Householder decomposition; given a matrix, the transformed hyperplane is obtained through the Householder vector orthogonal to the hyperplane; the reflection hyperplane is defined as:

[0024]

[0025] H is the Householder matrix, H tA key characteristic of H is that it is an orthogonal matrix, therefore H t The Jacobian determinant is 1;

[0026] S4: Use an image reconstruction decoder to decode the predicted features:

[0027] The features of the missing image obtained through the streaming model are input into the decoder. The decoder's role is to reconstruct the feature mapping from the original image to the original image. The decoder consists of two transposed convolutional layers and an upsampling layer. The first transposed convolutional layer uses 64 3x3 convolutional kernels with a stride of 2x2 and a padding pattern of 2. The second transposed convolutional layer uses 32 3x3 convolutional kernels with a stride of 2x2 and a padding pattern of 2. An upsampling layer follows the transposed convolutional layers. By magnifying the image through a series of transposed convolutional layers, more detailed information in the original image is gradually recovered. This process can be seen as restoring abstract features to the original image.

[0028] Furthermore, in steps S2, S3, and S4, if the reconstruction error between the reconstructed image and the original input image is sufficiently small, the reconstruction capability of the autoencoder is deemed sufficient to obtain the feature map of the CT image. The autoencoder is trained simultaneously with the feature map encoder and the image reconstruction decoder. After obtaining features that best represent the original CT image, the reconstructed image is separated from the original input image, and a streaming model is added to predict the missing image. To obtain a reconstructed image with the same dimensions as the original image, the feature map encoder structure also includes convolutional layers, and pooling layers are replaced with upsampling layers to restore the image size and retain information from the original image while maintaining a close resemblance. The decoder uses the same hyperparameters as the encoder.

[0029] Compared with existing technologies, this invention has the following advantages: The long-term medical image interpolation method based on Haushold decomposition and a flow model provided by this invention uses an improved flow model to interpolate longitudinal image data. The effectiveness and accuracy of the algorithm are verified by comparing the results of interpolation of other missing images. Experimental results demonstrate that this invention can reconstruct good results without a cumbersome process, and the predicted impact is also beneficial for physicians' auxiliary diagnosis. This invention can effectively extract and reconstruct features from images, and effectively restore the detailed information of lesions in the original images, providing decision support for doctors' longitudinal research on lung cancer and playing an objective role in auxiliary diagnosis. Different medical image interpolation results are shown in Table 1.

[0030] Table 1. Prediction results of different medical image interpolation models

[0031] Attached Figure Description

[0032] Figure 1 This is a flowchart of the long-term medical image interpolation method of the present invention: the overall process can be divided into four steps: image registration at different time periods, image feature encoding, prediction, and missing image decoding.

[0033] Figure 2 The process of generating missing images in this invention is as follows: that is, using the posterior distribution of the image to generate the sample distribution of the missing image.

[0034] Figure 3 It's an affine-coupled architecture: the sample is divided into two parts z. 1:d and z d+1:D The first part z 1:d Copy the output directly, the latter part z d+1:D Output is calculated using affine calculations.

[0035] Figure 4 This is the prediction result of a missing sample in the test set of a specific embodiment of the present invention: the first row shows the images of the sample collected at different times; the second row shows the missing image predicted using the method of the present invention; the third row shows the magnified result of the lesion area in the image.

[0036] Figure 5 This is the visualization result of the prediction results on the lesion site in this invention: the first row is the medical images and original images obtained by different methods, and the second row is the magnified result of the lesion area in the image.

[0037] Figure 6 This is the grayscale histogram of the predicted image in this invention: the grayscale histograms of the prediction results of different methods reflect the frequency of a certain grayscale value in the image. Detailed Implementation

[0038] The present invention will be further described below with reference to specific embodiments.

[0039] This implementation example Figure 1 and Figure 2 As shown, a long-term medical image interpolation method based on Haushold decomposition and flow model includes the following steps:

[0040] S1: Due to the varying acquisition times of long-term medical images, the images to be predicted are first preprocessed. The main purpose is to ensure that all anatomical points on the long-term images are matched. The original long-term medical images are then registered using regional differential homeomorphic metric mapping.

[0041] S2: A convolutional encoder is used to reconstruct long-term medical images. The registered images are used as input to the autoencoder to obtain a two-dimensional feature map. The encoder mainly consists of two convolutional layers and two pooling layers. The convolutional layers are used to extract key features from the original images, while the pooling layers improve the model's generalization ability while maintaining image feature invariance. During convolution, a classic 3×3 kernel is selected. The first convolutional layer contains 32 kernels with ReLU activation function, stride of 2, and same padding. The second convolutional layer contains 64 kernels with ReLU activation function, stride of 2, and same padding.

[0042] S3: Predicting long-term medical images using Haushold decomposition and a flow model. This model is primarily based on a flow structure, with each flow operation consisting of affine coupling, layer regularization, and Haushold decomposition. Affine coupling uses stacked simple bijections to establish a flexible and easily solvable transformation function. In each single-step bijection, an easily invertible function is used to update the input vector, such as... Figure 3 As shown. Layer regularization, by normalizing the features of the same layer across different samples, can make training more stable and also acts as a regularization mechanism to prevent network overfitting. Haushold decomposition can accelerate the computation of flow models.

[0043] S4: The decoder reconstructs the predicted features. To obtain a reconstructed image with the same dimensions as the original image, the decoder uses transposed convolutional layers with the same parameters as the encoder. The first transposed convolutional layer contains 64 kernels, with ReLU activation, a stride of 2, and same padding. The second transposed convolutional layer contains 32 kernels, with ReLU activation, a stride of 2, and same padding. Pooling layers are replaced with upsampling layers to restore the image size and preserve as much of the original information as possible. The results are as follows... Figure 4 and 5 As shown.

[0044] Figure 6 This demonstrates that the predicted image is structurally similar to the original image. A grayscale histogram is a statistical representation of the frequency of occurrence of all pixel values ​​in a medical image, based on their grayscale values. It indicates the number of pixels with a specific grayscale level in the image and reflects the frequency of that grayscale value.

[0045] The scope of protection claimed by this invention is not limited to the specific embodiments described above. Moreover, for those skilled in the art, this invention can have various modifications and alterations. Any modifications, improvements, and equivalent substitutions made within the concept and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A long-term medical image interpolation method based on Haushold decomposition and flow model, characterized in that: comprising the following steps: S1: Acquire images of the same patient at different pathological stages. Because images deform as the pathology progresses, regional differential homeomorphic metric mapping is first used to map long-term medical images from multiple stages. Registration is performed to obtain images of the same anatomical point; for the reference image With floating images In order to make and Similarity and the entire transformation can achieve spatial and temporal continuity and smoothness. The domain differential homeomorphism metric mapping utilizes the continuously differentiable time-dependent velocity field of ordinary differential equations. We construct a differential homeomorphism group, then generate the velocity space V through Hilbert space, and finally solve for the optimal solution of energy E in velocity space V using optimization methods including variational methods, ultimately achieving the reference image. With floating images The registration process between regions using the differential homeomorphic metric mapping is as follows: The registered images are then used as the image library for the next step, preparing for the interpolation of missing images. S1-1: Define the transformation region For a bounded open set, and Let V be the time-dependent velocity field, where V is The Hilbert space is formed by continuously differentiable vector fields. When constructing a differential homeomorphism using velocity vector fields, the principle of identity transformations indicates that the velocity vector field... Satisfies the ordinary differential equation: y is the transformed coordinate; S1-2: When an element of a differential homeomorphic transformation group acts on an image This yields the deformable object that satisfies the differential homeomorphic non-rigid transformation. At this point, the matching problem based on the differential homeomorphic transformation becomes solving the corresponding energy functional minimization problem, with the calculation formula as follows: （1） In formula (1), the first term on the right side of the equals sign is the regularization term, and the second term is the similarity, where the velocity field... The image was determined How to flow ; The goal of the regional differential homeomorphism metric mapping is to find To minimize the objective function above; by minimizing the objective function, the floating image is transformed into a registered image. ; S2: performing feature extraction on the registered image by using a feature mapping encoder, where the feature mapping encoder mainly includes two convolutional layers and two pooling layers; the convolutional layer is composed of a plurality of convolutional units, parameters of each convolutional unit are obtained through back propagation optimization, and the convolutional layer is mainly used for extracting local features of an original CT image; the first convolutional layer uses 32 3x3 convolution kernels, the activation function is ReLU, the stride is 2, and the padding mode is same; the second convolutional layer uses 64 3x3 convolution kernels, the activation function is ReLU, the stride is 2, and the padding mode is same; the pooling layer is configured to reduce feature dimensions, compress the scale of data and parameters, reduce overfitting, and improve the fault tolerance of the model at the same time; the pooling layer uses a 2x2 max pooling kernel; then the image feature map extracted by the encoder is input into a flow model to predict the missing image; after multiple convolutions, the size of the obtained feature map will become smaller; S3: Employing a flow model to utilize extracted image features To predict missing image features To improve the convergence and computational speed of the model, Haushold decomposition was introduced into the original flow model; the specific calculation process is as follows: S3-1 affine coupling: constructing a flexible and easy-to-solve transformation function by stacking simple bijections; in each single-step bijection, using an easily invertible function to update the input vector; therefore, using affine coupling to update the input vector; given a D-dimensional input x and a division interval d<D, the calculation formula for the output z of the corresponding affine coupling layer is as follows: （2） （3） and These represent scaling transformation and offset transformation, respectively. Represents the Hadamard product between vectors. This represents element-wise addition. This indicates the decomposition of Haushold; S3-2: Layer Normalization; For samples of different lengths, layer normalization is used to regularize the model; layer normalization normalizes the scale and bias of all neuron nodes in each layer of samples, rather than normalizing channels; Let H be the number of nodes in a hidden layer and l be the number of layers in the multilayer perceptron, and calculate the scaling parameter of layer normalization. and bias parameters : （4） （5） It is reflected by formulas (4) and (5) that the calculation of the statistic is independent of the number of samples, and only depends on the number of hidden layer nodes; （6） In formula (6), For the features of each layer after normalization, For the original features of each layer, For the scaling parameters of each layer, The bias parameters for each layer; After normalization, the scaling and bias parameters are treated as regular trainable parameters independent of the data. This is to prevent the denominator from being zero; S3-3: Householder decomposition; given a matrix, the transformed hyperplane is obtained through a Householder vector orthogonal to the hyperplane; the reflection hyperplane is defined as: （7） It is the Haushold matrix. The key characteristic is orthogonality, and the corresponding Jacobian determinant is 1; S4: decoding the predicted features by using an image reconstruction decoder: The features of the missing image obtained by the flow model will be input into the decoder; the function of the decoder is to reconstruct the image to the feature map of the original image; the decoder part includes two transposed convolutional layers and an upsampling layer; the first transposed convolutional layer uses 64 3x3 convolution kernels, with a stride of 2x2, a stride of 2, and a padding mode of same; the second transposed convolutional layer uses 32 3x3 convolution kernels, with a stride of 2x2, a stride of 2, and a padding mode of same; an upsampling layer is also arranged after the transposed convolutional layer; the image is amplified through a series of transposed convolutional layers, and more detailed information in the original image is gradually restored, so that the abstract features are restored to the original image.

Images