A medical in-situ hyperspectral image reconstruction method based on spectral manifold constraint rectilinear flow

Abstract

This invention discloses a medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified flow, comprising the following steps: Step S1: Preprocessing data and constructing a model; Step S2: Learning the spectral manifold a priori based on the constructed model; Step S3: Initializing the observation-induced source distribution based on the prior learned spectral manifold flow; Step S4: Training the manifold-constrained flow matching based on the prior learned spectral manifold flow and the initialized observation-induced source distribution; Step S5: In the inference stage, reconstructing the ordinary differential equation based on the trained manifold-constrained flow matching. This invention solves the limitations of deterministic regression, geometric mismatch of the generative model, and low computational efficiency of existing technologies.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology, and in particular to a method for in-situ hyperspectral image reconstruction in medicine based on spectral manifold-constrained rectified flow. Background Technology

[0002] Medical hyperspectral imaging (MHSI) can acquire spectral information of biological tissues across multiple consecutive narrow bands, which is of great significance for cancer edge detection (such as cholangiocarcinoma and glioma). To achieve real-time imaging, a spectral filter array (MSFA) snapshot camera is often used. However, MSFA imaging leads to a severe loss of spatial resolution, necessitating the reconstruction of a full-resolution hyperspectral data frame from sparse observations using a "demosaic" algorithm.

[0003] Existing technologies mainly include interpolation-based methods, regression methods based on convolutional neural networks (CNNs), and deep learning methods based on Transformers. While these methods improve reconstruction quality to some extent, they still suffer from the following problems: Limitations of deterministic regression: Most existing deep learning methods are deterministic regression models, tending to produce smooth spectral curves and losing high-frequency texture and microstructural information crucial for pathological diagnosis. Geometric mismatch of generative models: Conventional generative models (such as the standard diffusion model) assume that the data generation process begins with informationless Gaussian noise and is transmitted in Euclidean space. For medical in-situ hyperspectral data strictly constrained by physical laws, this assumption leads to transmission paths traversing non-physical regions, producing "spectral hallucinations," i.e., generating spectral features that do not exist biologically. Low computational efficiency: Standard diffusion model inference requires hundreds of iterations, making it difficult to meet the real-time requirements of clinical medicine. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified flow, which solves the limitations of deterministic regression, geometric mismatch of the generative model, and low computational efficiency in the prior art.

[0005] The specific technical solution of the present invention is as follows:

[0006] A medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified flow includes the following steps:

[0007] Step S1: Preprocessing data and building the model;

[0008] Step S2: Based on the constructed model, learn the spectral manifold prior.

[0009] Step S3: Initialize the observation-induced source distribution based on prior learned spectral flow;

[0010] Step S4: Train the flow matching of manifold constraints based on prior learning of spectral flow and initialization of observation-induced source distribution;

[0011] Step S5: In the inference phase, based on the flow matching of the training manifold constraints, the ordinary differential equations are reconstructed through inference.

[0012] Preferably, the model construction in step S1 is specifically as follows:

[0013] We construct a model architecture that includes a spectral variational autoencoder Spectral VAE and a flow matching backbone network UNet.

[0014] Preferably, step S2 specifically includes:

[0015] Spectral VAE is pre-trained using a full-resolution hyperspectral medical dataset. Low-dimensional latent representations of biological tissue spectra are learned through 1×1 convolutional layers to obtain the tangent space features of the data manifold.

[0016] Preferably, step S3 specifically involves: constructing a deterministic initial hyperspectral cube x0 based on the input MSFA original mosaic image through a physically consistent mapping, serving as the starting point for probability stream transmission.

[0017] Preferably, step S4 includes the following steps:

[0018] Step S41: During the training phase, construct the transmission path from x0 to the real hyperspectral image x1;

[0019] Step S42: Use the pre-trained VAE from step S2 to generate random noise in the tangent space of the spectral manifold, inject the noise into the transmission path, and train the backbone network using the Fourier domain FFT loss function so that it can learn to recover the velocity field of high-frequency details.

[0020] Preferably, step S5 specifically includes:

[0021] During the inference phase, the MSFA image to be reconstructed is input, an initialization x0 is generated, the ODE is solved using the trained backbone network, and a high-fidelity reconstructed image is obtained through a small number of iterations.

[0022] The beneficial effects of the medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified flow of the present invention are as follows:

[0023] This invention can effectively restore the high-frequency texture and microstructure information required for pathological diagnosis. The established geometric model has high accuracy and high computational efficiency. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Referring to the accompanying drawings will provide a clearer understanding of the features and advantages of the present invention. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. Wherein:

[0025] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0026] Figure 2 This is a schematic diagram illustrating the principle of observation-induced initialization in this invention.

[0027] Figure 3 This is a schematic diagram of the structure for generating spectral manifold-constrained noise in this invention.

[0028] Figure 4 This is a logic block diagram of flow matching and frequency domain constraints during the training phase of this invention. Detailed Implementation

[0029] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0030] Example 1:

[0031] For 9-band in situ hyperspectral data of brain tumors (MSFA pattern size 3×3), the implementation steps of this invention are as follows:

[0032] 1. Pre-trained spectral VAE:

[0033] 1) Construct a lightweight VAE consisting only of 1×1 convolutions.

[0034] 2) Encoder: Input 16-channel spectral data, which is then mapped to 8-channel mean μ and logarithmic variance logσ through 3 layers of 1×1 convolution. 2 .

[0035] 3) Decoder: The sampled latent vector z is mapped back to the 9-channel spectral space through 3 layers of 1×1 convolution.

[0036] 4) Training objective: Minimize the reconstruction error (MSE) and KL divergence within the band-level Z-score normalized space. This step enables the VAE to learn the intrinsic manifold structure of biological tissue spectra.

[0037] 2. Construct the rectified flow model:

[0038] 1) Input end: Design initialization module. For the input B × 1 × H × W mosaic image, according to the 3 × 3 MSFA pattern, it is spatially unfolded and rearranged to fill a coarse hyperspectral image x0 of B × 9 × H × W. Although this x0 has block artifacts, it retains the accurate intensity of physical measurements and serves as an "anchor point" for the flow model.

[0039] 2) Backbone Network: A denoising network based on the U-Net architecture is used, with the input being the noisy image x at the current time step. t The time embedding t results in a velocity field v.

[0040] 3. Training process (manifold constraints and frequency domain regularization):

[0041] In each training iteration:

[0042] 1) Sampling time t: Sampling from a uniform distribution in [0, 1].

[0043] 2) Generate tangent space noise S noise S is obtained by sampling z from a standard normal distribution and passing it through a pre-trained VAE decoder with frozen parameters. z And then denormalize it back to the original data space. This noise S noise It represents "spectral perturbations that conform to biophysical characteristics," rather than meaningless Gaussian white noise.

[0044] 3) Construct intermediate state x t : Here, x1 is the ground truth label.

[0045] 4) Calculate the loss:

[0046] Flow matching loss: Calculates the network prediction speed v pred The mean square error (MSE) between the target velocity (x1 - x0) and the target velocity.

[0047] FFT loss: predicts and reconstructs the image. Perform a Fast Fourier Transform on the real image x1 and calculate the L1 distance of its amplitude spectrum to constrain the consistency of high-frequency texture.

[0048] Total loss = Flow matching loss + λ FFT loss.

[0049] 4. Reasoning process:

[0050] For the new test data, first generate x0, then solve the ODE using the Euler method. Extend the time t from 0 to 1, setting the number of steps to 50.

[0051] The final output x1 is the reconstructed high-fidelity hyperspectral image.

[0052] Example 2:

[0053] For 16-band in situ hyperspectral data of brain tumors (MSFA pattern size of 4×4), the implementation steps of this invention are as follows:

[0054] 1. Pre-trained spectral VAE:

[0055] Construct a lightweight VAE consisting only of 1×1 convolutions.

[0056] 1) Encoder: Input 16-channel spectral data, which is then mapped to 8-channel mean μ and log-variance logσ through 3 layers of 1×1 convolution. 2 .

[0057] 2) Decoder: The sampled latent vector z is mapped back to the 16-channel spectral space through 3 layers of 1×1 convolution.

[0058] 3) Training objective: Minimize reconstruction error (MSE) and KL divergence within the band-level Z-score normalized space. This step enables the VAE to learn the intrinsic manifold structure of biological tissue spectra.

[0059] 2. Construct the rectified flow model:

[0060] 1) Input end: Design initialization module. For the input B × 1 × H × W mosaic image, according to the 4 × 4 MSFA mode, it is spatially unfolded and rearranged to fill a coarse hyperspectral image x0 of B × 16 × H × W. Although this x0 has block artifacts, it retains the accurate intensity of physical measurements and serves as the "anchor point" for the flow model.

[0061] 2) Backbone Network: A denoising network based on the U-Net architecture is used, with the input being the noisy image x at the current time step. t The time embedding t results in a velocity field v.

[0062] 3. Training process (manifold constraints and frequency domain regularization):

[0063] In each training iteration:

[0064] 1) Sampling time t: Sampling from a uniform distribution in [0, 1].

[0065] 2) Generate tangent space noise S noise S is obtained by sampling z from a standard normal distribution and passing it through a pre-trained VAE decoder with frozen parameters. z And then denormalize it back to the original data space. This noise S noise It represents "spectral perturbations that conform to biophysical characteristics," rather than meaningless Gaussian white noise.

[0066] 3) Construct intermediate state x t : Here, x1 is the ground truth label.

[0067] 4) Calculate the loss:

[0068] Flow matching loss: Calculates the network prediction speed v pred The mean square error (MSE) between the target velocity (x1 - x0) and the target velocity.

[0069] FFT loss: predicts and reconstructs the image. Perform a Fast Fourier Transform on the real image x1 and calculate the L1 distance of its amplitude spectrum to constrain the consistency of high-frequency texture.

[0070] Total loss = Flow matching loss + λ FFT loss.

[0071] 5. Reasoning process:

[0072] For the new test data, first generate x0, then solve the ODE using the Euler method. Extend the time t from 0 to 1, setting the number of steps to 50.

[0073] The final output x1 is the reconstructed high-fidelity hyperspectral image.

Claims

1. A medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified flow, characterized in that, Includes the following steps: Step S1: Preprocessing data and building the model; Step S2: Based on the constructed model, learn the spectral manifold prior. Step S3: Initialize the observation-induced source distribution based on prior learned spectral flow; Step S4: Train the flow matching of manifold constraints based on prior learning of spectral flow and initialization of observation-induced source distribution; Step S5: In the inference phase, based on the flow matching of the training manifold constraints, the ordinary differential equations are reconstructed through inference.

2. The medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified flow according to claim 1, characterized in that, The specific model construction in step S1 is as follows: We construct a model architecture that includes a spectral variational autoencoder Spectral VAE and a flow matching backbone network UNet.

3. The medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified flow according to claim 1, characterized in that, Step S2 specifically involves: Spectral VAE is pre-trained using a full-resolution hyperspectral medical dataset. Low-dimensional latent representations of biological tissue spectra are learned through 1×1 convolutional layers to obtain the tangent space features of the data manifold.

4. The medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified current according to claim 1, characterized in that, Specifically, step S3 involves constructing a deterministic initial hyperspectral cube x0 based on the input MSFA original mosaic image through a physically consistent mapping, serving as the starting point for probability stream transmission.

5. The medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified current according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: During the training phase, construct the transmission path from x0 to the real hyperspectral image x1; Step S42: Use the pre-trained VAE from step S2 to generate random noise in the tangent space of the spectral manifold, inject the noise into the transmission path, and train the backbone network using the Fourier domain FFT loss function so that it can learn to recover the velocity field of high-frequency details.

6. The medical in-situ hyperspectral image reconstruction method based on spectral manifold-constrained rectified flow according to claim 1, characterized in that, Step S5 specifically involves: During the inference phase, the MSFA image to be reconstructed is input, an initialization x0 is generated, the ODE is solved using the trained backbone network, and a high-fidelity reconstructed image is obtained through a small number of iterations.

Images