Photoacoustic image enhancement method based on time-driven Transform diffusion model

Abstract

The invention relates to the technical field of medical image enhancement, and discloses a time-driven Transform diffusion model-based photoacoustic image enhancement method, which comprises the following steps of: constructing a diffusion model based on a stochastic differential equation; based on the forward process of a diffusion model, noise is added into the photoacoustic image reconstructed under the non-ideal condition, and the photoacoustic image reconstructed under the non-ideal condition is converted into a known prior distribution image; the prior information is continuously updated through the reverse process of the diffusion model, and the noise is predicted by using a time-related network, so that the training of the diffusion model is realized; sampling the noise state of the photoacoustic image reconstructed under the non-ideal condition through the trained diffusion model, and iteratively solving a reverse stochastic differential equation by using a numerical scheme so as to enhance the photoacoustic image reconstructed under the non-ideal condition to obtain an enhanced photoacoustic image; distortion and artifacts in the photoacoustic image can be suppressed, and detail information of the photoacoustic image can be recovered to the greatest extent.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image enhancement, and in particular to a photoacoustic image enhancement method based on a time-driven Transformer diffusion model. Background Art

[0002] Photoacoustic computed tomography (PACT) is a novel, non-invasive, and low-cost biomedical imaging modality that combines rich optical contrast with high acoustic resolution. Irradiated by nanosecond laser pulses, biological tissue absorbs laser energy and converts it into heat, causing the temperature to rise, thereby emitting ultrasound signals through the thermoelastic effect. These signals, called photoacoustic (PA) signals, are received by the surrounding sensor array and an image of the optical absorption distribution is generated through a reconstruction algorithm. After two decades of development, PACT has shown great potential in animal and clinical imaging applications and has broad prospects.

[0003] As a well-studied inverse problem, traditional PACT reconstruction algorithms can be divided into three categories: analytical solutions, numerical methods, and model-based iterative methods. In general, if the close-range detection transducer array has enough elements, high-quality PACT images can be accurately reconstructed. However, in practice, it is usually too expensive to build an imaging system that meets these ideal conditions, and in many in vivo scenarios, the sensor array usually only partially wraps the tissue. At the same time, as the number of elements decreases (sparse and limited viewing angles), PACT images are susceptible to stripe artifacts and structural distortion. As analytical and numerical methods, they are more susceptible to stripe artifacts due to prior information. In addition, although model-based methods combine acoustic forward models with regularization (such as total variation regularization) to limit artifacts, the reconstructed image is still distorted in the case of limited views. Deep learning has shown great potential in PACT image processing, especially in sparse and limited view problems. But at the same time, due to the convergence problem in deep learning networks, it has little effect on processing photoacoustic reconstruction images under extremely sparse and limited viewing angles. Therefore, there is an urgent need for a deep learning network model that can provide a photoacoustic image enhancement method with better performance to improve the image quality of photoacoustic imaging of PACT systems under sparse and limited viewing angles. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a photoacoustic image enhancement method based on a time-driven Transformer diffusion model, which can not only suppress distortion and artifacts in the photoacoustic image, but also restore the detail information of the photoacoustic image to the greatest extent.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A photoacoustic image enhancement method based on a time-driven Transformer diffusion model comprises the following steps:

[0007] Step 1: construct a diffusion model based on stochastic differential equations;

[0008] Based on the forward process of the diffusion model, the photoacoustic image reconstructed under non-ideal conditions is converted into a known prior distribution image by adding noise to the photoacoustic image reconstructed under non-ideal conditions;

[0009] Through the inverse process of the diffusion model, the prior information is continuously updated, and the noise is predicted using a time-dependent neural network to achieve the training of the diffusion model;

[0010] Step 2: The noise state of the photoacoustic image reconstructed under non-ideal conditions is sampled by the trained diffusion model, and the inverse stochastic differential equation is iteratively solved using a numerical scheme, so as to enhance the photoacoustic image reconstructed under non-ideal conditions and obtain an enhanced photoacoustic image;

[0011] The photoacoustic image reconstructed under non-ideal conditions includes a sparse viewing angle photoacoustic image and a limited viewing angle photoacoustic image.

[0012] Furthermore, the forward process based on the diffusion model, by adding noise to the photoacoustic image reconstructed under non-ideal conditions, converts the photoacoustic image reconstructed under non-ideal conditions into a known prior distribution image, specifically includes:

[0013] The forward process of the diffusion model is expressed as follows: dx = θ t (μ-x)dt+σ t dw;

[0014] Where x represents the state feature, t represents the index of the tth noise addition step in the forward diffusion process; θ t and σ t represents the preset time-varying parameters that define the mean regression rate and the random fluctuation level, w represents the standard Wiener process; μ is the state mean; the state mean μ and the initial conditions of the forward process of the diffusion model are set to the photoacoustic image reconstructed under non-ideal conditions and the corresponding full-view photoacoustic image, respectively.

[0015] Furthermore, the inverse process of the diffusion model is used to continuously update the prior information and use the time-dependent neural network to predict the noise to realize the training of the diffusion model, which specifically includes:

[0016] The inverse process of the diffusion model is expressed as follows: x represents the state characteristics; t represents the index of the tth noise-added time step in the forward diffusion process; x(t) represents the state characteristics of the tth noise-added time step; p t(x) represents the marginal probability density function of x(t), represents the reverse time Wiener process; θ t and σ t represents the preset time-varying parameters that define the mean reversion rate and the level of stochastic fluctuations; μ is the state mean;

[0017] Finally, using the time-dependent neural network s φ (x(t),μ,t) to approximate the score function of each state, and train the diffusion model by using a noise network to approximate the noise:

[0018]

[0019] Where L(·) represents the loss function, φ represents the parameters of the neural network used in each state; ε t The mean is 0 and the variance is σ t Gaussian noise.

[0020] Furthermore, the time-dependent neural network adopts a time-driven Transformer network; the time-driven Transformer network includes an encoder, a decoder and a size adjustment layer; the encoder and the decoder both include multiple time-driven Transformer modules, and the time-driven Transformer network links the corresponding time-driven Transformer modules in the encoder and the decoder by using jump links; the time-driven Transformer module includes a time-driven multi-head transposition attention and a gated convolutional feedforward network.

[0021] Furthermore, the time-driven multi-head transposition attention is obtained by improving the Transformer module as follows:

[0022] Use convolution and depth-wise convolution to calculate the query matrix, key matrix, and value matrix;

[0023] Before calculating the query matrix, key matrix, and value matrix, the time step in the diffusion model is embedded in the calculation of the query matrix Q, key matrix Q, and value matrix V:

[0024]

[0025] Input features representing time-driven multi-head transposed attention; Both represent 3×3 depth-wise convolution; and Represents 1×1 convolution, uniformly denoted as W p At the same time, an additional multi-layer perceptron is inserted before the depth-wise convolution to convert the temporal embedding features into channel scaling γ Q, γ K , γ V and translation beta Q , β K , β V ;

[0026] Use channel attention mechanism to replace the self-attention mechanism in the Transformer module;

[0027] The time-driven multi-head transposed attention is defined as:

[0028]

[0029] in, Represents the output features of time-driven multi-head transposed attention; and denote the reshaped query matrix, key matrix, and value matrix respectively; α represents a learnable scaling parameter used to constrain the magnitude of the dot product before the softmax activation function softmax(·); Attention(·) denotes the self-attention feature matrix of the input image.

[0030] Furthermore, the gated convolutional feedforward network replaces the nonlinear activation and convolution in the Transformer module through a gating mechanism and depth-wise convolution:

[0031]

[0032] X and Y represent the input features and output features of the gated convolutional feedforward network, respectively; All represent 1×1 convolution; and All represent 3×3 depth-wise convolution; GELU(·) represents the Gelu activation function; Gating(·) represents the output features after the gating mechanism.

[0033] Compared with the prior art, the beneficial technical effects of the present invention are:

[0034] When a high-quality full-view image degenerates into a low-quality sparse or limited-view image, the diffusion model based on stochastic differential equations proposed in the present invention can build a series of prior images in intermediate states by continuously adding noise through the forward process of the stochastic differential equations. In the image enhancement process, the prior information can be continuously updated through the reverse process of the stochastic differential equations, and the noise can be predicted using a time-dependent network, thereby enhancing the convergence ability of the model and improving the quality of the enhanced image.

[0035] The time-driven Transformer network proposed in the present invention not only maintains the powerful feature extraction capability of the traditional Transformer for photoacoustic images, but also takes into account the improvement of computational efficiency and the reduction of computational costs. In actual use, the time-driven Transformer module, as the main module of the time-driven Transformer network, is composed of a time-driven multi-head transposition attention and a gated convolution feedforward network. Among them, the former uses depth-by-depth convolution to calculate the query, key and value matrix, and at the same time integrates the prior information of the time step in the diffusion model into the calculation of the three matrices, thereby reducing the number of model parameters and enhancing the model's performance. At the same time, the channel attention mechanism is used to replace the original self-attention mechanism, thereby enhancing the network's ability to extract features from images. The gated convolution feedforward network uses depth-by-depth convolution and GELU gate mechanisms to reduce the number of network parameters and improve computational efficiency while controlling the output image dimension and feature changes.

[0036] The present invention constructs a bidirectional mapping relationship from sparse or limited perspective to full perspective images by using stochastic differential equations, and gradually enhances the image using a time-driven Transformer network, thereby improving the convergence of the network and obtaining higher image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of the architecture of a diffusion model based on stochastic differential equations in an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the structure of a time-driven Transformer network in an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the structure of a time-driven Transformer module in an embodiment of the present invention;

[0040] Figure 4 (a) to Figure 4 (d) is a comparison diagram of the enhanced image of the photoacoustic image prediction model in the embodiment of the present invention when inputting a limited-viewing-angle photoacoustic image and the reconstructed image of the filter back projection (FBP) algorithm;

[0041] Figure 5 (a) to Figure 5 (d) is a comparison diagram of the enhanced image of the photoacoustic image prediction model in an embodiment of the present invention when a sparse viewing angle photoacoustic image is input and the reconstructed image of the FBP algorithm. DETAILED DESCRIPTION

[0042] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0043] The photoacoustic image enhancement method based on a time-driven Transformer diffusion model proposed in the present invention includes a diffusion model based on a stochastic differential equation and a time-driven Transformer network.

[0044] 1. Diffusion model based on stochastic differential equations

[0045] First, we introduce the image denoising model based on stochastic differential equation (SDE) proposed by Song et al. in 2020. Its essence is to transform the image distribution into a known prior distribution by slowly adding disturbance noise, and the corresponding inverse process can transform the prior distribution back into the image distribution through gradual denoising. Therefore, the entire diffusion process It can be expressed as:

[0046] dx=f(x,t)dt+g(t)dw; (1)

[0047] Where x(0) represents the initial condition, f and g are the drift function and the diffusion function respectively, and w represents the standard Wiener process. In general, the final state x(T) can be described as a Gaussian noise distribution with a preset mean and variance.

[0048] Subsequently, based on the inverse process of the stochastic differential equation, the image data can be sampled from the noise data, and the process is expressed as:

[0049]

[0050] in, represents the reverse time Wiener process, and p t (x) represents the marginal probability density function of x(t) at time t. When formula (2) is actually used, although due to The score function of the representative is unknown, but the score-based diffusion model can be used to train a time-dependent neural network using a score matching algorithm. φ (x(t),t) to approximate it.

[0051] In this invention, in order to convert sparse or limited-view images into high-quality full-view images, special drift and diffusion functions are constructed as follows:

[0052] dx=θ t (μ-x)dt+σ t dw; (3)

[0053] Among them, μ is the state mean, θ t and θ t Represents the preset time-varying parameters that define the mean reversion rate and the stochastic volatility level.

[0054] In actual use, the state mean and initial conditions are set to the sparse or limited-view image and its corresponding image under full view. Therefore, the image can be enhanced by the inverse process of the corresponding stochastic differential equation as follows:

[0055]

[0056] Finally, using the time-dependent neural network s φ (x(t),μ,t) to approximate the score function of each state, and use the noise network to approximate the noise instead of the image data, as shown below:

[0057]

[0058] Where L represents the loss function, φ represents the parameters of the network used in each state; ε t The mean is 0 and the variance is σ t After training, the network can enhance sparse and limited-view images by sampling the noise state and iteratively solving the inverse stochastic differential equation using a numerical scheme.

[0059] 2. Time-Driven Transformer Network

[0060] In order to maintain the effectiveness of the Transformer module while reducing the computational cost, this paper introduces a time-driven transformer network as a time-dependent neural network. φ (x(t),μ,t) is used to approximate the score function of each state. The time-driven Transformer network consists of an encoder, a decoder, and a size adjustment layer, where the encoder and decoder are composed of multiple time-driven Transformer modules. The time-driven Transformer module consists of two parts: time-driven multi-head transposed attention (TMTA) and a gated convolutional feed-forward network (GCFN).

[0061] Time-driven multi-head transposed attention: Compared with the traditional Transformer module, there are three main changes. First, convolution and depth-wise convolution (Dconv) are used to calculate the query matrix, key matrix, and value matrix. Second, before calculating the query matrix, key matrix, and value matrix, the time step in the diffusion model is embedded in the calculation of the above three matrices, as shown below:

[0062]

[0063] Among them, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; represents the input feature; W d represents 3×3 depth-wise convolution, W p represents a 1×1 convolution. At the same time, an additional multi-layer perceptron is inserted before the depth-wise convolution to convert the temporal embedding features into channel scaling γ and translation β. Finally, the channel attention mechanism is used to replace the self-attention mechanism in the traditional Transformer. In general, the multi-head transposed attention can be defined as:

[0064]

[0065] in, represents the output features; and denote the reshaped query matrix, key matrix, and value matrix respectively, and α represents a learnable scaling parameter used to constrain the magnitude of the dot product before the softmax activation function.

[0066] Gated Convolutional Feedforward Network: In this network module, a gating mechanism (i.e., GELU) and depth-wise convolution replace the nonlinear activation and convolution in the traditional Transformer, thereby enhancing the model's expressiveness and improving the independence between positions, allowing the model to better capture the features and patterns in the input image. In general, the gated convolutional feedforward network can be expressed as:

[0067]

[0068] Finally, the time-driven Transformer network uses skip connections to link corresponding modules in the encoder and decoder, thereby better extracting features from the input image and improving network performance.

[0069] The following two examples are provided to help those skilled in the art further understand the present invention. The two examples are run on the same computer, whose main configuration is: PyTorch 3.8.10 version, Microsoft Windows 10 operating system, InterXeon Gold 6226R CPU @ 3.5Hz and a single Nvidia TITAN GPU. At the same time, in the training network process of the two examples, all training parameters are kept consistent: the batch size is set to 2. The Adam optimizer is used, and the initial learning rate is set to 3 × 10-5 with a cosine scheduler decayed to 1e-7.

[0070] Meanwhile, in specific applications, the reconstruction algorithm for photoacoustic image input may adopt a filtered back-projection algorithm, as shown below:

[0071]

[0072] Among them, r s and r d represents the position of the detection surface and the element, Ω represents the detection surface; dΩ represents the solid angle of the detection surface, and the projection term b(r d ,t) is given by:

[0073]

[0074] The common method for calculating the time domain derivative in equation (10) is to multiply the signal by jω in the frequency domain, so In practical applications, it can be equivalent to a high-pass ramp filter. However, using the filter directly may cause artifacts and more high-frequency noise. Therefore, a bandpass filter is used in signal preprocessing to solve this problem, that is, a 3rd-order Butterworth filter with a bandwidth of 0.5 to 10 MHz. In general, equation (10) can be converted to:

[0075]

[0076] in, and They are represented as Fourier transform and its inverse transform respectively.

[0077] Example 1

[0078] For the image enhancement of living mice, this embodiment is intended to illustrate the effectiveness of the present invention for image enhancement with limited viewing angle.

[0079] In this embodiment, the geometry of the experimental transducer array is as follows: Figure 4(a) shows that the detection angles of the four groups of limited viewing angles are: π / 4, π / 2, 3π / 4 and π. The photoacoustic image reconstructed under limited viewing angle is used as the network input. At the same time, the image reconstructed from the photoacoustic signal collected by 512 channels under full viewing angle is used as the reference image, as shown in Figure 4 (b) shows the limited viewing angle image reconstructed by the FBP algorithm (i.e. Figure 4 As shown in (c), due to limited sampling, some signals of biological tissues are missing, resulting in serious distortion of the reconstructed image and loss of some detail information. Figure 4 As shown in (d), the photoacoustic image prediction model provided by this embodiment can effectively alleviate these problems and show expected improvements in the restoration of blood vessels and boundary details.

[0080] Table 1 shows the comparison of the quantitative evaluation results of the reconstructed image of the photoacoustic image prediction model of this embodiment with the input of limited-view photoacoustic data and the FBP algorithm. It can be seen that the effect of the photoacoustic image prediction model provided by this embodiment with the input of limited-view photoacoustic images is better than the FBP algorithm in terms of the two evaluation indicators of Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM).

[0081] Table 1 Average PSNR and SSIM of the live mouse dataset under limited viewing angle

[0082]

[0083]

[0084] Example 2

[0085] For finger image enhancement, this embodiment is intended to illustrate the effectiveness of the present invention for sparse view image enhancement.

[0086] In this embodiment, the geometry of the experimental transducer array is as follows: Figure 5 (a) shows that the number of transducers in the four groups of sparse view angles are 256, 128, 64 and 32 respectively. The photoacoustic image reconstructed under sparse view angle is used as the network input. At the same time, similar to the limited view angle, the image reconstructed by the full view angle 512 channels is used as the reference image, as shown in Figure 5 (b) shows the sparse view image reconstructed by the FBP algorithm (i.e. Figure 5 As shown in (c), due to the influence of spatial subsampling, a large number of stripe artifacts appear in the reconstructed image, which submerges the image and loses a lot of image details. Figure 5As shown in (d), the photoacoustic image prediction model provided by this embodiment can effectively suppress stripe artifacts and restore detail information in the image.

[0087] Table 2 shows the comparison of the quantitative evaluation results of the reconstructed image of the photoacoustic image prediction model in this embodiment when inputting sparse perspective photoacoustic data and the FBP algorithm. It can be seen that the effect of the photoacoustic image prediction model provided by this embodiment when inputting limited perspective photoacoustic images is better than the FBP algorithm in terms of the two evaluation indicators of PSNR and SSIM.

[0088] Table 2 Average PSNR and SSIM of finger dataset under sparse view

[0089]

[0090] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0091] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A photoacoustic image enhancement method based on a time-driven Transformer diffusion model, characterized in that: The following steps are involved: Step 1: construct a diffusion model based on stochastic differential equations; Based on the forward process of the diffusion model, the photoacoustic image reconstructed under non-ideal conditions is converted into a known prior distribution image by adding noise to the photoacoustic image reconstructed under non-ideal conditions; Through the inverse process of the diffusion model, the prior information is continuously updated, and the noise is predicted using a time-dependent neural network to achieve the training of the diffusion model; Step 2: The noise state of the reconstructed photoacoustic image under non-ideal conditions is sampled by the trained diffusion model, and the inverse stochastic differential equation is iteratively solved using a numerical scheme, so as to enhance the photoacoustic image reconstructed under non-ideal conditions and obtain an enhanced photoacoustic image; The photoacoustic image reconstructed under non-ideal conditions includes a sparse viewing angle photoacoustic image and a limited viewing angle photoacoustic image.

2. The photoacoustic image enhancement method based on the time-driven Transformer diffusion model according to claim 1, characterized in that: The forward process based on the diffusion model, by adding noise to the photoacoustic image reconstructed under non-ideal conditions, converts the photoacoustic image reconstructed under non-ideal conditions into a known prior distribution image, specifically includes: The forward process of the diffusion model is expressed as follows: dx = θ t (μ-x)dt+σ t dw; Where x represents the state feature, t represents the index of the tth noise addition step in the forward diffusion process; θ t and σ t represents the preset time-varying parameters that define the mean regression rate and the random fluctuation level, w represents the standard Wiener process; μ is the state mean; the state mean μ and the initial conditions of the forward process of the diffusion model are set to the photoacoustic image reconstructed under non-ideal conditions and the corresponding full-view photoacoustic image, respectively.

3. The photoacoustic image enhancement method based on the time-driven Transformer diffusion model according to claim 1, characterized in that: The inverse process of the diffusion model is used to continuously update the prior information and use the time-dependent neural network to predict the noise to realize the training of the diffusion model, which specifically includes: The inverse process of the diffusion model is expressed as follows: x represents the state characteristics; t represents the index of the tth noise-added time step in the forward diffusion process; x(t) represents the state characteristics of the tth noise-added time step; p t (x) represents the marginal probability density function of x(t), represents the reverse time Wiener process; θ t and σ t represents the preset time-varying parameters that define the mean reversion rate and the level of stochastic fluctuations; μ is the state mean; Finally, using the time-dependent neural network s φ (x(t),μ,t) to approximate the score function of each state, and train the diffusion model by using a noise network to approximate the noise: Where L(·) represents the loss function, φ represents the parameters of the neural network used in each state; ε t The mean is 0 and the variance is σ t Gaussian noise.

4. The photoacoustic image enhancement method based on the time-driven Transformer diffusion model according to claim 3, characterized in that: The time-dependent neural network adopts a time-driven Transformer network; the time-driven Transformer network includes an encoder, a decoder and a size adjustment layer; the encoder and the decoder both include multiple time-driven Transformer modules, and the time-driven Transformer network links the corresponding time-driven Transformer modules in the encoder and the decoder by using jump links; the time-driven Transformer module includes a time-driven multi-head transposition attention and a gated convolutional feedforward network.

5. The photoacoustic image enhancement method based on the time-driven Transformer diffusion model according to claim 4, characterized in that: The time-driven multi-head transposition attention is obtained by improving the Transformer module as follows: Use convolution and depth-wise convolution to calculate the query matrix, key matrix, and value matrix; Before calculating the query matrix, key matrix, and value matrix, the time step in the diffusion model is embedded in the calculation of the query matrix Q, key matrix Q, and value matrix V: Input features representing time-driven multi-head transposed attention; Both represent 3×3 depth-wise convolution; and Represents 1×1 convolution, uniformly denoted as W p At the same time, an additional multi-layer perceptron is inserted before the depth-wise convolution to convert the temporal embedding features into channel scaling γ Q , γ J , γ V and translation beta Q , β K , β V ; Use channel attention mechanism to replace the self-attention mechanism in the Transformer module; The time-driven multi-head transposed attention is defined as: in, Represents the output features of time-driven multi-head transposed attention; and denote the reshaped query matrix, key matrix, and value matrix respectively; α represents a learnable scaling parameter used to constrain the magnitude of the dot product before the softmax activation function softmax(·); Attention(·) denotes the self-attention feature matrix of the input image.

6. The photoacoustic image enhancement method based on the time-driven Transformer diffusion model according to claim 4, characterized in that: The gated convolutional feedforward network replaces the nonlinear activation and convolution in the Transformer module through a gating mechanism and depth-wise convolution: X and Y represent the input features and output features of the gated convolutional feedforward network, respectively; All represent 1×1 convolution; and All represent 3×3 depth-wise convolution; GELU(·) represents the Gelu activation function; Gating(·) represents the output features after the gating mechanism.

Images