Enhanced photoacoustic microscopy method based on physical degradation learning

Abstract

The application discloses an enhanced photoacoustic microscopic imaging method based on physical degradation learning, and comprises the following steps: based on the photoacoustic imaging principle, the image of real photoacoustic microscopic imaging is degraded into low-quality photoacoustic microscopic imaging through a physical model with variability degradation learning; a high-quality photoacoustic microscopic imaging image is used as a supervised training target; then, based on deep learning, the image is input into a convolutional neural network with an attention mechanism designed by the application for network training; the image quality of the reconstructed image is evaluated until the reconstruction quality meets the standard, otherwise the above steps are re-executed; and then, a real low-quality photoacoustic image is input into the trained neural network to obtain a reconstructed high-resolution photoacoustic image. The application can effectively improve the resolution and contrast of deep photoacoustic microscopic images, and can also denoise the photoacoustic microscopic images; and because the data set is acquired through a physical model, the application can effectively reduce the dependence on the data set, and is beneficial to improving the reconstruction effect under different environments.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of photoacoustic microscopy imaging, and particularly relates to an enhanced photoacoustic microscopy imaging method based on physical degradation learning. BACKGROUND

[0002] Photoacoustic imaging technology is a new biomedical imaging method with great clinical application value developed in recent years. It can specifically detect the absorption information of biological tissues based on the photoacoustic effect. Compared with other biomedical imaging technologies, photoacoustic imaging can realize specific molecular or cellular imaging, and provide accurate imaging information for the diagnosis and evaluation of diseases, and has the advantages of non-invasiveness, high resolution, high contrast, low cost and specific spectral recognition capability. Photoacoustic microscopy (PAM) is a high-resolution non-invasive imaging method developed on the basis of photoacoustic imaging, and has shown a wide application prospect in the field of biomedical imaging diagnosis.

[0003] Under the condition of safe limited laser fluence density, the penetration depth of photoacoustic microscopy is limited by the strong scattering of biological tissues, and is basically within one optical transport free path (about 1mm for biological tissues). Photoacoustic microscopy in the shallower region can obtain better image resolution and contrast. If the focusing depth of photoacoustic microscopy is improved, the spatial resolution will be reduced, the background noise will be enhanced, and the image contrast and quality will be affected. In recent years, with the rise of deep learning, the powerful function of deep learning can well solve the problems that are difficult to solve by traditional methods. However, the supervised training deep learning method is limited by the high cost of data set acquisition, and it is difficult to obtain. The reconstruction image quality of unsupervised training deep learning is not as good as that of supervised training. SUMMARY

[0004] The application provides an enhanced photoacoustic microscopy imaging method based on physical degradation learning, so as to solve the problem of reduced photoacoustic microscopy image resolution and contrast caused by limited penetration depth in photoacoustic microscopy imaging, and effectively solve the problem of limited data set of deep learning.

[0005] The technical scheme for achieving the object of the application is an enhanced photoacoustic microscopy imaging method based on physical degradation learning, comprising the following steps:

[0006] Step 1: Based on the principle of photoacoustic imaging, a plurality of groups of high-quality ground-truth photoacoustic microscopy images are obtained, and the ground-truth photoacoustic microscopy images are degraded into low-quality photoacoustic microscopy images through a physical degradation model with variability;

[0007] Step 2: The high-quality ground-truth photoacoustic microscopy images not subjected to the physical degradation model are used as a training target;

[0008] Step 3: match the low-quality photoacoustic microscopic image subjected to the physical degradation model and the high-quality ground-truth photoacoustic microscopic image and input them into the convolutional neural network with attention mechanism for network training, and obtain the corresponding mapping relationship through training;

[0009] Step 4: reconstruct the low-quality photoacoustic image through the mapping relationship obtained through training, and evaluate whether the reconstructed image quality meets the standard, if yes, proceed to Step 5, otherwise, re-execute Step 3;

[0010] Step 5: input the real-time obtained photoacoustic microscopic image into the trained convolutional neural network with attention mechanism for processing to obtain a high-quality image.

[0011] Preferably, based on the principle of photoacoustic imaging, the specific method for obtaining multiple sets of high-quality ground-truth photoacoustic microscopic images is as follows:

[0012] The pulsed light is irradiated to the biological tissue, and the ultrasonic transducer is used to sample the ultrasonic signal generated by the light-absorbing region in the tissue;

[0013] The ground-truth photoacoustic microscopic image is reconstructed by maximum value projection according to the sampling information.

[0014] Preferably, the specific formula for degrading the ground-truth photoacoustic microscopic image into a low-quality photoacoustic microscopic image through the physical degradation model with variability is as follows:

[0015]

[0016] In the formula, I m1 is the image reconstructed by maximum value projection, I m is the output image after the physical model with variability, h is the PSF kernel, α is the compensation function, and η is the total noise.

[0017] Preferably, the compensation function is specifically as follows:

[0018] α=μm 1+ξz

[0019] wherein μ is the attenuation coefficient, ξ is the compensation coefficient, m is the intensity interference term of the attenuation coefficient, and z is the intensity interference term of the compensation coefficient.

[0020] Preferably, the PSF kernel is specifically as follows:

[0021]

[0022] In the formula, T(ω) is the spherical focusing transducer frequency, is the field pattern of the spherical focusing transducer.

[0023] Preferably, when the convolutional neural network with attention mechanism is trained, the processing procedure of the image is as follows:

[0024] The low-quality photoacoustic microscopic image subjected to the physical degradation model is input into the convolutional neural network, in which the photoacoustic microscopic image is subjected to a first convolutional block and then a plurality of residual blocks;

[0025] The image after the first residual block is input into a U-shaped network, which is subjected to three times of down-sampling encoding, and after each time of down-sampling, the image is subjected to two second convolutional blocks and then a GC attention module;

[0026] The image after the down-sampling encoding is subjected to two times of up-sampling decoding, and the result of each time of up-sampling decoding is concatenated with the output result of the corresponding size in the down-sampling encoding process in the channel dimension, and then the concatenated result is subjected to a third convolutional block and then used as the input of the next up-sampling decoding; the image after the last up-sampling is subjected to a fourth convolutional block and a convolutional layer, and then added with the original image to output the image.

[0027] Preferably, the first residual block is composed of two convolutional blocks and a skip connection, and the specific formula is as follows:

[0028] f(x)=g(x)+x

[0029] wherein x is the input image, g(x) is the image subjected to two convolutional operations, and f(x) is the input image. Preferably, the loss function when the convolutional neural network with attention mechanism is trained is as follows:

[0030] L=αL mse +βL p

[0031] wherein L mse represents the mean square error loss, L p represents the perceptual loss, L represents the total generator loss, and α and β are the weights of the mean square error loss and the perceptual loss;

[0032] The specific calculation formula of the mean square error loss is as follows:

[0033]

[0034] wherein Y i represents the observation value of the actual photoacoustic image, Y′ i is the prediction value of the reconstructed photoacoustic image of the model, and n is the number of input samples;

[0035] The specific calculation formula of the perceptual loss is as follows:

[0036]

[0037] wherein qi (x) represents the real photoacoustic image in the i-th convolution layer of the VGG network, q i (G(z)) represents the generated image in the i-th convolution layer of the VGG network, and N represents the number of feature layers.

[0038] Preferably, the specific formula of the GC attention module is as follows:

[0039] z = x + W v2 ReLU(LN(W v1 ·y))

[0040]

[0041] wherein x is an input image, W v2 ReLU(LN(W v1 (·)) is a transformation module, y is a weight module, W v2 , W v1 is a 1x1 convolution used to capture the inter-channel dependency, ReLU is an activation function, LN is a normalization function, N p is the total number of positions in the feature map, W k is a linear transformation matrix, and z is an output image.

[0042] Preferably, the peak signal-to-noise ratio and the structural similarity coefficient are used to evaluate whether the quality of the reconstructed image meets the standard, and when the peak signal-to-noise ratio is greater than a first threshold and the structural similarity coefficient is greater than a second threshold, it is considered that the quality meets the standard.

[0043] Compared with the prior art, the present application has the following advantages: 1) in the quality direction of imaging, the present application can well handle the reduction of photoacoustic microscopic image resolution and contrast caused by scattering. The present application can well improve the quality of low-quality images; 2) in the depth direction of imaging, the present application can further strengthen the effective detection depth of photoacoustic microscopic imaging because it can improve the difficult-to-identify images in the deeper layer; 3) in the data set collection, the present application can effectively solve the problem of limited data set of deep learning because it solves the problem of difficult collection of supervised training data set; 4) in the equipment cost, the present application can reduce the requirement for laser conditions to some extent, thereby reducing the equipment cost of the laser.

[0044] The present application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of the present application based on physical degradation learning enhanced photoacoustic microscopic imaging method.

[0046] Figure 2 The schematic diagram of the whole network framework of the enhanced photoacoustic microscopic imaging method based on physical degradation learning of the application.

[0047] Figure 3 The schematic diagram of each network module in the enhanced photoacoustic microscopic imaging method based on physical degradation learning of the application.

[0048] Figure 4 The effect schematic diagram of the enhanced photoacoustic microscopic imaging method based on physical degradation learning of the application. DETAILED DESCRIPTION

[0049] An enhanced photoacoustic microscopic imaging method based on physical degradation learning solves the problem of difficult data set acquisition through a physical degradation model, and improves the image quality of photoacoustic microscopic imaging through a deep learning-based method, and the parameters of the physical degradation model are full of randomness and within a reasonable range in the training process, so as to improve the generalization of the method. Figure 1 As shown in the figure, the specific steps of the application are as follows:

[0050] Step 1: Based on the principle of photoacoustic imaging, a plurality of groups of high-quality ground-truth photoacoustic microscopic imaging results are obtained, and a physical degradation model with variability is used to degrade the ground-truth photoacoustic microscopic image into a low-quality photoacoustic microscopic image.

[0051] Specifically, pulsed laser irradiates biological tissue, and an ultrasonic transducer is used to sample the ultrasonic signal generated by the light absorption area in the tissue, wherein the ultrasonic signal generated by the light excitation is a photoacoustic signal. The above-mentioned method is used to collect and reconstruct the ground-truth photoacoustic microscopic image by using maximum projection.

[0052] In a further embodiment, the specific formula for reconstructing the ground-truth photoacoustic microscopic image by maximum projection is:

[0053] M(i,j)=max(P i,j (t:t+Δt))

[0054] Where (i,j) is the coordinate of the data collected in the surface scanning excitation mode, M(i,j) is the maximum projection at the collection position (i,j), P i,j is the photoacoustic time domain signal at the collection position (i,j), t is the start time of the selected interval, and Δt is the interval of the selected projection. By selecting the interval, the maximum projection of different depth layers can be realized.

[0055] In a further embodiment, the construction process of the physical degradation learning model is as follows:

[0056] The role of the convolution model is to reduce the resolution of the photoacoustic microscopic image by using the Rayleigh-Sommerfeld diffraction formula. The simplified formula is as follows:

[0057]

[0058] is the wave field is the Fourier transform of the spatial point, is the aperture weighting function The above formula can also be expressed as the field mode of a planar circular transducer.

[0059] By combining the field mode of a planar circular transducer with a spherical compensation function, we get where the spherical compensation function is:

[0060]

[0061] Then the spherical focusing transducer integrates to the sensor frequency T(ω). The integration range is the bandwidth of the field mode of the spherical focusing transducer of the ultrasound transducer, so the point spread function PSF kernel is obtained as follows:

[0062]

[0063] Adding a compensation function makes the image data set need to be attenuated to different degrees when reconstructed to different depths. The compensation function is specifically:

[0064] α = μm 1+ξz

[0065] where μ is the attenuation coefficient and ξ is the compensation coefficient.

[0066] The role of the noise model is to add noise in the degradation process to simulate the real environment. The specific formula is as follows:

[0067] η = η1+ η2+ η3

[0068] where η is the total noise, η1 is the Gaussian noise, η2 is the Rayleigh noise, and η3 is the Poisson noise. The size of the noise is not fixed, but full of randomness. Integrating the above formula, we get:

[0069]

[0070] I m1 is the image reconstructed by maximum value projection, I m is the output image after the physical model with variability.

[0071] Step 2: The high-quality photoacoustic microscopic image of the earth without physical degradation model is used as the network training target.

[0072] Specifically, the photoacoustic microscopic images are manually selected, and high-resolution and high-contrast photoacoustic microscopic images are selected as training targets to ensure the training quality of the convolutional neural network with attention mechanism.

[0073] Step 3: Matching the low-quality photoacoustic microscopic images and the high-quality photoacoustic microscopic images and inputting them into the convolutional neural network with attention mechanism for network training to obtain the corresponding mapping relationship through training.

[0074] As shown in Figure 2 and Figure 3 , the processing process of the convolutional neural network with attention mechanism for images is as follows:

[0075] Step 301: Input the low-quality photoacoustic microscopic images subjected to the physical degradation model into the convolutional neural network, and in the convolutional neural network, the photoacoustic images first pass through a convolution block and then pass through multiple residual blocks; the convolution block is composed of a convolution layer and an activation function, and the residual block is composed of two convolution blocks, two batch normalization layers and an activation function.

[0076] Step 302: The images after passing through the residual block are input into the U-shaped network, and the U-shaped network is subjected to three times of down-sampling encoding, and after each down-sampling, two convolution blocks are used, which are composed of a convolution layer, a batch normalization layer and an activation function, and then a GC attention module is used.

[0077] Step 303: After down-sampling, it is a process of twice up-sampling decoding, and the result of each up-sampling decoding is spliced with the corresponding size output result in the channel dimension in the down-sampling process, and then the spliced result is subjected to a same convolution block, and then used as the input of the next up-sampling decoding. After the last up-sampling is subjected to a convolution block, it is added to the original image through a convolution layer, and then the image is output. Finally, the loss function is used to control the convergence of the entire network model.

[0078] Specifically, the residual block is composed of two convolution blocks and a jump connection, and the specific formula is as follows:

[0079] f(x)=g(x)+x

[0080] Wherein, x is the input image, g(x) is the image after two convolution operations, and f(x) is the input image. The main function is to improve the depth of the convolutional neural network while maintaining the convergence of the neural network.

[0081] Specifically, the attention module is a GC attention module, and the specific formula is as follows:

[0082] z=x+W v2 ReLU(LN(W v1 ·y))

[0083]

[0084] where x is the input image, W v2 ReLu(LN(W v1 is the transformation module, y is the weight module, and the main function is to strengthen the analysis and identification of microvessels to provide the possibility for subsequent image reconstruction.

[0085] Specifically, the loss function of the convolutional neural network with attention mechanism is:

[0086] L=αL mse +βL p

[0087] where L mse represents the mean square error loss, L p represents the perceptual loss, L represents the total generator loss, and α and β are the weights of the mean square error loss and the perceptual loss.

[0088] Preferably, the physical degradation learning enhanced photoacoustic microscopic imaging quality method, the specific calculation formula of the mean square error loss is:

[0089]

[0090] where Y i represents the observed value of the actual photoacoustic image, Y′ i is the predicted value of the reconstructed photoacoustic image of the model, and n is the number of input samples.

[0091]

[0092] where q i (x) represents the feature of the real photoacoustic image in the i-th convolutional layer of the VGG network, q i (G(z)) represents the feature of the generated image in the i-th convolutional layer of the VGG network, and N represents the number of feature layers.

[0093] Step 4: Reconstruct the low-quality photoacoustic image by the mapping relationship obtained by training, and evaluate the reconstructed image quality until the reconstructed quality meets the standard, otherwise re-execute step 3.

[0094] Specifically, combined with the attached Figure 4 , the reconstructed image can well improve the resolution and contrast of the image. The peak signal-to-noise ratio and the structural similarity coefficient are used to evaluate the quality of the reconstructed image. When the peak signal-to-noise ratio is greater than 30 and the structural similarity coefficient is greater than 0.9, it is considered that the quality meets the standard.

[0095] Step 5: input the real-time obtained photoacoustic microscopic image into the trained convolutional neural network with attention mechanism for processing to obtain a high-quality image.

[0096] As described in the background section, since the penetration depth of photoacoustic microscopic imaging is limited within one optical transport free path (about 1 mm for biological tissue) under the condition of safe limited laser fluence density, the present application provides an enhanced photoacoustic microscopic imaging method based on physical degradation learning to solve the problems of poor resolution and contrast encountered by photoacoustic microscopy in deep imaging, and also can reduce the requirements for laser equipment. The present application provides an enhanced photoacoustic microscopic imaging method based on physical degradation learning, which realizes good imaging quality of photoacoustic microscopic imaging of deep tissue, and reduces the laser energy to obtain good imaging quality, so it can also reduce the requirements for laser equipment.

Claims

1. An enhanced photoacoustic microscopic imaging method based on physical degradation learning, characterized in that, Comprise the following steps: Step 1: Based on the principle of photoacoustic imaging, a plurality of groups of high-quality ground-truth photoacoustic microscopic images are obtained, and the ground-truth photoacoustic microscopic images are degraded into low-quality photoacoustic microscopic images through a physical degradation model with variability, and the specific formula is: In the formula, I m1 is an image reconstructed by maximum projection, I m is an output image after a physical model with variability, h is a PSF kernel, α is a compensation function, and η is total noise. The compensation function is specifically: α = μm 1+ξz Wherein, mu is the attenuation coefficient, xi is the compensation coefficient, m is the intensity interference term of the attenuation coefficient, and z is the intensity interference term of the compensation coefficient; Step 2: The high-quality ground-truth photoacoustic microscopic images not subjected to the physical degradation model are taken as the training target; Step 3: The low-quality photoacoustic microscopic images subjected to the physical degradation model and the high-quality ground-truth photoacoustic microscopic images are matched and input into the convolutional neural network with attention mechanism for network training, and the corresponding mapping relationship is obtained through training, and the processing process of the convolutional neural network with attention mechanism during training is: The low-quality photoacoustic microscopic images subjected to the physical degradation model are input into the convolutional neural network, and in the convolutional neural network, the photoacoustic microscopic images are subjected to a first convolutional block and then a plurality of residual blocks; the first residual block is composed of two convolutional blocks and a jump connection, and the specific formula is as follows: f(x) = g(x) + x Wherein, x is the input image, g(x) is the image after two convolutional operations, and f(x) is the input image; The image after the first residual block is input into a U-shaped network, and the U-shaped network is subjected to three times of down-sampling encoding, and after each down-sampling, the image is subjected to two second convolutional blocks and then a GC attention module; The image after down-sampling encoding is subjected to two times of up-sampling decoding, and the result of each up-sampling decoding is spliced with the output result of the corresponding size in the down-sampling encoding process in the channel dimension, and the spliced result is subjected to a third convolutional block and then input into the next up-sampling decoding; the image after the last up-sampling is subjected to a fourth convolutional block and a convolutional layer, and then added to the original image to output the image; Step 4: The mapping relationship obtained through training is used to reconstruct the low-quality photoacoustic image, and whether the quality of the reconstructed image meets the standard is evaluated, if yes, step 5 is performed, otherwise, step 3 is re-executed; Step 5: The real-time obtained photoacoustic microscopic image is input into the trained convolutional neural network with attention mechanism for processing to obtain a high-quality image.

2. The method of claim 1, wherein the physical degradation learning based enhanced photoacoustic microscopy is characterized by, Based on the principle of photoacoustic imaging, the specific method for obtaining a plurality of groups of high-quality ground-truth photoacoustic microscopic images is: The impulsive light is irradiated to the biological tissue, and the ultrasonic transducer is used to sample the ultrasonic signal generated by the light absorption area in the tissue; The ground-truth photoacoustic microscopic image is reconstructed by maximum value projection according to the sampling information.

3. The method of claim 1, wherein the physical degradation learning based enhanced photoacoustic microscopy is characterized by, The PSF kernel is specifically: where T(ω) is the frequency response of the spherical focusing transducer, is the field pattern of the spherical focusing transducer.

4. The method of claim 1, wherein the physical degradation learning based enhanced photoacoustic microscopy is characterized by, The loss function during training of the convolutional neural network with attention mechanism is: L = aL + βL mse L = aL + βL p where L mse represents the mean square error loss, L p represents the perceptual loss, L represents the total generator loss, and a and β are the weights of the mean square error loss and the perceptual loss, respectively; The specific calculation formula of the mean square error loss is: where Y i represents the observed value of the actual photoacoustic image, Y′ i is the predicted value of the reconstructed photoacoustic image of the model, and n is the number of input samples. The specific calculation formula of the perception loss is: where q j (x) represents the features of the real photoacoustic image in the jth convolutional layer of the VGG network, q j (G(z)) represents the features of the generated image in the jth convolutional layer of the VGG network, and N represents the number of feature layers.

5. The method of claim 1, wherein the physical degradation learning based enhanced photoacoustic microscopy is characterized by, The specific formula of the GC attention module is as follows: z = x + W v2 ReLU(LN(W v1 · y)) where x is the input image, W v2 ReLU(LN(W v1 )) is the transformation module, y is the weight module, W v2 , W v1 is 1x1 convolution for capturing inter-channel dependencies, ReLU is the activation function, LN is the normalization function, N p is the total number of positions in the feature map, W k is the linear transformation matrix, and z is the output image.

6. The method of claim 1, wherein the physical degradation learning based enhanced photoacoustic microscopy is characterized by, When evaluating whether the quality of the reconstructed image meets the standard, the peak signal-to-noise ratio and the structural similarity coefficient are used for evaluation, and when the peak signal-to-noise ratio is greater than a first threshold value and the structural similarity coefficient is greater than a second threshold value, it is considered that the quality meets the standard.

Images