Enhanced photoacoustic microscopic imaging method based on physical degradation learning

Abstract

The invention discloses an enhanced photoacoustic microscopic imaging method based on physical degradation learning, which comprises the following steps of: on the basis of a photoacoustic imaging principle, degrading an image of a real photoacoustic microscopic image to the ground into a low-quality photoacoustic microscopic image through physical model degradation learning with variability. A high-quality photoacoustic microscopic imaging image is used as a target of supervised training, deep learning is used as a basis, the high-quality photoacoustic microscopic imaging image is input into a self-designed convolutional neural network with an attention mechanism for network training, the quality of a reconstructed image is evaluated until the reconstruction quality reaches the standard, and otherwise, the above steps are executed again. And inputting a real low-quality photoacoustic image into the trained neural network to obtain a reconstructed high-resolution photoacoustic image. According to the method, the resolution ratio and the contrast ratio of the deep photoacoustic microscopic image can be effectively improved, meanwhile, the photoacoustic microscopic image can be denoised, the dependence on the data set can be effectively reduced due to the fact that the data set is obtained through the physical model, and the reconstruction effect under different environments can be improved.

Description

Technical Field

[0001] The invention belongs to the technical field of photoacoustic microscopy, and in particular is an enhanced photoacoustic microscopy method based on physical degradation learning. Background Art

[0002] Photoacoustic imaging technology is a new biomedical imaging method with great clinical application value developed in recent years. It is based on the photoacoustic effect and can specifically detect the absorption information of biological tissues. Compared with other biomedical imaging technologies, photoacoustic imaging can achieve specific molecular or cell imaging, provide accurate imaging information for disease diagnosis and evaluation, and has non-destructive, high-resolution, high-contrast, low-cost, and specific spectral recognition capabilities. Photoacoustic Microscopy (PAM) is a high-resolution non-invasive imaging method developed on the basis of photoacoustic imaging, and has shown broad application prospects in the field of biomedical imaging diagnosis.

[0003] Under the condition of safe laser energy flux density, the penetration depth of photoacoustic microscopy is limited by the strong scattering of biological tissues and is basically within one free path of light transmission (about 1mm for biological tissues). Photoacoustic microscopy in shallower areas can obtain better image resolution and contrast. If the focusing depth of photoacoustic microscopy is increased, the spatial resolution will be reduced, the background noise will be enhanced, and the image contrast and quality will be affected. With the rise of deep learning in recent years, the powerful functions of deep learning can well solve problems that are difficult to solve with traditional methods. However, the use of supervised training deep learning methods is limited by the high cost of obtaining data sets and the difficulty of obtaining them. The quality of reconstructed images by unsupervised deep learning is not as good as that of supervised training. Summary of the invention

[0004] The present invention proposes an enhanced photoacoustic microscopy imaging method based on physical degradation learning to solve the problem of reduced resolution and contrast of photoacoustic microscopy images due to limited penetration depth in photoacoustic microscopy imaging and can also effectively solve the problem of limited data sets in deep learning.

[0005] The technical solution to achieve the purpose of the present invention is an enhanced photoacoustic microscopy imaging method based on physical degradation learning, comprising the following steps:

[0006] Step 1: Based on the principle of photoacoustic imaging, multiple sets of high-quality real photoacoustic microscopy images of the earth are obtained, and the real photoacoustic microscopy images of the earth are degraded into low-quality photoacoustic microscopy images through a physical degradation model with variability;

[0007] Step 2: Use high-quality ground-truth photoacoustic microscopy images that have not been subjected to the physical degradation model as training targets;

[0008] Step 3: Match the low-quality photoacoustic microscopy image after the physical degradation model with the high-quality ground-truth photoacoustic microscopy image and input them into the convolutional neural network with attention mechanism for network training, and obtain the corresponding mapping relationship through training;

[0009] Step 4: Reconstruct the low-quality photoacoustic image through the mapping relationship obtained through training, and evaluate whether the reconstructed image quality meets the standard. If it meets the standard, proceed to step 5, otherwise, re-execute step 3;

[0010] Step 5: Use the real-time photoacoustic microscopy images to input the trained convolutional neural network with attention mechanism for processing to obtain high-quality images.

[0011] Preferably, based on the principle of photoacoustic imaging, the specific method for obtaining multiple sets of high-quality real photoacoustic microscopic images of the earth is:

[0012] The pulse laser is irradiated onto the biological tissue, and the ultrasonic transducer is used to sample the ultrasonic signal generated by the light absorption area in the tissue;

[0013] The real photoacoustic microscopy image of the ground is reconstructed by maximum projection based on the sampling information.

[0014] Preferably, the specific formula for degrading the real photoacoustic microscopy image of the earth into a low-quality photoacoustic microscopy image through a physical degradation model with variability is:

[0015]

[0016] In the formula, I m1 is the image reconstructed using maximum projection, I m is the output image after the physical model with variability, h refers to the PSF kernel, α is the compensation function, and η is the total noise.

[0017] Preferably, the compensation function is specifically:

[0018] α=μm 1+ξz

[0019] Among them, μ is the attenuation coefficient, ξ is the compensation coefficient, m is the intensity interference term of the attenuation coefficient, and z is the intensity interference term of the compensation coefficient.

[0020] Preferably, the PSF kernel is specifically:

[0021]

[0022] Where T(ω) is the frequency of the spherical focusing transducer to the sensor, Field pattern of a spherically focused transducer.

[0023] Preferably, when training a convolutional neural network with an attention mechanism, the image processing process is:

[0024] The low-quality photoacoustic microscopy image after the physical degradation model is input into the convolutional neural network, where the photoacoustic microscopy image passes through the first convolutional block and then passes through multiple residual blocks;

[0025] The image after the first residual block is input into the U-type network, which performs three downsampling encodings. After each downsampling, it passes through two second convolution blocks and then the GC attention module.

[0026] The downsampled encoded image undergoes two upsampling decodings. The result of each upsampling decoding is spliced ​​with the output result of the corresponding size in the downsampling encoding process in the channel dimension. The spliced ​​result passes through a third convolution block and then serves as the input for the next upsampling decoding. The image after the last upsampling passes through a fourth convolution block and a convolution layer, and is added to the original image to output the image.

[0027] Preferably, the first residual block is composed of two convolution blocks and a skip connection, and the specific formula is as follows:

[0028] f(x)=g(x)+x

[0029] Where x is the input image, g(x) is the image after two convolution operations, and f(x) is the input image. Preferably, the loss function of the convolutional neural network with attention mechanism during training is:

[0030] L=αL mse +βL p

[0031] Among them, L mse represents the mean square error loss, L p represents the perceptual loss, L represents the total generator loss, α and β are the weights of the mean square error loss and the perceptual loss;

[0032] The specific calculation formula of mean square error loss is:

[0033]

[0034] Among them, Y i represents the observed value of the actual photoacoustic image, Y′ i is the predicted value of the reconstructed photoacoustic image of the model, and n is the number of input samples;

[0035] The specific calculation formula of perceptual loss is:

[0036]

[0037] Among them, qi (x) represents the features of the real photoacoustic image in the i-th convolutional layer of the VGG network, q i (G(z)) represents the features of the generated image in the i-th convolutional layer of the VGG network, and N represents the number of feature layers.

[0038] Preferably, the specific formula of the GC attention module is as follows:

[0039] z=x+W v2 ReLU(LN(W v1 ·y))

[0040]

[0041] Where x is the input image, W v2 ReLU(LN(W v1 (·)) is the transformation module, y is the weight module, W v2 , W v1 It is a 1x1 convolution used to capture the dependencies between channels, ReLU is the activation function, LN is the normalization function, N p The total number of positions in the feature map, W k is the linear transformation matrix, z is the output image.

[0042] Preferably, when evaluating whether the reconstructed image quality meets the standard, the peak signal-to-noise ratio and the structural similarity coefficient are used for evaluation. When the peak signal-to-noise ratio is greater than a first threshold and the structural similarity coefficient is greater than a second threshold, it is considered that the quality meets the standard.

[0043] Compared with the prior art, the present invention has the following significant advantages: 1) In terms of imaging quality, the present invention can well handle the reduction in resolution and contrast of photoacoustic microscopy images caused by scattering. The present invention can well improve the quality of low-quality images; 2) In terms of imaging depth, the present invention can improve deeper and difficult-to-identify images, so the present invention can further enhance the effective detection depth of photoacoustic microscopy imaging; 3) In terms of data set collection, the present invention solves the problem of difficulty in collecting supervised training data sets, so the present invention can effectively solve the problem of limited data sets for deep learning; 4) In terms of equipment cost, the present invention can reduce the equipment cost of the laser because it can reduce the requirements for laser conditions to a certain extent.

[0044] The present invention is described in further detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The flowchart of the enhanced photoacoustic microscopy imaging method based on physical degradation learning of the present invention.

[0046] Figure 2 Schematic diagram of the overall network framework of the enhanced photoacoustic microscopy imaging method based on physical degradation learning of the present invention.

[0047] Figure 3 Schematic diagram of each network module in the enhanced photoacoustic microscopy imaging method based on physical degradation learning of the present invention.

[0048] Figure 4 Schematic diagram of the effect of the enhanced photoacoustic microscopy imaging method based on physical degradation learning of the present invention. DETAILED DESCRIPTION

[0049] A method for enhanced photoacoustic microscopy based on physical degradation learning solves the problem of difficulty in obtaining data sets through a physical degradation model, and improves the image quality of photoacoustic microscopy through a deep learning-based method. In the training process, the parameters of the physical degradation model are full of randomness and within a reasonable range to improve the generalization of the method. Figure 1 As shown, the specific steps of the present invention are as follows:

[0050] Step 1: Based on the principle of photoacoustic imaging, multiple sets of high-quality ground-based photoacoustic microscopy imaging results are obtained, and the ground-based photoacoustic microscopy images are degraded into low-quality photoacoustic microscopy images through a variable physical degradation model;

[0051] Specifically, pulsed laser is irradiated to biological tissue, and ultrasonic transducer is used to sample ultrasonic signals generated by light absorption area in the tissue, wherein the ultrasonic signals generated by light excitation are photoacoustic signals. The above method is used to collect and reconstruct the true photoacoustic microscopic image of the earth by maximum projection.

[0052] In a further embodiment, the specific formula for reconstructing the true photoacoustic microscopic image of the earth using maximum projection is:

[0053] M(i,j)=max(P i,j (t:t+Δt))

[0054] Where (i, j) is the coordinate of the data collected in the surface scanning excitation mode, M(i, j) is the maximum projection at the acquisition position (i, j), P i,j is the photoacoustic time domain signal at the acquisition position (i, j), t is the starting time of the selected interval, Δt is the interval of the selected projection, and the maximum value projection of different depth layers can be achieved by selecting the interval.

[0055] In a further embodiment, the construction process of the physical degradation learning model is:

[0056] The role of the convolution model is to reduce the resolution of the photoacoustic microscopy image using the Rayleigh-Sommerfeld diffraction formula. The simplified formula is as follows:

[0057]

[0058] It's TRON The Fourier transform at a point in space, is the aperture weighting function The above formula can also be expressed as the field pattern of a planar circular transducer.

[0059] By combining the field pattern of the planar circular transducer with the spherical compensation function, we obtain The spherical compensation function is:

[0060]

[0061] The spherical focusing transducer is then integrated toward the sensor frequency T(ω). The integration range is the bandwidth of the field pattern of the ultrasonic transducer spherical focusing transducer, so the point spread function PSF kernel is obtained as follows:

[0062]

[0063] When adding a compensation function to reconstruct the image data set to different depths, the image needs to be attenuated to different degrees. The specific compensation function is:

[0064] α=μm 1+ξz

[0065] Among them, μ is the attenuation coefficient and ξ is the compensation coefficient.

[0066] The role of the noise model is to add noise to the degradation process to simulate the real environment. The specific formula is as follows:

[0067] η=η1+η2+η3

[0068] Where η is the total noise, η1 is Gaussian noise, η2 is Rayleigh noise, and η3 is Poisson noise. And the size of the noise is not fixed, it is full of randomness. Integrating the above formulas, we can get:

[0069]

[0070] I m1 is the image reconstructed using maximum projection, I m It is the output image after the physical model with variability.

[0071] Step 2: Use high-quality ground-truth photoacoustic microscopy images that have not been subjected to the physical degradation model as network training targets;

[0072] Specifically, photoacoustic microscopy images are manually selected, and high-resolution and high-contrast photoacoustic microscopy images are selected as training targets to ensure the training quality of the convolutional neural network with an attention mechanism.

[0073] Step 3: Match the low-quality photoacoustic microscopy image with the high-quality photoacoustic microscopy image and input them into a convolutional neural network with an attention mechanism for network training, and obtain the corresponding mapping relationship through training;

[0074] like Figure 2 and Figure 3 As shown in the figure, the image processing process of the convolutional neural network with attention mechanism is as follows:

[0075] Step 301: Input the low-quality photoacoustic microscopy image that has passed the physical degradation model into a convolutional neural network. In the convolutional neural network, the photoacoustic image first passes through a convolution block and then passes through multiple residual blocks; the convolution block is composed of a convolution layer and an activation function, and the residual block is composed of two convolution blocks, two batch normalization layers and an activation function.

[0076] Step 302: The image after the residual block is input into the U-type network. The U-type network performs three downsampling encodings. After each downsampling, it passes through two convolution blocks, which are composed of a convolution layer, a batch normalization layer and an activation function, and then passes through the GC attention module.

[0077] Step 303: After downsampling, there are two upsampling decoding processes. The result of each upsampling decoding is spliced ​​with the output result of the corresponding size in the downsampling process in the channel dimension. The spliced ​​result passes through the same convolution block and then serves as the input for the next upsampling decoding. The last upsampling passes through a convolution block and then passes through a convolution layer and then adds it to the original image, and then outputs the image. Finally, the convergence of the entire network model is controlled by the loss function.

[0078] Specifically, the residual block consists of two convolution blocks and a jump connection. The specific formula is as follows:

[0079] f(x)=g(x)+x

[0080] Among them, x is the input image, g(x) is the image after two convolution operations, and f(x) is the input image. The main function is to increase the depth of the convolutional neural network while maintaining the convergence of the neural network.

[0081] Specifically, the attention module is a GC attention module, and the specific formula is as follows:

[0082] z=x+W v2 ReLU(LN(W v1 ·y))

[0083]

[0084] Where x is the input image, W v2 ReLu(LN(W v1 (·)) is the conversion module, and y is the weight module. Its main function is to enhance the analysis and recognition of microvessels and provide the possibility for subsequent image reconstruction.

[0085] Specifically, the loss function of the convolutional neural network with attention mechanism is:

[0086] L=αL mse +βL p

[0087] Where L mse represents the mean square error loss, L p represents the perceptual loss, L represents the total generator loss, and α and β are the weights of the mean square error loss and the perceptual loss.

[0088] Preferably, in the method for enhancing the quality of photoacoustic microscopy imaging by physical degradation learning, the specific calculation formula of mean square error loss is:

[0089]

[0090] Among them, Y i represents the observed value of the actual photoacoustic image, Y′ i is the predicted value of the reconstructed photoacoustic image of the model, and n is the number of input samples.

[0091]

[0092] Among them, q i (x) represents the features of the real photoacoustic image in the i-th convolutional layer of the VGG network, q i (G(z)) represents the features of the generated image in the i-th convolutional layer of the VGG network, and N represents the number of feature layers.

[0093] Step 4: Reconstruct the low-quality photoacoustic image through the mapping relationship obtained through training, and evaluate the quality of the reconstructed image until the reconstruction quality meets the standard, otherwise re-execute step 3.

[0094] Specifically, in combination with Figure 4 The reconstructed image can improve the resolution and contrast of the image. The peak signal-to-noise ratio and structural similarity coefficient are used to evaluate the quality of the reconstructed image. When the peak signal-to-noise ratio is greater than 30 and the structural similarity coefficient is greater than 0.9, the quality is considered to be up to standard.

[0095] Step 5: Use the real-time photoacoustic microscopy images to input the trained convolutional neural network with attention mechanism for processing to obtain high-quality images.

[0096] As described in the background technology section, given that under the condition of safety-limited laser energy flux density, the penetration depth of photoacoustic microscopy is limited by the strong scattering of biological tissues, which is basically within one free path of light transmission (about 1 mm for biological tissues). The present invention provides an enhanced photoacoustic microscopy imaging method based on physical degradation learning to solve the problems of poor resolution and contrast encountered when performing deep imaging with photoacoustic microscopy, and can also reduce the requirements for laser equipment. The present invention provides an enhanced photoacoustic microscopy imaging method based on physical degradation learning, which achieves better imaging quality of deep tissues in photoacoustic microscopy imaging, and reducing laser energy can also obtain good imaging quality, so it can also reduce the requirements for laser equipment.

Claims

1. An enhanced photoacoustic microscopy imaging method based on physical degradation learning, characterized in that: The following steps are involved: Step 1: Based on the principle of photoacoustic imaging, multiple sets of high-quality real photoacoustic microscopy images of the earth are obtained, and the real photoacoustic microscopy images of the earth are degraded into low-quality photoacoustic microscopy images through a physical degradation model with variability; Step 2: Use high-quality ground-truth photoacoustic microscopy images that have not been subjected to the physical degradation model as training targets; Step 3: Match the low-quality photoacoustic microscopy image after the physical degradation model with the high-quality ground-truth photoacoustic microscopy image and input them into the convolutional neural network with attention mechanism for network training, and obtain the corresponding mapping relationship through training; Step 4: Reconstruct the low-quality photoacoustic image through the mapping relationship obtained through training, and evaluate whether the reconstructed image quality meets the standard. If it meets the standard, proceed to step 5, otherwise, re-execute step 3; Step 5: Use the real-time photoacoustic microscopy images to input the trained convolutional neural network with attention mechanism for processing to obtain high-quality images.

2. The enhanced photoacoustic microscopy imaging method based on physical degradation learning according to claim 1, characterized in that: Based on the principle of photoacoustic imaging, the specific method for obtaining multiple sets of high-quality real photoacoustic microscopic images of the earth is: The pulse laser is irradiated onto the biological tissue, and the ultrasonic transducer is used to sample the ultrasonic signal generated by the light absorption area in the tissue; The real photoacoustic microscopy image of the ground is reconstructed by maximum projection based on the sampling information.

3. The enhanced photoacoustic microscopy imaging method based on physical degradation learning according to claim 2, characterized in that: The specific formula for degrading the ground-truth photoacoustic microscopy image into a low-quality photoacoustic microscopy image through a physical degradation model with variability is: In the formula, I m1 is the image reconstructed using maximum projection, I m is the output image after the physical model with variability, h refers to the PSF kernel, α is the compensation function, and η is the total noise.

4. The enhanced photoacoustic microscopy imaging method based on physical degradation learning according to claim 3, characterized in that: The compensation function is specifically: α=μm 1+ξz Among them, μ is the attenuation coefficient, ξ is the compensation coefficient, m is the intensity interference term of the attenuation coefficient, and z is the intensity interference term of the compensation coefficient.

5. The enhanced photoacoustic microscopy imaging method based on physical degradation learning according to claim 1, characterized in that: The PSF kernel is specifically: Where T(ω) is the frequency of the spherical focusing transducer to the sensor, Field pattern of a spherically focused transducer.

6. The enhanced photoacoustic microscopy imaging method based on physical degradation learning according to claim 5, characterized in that: When training a convolutional neural network with an attention mechanism, the image processing process is: The low-quality photoacoustic microscopy image after the physical degradation model is input into the convolutional neural network, where the photoacoustic microscopy image passes through the first convolutional block and then passes through multiple residual blocks; The image after the first residual block is input into the U-type network, which performs three downsampling encodings. After each downsampling, it passes through two second convolution blocks and then the GC attention module. The downsampled encoded image undergoes two upsampling decodings. The result of each upsampling decoding is spliced ​​with the output result of the corresponding size in the downsampling encoding process in the channel dimension. The spliced ​​result passes through a third convolution block and then serves as the input for the next upsampling decoding. The image after the last upsampling passes through a fourth convolution block and a convolution layer, and is added to the original image to output the image.

7. The enhanced photoacoustic microscopy imaging method based on physical degradation learning according to claim 6, characterized in that: The first residual block consists of two convolutional blocks and a jump connection. The specific formula is as follows: f(x)=g(x)+x Among them, x is the input image, g(x) is the image after two convolution operations, and f(x) is the input image.

8. The enhanced photoacoustic microscopy imaging method based on physical degradation learning according to claim 6, characterized in that: The loss function of the convolutional neural network with attention mechanism during training is: L=αL mse +βL p Among them, L mse represents the mean square error loss, L p represents the perceptual loss, L represents the total generator loss, α and β are the weights of the mean square error loss and the perceptual loss; The specific calculation formula of mean square error loss is: Among them, Y i represents the observed value of the actual photoacoustic image, Y′ i is the predicted value of the reconstructed photoacoustic image of the model, and n is the number of input samples; The specific calculation formula of perceptual loss is: Among them, q i (x) represents the features of the real photoacoustic image in the i-th convolutional layer of the VGG network, q i (G(z)) represents the features of the generated image in the i-th convolutional layer of the VGG network, and N represents the number of feature layers.

9. The enhanced photoacoustic microscopy imaging method based on physical degradation learning according to claim 6, characterized in that: The specific formula of the GC attention module is as follows: z=x+W v2 ReLU(LN(W v1 ·y)) Where x is the input image, W v2 ReLU(LN(W v1 (·)) is the transformation module, y is the weight module, W v2 , W v1 It is a 1x1 convolution used to capture the dependencies between channels, ReLU is the activation function, LN is the normalization function, N p The total number of positions in the feature map, W k is the linear transformation matrix, z is the output image.

10. The enhanced photoacoustic microscopy imaging method based on physical degradation learning according to claim 1, characterized in that: The peak signal-to-noise ratio and the structural similarity coefficient are used to evaluate whether the reconstructed image quality meets the standard. When the peak signal-to-noise ratio is greater than the first threshold and the structural similarity coefficient is greater than the second threshold, the quality is considered to meet the standard.

Images