Deep neural network based super-resolution photoacoustic microscopy imaging method

Abstract

This invention discloses a super-resolution photoacoustic microscopy imaging method based on a deep neural network, comprising: acquiring two sets of photoacoustic signal data at high and low resolution using an optical resolution photoacoustic microscopy imaging system; reconstructing images from the two sets of acquired photoacoustic signal data; preprocessing the two sets of reconstructed original photoacoustic images to form an input sample set; building and training a deep neural network model, specifically a DenseRes-SRGAN adversarial neural network architecture optimized based on photoacoustic images; and inputting the acquired high and low resolution images into the trained deep neural network to obtain super-resolution results. The high and low resolution sampling of this invention does not skip information within the field of view. Compared with sparse data, the representative pixels are a weighted representation of each small pixel in a large step scan mode, thereby reproducing low-scale information that cannot be resolved by low resolution and large focal diameter. Compared with high-resolution reconstruction of sparse data, it has improved reconstruction generalization ability and realism.

Description

Technical Field

[0001] This invention belongs to the field of photoacoustic microscopy imaging technology, specifically a super-resolution photoacoustic microscopy imaging method based on deep neural networks. Background Technology

[0002] Photoacoustic imaging is a novel biomedical imaging technology that has been widely used in clinical practice in recent years. Compared with other biomedical imaging technologies, photoacoustic images provide specific molecular or cellular images, offering precise image information for disease diagnosis and evaluation. Photoacoustic microscopy (PAM) is a high-resolution, non-invasive imaging technique based on photoacoustic technology, and it has broad application prospects in biomedical image diagnosis.

[0003] Over the past few years, the imaging speed of optical resolution optical microscopy (OR-PAM) has been increased several times over. However, the maximum imaging speed achieved is still insufficient for many applications, such as the study of neuronal activity across the brain, where the imaging speed must be at least ten to several hundred times faster than existing methods.

[0004] Existing photoacoustic image processing methods struggle to effectively improve imaging speed. For example, modifying the scanning module and expanding the sampling interval to allow low-pixel images to learn from high-pixel images, while achieving imaging speeds several times faster than conventional photoacoustic microscopy using deep neural networks, suffers from inherent problems. The high-level feature information skimmed by interpolation sampling, when reconstructed using a trained network, often suffers from over-regularization, resulting in the large-scale omission of deep feature information. Furthermore, when the imaging objects differ significantly, the generalization ability drops drastically, weakening the effectiveness and realism of image reconstruction. Moreover, photoacoustic microscopy systems using photoacoustic detector arrays are extremely costly due to the complexity of the arrays. Summary of the Invention

[0005] To address the aforementioned technical deficiencies in the existing technology, this invention proposes a deep learning-based super-resolution photoacoustic microscopy imaging method and system.

[0006] The technical solution to achieve the objective of this invention is: a super-resolution photoacoustic microscopy imaging method based on deep neural networks, the specific steps of which are as follows:

[0007] S1: Obtain two sets of photoacoustic signal data with high and low resolution through an optical resolution photoacoustic microscopy system;

[0008] S2: Reconstruct the image from the two sets of acquired photoacoustic signal data;

[0009] S3: Preprocess the two sets of reconstructed original photoacoustic images to form the input sample set;

[0010] S4: Build and train a deep neural network model, specifically a DenseRes-SRGAN adversarial neural network architecture optimized based on photoacoustic images;

[0011] S5: Input the acquired high- and low-resolution images into the trained deep neural network to obtain super-resolution results.

[0012] Preferably, the optical resolution photoacoustic microscopy imaging system includes:

[0013] The excitation module is used to focus short-pulse laser light onto the imaging sample;

[0014] The scanning module is used to perform a full scan of the imaging range;

[0015] The acquisition module is used to acquire and amplify photoacoustic signals and transmit the data to a computer.

[0016] Preferably, the excitation module includes a pulsed laser, a pinhole filter system, two sets of mirrors, a collimating and beam expanding device, a light shield, and an objective lens. When acquiring photoacoustic signals with a large focal radius, after laser excitation, the optical path passes through the pinhole filter system, two sets of mirrors, and the objective lens, and is finally focused onto the imaging sample. When acquiring photoacoustic signals with a small focal radius, after laser excitation, the optical path passes through the pinhole filter system, two sets of mirrors, a collimating and beam expanding device, a light shield, and the objective lens, and is focused onto the imaging sample.

[0017] Preferably, the scanning step of the scanning module is different when acquiring photoacoustic signals at different focal radii.

[0018] Preferably, maximum projection is used for image reconstruction. The formula for calculating maximum projection is:

[0019] M(i,j)=max(P i,j (t:t+Δt))

[0020] Where M(i,j) is the projection value at the acquisition position (i,j), and P i,j It is the photoacoustic time-domain signal at the acquisition location (i,j), and Δt is the selected projection interval.

[0021] Preferably, the specific method for preprocessing the two sets of reconstructed original photoacoustic images is as follows:

[0022] The two reconstructed images with different resolutions are pixel-shifted along the x-axis by even-numbered rows of pixels.

[0023] The two sets of photoacoustic images with different resolutions after shifting are input into a pre-trained U-Net network to obtain images with image misalignment and noise removed.

[0024] Preferably, the DR-SRGAN adversarial neural network architecture includes a generator and a discriminator, wherein the generator includes:

[0025] (1) Photoacoustic image low-level feature extraction module: The input single-channel image is uniformly segmented and arranged in order. The segmented image is then passed through a convolutional layer and a PReLU activation function layer to obtain image features based on the parameters of the convolutional layer.

[0026] (2) Photoacoustic image high-level feature extraction module: Image features based on convolutional layer parameters are processed through five residual blocks of the same size, each residual block consisting of two sets of convolutional layers and an intermediate PReLU activation function layer; then a dense block is connected, which consists of several dense layers, each consisting of two sets of normalization layers, activation layers, and convolutional layers, and the dense layers are densely connected.

[0027] (3) Deconvolution layer: The image output from the last dense block is passed through a convolutional layer, a regularization layer, and a pointwise addition layer; then the image size is expanded by a deconvolutional layer composed of two sets of convolutional blocks, where the convolutional block is composed of a convolutional layer, a pixel shuffle layer, and a PReLU activation function layer;

[0028] (4) Reconstruction layer: Reconstruction is achieved through a single convolutional layer;

[0029] The discriminator structure is a CNN network, in which the activation function is Leaky-ReLU; the end of the CNN network uses dense blocks followed by a sigmoid function to perform binary classification and score the reconstructed image generated by the generator.

[0030] Preferably, the DenseRes-Net adversarial neural network parameter loss function includes the generator loss function and the discriminator loss function;

[0031] The generator's loss function is as follows:

[0032]

[0033] Among them, l SR For the perceptual loss function, θ G Let be the generator loss function. For a parameterized generator model, For low-resolution images with a batch size of N, For high-resolution images with a batch size of N.

[0034]

[0035] K A K B These are the modified perceptual loss ratios. For content loss function, For adversarial loss function;

[0036] The loss function of the discriminator is the binary cross-entropy loss function.

[0037] Preferably, the content loss function is as follows:

[0038]

[0039] The adversarial loss function is specifically as follows:

[0040]

[0041] in Here, W and H represent the content loss function, corresponding to the image width and height, and φ is the feature function operation performed by substituting it into a certain layer of VGG19. To counter the loss function, This is a discriminator model with parameters.

[0042] Compared with the prior art, the significant advantages of this invention are: (1) The neural network of this invention adopts a perceptual loss weighted by feature domain loss and adversarial loss, and while removing the BN layer to accelerate network training, it adds dense blocks to ensure the optimization of deep network parameters; the improved network structure has a good extraction effect on the features of each level of photoacoustic image, and based on the characteristics of this information mining technology of deep learning, high-speed super-resolution imaging can be achieved; (2) This invention is significantly different from sparse data high-resolution reconstruction. The high and low resolution sampling of this invention does not skip the information in the field of view. Compared with sparse data, the representative pixel is a weighted representation of each small pixel in the large step scanning mode, thereby reproducing the low-scale information that cannot be distinguished by low resolution and large focal diameter. Compared with sparse data high-resolution reconstruction, it has the reconstruction generalization ability and the improvement of realism; (3) This invention can make up for the objective lens experimental conditions and realize the high-resolution reconstruction that low numerical aperture objective lenses cannot achieve.

[0043] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0044] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0045] Figure 1 This is a flowchart of the high-resolution photoacoustic microscopy imaging method based on deep learning neural networks of the present invention.

[0046] Figure 2 This is a schematic diagram of the optical resolution photoacoustic microscopy imaging system of the present invention.

[0047] Figure 3 This is a schematic diagram of the network generator for the high-resolution photoacoustic microscopy imaging method based on deep learning neural networks of the present invention. Detailed Implementation

[0048] It is readily understood that, based on the technical solution of this invention, various embodiments of the invention can be conceived by those skilled in the art without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention. Rather, these embodiments are provided to enable those skilled in the art to gain a more thorough understanding of the invention. Preferred embodiments of the invention are described below in conjunction with the accompanying drawings, which form part of this application and, together with the embodiments of the invention, serve to illustrate the innovative concept of the invention.

[0049] like Figure 1 As shown, the present invention is conceived as a super-resolution photoacoustic microscopy imaging method based on deep learning, the specific steps of which are as follows:

[0050] S1: Obtain two sets of photoacoustic signal data containing information at different resolutions using an optical resolution photoacoustic microscopy system;

[0051] like Figure 2 As shown, the optical resolution photoacoustic microscopy imaging system includes:

[0052] The excitation module is used to focus a short-pulse laser onto the imaging sample to induce the photoacoustic effect; after passing through the excitation module, the system focuses the excitation light onto the imaging plane.

[0053] The scanning module is used to perform a full scan of the sample's imaging range.

[0054] The acquisition module is used to acquire and amplify photoacoustic signals and transmit the data to a computer. The acquisition module uses a PC to connect to an acquisition card to receive the photoacoustic signals from the amplifier, and performs real-time detection and acquisition on the PC.

[0055] Specifically, the excitation module includes a pulsed laser, a pinhole filter system, two sets of mirrors, a collimating and beam expanding device, a light shield, and an objective lens.

[0056] When acquiring photoacoustic signals with a large focal radius, which corresponds to a low-resolution image, after laser excitation, the optical path passes through optical devices such as a pinhole filter system, two sets of mirrors, and objective lenses, and is finally focused onto the imaging sample.

[0057] When acquiring photoacoustic signals with a small focal radius, which corresponds to a high-resolution image, after laser excitation, the optical path passes through optical devices such as a pinhole filter system, two sets of mirrors, a collimating and beam expanding device, a light shield, and an objective lens, and is finally focused onto the imaging sample.

[0058] In a further embodiment, the scanning module includes an FPGA, a servo motor, and a sample stage. The annular ultrasonic transducer and the objective lens are concentric, and the objective lens is placed at a predetermined height above the sample and the imaging plane. The sample and the imaging plane are placed on the sample stage mounted on the two-dimensional servo motor.

[0059] The laser output pulses are transmitted to the FPGA for modulation (i.e., digital processing), and then the FPGA electrical signals are transmitted to the servo motor. The servo motor controls the movement of the imaging platform mounted on the motor platform. Simultaneously, a photoacoustic signal is collected once via an ultrasonic transducer for each movement. In other words, the servo motor connects to a glass dish positioned directly below the four-corner cage objective lens, and the laser output pulses are triggered by the FPGA pulse detection function to achieve programmable control of the scanning motor.

[0060] Specifically, the servo motor external drive uses high-level trigger control. Taking the x and y axes as an example, dual FPGA pin outputs are set. When the corresponding number of scan pulses in one x-axis row triggers the servo motor, the x-axis pin for the next pulse duration is set to low, and the y-axis pin output is set to high for one step cycle pulse. This process continues until the entire imaging plane is scanned, acquiring the photoacoustic pressure signal of the two-dimensional plane containing depth information for subsequent image reconstruction.

[0061] When acquiring photoacoustic signals with a large focal radius, the focal radius is calculated by the excitation wavelength and the system NA, and the dual-axis stepping of the scanning motor is determined based on the focal radius.

[0062] When acquiring photoacoustic signals with a small focal radius, the focal radius is calculated by the excitation wavelength, the diameter of the collimating device, and the focal length of the focusing lens of the beam expander and collimator, and the dual-axis stepping of the scanning motor is determined based on the focal radius.

[0063] In this invention, the scanning module employs a combination of zoom technology and optical acoustic signals from high- and low-resolution images. Zoom technology is used to change the focal diameter, and the ideal resolution change is calculated based on prior conditions. The number of step-period pulses in the scanning module is then adjusted according to the calculated ideal resolution. The zoom technology can be interpreted as using a collimating and beam-expanding device to obtain a smaller focal diameter; essentially, it means...

[0064] d=2fλ / D

[0065] In the formula: f is the focal length of the focusing lens; λ is the wavelength of the incident beam; D is the diameter of the incident beam; d is the diameter of the zoom focal point. As can be seen from the formula, the proportional relationship between the focal point diameter d and the focal length f of the focusing lens and the laser wavelength λ is inversely proportional to the diameter D of the incident beam.

[0066] S2: Reconstruct the image from the two sets of acquired photoacoustic signal data; the present invention adopts the maximum projection image reconstruction method, the principle of which is to project the photoacoustic pressure signal of the two-dimensional plane containing depth information onto the two-dimensional plane to obtain the two-dimensional reconstructed image.

[0067] Specifically, the formula for calculating the maximum projection is:

[0068] M(i,j)=max(P i,j (t:t+Δt))

[0069] Where M(i,j) is the projection value at the acquisition position (i,j), and P i,j It is the photoacoustic time-domain signal acquired at position (i,j), and Δt is the selected projection interval. By selecting the interval, projection of different depth layers can be achieved.

[0070] S3: Preprocess the two sets of reconstructed original photoacoustic images to form the input sample set;

[0071] The photoacoustic image preprocessing method combines a trained U-Net network with pixel shifting. Because photoacoustic microscopy systems operate in conjunction with short-pulse lasers and immersion scanning equipment, they are prone to problems such as thermal expansion, physical fluctuations, mechanical noise, and difficulties in linear motor timing control, often resulting in mechanical misalignment, artifacts, and Gaussian noise affecting the photoacoustic reconstructed image. For the resulting misalignment and noise, large-scale misalignment can be restored using pixel shifting, and then U-Net is used to optimize and denoise the edges of the photoacoustic image (i.e., small-scale misalignment). The specific process of this preprocessing method is as follows:

[0072] The two reconstructed images with different resolutions (high-resolution and low-resolution photoacoustic signal images) are pixel-shifted along the x-axis by even-numbered rows of pixels. The shift distance is determined by minimizing the MSE of the prior ground-based real photoacoustic image and the misaligned image.

[0073] The shifted high- and low-resolution photoacoustic images are input into a pre-trained U-Net network to obtain preprocessed results that remove image misalignment and noise. The U-Net network is trained using the high- and low-resolution photoacoustic image datasets obtained by this system, and Gaussian noise is added to the low-resolution photoacoustic images.

[0074] S4: Build and train a deep neural network model, specifically a DenseRes-SRGAN (DR-SRGAN) adversarial neural network architecture based on photoacoustic image optimization.

[0075] The DR-SRGAN adversarial neural network architecture is based on CNN and trained using the GAN method to achieve image super-resolution reconstruction. The DR-SRGAN architecture includes a generator and a discriminator. The generator's main body consists of a series of residual blocks and dense blocks, with a subpixel module added at the end of the generator model.

[0076] In a further embodiment, the generator network includes:

[0077] (1) Photoacoustic image low-level feature extraction module: The input single-channel image is uniformly segmented and arranged in order. The segmented image is then passed through a convolutional layer and a PReLU activation function layer to obtain image features based on the parameters of the convolutional layer. The convolutional layer and the activation function layer together constitute an image feature extraction convolutional block without a pooling layer.

[0078] (2) Photoacoustic image high-level feature extraction module: Image features based on convolutional layer parameters are processed through five residual blocks of the same size, each residual block consisting of two sets of convolutional layers and an intermediate PReLU activation function layer; then a dense block is connected, which consists of several dense layers, each consisting of two sets of normalization layers, activation layers, and convolutional layers, and the dense layers are densely connected; the residual blocks and the dense blocks together constitute the high-level feature extraction layer; since the residual blocks remove the BN layer, the imaging speed is improved, and the dense layers ensure the optimization of network parameter gradients;

[0079] (3) Deconvolution layer: The image output from the last dense block is passed through a convolutional layer, a regularization layer, and a pointwise addition layer; then the image size is expanded by a deconvolutional layer composed of two sets of convolutional blocks, where the convolutional block is composed of a convolutional layer, a pixel shuffle layer, and a PReLU activation function layer;

[0080] (4) Reconstruction layer: Finally, reconstruction is performed through a single convolutional layer.

[0081] The generator architecture is primarily a Res-Net residual network. See the generator architecture below. Figure 3 .

[0082] In a further embodiment, the discriminator structure is a CNN network, where the activation function uses Leaky-ReLU; the end of the CNN network uses dense blocks followed by a sigmoid function to perform binary classification on the reconstructed image I generated by the generator. SR Scoring is performed. Convolutional layers that do not change the matrix size are added to the dense blocks to further implement fully connected functionality.

[0083] In a further embodiment, the DenseRes-Net adversarial neural network parameter loss function includes the generator loss function and the discriminator loss function.

[0084] The generator's loss function is as follows:

[0085] Among them, l SR The perceptual loss function is used.

[0086] K A K B The modified perceived loss ratio.

[0087] Content loss:

[0088] Combat losses:

[0089] The loss function of the discriminator is the binary cross-entropy loss function.

[0090] The adversarial loss function is basically the same as the discriminator's loss function, except that the '0' and '1' are swapped when calculating the binary cross-entropy. The purpose of this is to optimize the performance of the generator and the discriminator respectively.

[0091] The training process of the DR-SRGAN adversarial neural network is divided into two stages. Preprocessed high- and low-resolution images are input into the DR-SRGAN adversarial neural network.

[0092] First, in the first stage, only the generator network is trained, while the parameters of the discriminator network are fixed. In this stage, the generator network is trained once using content loss and adversarial loss, while simultaneously outputting a super-resolution image corresponding to the low-resolution image. Then, the process jumps to the second stage. In the second stage, the parameters of the generator network are fixed, and the discriminator network is trained. In this stage, the output from the first stage, i.e., the super-resolution image, is passed through the discriminator network, and a numerical value ranging from [0,1] is output, representing the similarity between the super-resolution image and the high-resolution image. Next, the discriminator network is trained using the discriminator loss function (the training method involves calculating the binary cross-entropy loss function by performing a batch of discriminator outputs with '0' values ​​of the same size. Because the discriminator network needs to identify the super-resolution image, [0,1] represents the gradual transition from 'unlike' to 'like'), and then the process jumps back to the first stage.

[0093] S5: After preprocessing the acquired high- and low-resolution images using the method in step S3, input them into the trained deep neural network model to obtain super-resolution results.

[0094] Example

[0095] A deep learning-based super-resolution photoacoustic microscopy imaging method, the specific steps of which are as follows:

[0096] S1: Obtain two sets of photoacoustic signal data containing information at different resolutions using an optical resolution photoacoustic microscopy system;

[0097] When acquiring photoacoustic signals with a large focal radius, after laser excitation, the optical path passes through optical devices such as a pinhole filter system, two sets of mirrors, and objective lenses, and is finally focused onto the imaging sample.

[0098] When acquiring photoacoustic signals with a small focal radius, after laser excitation, the optical path passes through optical devices such as a pinhole filter system, two sets of mirrors, a collimating and beam expanding device, a light shield, and an objective lens, and is finally focused onto the imaging sample.

[0099] S2: Reconstruct the image from the two sets of acquired photoacoustic signal data; the present invention adopts the maximum projection image reconstruction method, the principle of which is to project the photoacoustic pressure signal of the two-dimensional plane containing depth information onto the two-dimensional plane to obtain the two-dimensional reconstructed image.

[0100] S3: Preprocess the two sets of reconstructed original photoacoustic images to form the input sample set;

[0101] The photoacoustic image preprocessing method combines a trained U-Net network with pixel shifting. Because photoacoustic microscopy systems operate in conjunction with short-pulse lasers and immersion scanning equipment, they are prone to problems such as thermal expansion, physical fluctuations, mechanical noise, and difficulties in linear motor timing control, often resulting in mechanical misalignment, artifacts, and Gaussian noise affecting the photoacoustic reconstructed image. For the resulting misalignment and noise, large-scale misalignment can be restored using pixel shifting, and then U-Net is used to optimize and denoise the edges of the photoacoustic image (i.e., small-scale misalignment). The specific process of this preprocessing method is as follows:

[0102] The two reconstructed images with different resolutions (high-resolution and low-resolution photoacoustic signal images) are pixel-shifted along the x-axis by even-numbered rows of pixels. The shift distance is determined by minimizing the MSE of the prior ground-based real photoacoustic image and the misaligned image.

[0103] The shifted high- and low-resolution photoacoustic images are input into a pre-trained U-Net network to obtain preprocessed results that remove image misalignment and noise. The U-Net network is trained using the high- and low-resolution photoacoustic image datasets obtained by this system, and Gaussian noise is added to the low-resolution photoacoustic images.

[0104] S4: Build and train a deep neural network model, specifically a DenseRes-SRGAN (DR-SRGAN) adversarial neural network architecture based on photoacoustic image optimization.

[0105] The DR-SRGAN adversarial neural network architecture is based on CNN and trained using the GAN method to achieve image super-resolution reconstruction. The DR-SRGAN architecture includes a generator and a discriminator. The generator's main body consists of a series of residual blocks and dense blocks, with a subpixel module added at the end of the generator model.

[0106] GeneratorNetwork structure:

[0107] (1) Low-level feature extraction module: The input single-channel image is uniformly divided into 32*32 pixel sizes and arranged in order. The resulting segmented image is passed through a convolutional layer with a kernel size of 9*9, 64 channels, and a stride of 1, and a PReLU activation function layer with a learning parameter α of 1 and an initial value of 0.25 to obtain image features based on the convolutional layer parameters. The convolutional layer and the activation function layer together constitute an image feature extraction convolutional block, without a pooling layer.

[0108] (2) High-level feature extraction module: Next, the network passes through five residual blocks of the same size. Each residual block consists of two sets of 3*3 kernel layers with 64 channels and a stride of 1 convolutional layer, and two sets of PReLU activation function layers in the middle. Then, it passes through four dense blocks with the number of dense layers [2, 4, 2, 4], and the number of features (channels) of each dense layer is 32. The dense layer consists of two sets of normalization layers, activation layers, and convolutional layers. The five residual blocks and the four dense blocks together constitute the high-level feature extraction layer. Since the BN layer is removed from the residual blocks, the imaging speed is improved. Then, the dense layers ensure the optimization of the network parameter gradient.

[0109] (3) Deconvolution layer: The last residual block is passed through a convolutional layer with a kernel size of 3*3, a number of channels of 64, and a stride of 1, a regularization layer, and a pointwise addition layer; then the image size is expanded by a deconvolutional layer composed of two sets of convolutional blocks, wherein the convolutional block consists of a convolutional layer with a kernel size of 3*3, a number of channels of 256, a stride of 1, a 2× upsampling pixel shuffle layer, and a PReLU activation function layer;

[0110] (4) Reconstruction layer: Finally, reconstruction is performed using a single convolutional layer with a kernel size of 9*9, a channel number of 3, and a step size of 1.

[0111] The generator architecture is primarily a Res-Net residual network. See the generator architecture below. Figure 3 .

[0112] Discriminator Network Structure:

[0113] The discriminator is structured as a CNN network, using Leaky-ReLU as the activation function. The network first passes through a convolutional layer with a kernel size of 3*3, 64 channels, and a stride of 1, followed by a Leaky ReLU activation function layer. Then, it passes through another convolutional layer with a kernel size of 3*3, 64 channels, and a stride of 2, a BN layer, and another Leaky ReLU activation function layer. The image features are extracted by alternating between two types of convolutional blocks. The first convolutional block consists of a convolutional layer with a kernel size of 3*3, 128 channels, and a stride of 1, a BN layer, and a Leaky ReLU layer. The second convolutional block has a convolutional layer with a kernel size of 3*3, 64 channels, and a stride of 2. After passing through each of the two convolutional layers, the number of channels in the next convolutional layer is doubled, adding downsampled information to the dimension. Finally, the network ends with a (1024-1) Dense layer, a Leaky ReLU activation function layer, followed by a (1-1) Dense layer and a sigmoid function to form a Dense block, which performs a binary classification for I. LR and I HR Scoring is performed for the discriminant regression of the discriminator.

[0114] S5: After preprocessing the acquired high- and low-resolution images using the method in step S3, input them into the trained deep neural network model to obtain super-resolution results.

[0115] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0116] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as including all features in the exemplary embodiments as essential technical features of the claims of this patent.

[0117] It should be understood that the modules, units, components, etc., included in the device of one embodiment of the present invention can be adaptively changed to be placed in a device different from that embodiment. Different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

Claims

1. A super-resolution photoacoustic microscopy method based on deep neural networks, characterized in that, The specific steps are as follows: S1: Obtain two sets of photoacoustic signal data with high and low resolution through an optical resolution photoacoustic microscopy system; S2: Reconstruct the image from the two sets of acquired photoacoustic signal data; S3: Preprocess the two sets of reconstructed original photoacoustic images to form the input sample set; S4: Build and train a deep neural network model. Specifically, the deep neural network model is a densely connected super-resolution generative adversarial neural network architecture optimized based on photoacoustic images. This architecture includes a generator and a discriminator. The generator includes: (1) Photoacoustic image low-level feature extraction module: The input single-channel image is uniformly segmented and arranged in order. The segmented image is then passed through a convolutional layer and a PReLU activation function layer to obtain image features based on the parameters of the convolutional layer. (2) Photoacoustic image high-level feature extraction module: Image features based on convolutional layer parameters are processed through five residual blocks of the same size, each residual block consisting of two sets of convolutional layers and an intermediate PReLU activation function layer; then a dense block is connected, which consists of several dense layers, each consisting of two sets of normalization layers, activation layers, and convolutional layers, and the dense layers are densely connected. (3) Deconvolution layer: The image output from the last dense block is passed through a convolutional layer, a regularization layer, and a pointwise addition layer; then the image size is expanded by a deconvolutional layer composed of two sets of convolutional blocks, where the convolutional block is composed of a convolutional layer, a pixel shuffle layer, and a PReLU activation function layer; (4) Reconstruction layer: Reconstruction is achieved through a single convolutional layer; The discriminator structure is a CNN network, in which the activation function is Leaky-ReLU; the end of the CNN network uses dense blocks followed by a sigmoid function to perform binary classification and score the reconstructed image generated by the generator. The loss function of the densely connected super-resolution generative adversarial neural network architecture parameters includes the loss function of the generator and the loss function of the discriminator; The generator's loss function is as follows: Among them, l SR For the perceptual loss function, θ G Let be the generator loss function. For a parameterized generator model, For low-resolution images with a batch size of N, For high-resolution images with a batch size of N; K A K B These are the modified perceptual loss ratios. For content loss function, For adversarial loss function; The loss function of the discriminator is the binary cross-entropy loss function; The content loss function is specifically as follows: The adversarial loss function is specifically as follows: in, Here, W and H represent the content loss function, respectively, and φ represents the image width and height. φ is the feature function operation performed by substituting it into a certain layer of VGG19. To counter the loss function, This is a discriminator model with parameters; S5: Input the acquired high- and low-resolution images into the trained deep neural network to obtain super-resolution results.

2. The super-resolution photoacoustic microscopy method based on deep neural networks according to claim 1, characterized in that, The optical resolution photoacoustic microscopy imaging system includes: The excitation module is used to focus short-pulse laser light onto the imaging sample; The scanning module is used to perform a full scan of the imaging range; The acquisition module is used to acquire and amplify photoacoustic signals and transmit the data to a computer.

3. The super-resolution photoacoustic microscopy method based on deep neural networks according to claim 2, characterized in that, The excitation module includes a pulsed laser, a pinhole filter system, two sets of mirrors, a collimating and beam expanding device, a light shield, and an objective lens. When acquiring photoacoustic signals with a large focal radius, after laser excitation, the optical path passes through the pinhole filter system, two sets of mirrors, and the objective lens, and is finally focused onto the imaging sample. When acquiring photoacoustic signals with a small focal radius, after laser excitation, the optical path passes through the pinhole filter system, two sets of mirrors, a collimating and beam expanding device, a light shield, and the objective lens, and is focused onto the imaging sample.

4. The super-resolution photoacoustic microscopy method based on deep neural networks according to claim 1, characterized in that, When acquiring photoacoustic signals at different focal radii, the scanning step of the scanning module is different.

5. The super-resolution photoacoustic microscopy method based on deep neural networks according to claim 1, characterized in that, Image reconstruction is performed using maximum projection. The formula for calculating maximum projection is as follows: M(i,j)=max(P i,j (t:t+Δt)) Where M(i,j) is the projection value at the acquisition position (i,j), and P i,j It is the photoacoustic time-domain signal at the acquisition location (i,j), and Δt is the selected projection interval.

6. The super-resolution photoacoustic microscopy method based on deep neural networks according to claim 1, characterized in that, The specific methods for preprocessing the two sets of reconstructed original photoacoustic images are as follows: The two reconstructed images with different resolutions are pixel-shifted along the x-axis by even-numbered rows of pixels. The two sets of photoacoustic images with different resolutions after shifting are input into a pre-trained U-Net network to obtain images with image misalignment and noise removed.

Images