Unsupervised image registration method for photoacoustic fast scanning system

Abstract

The invention discloses an unsupervised image registration method for a photoacoustic fast scanning system. The unsupervised image registration method comprises the following steps: acquiring data by using a fast scanning photoacoustic microscope; carrying out preprocessing steps of hooping, normalization, filtering, affine transformation and the like on the data to form a photoacoustic image data set; constructing a registration model based on a convolutional neural network and a space conversion module, fusing an attention mechanism to distribute weights for different pixels, adopting a two-stage training strategy, and introducing mutual information and a smooth item as a loss function; and predicting the registered photoacoustic image based on the training weight. According to the method, the registered image can be predicted in an extremely short time under the condition that a truth value is not input, an attention mechanism is introduced into the convolutional network, key factors are endowed with higher weights during feature extraction, the registration efficiency is effectively improved, mutual information is introduced as a loss function, the probability relation between pixel intensities is effectively modeled, and the registration accuracy is improved. And the similarity of the photoacoustic images acquired at different time points can be evaluated more accurately.

Description

Technical Field

[0001] The invention relates to the technical field of photoacoustic image registration, and in particular to an unsupervised image registration method for a photoacoustic rapid scanning system. Background Art

[0002] Fast scanning is of great significance for photoacoustic microscopy systems to improve imaging speed and realize real-time monitoring of biological activities. However, fast scanners are inherently susceptible to interference, and various factors such as scanner manufacturing accuracy, installation errors, and material fatigue can affect the consistency of the scanning trajectory. Under harsh working conditions, such as when using fiber-optic photoacoustic probes or handheld imaging systems for high-speed water immersion scanning, the scanner is subject to continuous random interference, which can cause image distortion and misalignment between multiple two-dimensional and three-dimensional images. Therefore, the registration and restoration of these images is crucial to accurately quantify dynamic information in long-term imaging.

[0003] Traditional registration methods are generally divided into two categories. One is to use scale-invariant feature transform (SIFT) and speeded up robust feature method (SURF) to extract the coordinates of the same features in two images to achieve correction of distortion. However, these two methods are not suitable for images with blurred feature points or a small number of features. When the intensity of the photoacoustic image changes over time, the traditional feature point-based registration method does not perform well. Another image intensity-based registration method determines the coordinate offset of each point by calculating the grayscale difference between the two images. The disadvantage is that this method is sensitive to grayscale distribution and performs poorly on photoacoustic images with uneven grayscale.

[0004] With the rapid development of deep learning, some studies have attempted to use deep learning technology to improve the quality of photoacoustic images. A method and system for photoacoustic image restoration based on a Transformer neural network (CN202410365893.3), the Transformer neural network used in this method is composed of a plurality of Transformer modules based on a transposed attention mechanism added to a U-shaped structure, which has the ability to learn the image globally, and restores the low-resolution photoacoustic image with severe artifacts and distortion obtained under ill-posed conditions (sparse, limited-viewing angle transducer distribution) to a high-quality photoacoustic image similar to that under high-density, large (or full) viewing angle transducer distribution conditions. The invention adds a plurality of Transformer modules based on a transposed attention mechanism to a U-shaped structure, which has the ability to learn globally, and can restore the distorted photoacoustic image to a high-quality image. The disadvantage is that the Transformer has a large number of parameters, requires a large number of data sets for training, and the model needs to input real labels, which is difficult to achieve in a fast scanning photoacoustic microscope system. A computational photoacoustic microscopy system and method based on unsupervised deep learning (CN 202410410793.8), the method prepares and fixes an imaging specimen, obtains the photoacoustic data of the specimen through a computational photoacoustic microscopy system, extracts effective layer data to construct a data set; establishes an image reconstruction network model based on unsupervised deep learning; obtains a trained image reconstruction network model after training; performs maximum projection on the effective layer information in the obtained photoacoustic data and inputs it into the S3 network to obtain a high-quality image. The image reconstruction network model includes an encoder and a decoder, the encoder includes 5 convolutional layers, each of which includes two convolutional units; the decoder includes 5 convolutional layers, each of which includes a convolutional unit and a LeakyRelu activation layer unit. The invention can retain image detail information and remove background noise without the need for true value input, but it cannot restore distorted image details. Summary of the invention

[0005] To solve the above technical problems, the purpose of the present invention is to provide an unsupervised image registration method for a photoacoustic rapid scanning system, introduce an attention mechanism in a convolutional network, give higher weights to key factors during feature extraction, effectively improve the efficiency of registration, introduce mutual information as a loss function, effectively model the probabilistic relationship between pixel intensities, and help to more accurately evaluate the similarity of photoacoustic images acquired at different time points.

[0006] The present invention is achieved by at least one of the following technical solutions.

[0007] An unsupervised image registration method for a photoacoustic rapid scanning system comprises the following steps:

[0008] A. Collect data using fast scanning photoacoustic microscopy;

[0009] B. performing preprocessing steps of hooping, filtering, normalization, and affine transformation on the data to form a photoacoustic image data set;

[0010] C. Construct a registration model based on convolutional neural network and spatial transformation module, integrate the attention mechanism to assign weights to different pixels, adopt a two-stage training strategy, and introduce mutual information and smoothness terms as loss functions;

[0011] D. Prediction of the registered photoacoustic image based on the trained weights.

[0012] Furthermore, in step A, the light beam of the optical resolution photoacoustic microscopy imaging system based on the water-immersed resonant mirror is reflected onto the sample to be tested through a uniaxial water-immersed resonant mirror coated with aluminum, and the generated ultrasonic wave is received by the piezoelectric sensor.

[0013] Furthermore, in step A, the signal intensity of the acquired data is projected along a direction perpendicular to the image plane, and the maximum amplitude value at each position is taken to form a maximum amplitude projection image.

[0014] Furthermore, in step B, the intensity values ​​of the image are clamped according to the first percentile and the 99th percentile of the image to remove outliers, and then the data is normalized to the range of [0, 1] and median filtered, and finally an initial affine alignment is performed to roughly align the image positions.

[0015] Furthermore, in step C, a registration model based on a convolutional neural network and a spatial transformation module is constructed, the attention mechanism is integrated to strengthen the weight of key pixels, a two-stage training strategy is adopted for odd-numbered and even-numbered frame images, mutual information loss and smoothing loss are introduced, and the network's registration effect on photoacoustic images with changing image intensity is improved.

[0016] Furthermore, the fixed image I is predicted by the convolutional neural network f With floating image I m The deformation field φ between them, the spatial transformation module resamples the floating image according to the deformation field to obtain the registered image

[0017] Furthermore, the spatial transformation module includes a grid generator and a sampler. For each pixel p, the sub-pixel position p′=p+g is calculated in the floating image. θ (p), g θ (p) represents the deformation of pixel p, using bilinear interpolation to interpolate eight neighboring pixels:

[0018]

[0019] In the formula is the pixel of the image after registration, q∈Z(p′) is the neighborhood of p′, Ω is the two-dimensional spatial domain, d∈[x,u] is the iterative calculation dimension of the image, and p′ d represents the sub-pixel in d dimension, q d Represents the neighborhood pixels in d dimension.

[0020] Furthermore, when training the registration model, all odd frames are treated as floating images and registered with the first frame, and then all even frames are treated as floating images and registered with their previous odd frames; when testing, all the registered odd frames are first predicted, and then all the even frames are aligned using the registered odd frames as fixed images.

[0021] Furthermore, the mutual information L is introduced MI and bending energy penalty L smooth As a loss function, the formula is as follows:

[0022]

[0023]

[0024]

[0025] In the formula, n represents the number of samples, M represents the absolute value of the distance between the predicted output and the target, and L smooth (φ) calculates the second-order gradient of the deformation field, u(p) is the displacement of pixel p, and λ is a hyperparameter. is the gradient operator, and σ is the standard deviation of the Gaussian kernel.

[0026] Furthermore, the ParZen window formula of the Gaussian function is used to calculate the mutual information. is the Gaussian kernel function.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention discloses an unsupervised image registration method for a photoacoustic rapid scanning system. After unsupervised network training, the model can predict the registered image in a very short time without inputting a true value. An attention mechanism is introduced into the convolutional network, and a higher weight is given to key factors during feature extraction, which effectively improves the efficiency of registration. Mutual information is introduced as a loss function, and the probability relationship between pixel intensities is effectively modeled, which helps to more accurately evaluate the similarity between photoacoustic images acquired at different time points. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of an unsupervised image registration method for a photoacoustic rapid scanning system according to an embodiment;

[0030] Figure 2 is a structural diagram of a registration model of an embodiment;

[0031] Figure 3 is a flow chart of network training and testing of an embodiment;

[0032] Figure 4 2 is a result diagram of the embodiment before and after visualization registration. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0034] like Figure 1 As shown, an unsupervised image registration method for a photoacoustic rapid scanning system of this embodiment includes the following steps:

[0035] Step 10, collecting data using a rapid scanning photoacoustic microscope;

[0036] Step 20, performing preprocessing steps such as hopping, filtering, normalization, and affine transformation on the data to form a photoacoustic image data set;

[0037] Step 30: Construct a registration model based on a convolutional neural network and a spatial transformation module, integrate the attention mechanism to assign weights to different pixels, adopt a two-stage training strategy, and introduce mutual information and smoothness terms as loss functions;

[0038] Step 40: predicting the registered photoacoustic image based on the training weights;

[0039] The above step 10 adopts an optical resolution photoacoustic microscopy imaging system based on a water-immersed resonant mirror. The light beam is reflected onto the sample to be tested through a uniaxial water-immersed resonant mirror coated with an aluminum coating. The generated ultrasonic wave is reflected by the resonant mirror, collimated by a plano-concave acoustic lens, then transmitted through two prisms, and finally received by a piezoelectric sensor.

[0040] The scanning system consists of two 532nm pump lasers, polarization-maintaining single-mode optical fiber, piezoelectric sensor (center frequency of 50MHz, 78% bandwidth), amplifier, data acquisition card, FPGA card, linear translation stage, etc.

[0041] As a specific embodiment, the fast scanning photoacoustic microscope uses a linear translation stage to drive the probe to scan along the slow axis direction, and the size of the scanning area is 2.5×6.7 mm 2, A scan rate is 3.2MHz, B scan rate is 2036Hz, C scan rate is 1.7Hz. During the experiment, 5 micrograms of adrenaline were injected into the hind leg muscles of mice to observe the changes in hemoglobin concentration in the mouse ear blood vessels.

[0042] In the above step 10, the signal intensity of the collected data is projected along a direction perpendicular to the image plane, and the maximum amplitude value at each position is taken to form a maximum amplitude projection image.

[0043] In the above step 20, the intensity value of the image is clamped according to the 1st percentile and the 99th percentile of the image to remove outliers. Then the data is normalized to the range of [0,1] and median filtered using a 3×3 filter, and finally an initial affine alignment is applied to roughly align the image positions.

[0044] In the above step 30, a registration network based on a convolutional neural network and a spatial transformation module is constructed, an attention mechanism is integrated to strengthen the weight of key pixels, a two-stage training strategy is adopted for odd-numbered and even-numbered frame images, mutual information loss and smoothing loss are introduced, and the registration effect of the network on photoacoustic images with changing image intensity is improved.

[0045] As a preferred embodiment, UNet is selected as the convolutional neural network in this embodiment, such as Figure 2 As shown in the figure. UNet consists of two parts: encoder and decoder. The encoder uses 2D convolution operation with a kernel size of 4, a stride of 2, and a padding of 1. After each convolution layer, a LayerNorm layer and a ReLU activation layer with a parameter of 0.2 are added. After each convolution operation, the output feature size is halved. After 4 downsampling operations, the size of the output feature is (1 / 16) of the input size. 2 .

[0046] In the decoding stage, the size of the output features is gradually increased through bilinear interpolation, the sampling coefficient is set to 2, and the final output deformation field has the same size as the input image. The jump connection directly integrates the features learned in the encoding stage with the corresponding features in the decoding stage, thereby helping the network to better preserve low-level features and spatial information.

[0047] A simplified attention module is introduced after each downsampling layer. This module consists of a convolution layer with a convolution kernel size of 1 and a sigmoid layer. The convolution operation and sigmoid function assign weights to the input feature map, improving the model performance by focusing on more useful features.

[0048] The optimized UNet is used to concatenate two single-channel grayscale images (fixed image and floating image) into a two-channel grayscale image as input, and output a three-channel deformation field. The spatial transformation module resamples the floating image according to the deformation field to obtain the registered image.

[0049] The spatial transformation module consists of a grid generator and a sampler. For each pixel p, the sub-pixel position p′=p+g is calculated in the floating image. θ (p), g θ (p) represents the deformation of pixel p, using bilinear interpolation to interpolate eight neighboring pixels:

[0050]

[0051] In the formula is the pixel of the image after registration, q∈Z(p′) is the neighborhood of p′, Ω is the two-dimensional spatial domain, d∈[x,u] is the iterative calculation dimension of the image, and p′ d represents the sub-pixel in d dimension, q d Represents the neighborhood pixels in d dimension.

[0052] In this embodiment, before correcting the distortion between the image sequences, the misalignment between the odd and even columns in the image is first restored. The process is as follows: Figure 3 As shown in (a). First, all odd columns in the preprocessed image are extracted, and then linear interpolation is performed between the columns to obtain an interpolated image of the same size as the input image. This process will cause the loss of even column information, but ensures that there is no misalignment between columns. The interpolated image is used as a fixed image, and the image that has not been extracted and interpolated is used as a floating image to input the network. The above process aligns the floating image to the fixed image while retaining the even column information. In this case, the interpolated image provides an easily obtained approximate baseline, allowing the floating image to be aligned with it without losing information. Ultimately, the output of this process is complete and there is no misalignment. From Figure 3 It can be observed in (a) that the input image has obvious jagged edges at the edges of blood vessels, but after correction, the jagged edges disappear, proving that the misalignment has been effectively restored.

[0053] The network training and testing process for image sequence registration is as follows Figure 3 As shown in (b), during training, all odd frames are treated as floating images and registered with the first frame, and then all even frames are treated as floating images and registered with their previous odd frames. During testing, the network first predicts all the registered odd frames, and then uses the registered odd frames as fixed images to align all even frames.

[0054] Introducing mutual information L MI and bending energy penalty L smoothAs a loss function, the formula is as follows:

[0055]

[0056]

[0057]

[0058] Where n represents the number of samples, u(p) is the displacement of pixel p, M represents the absolute value of the distance between the predicted output and the target, and L smooth What is calculated is the second-order gradient of the deformation field, and λ is a hyperparameter.

[0059] Since the probability calculation method based on histogram is not differentiable, the Parzen window formula of Gaussian function is used to calculate the mutual information. is the Gaussian kernel function, and σ is the standard deviation of the Gaussian kernel.

[0060] In the above step 40, the registered photoacoustic image is predicted based on the training weights. The training data set consists of 399 continuously acquired photoacoustic images of mouse ears. The image size is 320×1024, 80% of the images are used as training sets, and 20% of the images are used as test sets. In order to increase the diversity of the data, rotation of 180° and flipping of 180° are applied to the training data set. First, the registration model is trained, using the Pytorch network framework equipped with NVIDIA GeForce RTX 4080, and the relevant training parameters are set: optimizer Adam, initial learning rate 0.001, number of iterations 6380, λ is set to 10, and σ in the mutual information loss is set to 0.6. By minimizing the loss function, the optimal training weights are saved. During testing, the registration model is configured using the optimal training weights. First, all odd frames in the test set are aligned with the first frame to obtain the registered odd frames. Then all even frames are used as floating images, and the registered adjacent odd frames are used as fixed images to obtain all the registered even frames. The mutual information matrix (MIM) is used to measure the similarity of images before and after registration. The higher the overall brightness of the MIM, the stronger the similarity between the images, and vice versa. Figure 4 As shown in (a) and (b), Figure 4 (b) is the image after registration, which is obviously brighter, meaning that the correlation between images is significantly enhanced.

[0061] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well.

Claims

1. An unsupervised image registration method for a photoacoustic rapid scanning system, characterized in that: The following steps are involved: A. Collect data using fast scanning photoacoustic microscopy; B. performing preprocessing steps of hooping, filtering, normalization, and affine transformation on the data to form a photoacoustic image data set; C. Construct a registration model based on convolutional neural network and spatial transformation module, integrate the attention mechanism to assign weights to different pixels, adopt a two-stage training strategy, and introduce mutual information and smoothness terms as loss functions; D. Prediction of the registered photoacoustic image based on the trained weights.

2. The unsupervised image registration method for a photoacoustic rapid scanning system according to claim 1, characterized in that: In step A, the light beam of the optical resolution photoacoustic microscopy imaging system based on a water-immersion resonant mirror is reflected onto the sample to be tested through a uniaxial water-immersion resonant mirror coated with aluminum, and the generated ultrasonic wave is received by a piezoelectric sensor.

3. The unsupervised image registration method for a photoacoustic rapid scanning system according to claim 1, characterized in that: In step A, the signal intensity of the acquired data is projected along a direction perpendicular to the image plane, and the maximum amplitude value at each position is taken to form a maximum amplitude projection image.

4. The unsupervised image registration method for a photoacoustic rapid scanning system according to claim 1, characterized in that: In step B, the intensity values ​​of the image are clamped according to the first percentile and the 99th percentile of the image to remove outliers, and then the data is normalized to the range of [0,1] and median filtered, and finally an initial affine alignment is performed to roughly align the image positions.

5. The unsupervised image registration method for a photoacoustic rapid scanning system according to claim 1, characterized in that: In step C, a registration model based on convolutional neural network and spatial transformation module is constructed, the attention mechanism is integrated to strengthen the weight of key pixels, a two-stage training strategy is adopted for odd-numbered and even-numbered frame images, mutual information loss and smoothing loss are introduced, and the network's registration effect on photoacoustic images with changing image intensity is improved.

6. The unsupervised image registration method for a photoacoustic rapid scanning system according to claim 5, characterized in that: Predict a fixed image I through a convolutional neural network f With floating image I m The deformation field φ between them, the spatial transformation module resamples the floating image according to the deformation field to obtain the registered image 7. The unsupervised image registration method for a photoacoustic rapid scanning system according to claim 6, characterized in that: The spatial transformation module includes a grid generator and a sampler. For each pixel p, the sub-pixel position p'=p+g is calculated in the floating image. θ (p), g θ (p) represents the deformation of pixel p, using bilinear interpolation to interpolate eight neighboring pixels: In the formula is the pixel of the image after registration, q∈Z(p') is the neighborhood of p', Ω is the two-dimensional spatial domain, d∈[x,y] is the iterative calculation dimension of the image, and p' d represents the sub-pixel in d dimension, q d Represents the neighborhood pixels in d dimension.

8. The unsupervised image registration method for a photoacoustic rapid scanning system according to claim 6, characterized in that: When training the registration model, all odd frames are registered with the first frame as floating images, and then all even frames are registered with the previous odd frame as floating images. When testing, all the registered odd frames are first predicted, and then all the even frames are aligned with the registered odd frames as fixed images.

9. The unsupervised image registration method for a photoacoustic rapid scanning system according to claim 6, characterized in that: Introducing mutual information L MI and bending energy penalty L smooth As a loss function, the formula is as follows: In the formula, n represents the number of samples, M represents the absolute value of the distance between the predicted output and the target, and L smooth (φ) calculates the second-order gradient of the deformation field, u(p) is the displacement of pixel p, and λ is a hyperparameter. is the gradient operator, and σ is the standard deviation of the Gaussian kernel.

10. The unsupervised image registration method for a photoacoustic rapid scanning system according to claim 9, characterized in that: The mutual information is calculated using the Parzen window formula of the Gaussian function. is the Gaussian kernel function.

Images