Photoacoustic microscopic image correction restoration method and device based on random subsampling and morphological filtering

Abstract

The application discloses a photoacoustic microscopic image correction and restoration method and device based on random subsampling and morphological filtering, and the method comprises the following steps. O Performing distortion estimation, estimating the maximum amplitude by solving the Euclidean distance of the blood vessel center line and edge with morphological characteristics, and obtaining a morphological filtering parameter d; collecting K relatively stable photoacoustic microscopic signals, and performing equal-interval sampling respectively to serve as a positive sample set I D ; fitting the image information of the positive sample I D with a multi-Gaussian distribution, simulating jitter, and serving as a negative sample set I d ; constructing a PADC network model based on random subsampling and morphological filtering, wherein the PADC network model comprises a PADC encoder structure, a PADC decoder structure and a skip layer; training the constructed PADC network model to obtain a trained PADC network model; and correcting and restoring the photoacoustic microscopic image to be processed by using the trained PADC network model. The application can effectively correct and restore the system motion artifacts caused by NURD in the maximum value image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of biomedicine, and particularly relates to a photoacoustic microscopic image correction and restoration method and device based on random subsampling and morphological filtering. BACKGROUND

[0002] In photoacoustic microscopic endoscopic imaging applications, the maximum projection mode can obtain the blood vessel morphological information between continuous frames, while mechanical point scanning imaging inevitably has system motion artifacts caused by non-uniform rotational disturbance (NURD) of the probe. NURD is unstable in time, because it depends not only on the specific structure of the probe, but also on the specific path of the curved cavity, both of which dynamically change with the motion of the living body experiment and the pullback position. In addition, due to the existence of coupling liquid such as white oil or physiological saline, there is a viscous state between the sheath and the probe, and sudden jumps may occur between the two angular positions, or the probe tip gradually increases or decreases due to friction, which will cause the occurrence of NURD. In addition, as an open-loop imaging system, vital signs such as breathing, heartbeat and involuntary movement of the living body will cause motion artifacts. The motion artifacts usually exhibit irregular sawtooth-like misplacement in amplitude and direction.

[0003] The catheter-based photoacoustic endoscopic microscopic imaging system has a three-dimensional scanning mechanism: longitudinal motion (pullback) and azimuthal (angular) rotational motion. The catheter needs to be highly flexible to be used in a tortuous path and to be operated during the operation process. NURD is a common problem for catheter-based imaging systems, such as intravascular photoacoustic endoscopic microscopy. In ordinary photoacoustic endoscopic scanning systems with an external rotational motor, the probe is driven at the proximal end. In these systems, NURD is observed in in vivo and in vitro studies, indicating that NURD exists due to mechanical friction between the catheter torque coil and the sheath, regardless of body movement or other physiological problems. The physiological motion and bending encountered in endoscopy and vascular imaging applications still have a great influence on the severity of rotational non-uniformity. Some of the above problems are related to proximal rotation, therefore, microelectromechanical catheter technology for distal rotation scanning can improve the imaging process. However, NURD still exists due to manufacturing defects and mechanical instability of the motor. Although this can be overcome by using any kind of closed-loop control, such as an encoder, it will increase the size of the probe, which may not allow it to be used in clinical applications.

[0004] In all cases, the uncertain inter-frame NURD variation inhibits the correction method. In axial reconstruction, this appears as the same oscillation mode as the heart beat. Algorithms can be developed to correct and restore these artifacts. At present, there are mainly two methods:

[0005] The first method includes using the method of adjacent Aline or cross-correlation between Aline to estimate NURD and angular distortion of different frames. All feature-based techniques have a common major drawback: they require a reference frame without NURD or at least zero-mean NURD in the entire dataset. Otherwise, they will not only correct the relative NURD between frames, but also propagate the NURD that occurs in the reference frame. In addition, it cannot be guaranteed that successful matching can be achieved at all azimuth positions to achieve tracking. Moreover, the method using cross-correlation or phase information usually requires highly correlated images. When tested in vivo, some methods need to completely disable the longitudinal scan (pullback) or only provide moderate improvement in rotational uniformity. When the patient or operator has significant motion, methods that do not rely on feature-based imaging methods for correction can be desirable due to changes in tissue features caused by physiological motion.

[0006] The second method belongs to the method of using reference features (spatially fixed) to correct NURD. Fixed structural reference features are used to assist registration, and then the dataset is obtained. These methods also include the reflection of the sheath of the catheter or the optical assembly, and the use of fixed pillars supporting the micro-mechanical structure as a reference for detecting NURD in the micro-mechanical catheter. For example, the fixed pillars supporting the micro-mechanical structure are used as reference features, and the cross-sectional images are resampled according to the feature positions. However, these methods can only provide a low temporal resolution velocity measurement, and have the disadvantages of longitudinal sampling oversampling and low velocity temporal resolution. In the field of photoacoustic microscopic endoscopic imaging, due to the non-rigid azimuth misalignment between image frames, the correction and restoration of the maximum value image cannot be completed by local angle registration, which is also a limitation of this method. SUMMARY

[0007] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a photoacoustic microscopic image correction and restoration method and device based on random subsampling and morphological filtering, which can correct and restore the system motion artifacts caused by NURD in the maximum value image without using reference features, using a small sample training set containing the same morphological characteristics, and without using reference features.

[0008] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0009] The present application provides a photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering, comprising the following steps:

[0010] The observed image I O Distortion estimation, coarse estimation of the maximum amplitude by solving the Euclidean distance of the blood vessel centerline and edge with morphological characteristics, and obtaining the morphological filtering parameter d;

[0011] K stable photoacoustic microscopic signals are collected and equally spaced sampling is performed respectively as a positive sample set I D ; image information of the positive sample I D is fitted by a multi-Gaussian distribution, simulated jitter is used as a negative sample set I d ;

[0012] A PADC network model based on random subsampling and morphological filtering is constructed, the PADC network model comprises a PADC encoder structure, a PADC decoder structure and a skip layer; each PADC encoder structure comprises a plurality of encoder sub-modules, each encoder sub-module is a first deep residual network ResNet, a morphological opening operation is performed on an image after subsampling of a previous layer by using the first deep residual network ResNet according to distortion estimation, and each encoder sub-module has a random subsampling realized by a maximum pooling layer; each PADC decoder structure comprises a plurality of decoder sub-modules, each decoder sub-module is a second deep residual network ResNet, a morphological opening operation is performed on an image of a current layer by using the second deep residual network ResNet according to distortion estimation, and each decoder sub-module has an up-sampling realized by a deconvolution layer; the skip layer connects outputs of encoder sub-modules with the same resolution in the encoder structure and up-sampling results, and the outputs and the up-sampling results are used as inputs of a next decoder sub-module in the decoder structure;

[0013] The constructed PADC network model is trained to obtain a trained PADC network model;

[0014] The photoacoustic microscopic image to be processed is corrected and restored by using the trained PADC network model.

[0015] As a preferred technical solution, the calculation formula of the negative sample set I d is as follows:

[0016]

[0017] I d =P(I D )(2)

[0018] Wherein P is a probability density function, x is an input image, μ represents an average value or an expected value of a gray value, and δ represents a standard deviation of the gray value.

[0019] As a preferred technical solution, the constructed PADC network model is trained to obtain a trained PADC network model, and specifically:

[0020] A PADC input layer is constructed, an image domain regularization is designed, a regularization term is used for constraint to prevent sample overfitting, and Gaussian random noise is added in the first layer image in the training process;

[0021] In each encoder submodule, a first deep residual network ResNet is used to perform a morphological opening operation on the image after sub-sampling of the previous layer according to the distortion estimation;

[0022] In each encoder submodule, a max-pooling layer is used to randomly sub-sample the image of each layer using a Gaussian pyramid;

[0023] In each decoder submodule, a second deep residual network ResNet is used to perform a morphological opening operation on the image of the current layer according to the distortion estimation;

[0024] An up-sampling is implemented using the de-convolutional layers designed between the decoder submodules, and the resolution is sequentially increased through the up-sampling operation. In the PADC decoder structure, a Laplacian pyramid is used to up-sample the image of each layer until the resolution is consistent with that of the input image;

[0025] The output of the encoder submodule with the same resolution in the skip-layer encoder structure is connected with the up-sampling result;

[0026] A target function is constructed, and the difference between the reconstructed image in the training model and the positive sample is continuously narrowed by solving a constrained optimization problem; that is:

[0027] I d argmin{||I D -I d ||2}+λTV(I d ) (8)

[0028] st.||I D -I d ||2≤ε (9)

[0029]

[0030] where I d is the input image of each layer in the decoder, ε represents the maximum value allowed for the reconstruction error, TV is the TVloss regular expression, which is used to square the gradient of the image, and can not only remove additive noise in the image, but also effectively preserve the edge information of the image, i and j represent the pixel coordinates of the image I d , and λ is a weighting parameter;

[0031] An iterator is constructed to determine whether the number of iterations is met. If not, the training process of the PADC encoder structure and the PADC decoder structure is repeated, and if met, the training is completed, and the trained PADC network model is output.

[0032] As a preferred technical solution, the morphological opening operation is specifically:

[0033]

[0034] wherein represents the image of each layer after morphological filtering, represents the input image of each layer; O represents a morphological opening operation, which is a process of first eroding and then dilating, represents an erosion operation, which is a process of eliminating boundary points and making the boundary shrink inward, represents a dilation operation, which is a process of merging all background points in contact with the object into the object and making the boundary expand outward.

[0035] As a preferred technical solution, the erosion operation is specifically: using a 3x3 kernel structure element, scanning each pixel of the image, performing an "and" operation on the kernel structure element and the binary image covered thereby, if both are 1, the pixel of the result image is 1, otherwise 0;

[0036] The dilation operation is specifically: using a 3x3 kernel structure element, scanning each pixel of the image, performing an "and" operation on the kernel structure element and the binary image covered thereby, if both are 0, the pixel of the result image is 0; otherwise, 1.

[0037] As a preferred technical solution, the max-pooling layer uses a Gaussian pyramid to randomly subsample each layer of the image, specifically:

[0038]

[0039] wherein, is the input image of the i-th layer, represents the i-1-th layer of the image after morphological filtering pyrDown is the subsampling operation in the image pyramid, first convolve the image with a Gaussian kernel, then delete all even rows and even columns, and the area of the new image will become one fourth of the source image;

[0040] The resolution is raised in turn by upsampling operation, and in the PADC decoder structure, the Laplacian pyramid is used to upsample each layer of the image until the resolution is consistent with the input image, specifically:

[0041]

[0042] wherein, L is the expression of the Laplacian pyramid, is the input image of the i-th layer, and pryUp operation is the upsampling operation.

[0043] As a preferred technical solution, the image upsampling is a process of continuously enlarging the image from a small image, specifically:

[0044] The image is enlarged by 2 times in each direction, the newly added rows and columns are filled with 0, and the same convolution kernel as "down sampling" is multiplied by 4, and then convolution operation is performed with the enlarged image to obtain the new value of the "new pixel".

[0045] Another aspect of the present application provides a photoacoustic microscopic image correction and restoration system based on random subsampling and morphological filtering, comprising a reference acquisition module, a sample acquisition module, a model construction module, a model training module and a correction and restoration module.

[0046] The reference acquisition module is used for observing an image I O , performing distortion estimation, coarsely estimating the maximum amplitude by solving the Euclidean distance of the blood vessel center line and edge with morphological characteristics, and obtaining a morphological filtering parameter d.

[0047] The sample acquisition module is used for collecting K relatively stable photoacoustic microscopic signals, and performing equal interval sampling respectively as a positive sample set I D ; by fitting the image information of the positive sample I D with a multi-Gaussian distribution, simulating jitter as a negative sample set I d .

[0048] The model construction module is used for constructing a PADC network model based on random subsampling and morphological filtering, wherein the PADC network model comprises a PADC encoder structure, a PADC decoder structure and a skip layer; each PADC encoder structure comprises a plurality of encoder submodules, each of which is a first deep residual network ResNet, and the first deep residual network ResNet is used to perform morphological opening operation on the image after subsampling of the previous layer according to the distortion estimation; each of the encoder submodules has a random subsampling realized by a maximum pooling layer; each PADC decoder structure comprises a plurality of decoder submodules, each of which is a second deep residual network ResNet, and the second deep residual network ResNet is used to perform morphological opening operation on the image of the current layer according to the distortion estimation, and the decoder submodules all have an up-sampling realized by a deconvolution layer; the skip layer connects the outputs of the encoder submodules with the same resolution in the encoder structure and the up-sampling results, and takes the outputs as the input of the next decoder submodule in the decoder structure.

[0049] The model training module is used for training the constructed PADC network model to obtain a trained PADC network model.

[0050] The correction and restoration module is used for correcting and restoring the photoacoustic microscopic image to be processed by using the trained PADC network model.

[0051] The application further provides an electronic device, which comprises:

[0052] at least one processor; and

[0053] a memory connected in communication with the at least one processor; wherein

[0054] the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering.

[0055] The application further provides a computer readable storage medium storing a program, and the program is executed by a processor to implement the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering.

[0056] Compared with the prior art, the application has the following advantages and beneficial effects:

[0057] Since each layer of the PADC network in the application in the scale space is a deep residual network (ResNet), the deep residual network (ResNet) structure can improve the convergence stability and generalization of the model, thereby adapting to small sample training; the structure can improve the algorithm's ability to capture details and has strong robustness, and can correct and restore photoacoustic microscopic images. Using high-quality photoacoustic microscopic images as training data sets, the correction and restoration effect of photoacoustic microscopic images will be better; in the photoacoustic microscopic endoscopic imaging process, due to the existence of non-uniform rotation distortion (NURD), it is difficult to collect photoacoustic microscopic endoscopic positive samples, the PADC network of the application uses an unsupervised learning method, and can train a new model using a photoacoustic microscopic image data set in the absence of sufficient prior knowledge, to correct and restore photoacoustic microscopic endoscopic images. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0059] Figure 1 The flowchart of the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering of the embodiments of the application;

[0060] Figure 2a The positive sample image of the embodiments;

[0061] Figure 2b a negative sample image for an embodiment;

[0062] Figure 3 a block diagram of a photoacoustic microscopic image correction and restoration system based on random subsampling and morphological filtering for another embodiment of the present application;

[0063] Figure 4 a structural diagram of an electronic device for another embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to enable persons skilled in the art to better understand the schemes of the present application, the technical schemes in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0065] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be contained in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean that it refers to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by persons skilled in the art that the embodiments described in the present application can be combined with other embodiments.

[0066] In order to effectively correct and restore photoacoustic microscopic images with motion artifacts, the present application provides a photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering. The PADC training network in the method can effectively utilize the photoacoustic microscopic training data set to generate a training model that can correct and restore the influence of photoacoustic NURD.

[0067] As shown in Figure 1 the present embodiment of a photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering includes the following steps:

[0068] Step 1: distortion estimation;

[0069] The observed image I O is subjected to distortion estimation, and the Euclidean distance of the blood vessel centerline and edge with morphological characteristics is solved to roughly estimate the maximum amplitude, and the morphological filtering parameter d is obtained.

[0070] Step 2: sample data set making;

[0071] The maximum projection data set of mouse brain blood vessels of photoacoustic microscopy published by the team of Dr. Junjie Yao of Duke University in the United States is used as the positive sample set I D; The positive sample I is fitted by a multi-Gaussian distribution D , the image information of the positive sample I is simulated to be jittered as a negative sample set I d ; As shown in Figure 2a and Figure 2b , Figure 2a is a positive sample set example, Figure 2b is a negative sample set example; that is:

[0072]

[0073] I d = P(I D ) (2)

[0074] Wherein, P is a probability density function, x is an input image, μ represents an average value or an expected value of a gray value, δ represents a standard deviation of the gray value, I d is a negative sample, and I D is a positive sample.

[0075] Step 3: using a PADC network model based on random subsampling and morphological filtering, and training the PADC network model; the specific steps are as follows:

[0076] Step 3-1: Gaussian noise is added in the first layer image during training, and since the high-dimensional input space corresponding to the small sample quantity is relatively sparse, the neural network is difficult to learn the mapping relationship therefrom, and by designing image domain regularization, the regularization term is used to prevent sample overfitting, so as to minimize the structural risk. That is:

[0077] I d = I d + n(I d ) (3)

[0078]

[0079] Wherein, x is an input image, n is Gaussian noise, μ is an expected value of the uniform distribution, and δ is a standard deviation. m numbers X are randomly taken in the interval, and n can be regarded as a standardized variable of , when m is large enough, n approximately obeys a normal distribution, is the average number of X.

[0080] Step 3-2: constructing a PADC encoder structure;

[0081] Step 3-2-1: the PADC encoder is divided into 4 encoder submodules, and each encoder submodule is a first deep residual network (ResNet); according to the distortion estimation, the first deep residual network ResNet is used to perform morphological open operation (erosion, expansion) on the image after subsampling of the previous layer; that is:

[0082]

[0083] wherein, represents the image of each layer after morphological filtering, represents the input image of each layer; O represents a morphological opening operation. represents an erosion operation, which is a process of eliminating boundary points and making the boundary shrink inward; represents an expansion operation, which is a process of merging all background points in contact with the object into the object, making the boundary expand outward.

[0084] Further, the erosion operation is specifically: using a 3x3 kernel structure element, scanning each pixel of the image, and performing an "and" operation with the binary image covered by the kernel structure element. If all are 1, the pixel of the result image is 1, otherwise 0.

[0085] Further, the expansion operation is specifically: using a 3x3 kernel structure element, scanning each pixel of the image, and performing an "and" operation with the binary image covered by the kernel structure element. If all are 0, the pixel of the result image is 0, otherwise 1.

[0086] Step 3-2-2: There is a random subsampling between each encoder sub-module through a max pooling layer, which realizes the max pooling layer by randomly subsampling each layer of image using a Gaussian pyramid. That is:

[0087]

[0088] wherein, is the input image of the i-th layer, represents the i-1-th layer of morphologically filtered image, pyrDown is the subsampling operation in the image pyramid, which is to downsample and blur each layer of image, first convolve the image with a Gaussian kernel, then delete all even rows and even columns, and the area of the new image will become one fourth of the source image.

[0089] Further, the encoder is a classic convolution structure, which repeatedly uses a padding-free 4x4 convolution followed by a leaky ReLU and a 2x2 max pooling operation with a stride of 2. In each random subsampling step, the number of feature map channels is doubled.

[0090] Step 3-3: Construct the PADC decoder structure;

[0091] Step 3-3-1: The PADC decoder structure is divided into 4 sub-modules, and each decoder sub-module is a second deep residual network (ResNet), which is used to perform morphological opening operation (erosion, dilation) on the image of the current layer according to the distortion estimation.

[0092] Step 3-3-2: There is a deconvolution layer between each decoder sub-module to realize upsampling, and the resolution is raised by upsampling operation. In the PADC decoder, the Laplacian pyramid is used to upsample each layer of image until the resolution is consistent with the input image. That is:

[0093]

[0094] Where L is the expression of the Laplacian pyramid, is the input image of the i-th layer, and the pryUp operation is the upsampling operation. The image is upsampled by continuously enlarging the small image. It expands the image to 2 times the original image in each direction, and the newly added rows and columns are filled with 0. The same convolution kernel as "downsampling" is multiplied by 4, and then the enlarged image is convolved to obtain the new value of the "new pixel".

[0095] Further, there are a large number of feature channels in the upsampling channel, which enables the network to pass context information to high-resolution layers. The resolution is raised by upsampling operation until it is consistent with the resolution of the input image. In the expansion path, there are many 2x2 up-convolution operations, which halve the number of channels and are spliced with the corresponding feature maps cut from the compression path. Then there are two 3x3 convolutions, each followed by a leaky linear rectified activation function for calculation. When the photoacoustic signal input value is negative, the gradient of the leaky linear rectified function is a constant, not 0. When the photoacoustic signal input value is positive, the leaky linear rectified function and the ordinary ramp function are consistent;

[0096] Step 3-4: Build a skip connection layer to connect the outputs of the sub-modules with the same resolution in the encoder and the upsampled results as the input of the next sub-module in the decoder.

[0097] Step 3-5: Build the objective function to continuously narrow the gap between the reconstructed image in the training model and the positive sample by solving the constrained optimization problem; that is:

[0098] I d " = argmin{||I D " - I d ||2} + λTV(I d ") (8)

[0099] st.||I S -I d "||2≤ε (9)

[0100]

[0101] where I d is the input image of each layer in the decoder, and ε represents the maximum value allowed for the reconstruction error. TV is the TVloss regularizer, which is mainly the sum of the square of the gradient of the image, and can not only remove the additive noise in the image, but also effectively preserve the edge information of the image, i and j represent the pixel coordinates of the image I d , and λ is the weighting parameter.

[0102] Step 3-6: Construct an iterator to determine whether the number of iterations is met. If not, repeat the process of the encoder and the decoder, and if met, the training is completed, and a new training model is output.

[0103] Step 3-7: input the observed image I O into the model trained by the above steps;

[0104] Step 3-8: output the corrected and restored image I r .

[0105] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other order or simultaneously.

[0106] Based on the same idea as the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering in the above embodiment, the present application also provides a photoacoustic microscopic image correction and restoration system based on random subsampling and morphological filtering, which can be used to execute the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering. For the convenience of description, in the structural schematic diagram of the photoacoustic microscopic image correction and restoration system embodiment based on random subsampling and morphological filtering, only the part related to the embodiment of the present application is shown, and those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and can include more or fewer components than the illustrated, or combine certain components, or different component arrangements.

[0107] As Figure 3As shown, in another embodiment of the present application, a photoacoustic microscopic image correction and restoration system 100 based on random subsampling and morphological filtering is provided, which comprises a reference acquisition module 101, a sample acquisition module 102, a model construction module 103, a model training module 104 and a correction and restoration module 105;

[0108] The reference acquisition module 101 is configured to acquire an observation image I O , perform distortion estimation, and obtain morphological filtering parameters d by solving the Euclidean distance between the blood vessel centerline and the edge with morphological characteristics to roughly estimate the maximum amplitude;

[0109] The sample acquisition module 102 is configured to collect K relatively stable photoacoustic microscopic signals, and perform equal-interval sampling on each of the K signals to obtain a positive sample set I D ; and simulate jitter by fitting the image information of the positive sample I D to obtain a negative sample set I d ;

[0110] The model construction module 103 is configured to construct a PADC network model based on random subsampling and morphological filtering, wherein the PADC network model comprises a PADC encoder structure, a PADC decoder structure and a skip layer; each PADC encoder structure comprises a plurality of encoder sub-modules, each of which is a first deep residual network ResNet, and each first deep residual network ResNet is configured to perform morphological opening operation on an image obtained after subsampling of a previous layer according to distortion estimation; each PADC decoder structure comprises a plurality of decoder sub-modules, each of which is a second deep residual network ResNet, and each second deep residual network ResNet is configured to perform morphological opening operation on an image of a current layer according to distortion estimation; and the skip layer is configured to connect outputs of encoder sub-modules with the same resolution in the encoder structure and up-sampling results to serve as inputs of a next decoder sub-module in the decoder structure.

[0111] The model training module 104 is configured to train the constructed PADC network model to obtain a trained PADC network model.

[0112] The correction and restoration module 105 is configured to correct and restore a photoacoustic microscopic image to be processed by using the trained PADC network model.

[0113] It should be noted that the photoacoustic microscopic image correction and restoration system based on random subsampling and morphological filtering of the present application corresponds to the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering of the present application, and the technical features and advantages described in the above embodiment of the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering are applicable to the embodiment of the photoacoustic microscopic image correction and restoration based on random subsampling and morphological filtering, and the specific content can be referred to the description in the method embodiment of the present application, which will not be described here again, and hereby declared.

[0114] In addition, in the embodiment of the photoacoustic microscopic image correction and restoration system based on random subsampling and morphological filtering of the above embodiment, the logical division of each program module is only illustrative, and in actual application, the above function allocation can be completed by different program modules according to needs, for example, the configuration requirements of the corresponding hardware or the convenience of software implementation, that is, the internal structure of the photoacoustic microscopic image correction and restoration system based on random subsampling and morphological filtering is divided into different program modules to complete all or part of the functions described above.

[0115] As shown in Figure 4 In one embodiment, an electronic device implementing the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering is provided, and the electronic device 200 can include a first processor 201, a first memory 202 and a bus, and can further include a computer program stored in the first memory 202 and executable on the first processor 201, such as a photoacoustic microscopic image correction and restoration program 203.

[0116] The first memory 202 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The first memory 202 can be an internal storage unit of the electronic device 200 in some embodiments, such as the mobile hard disk of the electronic device 200. The first memory 202 can also be an external storage device of the electronic device 200 in other embodiments, such as the plug-in mobile hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the electronic device 200. Further, the first memory 202 can include both the internal storage unit and the external storage device of the electronic device 200. The first memory 202 can be used not only to store application software and various data installed on the electronic device 200, such as the code of the photoacoustic microscopic image correction and restoration program 203, but also to temporarily store data that has been output or will be output.

[0117] The first processor 201 may, in some embodiments, be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits of the same function or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connects various components of the entire electronic device through various interfaces and lines, executes programs or modules stored in the first memory 202, and calls data stored in the first memory 202 to perform various functions and process data of the electronic device 200.

[0118] Figure 4 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 4 The structure shown does not constitute a limitation on the electronic device 200, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0119] The photoacoustic microscopic image correction and restoration program 203 stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions, which, when running in the first processor 201, can achieve:

[0120] distortion estimation on the observation image I O , estimate the maximum amplitude by solving the Euclidean distance of the blood vessel centerline and edge with morphological characteristics, and obtain the morphological filtering parameter d;

[0121] Collect K relatively stable photoacoustic microscopic signals, and perform equal-interval sampling respectively as a positive sample set I D ; fit the image information of the positive sample I D with a multi-Gaussian distribution, simulate jitter, and obtain a negative sample set I d ;

[0122] A PADC network model based on random subsampling and morphological filtering is constructed, the PADC network model comprising a PADC encoder structure, a PADC decoder structure and a skip layer; each PADC encoder structure comprises a plurality of encoder sub-modules, each encoder sub-module being a first deep residual network ResNet, a morphological opening operation being performed on an image subsampled by a previous layer by using the first deep residual network ResNet according to distortion estimation, and there being a random subsampling implemented by a maximum pooling layer between each of the encoder sub-modules; each PADC decoder structure comprises a plurality of decoder sub-modules, each decoder sub-module being a second deep residual network ResNet, a morphological opening operation being performed on an image of a current layer by using the second deep residual network ResNet according to distortion estimation, and there being an up-sampling implemented by a deconvolution layer between each of the decoder sub-modules; the skip layer connects outputs of the encoder sub-modules having the same resolution in the encoder structure with up-sampling results, and takes the outputs as inputs of a next decoder sub-module in the decoder structure.

[0123] The constructed PADC network model is trained to obtain a trained PADC network model.

[0124] The photoacoustic microscopic image to be processed is corrected and restored by using the trained PADC network model.

[0125] Further, the modules / units of the electronic device 200, if realized in the form of software function units and sold or used as independent products, can be stored in a non-volatile computer readable storage medium. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0127] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0128] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are included in the protection scope of the present application.

Claims

1. A photoacoustic microscopic image correction restoration method based on random subsampling and morphological filtering, characterized by, The method comprises the following steps: On the observed image I o Performing distortion estimation, obtaining morphological filtering parameter d by solving the Euclidean distance between the blood vessel centerline and edge with morphological characteristics to roughly estimate the maximum amplitude. Collect K stable photoacoustic microscopic signals, and respectively carry out equal interval sampling as positive sample set I D ; Through multi-Gaussian distribution fitting image information of the positive sample I D , simulate jitter as negative sample set I d ; The PADC network model is constructed based on random subsampling and morphological filtering, and the PADC network model comprises a PADC encoder structure, a PADC decoder structure and a skip layer; each PADC encoder structure comprises a plurality of encoder submodules, each of which is a first deep residual network ResNet, and the first deep residual network ResNet is used to perform a morphological opening operation on an image subsampled by a previous layer according to distortion estimation; each of the encoder submodules has a random subsampling implemented by a maximum pooling layer; each PADC decoder structure comprises a plurality of decoder submodules, each of which is a second deep residual network ResNet, and the second deep residual network ResNet is used to perform a morphological opening operation on an image of a current layer according to distortion estimation, and the decoder submodules have an up-sampling implemented by a deconvolution layer; The skip layer connects the outputs of the encoder submodules with the same resolution in the encoder structure and the up-sampling result, and serves as the input of a next decoder submodule in the decoder structure; The PADC network model is trained to obtain a trained PADC network model; The photoacoustic microscopic image to be processed is corrected and restored by using the trained PADC network model. 2.The photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering according to claim 1, wherein, Negative sample set I d The calculation formula is as follows: ；(1) ；(2) wherein P is a probability density function, x is an input image, μ denotes the mean or expected value of the gray value, δ denotes the standard deviation of the gray value. 3.The photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering according to claim 1, wherein, The PADC network model is trained to obtain a trained PADC network model, and the training comprises the following steps: A PADC input layer is constructed, a Gaussian random noise is added to a first layer image in the training process by designing an image domain regularization and using a constraint of a regularization term to prevent sample overfitting; In each of the encoder submodules, the first deep residual network ResNet is used to perform a morphological opening operation on an image subsampled by a previous layer according to distortion estimation; In each of the encoder submodules, the maximum pooling layer uses a Gaussian pyramid to perform random subsampling on each layer image; In each of the decoder submodules, the second deep residual network ResNet is used to perform a morphological opening operation on an image of a current layer according to distortion estimation; An up-sampling is implemented by using a deconvolution layer designed between the decoder submodules, and the resolution is sequentially increased by the up-sampling operation, and a Laplacian pyramid is used to perform the up-sampling on each layer image in the PADC decoder structure until the resolution is consistent with that of the input image; The skip layer is used to connect the outputs of the encoder submodules with the same resolution in the encoder structure and the up-sampling result; An objective function is constructed, and a reconstruction picture in the training model is constantly narrowed down to the difference between the positive sample by solving a constrained optimization problem; that is: ； (8) ； (9) ； (10) wherein for each layer of input image in the decoder, represents the maximum value allowed for the reconstruction error, TV is the TVloss regular term, which is used to square the gradient of the image, not only can remove the additive noise in the image, but also can effectively preserve the edge information of the image, i, j represent the pixel coordinates of the image I d , and λ is a weighting parameter; An iterator is constructed, and whether the number of iterations is satisfied is judged; if not, the training process of the PADC encoder structure and the PADC decoder structure is repeated, if yes, the training is completed, and the trained PADC network model is output.

4. The photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering according to claim 3, characterized in that, The morphological opening operation specifically comprises: ； wherein represents the image of each layer after morphological filtering, represents the input image of each layer; O represents a morphological opening operation, which is a first erosion followed by a dilation, represents an erosion operation, which is a process that eliminates boundary points, making the boundary shrink inward, represents a dilation operation, which is a process that merges all background points that touch the object into the object, making the boundary expand outward.

5. The photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering according to claim 4, characterized in that, The erosion operation specifically comprises: a 3x3 kernel structure element is used to scan each pixel of an image, and a "and" operation is performed on the kernel structure element and a binary image covered by the kernel structure element; if both are 1, the pixel of a result image is 1, otherwise, the pixel of the result image is 0. The dilation operation is specifically: using a 3x3 kernel structure element, scanning each pixel of the image, using the kernel structure element and the binary image covered thereby to perform an AND operation, if both are 0, the pixel of the result image is 0; otherwise, 1.

6. The photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering according to claim 3, wherein, The max pooling layer utilizes a Gaussian pyramid to randomly sub-sample each layer of the image, specifically: ； wherein, is the input image for the i-th layer, represents the morphologically filtered image for the i-1-th layer pyrDown is the sub-sampling operation in the image pyramid, which first convolves the image with a Gaussian kernel and then removes all even rows and even columns, so that the area of the new image becomes one quarter of the source image; The resolution is sequentially increased through upsampling operations, and in the PADC decoder structure, a Laplacian pyramid is used to upsample each layer of the image until the resolution is consistent with that of the input image, specifically: ； where L is a Laplacian pyramid expression, is the input image of the i-th layer, and pryUp is an up-sampling operation.

7. The photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering according to claim 6, wherein, The upsampling operation is a process of continuously enlarging the small image, specifically: The image is enlarged to twice the size of the original image in each direction, the newly added rows and columns are filled with 0, and the same convolution kernel as "downsampling" is multiplied by 4, and then a convolution operation is performed with the enlarged image to obtain the new value of the "new pixel".

8. A photoacoustic microscopic image correction restoration system based on random subsampling and morphological filtering, characterized by, It comprises a reference acquisition module, a sample acquisition module, a model construction module, a model training module, and a correction and restoration module. The reference acquisition module is configured to perform distortion estimation on the observation image I o The maximum amplitude is coarsely estimated by solving the Euclidean distance between the blood vessel centerline and the edge with morphological characteristics, and a morphological filtering parameter d is acquired. The sample acquisition module is configured to collect K stable photoacoustic microscopic signals, and perform equal-interval sampling on the K stable photoacoustic microscopic signals respectively to obtain a positive sample set I D ; image information of the positive sample I D is fitted by a multi-Gaussian distribution, and simulated jitter is used as a negative sample set I d ; The model construction module is configured to construct a PADC network model based on random subsampling and morphological filtering, wherein the PADC network model comprises a PADC encoder structure, a PADC decoder structure, and a skip layer; each PADC encoder structure comprises a plurality of encoder sub-modules, each of which is a first deep residual network ResNet, and each first deep residual network ResNet is configured to perform a morphological opening operation on an image sub-sampled by a previous layer according to a distortion estimation; each PADC decoder structure comprises a plurality of decoder sub-modules, each of which is a second deep residual network ResNet, and each second deep residual network ResNet is configured to perform a morphological opening operation on an image of a current layer according to a distortion estimation; and each of the decoder sub-modules has a deconvolution layer for upsampling. The skip layer connects outputs of encoder sub-modules with the same resolution in the encoder structure and the upsampling result, and uses the connection result as an input of a next decoder sub-module in the decoder structure. The model training module is configured to train the constructed PADC network model to obtain a trained PADC network model. The correction and restoration module is configured to correct and restore a photoacoustic microscopic image to be processed by using the trained PADC network model.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering according to any one of claims 1-7.

10. A computer-readable storage medium storing a program, the program comprising instructions that cause a computer to: 10 5 The program is executed by the processor to implement the photoacoustic microscopic image correction and restoration method based on random subsampling and morphological filtering according to any one of claims 1-7.

Images