PG-SPECT image reconstruction method based on anisotropic diffusion network

Abstract

The present invention proposes a PG-SPECT image reconstruction method based on anisotropic diffusion network, and the implementation steps are: obtaining a training sample set and a test sample set; constructing anisotropic diffusion neural network model O; iteratively training anisotropic diffusion neural network model O; and obtaining image reconstruction results. In the process of training the reconstruction model, the smooth particle network in the constructed PG-SPECT image super-resolution reconstruction model based on anisotropic diffusion network can effectively retain the context information and content details of the PG-SPECT image by expanding the convolution receptive field; the anisotropic diffusion neural network guides the PG-SPECT image to perform hierarchical diffusion of features during the image reconstruction process, thereby generating an image with finer structure and less noise; experimental results show that the present invention can effectively improve the resolution of PG-SPECT image reconstruction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to an image super-resolution reconstruction method, and specifically to a PG-SPECT image reconstruction method based on an anisotropic diffusion network, which can be applied to fields such as medical imaging. Background Art

[0002] Image reconstruction refers to the reconstruction of degraded images, which can only make the image as close to the original image as possible, but it is difficult to restore it accurately due to factors such as noise interference and imaging system limitations. Image super-resolution reconstruction (SR) refers to the reconstruction of higher-resolution images using existing images, aiming to overcome or compensate for problems such as image blur and poor quality caused by the limitations of the image acquisition system or the acquisition environment itself. Compared with low-resolution images, high-resolution images usually contain richer texture details, greater pixel density, and higher reliability.

[0003] The types of medical imaging widely used in clinical practice mainly include X-ray imaging, magnetic resonance imaging, nuclear medicine imaging, and ultrasonic imaging. PG-SPECT images are a type of nuclear medicine image formed using a photon counting detector. Its characteristics are that it contains more additional information and its noise characteristics are different from those of SPECT images. Single photon emission tomography SPECT is a nuclear medicine imaging technology. Radioactive tracers are injected into the patient's body, and then the gamma rays emitted by the radioactive tracers are measured. The distribution map of the radioactive tracers in the human body can be reconstructed. This map can reflect the structure of human tissues and their activity functions. In SPECT technology, image reconstruction is the most core link. Generally speaking, due to the limitations of the detection instrument itself and the imaging system, the noise of the collected raw data is very large. Therefore, when performing reconstruction processing, the raw data needs to be super-resolution reconstructed to obtain a high-resolution image. Super-resolution reconstruction technology is applied to medical images, which can reconstruct higher-resolution images to improve image quality and add more detailed information.

[0004] In order to obtain super-resolution PG-SPECT reconstructed images, for example, a patent application with application publication number CN117541471A and titled “PG-SPECT image super-resolution reconstruction method based on SPH inspiration” discloses a PG-SPECT image super-resolution reconstruction method. The smooth particle sub-network in the invention can effectively retain the temporal and spatial detail information of the PG-SPECT image by expanding the convolution receptive field; the NS guided sub-network guides the PG-SPECT pixels to move in the desired direction during the image reconstruction process, thereby producing clear edges in the image, thereby improving the resolution of PG-SPECT image reconstruction. However, because pixel interactions are not constrained during the model training process, there are more speckle noise and fewer detailed structures, which affects the further improvement of the reconstructed image resolution. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and propose a PG-SPECT image reconstruction method based on anisotropic diffusion network to solve the technical problem of low reconstructed image resolution in the prior art.

[0006] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0007] (1) Obtain training sample set and test sample set:

[0008] After preprocessing the acquired K PG-SPECT images, N preprocessed PG-SPECT images are downsampled, and the downsampled PG-SPECT images are used as labels of the corresponding preprocessed PG-SPECT images. Then, the N preprocessed PG-SPECT images and their labels form a training sample set R1, and the remaining preprocessed PG-SPECT images form a test sample set E1, where K ≥ 600,

[0009] (2) Constructing a reconstruction model O based on anisotropic diffusion network:

[0010] A reconstruction model O is constructed, which includes a cascade of supramolecular networks composed of parallel smooth particle networks and anisotropic diffusion neural networks, a fusion module and an upsampling module; wherein the smooth particle network is used to perform preliminary reconstruction of each sample; the anisotropic diffusion neural network is used to perform feature hierarchical diffusion of each sample; the fusion module is used to perform element-by-element fusion of the reconstruction result of each sample with the feature hierarchical diffusion result; and the upsampling module is used to upsample the reconstructed feature map rich in detail information obtained by fusion;

[0011] (3) Iteratively train the reconstruction model O:

[0012] The training sample set R1 is used as the input of the reconstruction model O based on the anisotropic diffusion network and it is iteratively trained to obtain the trained reconstruction model O * ;

[0013] (4) Obtain the reconstruction results of PG-SPECT images:

[0014] The test sample set E1 is used as the input of the trained reconstruction model O* for forward propagation to obtain the PG-SPECT reconstructed images corresponding to all test samples.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] In the process of training the reconstruction model, the smooth particle network in the PG-SPECT image reconstruction model based on the anisotropic diffusion network constructed by the present invention can effectively retain the context information and content details of the PG-SPECT image by expanding the convolution receptive field; the anisotropic diffusion neural network guides the PG-SPECT image to perform hierarchical diffusion of features during the image reconstruction process, thereby generating an image with finer structure and less noise; experimental results show that the present invention can effectively improve the resolution of PG-SPECT image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart for realizing the present invention;

[0018] Figure 2 It is a structural schematic diagram of a reconstruction model based on anisotropic diffusion network of the present invention;

[0019] Figure 3 It is a schematic diagram of the structure of the smooth particle network of the present invention. DETAILED DESCRIPTION

[0020] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0021] Reference Figure 1 , the present invention comprises the following steps:

[0022] Step 1) Get the training sample set R1 and the test sample set E1:

[0023] Step 1a) obtaining K PG-SPECT images from the SPECT data set, where K ≥ 600. In this embodiment, K = 1000;

[0024] Step 1b) After preprocessing the acquired K PG-SPECT images, downsample the N preprocessed PG-SPECT images, and use the downsampled PG-SPECT images as labels of the corresponding preprocessed PG-SPECT images, then form a training sample set R1 with the N preprocessed PG-SPECT images and their labels, and form a test sample set E1 with the remaining preprocessed PG-SPECT images, wherein K≥600,

[0025] The preprocessing steps are as follows: randomly flip each PG-SPECT image horizontally, then rotate it 90 degrees, enhance the data information of the PG-SPECT image, and crop each data-enhanced PG-SPECT image to obtain K preprocessed PG-SPECT images of size H×H, where H≥512;

[0026] The downsampling implementation steps are as follows: downsample each preprocessed PG-SPECT image by 1 / 4 to obtain K images of size PG-SPECT images.

[0027] Step 2) construct a reconstruction model O based on anisotropic diffusion network, whose structure is as follows: Figure 2 As shown:

[0028] Constructing a reconstruction model O including a supramolecular network, a fusion module and an upsampling module connected in sequence; wherein the supramolecular network includes two branches, namely, a parallel arranged smooth particle network for image reconstruction of input samples and an anisotropic diffusion neural network for feature hierarchical diffusion of input samples;

[0029] The structure of the smooth particle network is as follows Figure 3As shown, it includes cascaded convolution modules and D hybrid difference equation networks, D≥2; each hybrid difference equation network includes an upper branch network and a lower branch network arranged in parallel, and a hybrid difference feature extraction layer connected to the output ends of the upper branch network and the lower branch network; the upper branch network includes a first convolution layer and a cross attention module connected in sequence; the lower branch network is composed of a cross attention module; the cross attention module includes a cross sub-network, a cross attention layer and a bottleneck module connected in sequence; the cross sub-network includes a second convolution layer, a normalization layer, a third convolution layer, a GeLU activation layer connected in sequence, and three mapping modules arranged in parallel connected to the output end of the GeLU activation layer; the mapping module includes a fourth convolution layer and a depth-separable convolution layer connected in sequence; the bottleneck module includes a fifth convolution layer and a normalization layer connected in sequence; the network parameters are set as follows: the convolution kernel size of the convolution layer in the convolution module is 3*3, and the number of channels is set to 64, and the convolution kernel size of the first, second, third, fourth and fifth convolution layers is 1*1, and the number of channels is set to 64;

[0030] Cross attention layer: After the feature map obtained by the convolution module passes through the first convolution layer, the high-level feature map F is obtained. high Input the upper branch cross attention module and map it through the cross sub-network to generate the corresponding query matrix Q X , key value matrix K X , value matrix V X ; At the same time, the low-level feature map F obtained by the convolution module low Input the cross attention module of the lower branch and map it through the cross sub-network to generate the corresponding query matrix Q Y , key value matrix K Y , value matrix V Y , where F low 、F high The outputs of the second convolutional layer are denoted as F C-low 、F C-high , F C-low 、F C-high The outputs after the GeLU activation layer are denoted as F G-low 、F G-high ,but:

[0031] F C-high ,F C-low =W C (F low ,F high ) (1)

[0032] F G-high ,F G-low =δ(W c (LN(F C-high ,F C-low))) (2)

[0033] Then the Q, K, and V matrices are obtained through the mapping module:

[0034] (Q X ,K X ,V X )=W dc (W c (F G-high )) (3)

[0035] (Q Y ,K Y ,V Y )=W dc (W c (F G-low )) (4)

[0036] Among them, W dc represents the depthwise separable convolutional layer, W c represents 1×1 convolution, δ represents GeLU activation layer, and LN(·) represents normalization layer;

[0037] The two Q matrices are swapped and then fed into the criss-cross attention layer to generate the corresponding weight values. The criss-cross attention layer operates as follows:

[0038] F att-high =F G-high ×CA(Q Y ,K X ,V X ) (5)

[0039] F att-low =F G-low ×CA(Q X ,K Y ,V Y ) (6)

[0040] High-level feature map F high The output of the upper branch is F finall-high , low-level feature map F low The output after the lower branch is F finall-low , F att-low 、F att-high The outputs of the bottleneck modules are denoted as F bottle-high 、F bottle-low ; respectively through F finall-low 、F finall-high With feature F C-low 、F C-high The residual connection is used to realize the fusion of feature information, and the high-resolution feature information after refinement of the upper branch network is input layer by layer into the corresponding lower branch network to supplement the original input:

[0041] F finall-high =F C-high +F bottle-high (7)

[0042] F finall-low =F C-low +F bottle-low +F finall-high (8)

[0043] In this way, the smooth particle network can realize the aggregation of temporal and spatial feature information at different scales, thereby improving the resolution of PG-SPECT images.

[0044] Hybrid differential feature extraction layer: The smooth particles in the fluid interact with each other under the guidance of specific forces and boundary conditions, resulting in the process of fluid flow, which can be compared to the process in which the pixels of the PG-SPECT image interact with each other under the constraints of the loss function, resulting in the rearrangement of the pixel values. Specifically, the continuous fluid system can be compared to the interaction between discrete smooth particles, which affects the smooth function values ​​of these particles and guides the fluid to flow forward. During the PG-SPECT image super-resolution training process, under the guidance of the loss function, the pixels in the PG-SPECT image drive the evolution of each feature map in the image super-resolution network, evolving from the shallow layer to the deep layer; therefore, the present invention seeks a solution to the PG-SPECT image super-resolution problem from the perspective of the theory related to discrete smooth particles.

[0045] In order to better fuse the image feature information, the present invention designs a connection structure in the form of a mixed difference equation and follows the graphical concept of the mixed difference equation. The characteristics of the time and space dimension information can be extracted through mixed derivatives, guiding the network to capture global and local information and encode the detailed content of the PG-SPECT image.

[0046] Second-order mixed partial derivatives It can be expressed as:

[0047]

[0048] Then a second-order central difference substitution is applied at the grid point (i+1,j)

[0049]

[0050] Then replace the difference at the grid point (i-1, j) with a similar

[0051]

[0052] Finally, the mixed difference equation is realized as:

[0053]

[0054] In the formula, This is the part of the hybrid second-order difference equation network to be learned. By simply transforming the equation, the final expression of the hybrid difference feature extraction layer is obtained as follows:

[0055]

[0056] This feature variation guided by the graphical concept of hybrid difference equations is continuous and stable, and the smooth particle network can obtain stronger feature extraction capabilities.

[0057] The structure of the anisotropic diffusion neural network includes a sixth convolutional layer and a PMD feature guidance layer in sequential cascade, where PMD refers to Perona-Malik diffusion; the network parameters are set as follows: the convolution kernel size of the sixth convolutional layer is 3*3, and the number of channels is set to 64;

[0058] PMD feature guidance layer: In the reconstruction model training process, the smooth particle network is mainly designed from the perspective of feature information aggregation. In addition to feature information aggregation, eliminating noise in smooth areas is another important information. In order to solve this problem, and using the analogy between the smooth particle fluid flow process and the PG-SPECT image training process, the PMD equation was constructed.

[0059] Guided branch inspired by PMD equation: Given an image or feature u, its PMD equation is:

[0060]

[0061]

[0062] in, is the diffusion coefficient, t is the diffusion step length, and k is the shape constant.

[0063] It is easy to see that in flat or smooth areas with small gradients The diffusion coefficient g is close to 1, and in areas with rich texture or structural details The diffusion coefficient is close to 0. With the spatial variation of the diffusion coefficient, PMD can preserve details while relocating noise in smooth regions.

[0064] By using the second-order approximation and u xx +u yy =u αα +u ββ By simple derivation, equation (1) can be rewritten as:

[0065]

[0066] Among them, u xx and u yy are the second-order partial derivatives of u along the x and y directions, respectively, u αα and u ββ are the second-order partial derivatives of u along the normal direction (image gradient direction) and the tangential direction (edge ​​direction). In order to better preserve structures such as edge details, it is necessary to perform as little smoothing as possible in the image gradient direction. To this end, we will αα The coefficient of is set to zero, that is This means If the diffusion step length Δt is set to 1, the following equation is obtained:

[0067]

[0068] Among them, u x and u y are the first-order partial derivatives of u along the x and y directions, respectively. The above formula can be expressed as a residual neural block, where the right side of the equation is the learned residual Δu. In fact, the residual neural block can be implemented as a number of convolutional layers with fixed kernels. We can obtain u after the network passes forward. x 、u xx 、u y and u yy and Δu, for example: u x =W x *u, where * indicates a convolution operation.

[0069] This anisotropic diffusion neural network can re-move noise in smooth areas while maintaining the geometric structure, thereby obtaining PG-SPECT images with finer structure and less noise.

[0070] The fusion module contains three alternating convolutional layers and nonlinear ReLU function layers, and the nonlinear ReLU function layer is implemented by the ReLU function;

[0071] The upsampling module is implemented by PixelShuffle with an amplification parameter of 4. PixelShuffle is an image upsampling method based on deep learning.

[0072] Step 3) Iteratively train the reconstruction model O based on the anisotropic diffusion network:

[0073] Step 3a) Initialize the number of iterations to s, the maximum number of iterations to S, S ≥ 10000, and the reconstructed model of the sth iteration is O s , O s The learnable weights and bias parameters are w s , b s , and let s = 1;

[0074] Step 3b) Randomly select N training samples from the training sample set R1 with replacement as the anisotropic diffusion neural network model O s Input;

[0075] Step 3c) The first convolutional layer of the smooth particle network convolves the input training sample to obtain the feature map Y1 of the input training sample, where Y1 m Represents the feature map after convolution of the mth training sample; input the feature map Y1 into the second convolution layer, and the second convolution layer further extracts features from the feature map Y1 to obtain the feature map Y2; the feature map Y2 is sequentially input into D hybrid difference equation networks, where the upper branch network in the d hybrid difference equations optimizes and enhances the features of the feature map Y2 to obtain the high-level feature map Y3, and the lower branch network extracts features from the feature map Y2 to obtain the feature map Y4, and then the high-level feature map Y3 is fused element by element with the feature map Y4 to guide the feature map Y4 to interact with high-level information to obtain the refined feature map Y5; the feature map Y3 and the feature map Y5 are respectively used as the input of the hybrid difference feature extraction layer, and the hybrid difference feature extraction layer can extract richer information in the time and space dimensions to obtain the feature map Y6; repeat D times to obtain the feature map Y7;

[0076] At the same time, the eighth convolution layer in the anisotropic diffusion neural network convolves the input training sample to obtain a feature map Y8 of the input training sample, and the PMD feature guidance layer extracts a feature map Y9 with less noise and finer structure from the feature map Y8;

[0077] The fusion module fuses the feature maps Y7 and Y9 element by element to obtain the reconstructed feature map Y 10 , the upsampling module performs the feature map Y 10 Upsampling is performed to obtain the reconstructed PG-SPECT image.

[0078] Step 3d) The loss function is calculated using the L1 norm, and O is calculated by reconstructing the image and its corresponding training sample label. s The loss value L s , and calculate the loss function L by the chain rule s The expression of Yes s 、b s The update formulas for updating are:

[0079]

[0080]

[0081]

[0082] in, represents the reconstructed image, I represents the label of the sample in the training sample set, and w s 、b s Indicates O s The weights and bias parameters of all learnable parameters, w s ', b s ' represents the updated learnable parameters, l r represents the learning rate, L s is the loss function, Represents a derivative operation.

[0083] Step 3e) Determine whether s≥S is true. If so, obtain the reconstruction model O* of the trained anisotropic diffusion neural network. Otherwise, set s=s+1, O s =0, and execute step (3c);

[0084] Step 3f) Obtain the image reconstruction result:

[0085] The test sample set E1 is used as the input of the trained anisotropic diffusion neural network model O* for forward propagation to obtain the reconstructed images corresponding to all test samples.

[0086] The smooth particle network proposed in the present invention can effectively learn rich detail information in time and space dimensions by establishing long-range dependencies between high-level features and low-level features; while the anisotropic diffusion neural network can achieve hierarchical diffusion of features, with the goal of re-moving noise in smooth areas while maintaining geometric structures, to obtain images with less noise and finer structures. The results on the SPECT dataset show that the network structure proposed in the present invention can generate PG-SPECT images with high resolution.

[0087] Combined with simulation experiments, the technical effects of the present invention are further explained.

[0088] 1. Simulation conditions and contents:

[0089] The hardware platform of the simulation experiment is: the processor is Intel(R) Core i9-9900K CPU, the main frequency is 3.6GHz, the memory is 32GB, and the graphics card is NVIDIA GeForce RTX 2080Ti. The software platform of the simulation experiment is: Ubuntu 16.04 operating system, Python version is 3.7, and Pytorch version is 1.7.1.

[0090] The hyperspectral image dataset used in the simulation experiment is the SPECT dataset. 1,000 heart disease images are selected from the SPECT dataset as the training dataset, and the CT dataset and PET dataset are used as the other two test sample sets, denoted as Test1 and Test2 respectively.

[0091] The PG-SPECT image reconstruction method based on anisotropic diffusion network proposed in the present invention mainly consists of two parts: smooth particle network (SPH) and anisotropic diffusion neural network (PMD). In order to verify the effectiveness of these two network structures, an ablation study was carried out; the results are listed in Table 1.

[0092] Table 1

[0093]

[0094] Based on the analysis of the results in the above simulation experiments, the PG-SPECT image reconstruction method based on anisotropic diffusion network proposed in the present invention can generate higher resolution PG-SPECT images thanks to the smooth particle network and anisotropic diffusion neural network.

Claims

1. A PG-SPECT image reconstruction method based on anisotropic diffusion network, characterized in that: The following steps are involved: (1) Obtain training sample set and test sample set: After preprocessing the acquired K PG-SPECT images, N preprocessed PG-SPECT images are downsampled, and the downsampled PG-SPECT images are used as labels of the corresponding preprocessed PG-SPECT images. Then, the N preprocessed PG-SPECT images and their labels form a training sample set R1, and the remaining preprocessed PG-SPECT images form a test sample set E1, where K ≥ 600, (2) Constructing a reconstruction model O based on anisotropic diffusion network: A reconstruction model O is constructed, which includes a cascade of supramolecular networks composed of parallel smooth particle networks and anisotropic diffusion neural networks, a fusion module and an upsampling module; wherein the smooth particle network is used to perform preliminary reconstruction of each sample; the anisotropic diffusion neural network is used to perform feature hierarchical diffusion of each sample; the fusion module is used to perform element-by-element fusion of the reconstruction result of each sample with the feature hierarchical diffusion result; and the upsampling module is used to upsample the reconstructed feature map rich in detail information obtained by fusion; A smooth particle network, comprising a cascaded convolutional module and D hybrid difference equation networks, D ≥ 2; each hybrid difference equation network comprises an upper branch consisting of a third convolutional layer and a cross-attention module stacked thereon, and a lower branch using the cross-attention module, wherein the cross-attention module comprises a cascaded cross-subnetwork, a cross-attention layer, and a bottleneck module; Anisotropic diffusion neural network, including cascaded convolutional layers and PMD feature guidance layers, where PMD refers to Perona-Malik diffusion; The fusion module includes multiple alternately stacked convolutional layers and nonlinear ReLU function layers; (3) Iteratively train the reconstruction model O: The training sample set R1 is used as the input of the reconstruction model O based on the anisotropic diffusion network for iterative training. The implementation steps are as follows: (3a) The number of initial iterations is s, the maximum number of iterations is S, S ≥ 10000, and the reconstructed model O of the sth iteration s The learnable weights and bias parameters are w s 、b s , and let s = 1; (3b) The smooth particle network performs preliminary image reconstruction on each training sample, and the anisotropic diffusion neural network performs feature hierarchical diffusion on each sample; the fusion module performs element-by-element fusion of the nth initially reconstructed feature map rich in structural details and its corresponding feature diffusion map after feature hierarchical diffusion; the upsampling module uses the PixelShuffle method to upsample each fused feature map containing rich structural details and less noise to obtain N reconstructed PG-SPECT images; (3c) The loss function is calculated using the L1 norm, and each PG-SPECT image obtained by reconstruction and its corresponding training sample label I n Calculate O s The loss value L s , then through L s Yes s 、b s Update to obtain the reconstructed network model O of this iteration s ; (3d) Determine whether s ≥ S. If so, obtain the trained reconstruction model O * , otherwise, let s = s + 1, O s =0, and execute step (3b); (4) Obtain the reconstruction results of PG-SPECT images: The test sample set E1 is used as the input of the trained reconstruction model O* for forward propagation to obtain the PG-SPECT reconstructed images corresponding to all test samples.

2. The method according to claim 1, characterized in that The preprocessing of the K acquired PG-SPECT images in step (1) is implemented by: Each PG-SPECT image is randomly flipped horizontally and then rotated 90 degrees to achieve data enhancement, and each data-enhanced PG-SPECT image is cropped into a PG-SPECT image of size H×H, where H≥512.

3. The method according to claim 2, characterized in that The step (1) of downsampling the N pre-processed PG-SPECT images is specifically as follows: downsampling each pre-processed PG-SPECT image by 1 / 4 to obtain K images of size PG-SPECT images.

4. The method according to claim 1, characterized in that: The smooth particle network described in step (3b) performs preliminary image reconstruction on each training sample, and the implementation steps are as follows: The smooth particle network includes cascaded convolution modules and D hybrid difference equation networks, each hybrid difference equation network includes an upper branch consisting of a third convolution layer and a cross-attention module stacked thereon arranged in parallel, and a lower branch using a cross-attention module, the cross-attention module includes a cascaded cross-subnetwork, a cross-attention layer and a bottleneck module; wherein the convolution module performs feature extraction, and the subsequent D hybrid difference equation networks extract context information to obtain a preliminary reconstructed image.

5. The method according to claim 1, characterized in that O described in step (3c) s The loss value L s , the calculation formula is: Among them, argmin is the value of the variable that makes the following formula reach the minimum value, and || || represents the norm.

6. The method according to claim 1, characterized in that The pair ω described in step (3c) s 、b s To update, the update formulas are: Among them, w s ', b s 'respectively represent w s 、b s The updated result, l r represents the learning rate, Respectively represent L s Yes s 、b s The partial derivative operation.

Images