Multi-modal efficient system matrix super-resolution calibration method based on graph convolutional neural network

Abstract

The invention belongs to the field of magnetic particle imaging system matrix super-resolution, particularly relates to a multi-modal efficient system matrix super-resolution calibration method based on a graph convolutional neural network, and aims to solve the technical problems that an existing system matrix calibration method is low in precision and slow in speed. The method comprises the following steps: acquiring a low-resolution system matrix of a target sample and a corresponding system receiving coil channel and a frequency index; constructing a graph structure by taking rows of the low-resolution system matrix as nodes of the graph structure and taking a receiving coil channel and a frequency index as node features of the graph structure; processing the graph structure through a mixed model of graph convolution and image convolution to obtain an updated low-resolution system matrix; and obtaining a high-resolution system matrix corresponding to the updated low-resolution system matrix through an up-sampling module and a convolution module. According to the method, the graph structure is combined with the multi-modal information, the calibration precision is improved, the convolutional neural network is used for processing and analyzing the data, and the calibration speed is increased.

Description

Background Art

[0002] In the field of magnetic particle imaging (MPI), the system matrix reconstruction method is one of the most widely used reconstruction methods, which obtains higher image quality compared with the X-space based method.

[0003] The system matrix reconstruction method uses the system matrix (SM) as the mapping between the MPI image and the MPI signal, and recovers the MPI image from the MPI signal. The linear relationship is:

[0004] Sc = u

[0005] where, represents the system matrix, represents the MPI image vector, represents the MPI signal vector. The integer N represents the number of pixels of the MPI image and also represents the gridding size of the system matrix. The integer M represents the number of frequency rows of the system matrix.

[0006] For the measurement of the system matrix, it is necessary to repeatedly move a delta MNP sample on each voxel in the field of view (FOV) and record the corresponding signal. Traditional SM calibration usually requires multiple measurement averages to improve the calibration quality, which significantly increases the calibration time. In recent years, research has proposed deep learning-based super-resolution techniques to accelerate SM calibration, so as to capture the high-resolution (HR) SM from the low-resolution (LR) SM measurement, thereby improving the resolution and quality of the subsequently reconstructed MPI image. Existing calibration methods often regard an SM row as an independent data point. This modeling method ignores the integrity of the system matrix and the relationship between frequency components. In fact, the frequency components of the system matrix are not completely independent. For example, each frequency component contains two additional information components: the coil channel and the frequency index. Existing research has shown that the images reconstructed from the same receiving coil or SM rows with similar frequencies are more similar. This shows that the coil channel and the frequency index will establish a certain connection between different SM rows, which can be regarded as additional multi-modal information. When performing the system matrix super-resolution task, considering these two hardware information of the coil channel and the frequency index and integrating them into the system matrix calibration model is beneficial to improving the calibration accuracy.

[0007] Convolutional Neural Networks (CNNs), as a powerful model, have been applied to many different problems related to 2D and 3D images, such as image segmentation and classification. A graph is a complex relational network composed of nodes and edges. Graph Convolutional Networks (GCNs) are a generalization of Convolutional Neural Networks (CNNs) on graph-structured data. GCNs can capture the complex relationships between nodes in a graph and utilize these relationships for tasks such as node classification and graph classification. Although there have been some studies on GCNs for disease prediction, medical image analysis, etc. in the medical field, no study has represented the relationships between the rows of a system matrix using a graph structure and constructed GCNs for system matrix super-resolution.

[0008] In summary, we consider using hardware information to model the graph relationships of SM row data, using CNNs to extract the feature information of the system matrix, and combining GCNs to fuse the corresponding hardware information, ultimately achieving high-quality super-resolution fast calibration of the system matrix. Summary of the Invention

[0009] To solve the above problems in the prior art, that is, the technical problems of low accuracy and slow speed in the existing system matrix calibration method, the present invention provides a multi-modal efficient system matrix super-resolution calibration method based on graph convolutional neural networks, including:

[0010] Obtain the low-resolution system matrix of the target sample, as well as the corresponding system receiving coil channels and frequency indices;

[0011] Use the rows of the low-resolution system matrix as the nodes of the graph structure, and the receiving coil channels and frequency indices as the node features of the graph structure to construct the graph structure;

[0012] Perform convolution operations on the graph structure through the multi-layer convolutional layers of the graph convolution and image convolution hybrid model, extract the deep graph features of the target sample, and obtain an updated low-resolution system matrix;

[0013] Obtain the high-resolution system matrix corresponding to the updated low-resolution system matrix through the upsampling module and the convolution module.

[0014] Preferably, the construction of the graph structure specifically includes:

[0015] Calculate the similarity γ of the receiving coil channels k and the similarity γ of the frequency indices f , and obtain the similarity index γ between the row S i of the low-resolution system matrix and its neighboring system matrix row S j ​s (S i , S j );

[0016] Based on the similarity index γ s (S i , S j ), calculate the graph edge weight A of the row S i with the surrounding neighbor matrix; ij ;

[0017] Based on the nodes of the graph structure, the row S i with the graph edge weight A of the surrounding neighbor matrix ij , complete the construction of the graph structure on the low-resolution system matrix.

[0018] Preferably, when the receiving coil channels corresponding to the row S i and the row S j in the low-resolution system matrix are the same, γ k = 1; when they are different, γ k = 0.

[0019] Preferably, the γ f is a unit step function with respect to the threshold ε, and the threshold ε is set according to the frequency index of the MPI data set corresponding to the target sample.

[0020] Preferably, when the absolute value of the difference between the frequency indices corresponding to the row S i and the row S j in the low-resolution system matrix is less than the threshold ε, γ f = 1; when it is greater than or equal to the threshold ε, γ f = 0.

[0021] Preferably, the similarity index γ i between the row S j and the row S s (S i , S j ) in the low-resolution system matrix is the weighted sum of the similarity γ k of the corresponding receiving coil channels and the similarity γ f of the frequency indices.

[0022] Preferably, the graph edge weight A of the row S i with the surrounding neighbor matrix ij , is the sum of the similarity indices ∑γ s (S i , S j ) of γ i and all rows of the low-resolution system matrix connected thereto s (Si , S j ) in the ratio. The convolution operation of the graph structure is performed through the multi-layer convolution layer of the graph convolution and image convolution hybrid model to extract the deep graph features of the target sample, and an updated low-resolution system matrix is obtained, including:

[0023] Through the multi-layer convolution layer, the row S in the low-resolution system matrix i and the neighbor matrix row S around it j are convolved to obtain the features Conv(S i ) and Conv(S j ) after the convolution operation;

[0024] The self-loop coefficient λ is preset for different MPI data sets, and at the same time, combined with the graph edge weight A ij the features Conv(S i ) and Conv(S j ) are weighted and summed, and corrected through the ReLU function to obtain the updated low-resolution system matrix row S' i , and the updated low-resolution system matrix is sorted out.

[0025] On the other hand, the present invention proposes an electronic device, including:

[0026] At least one processor; and

[0027] A memory communicatively connected to at least one of the processors;

[0028] Wherein, the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned multi-modal efficient system matrix super-resolution calibration method based on the graph convolutional neural network.

[0029] In the third aspect of the present invention, a computer-readable storage medium is proposed. The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned multi-modal efficient system matrix super-resolution calibration method based on the graph convolutional neural network.

[0030] Advantages of the present invention:

[0031] (1) When calibrating the system matrix, the method of the present invention considers the relationship generated between different system matrix rows by the coil channels and frequency indices, regards them as additional multi-modal information, models and calibrates the system matrix, and improves the calibration accuracy compared with the traditional system matrix calibration.

[0032] (2) The method of the present invention represents the characteristic information of the system matrix through a graph structure, and uses a graph convolutional neural network to process the graph structure data for system matrix super-resolution. It can quickly capture the complex relationships between nodes in the graph, and has a faster speed and higher quality compared with traditional system matrix calibration. Description of the Drawings

[0033] Other features, objects, and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0034] Figure 1 is a schematic flowchart of the method of the present invention;

[0035] Figure 2 is a schematic diagram of the graph structure of the rows of the constructed system matrix;

[0036] Figure 3 is a schematic flowchart of the system matrix super-resolution based on the hybrid model of graph convolution and image convolution; Detailed Embodiments

[0037] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention and are not intended to limit the invention. Additionally, it should be noted that for the sake of description, only the parts related to the relevant invention are shown in the drawings.

[0038] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0039] See Figure 1 , which is a schematic flowchart of the method of the present invention. The present invention provides a multi-modal efficient system matrix super-resolution calibration method based on a graph convolutional neural network. The calibration method includes:

[0040] Obtain the low-resolution system matrix of the target sample, as well as the corresponding system receiving coil channels and frequency indices;

[0041] Use the rows of the low-resolution system matrix as the nodes of the graph structure, and use the receiving coil channels and frequency indices as the node features of the graph structure to construct a graph structure;

[0042] Perform convolution operations on the graph structure through the multi-layer convolution layers of the hybrid model of graph convolution and image convolution, extract the deep graph features of the target sample, and obtain an updated low-resolution system matrix;

[0043] Obtain the high-resolution system matrix corresponding to the updated low-resolution system matrix through an upsampling module and a convolution module.

[0044] The method of the present invention is an improvement based on the traditional system matrix reconstruction method. When calibrating the system matrix, the coil channels and frequency index information of the target sample receiving coil are used as multi-modal information, and a graph structure is constructed in combination with the low-resolution system matrix. The graph structure is processed through graph convolution operations to extract deep feature information. Combining the upsampling module and the convolution module, the details of the feature map image are retained, and the accuracy of super-resolution is improved. Finally, it is converted into a high-resolution system matrix. In the traditional system matrix reconstruction method, the integrity of the system matrix and the relationship between the coil channels and frequency indices are ignored. Existing research shows that the images reconstructed from the same receiving coil or SM rows with similar frequencies are more similar, that is, the coil channels and frequency indices will establish a certain connection between different SM rows. The method of the present invention optimizes this point. When performing the system matrix super-resolution task, the coil channels and frequency indices are regarded as additional multi-modal information for modeling and calibration, effectively improving the calibration accuracy.

[0045] See Figure 2 , which is a schematic diagram of the graph structure of the system matrix row constructed by the method of the present invention, and its construction method is as follows:

[0046] Obtain the low-resolution system matrix of the target sample and the corresponding system receiving coil channels and frequency indices;

[0047] Using the rows of the low-resolution system matrix as the nodes of the graph structure and the receiving coil channels and frequency indices as the node features of the graph structure, construct a graph structure;

[0048] Among them, constructing the graph structure means calculating the similarity γ k of the receiving coil channels and the similarity γ f of the frequency indices, and obtaining the similarity index γ i between the row S j in the low-resolution system matrix and its neighboring system matrix row S s (S i , S j );

[0049] Based on the similarity index γ s (S i , S j ), calculate the graph edge weight A i of the row S ij with the surrounding neighbor matrices;

[0050] Based on the nodes of the graph structure, the graph edge weight A i of the row S ij with the surrounding neighbor matrices, complete the construction of the graph structure on the low-resolution system matrix.

[0051] Although in the medical field, some studies have applied graph convolutional neural networks to capture the complex relationships between nodes in the graph structure, in the field of matrix super-resolution of magnetic particle imaging systems, no research has represented the relationships between the rows of the system matrix using a graph structure. In the present invention, by taking the rows of the system matrix as graph nodes, the frequency index and coil channels as node features, and the edges of the graph being jointly determined by the similarity of the frequency index and coil channels, it is possible to utilize the powerful ability of the graph structure to express complex relationships while considering the connection between the frequency index and coil channels for the rows of the system matrix.

[0052] See Figure 3 , which is a schematic diagram of the system matrix super-resolution process based on the hybrid model of graph convolution and image convolution of the present invention, and its process includes:

[0053] Perform convolution operations on the graph structure through the multi-layer convolution layer of the hybrid model of graph convolution and image convolution to extract the deep graph features of the target sample and obtain an updated low-resolution system matrix;

[0054] Through the upsampling module and the convolution module, improve the resolution of the updated low-resolution system matrix to generate a high-resolution system matrix, and calculate the Train the model parameters between the generated high-resolution system matrix and the ground truth of the dataset.

[0055] Convolutional neural networks are applied in many different problems related to 2D and 3D images, such as image segmentation and classification. Graph convolutional neural networks are a generalization of convolutional neural networks on graph-structured data and can capture the complex relationships between nodes in the graph. In the present invention, for the constructed graph structure, convolutional neural networks are used to extract the feature information of the system matrix, combined with graph convolutional neural networks to fuse the corresponding hardware information, and finally achieve high-quality super-resolution and fast calibration of the system matrix.

[0056] In order to more clearly illustrate a multi-modal efficient system matrix super-resolution calibration method based on graph convolutional neural networks of the present invention, each module in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0057] A multi-modal efficient system matrix super-resolution calibration method based on graph convolutional neural networks according to the first embodiment of the present invention includes the following steps:

[0058] Step S100, obtain the low-resolution system matrix of the target sample and the corresponding system receiving coil channels and frequency indices.

[0059] Step S200, take the rows of the low-resolution system matrix as the nodes of the graph structure, and the receiving coil channels and frequency indices as the node features of the graph structure to construct the graph structure.

[0060] Step S210, calculate the similarity γ of the receiving coil channels respectivelyk and the similarity γ of the frequency index f , and obtain the row S in the low-resolution system matrix i and the similarity index γ between its neighboring system matrix rows S j (S s (S i , S j ).

[0061] For two SM rows S i , S j , define the function γk as the similarity measure of the coil channel information. If the coil channels are the same, γk is 1; otherwise, it is 0. Define γf as the similarity measure of the frequency index information. γf is a unit step function with respect to the threshold ε. For different data sets, the value of the threshold ε is different. Taking the OpenMPI data set as an example, its frequency index is a continuous variable from 25 kHz to 500 kHz, and ε can be set to 10 kHz. As shown in Equations (1) and (2):

[0062]

[0063]

[0064] When the frequency indices between S i , S j are close or the coil channels are the same, their similarity increases accordingly. The similarity index γs between every two SM rows is as shown in Equation (3):

[0065] γ s (S i , S j ) = γ k + γ f (3)

[0066] Step S220, based on the similarity index γ s (S i , S j ), calculate the graph edge weight A i of row S ij with the surrounding neighbor matrix. Based on the nodes of the graph structure, the graph edge weight A i of row S ij with the surrounding neighbor matrix, complete the construction of the graph structure on the low-resolution system matrix.

[0067] Among them, considering the SM with M rows, the graph edge weight A i of S ij with the surrounding neighbor matrix is defined as shown in Equation (4):

[0068]

[0069] Among them, ∑γ s (S i , S j ) represents the sum of the similarity indices between S i and all the SM rows connected to it.

[0070] Step S300: Perform a convolution operation on the graph structure through the multi-layer convolutional layers of the graph convolution and image convolution hybrid model, extract the deep graph features of the target sample, and obtain an updated low-resolution system matrix.

[0071] Among them, the graph convolution and image convolution hybrid model includes multi-layer convolutional layers based on graph convolutional neural networks, an upsampling module, and a convolutional module, which are used to extract the deep graph features of the target sample, update the low-resolution system matrix, and super-resolve the updated low-resolution system matrix to obtain the corresponding high-resolution system matrix.

[0072] Step S301: Through the multi-layer convolutional layers, perform convolution on the row S i of the low-resolution system matrix and the neighboring matrix row S j around it, and obtain the features Conv(S i ), Conv(S j ) after the convolution operation.

[0073] Step S302: Preset a self-loop coefficient λ for different MPI datasets, and at the same time, combine the graph edge weight A ij to perform weighted summation on the features Conv(S i ), Conv(S j ), and correct it through the ReLU function to obtain the updated row S′ i of the low-resolution system matrix, and organize it to obtain the updated low-resolution system matrix.

[0074] Among them, for a specific SM row S i , its features are jointly determined by hardware information (frequency index, coil channel) and software information. After constructing the graph structure, the graph convolutional neural network performs a convolution operation on the graph structure, which can simultaneously utilize the structural information of the graph and the node feature information, and extract the deep features of the graph through multi-layer convolutional layers. The new SM row S′ i can be calculated by the following formula (5):

[0075] S′ i = ReLU(∑A ij Conv(S j ) + λConv(S i )) (5)

[0076] Among them, λ is the self-loop coefficient preset according to different MPI datasets, which is used to adjust Si The weight of λ varies for different datasets.

[0077] Step S400: Through the upsampling module and the convolutional module, improve the resolution of the updated low-resolution system matrix to generate a high-resolution system matrix, and calculate the training model parameters between the generated high-resolution system matrix and the dataset ground truth.

[0078] In the present invention, a graph structure is constructed based on the hardware information (frequency index, coil channels) and software information (low-resolution system matrix) of the system matrix, a hybrid convolutional neural network, and a graph convolution and image convolution hybrid model is constructed to generate a high-resolution system matrix. By using the graph structure and considering the hardware information of the system matrix for modeling, and using the neural network to analyze and process information, while improving the resolution of the system matrix, the calibration accuracy and speed of the system matrix are also improved.

[0079] An electronic device according to the second embodiment of the present invention includes:

[0080] At least one processor;

[0081] And a memory communicatively connected to at least one of the processors;

[0082] Wherein, the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned multi-modal efficient system matrix super-resolution calibration method based on a graph convolutional neural network.

[0083] A computer-readable storage medium according to the third embodiment of the present invention stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned multi-modal efficient system matrix super-resolution calibration method based on a graph convolutional neural network.

[0084] Those skilled in the art of the present technology can clearly understand that for the convenience and brevity of description, the specific working processes and related descriptions of the above-described storage device and processing device can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0085] Those skilled in the art should be able to realize that the modules and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field. To clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in the form of electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0086] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or represent a specific order or sequence.

[0087] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, so that a process, method, article, or device / apparatus comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in these processes, methods, articles, or devices / apparatuses.

[0088] So far, the technical solution of the present invention has been described in combination with the preferred embodiments shown in the drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims

1. A multi-modal efficient system matrix super-resolution calibration method based on graph convolutional neural network, characterized in that: The calibration method comprises: Obtain a low-resolution system matrix of the target sample and the corresponding system receiving coil channel and frequency index; Using the rows of the low-resolution system matrix as nodes of a graph structure, and using the receiving coil channels and frequency indexes as node features of the graph structure, to construct a graph structure; Performing a convolution operation on the graph structure through a multi-layer convolution layer of a hybrid model of graph convolution and image convolution, extracting deep graph features of the target sample, and obtaining an updated low-resolution system matrix; A high-resolution system matrix corresponding to the updated low-resolution system matrix is ​​obtained through an upsampling module and a convolution module.

2. The calibration method according to claim 1, characterized in that: The constructing of the graph structure specifically includes: Calculate the similarity γ of the receiving coil channels respectively k and the similarity of the frequency index γ f , and obtain row S in the low-resolution system matrix i and its neighbor system matrix row S j The similarity index γ between s (S i ,S j ); Based on the similarity index γ s (S i ,S j ), calculate the row S i The graph edge weights A with the surrounding neighbor matrix ij ; Based on the nodes of the graph structure, the row S i The graph edge weights A with the surrounding neighbor matrix ij , completing the graph structure construction on the low-resolution system matrix.

3. The calibration method according to claim 2, characterized in that: When the row S in the low-resolution system matrix i and line S j When the corresponding receiving coil channels are the same, γ k =1, different, γ k =0.

4. The calibration method according to claim 3, characterized in that: The gamma f is a unit step function about a threshold ε, where the threshold ε is set according to the frequency index of the MPI data set corresponding to the target sample.

5. The calibration method according to claim 4, characterized in that: When the row S in the low-resolution system matrix i and line S j When the absolute value of the difference of the corresponding frequency index is less than the threshold ε, γ f =1, when it is greater than or equal to the threshold ε, γ f =0.

6. The calibration method according to claim 2, characterized in that: The low-resolution system matrix has rows S i and line S j The similarity index γ between s (S i ,S j ) is the similarity γ of the corresponding receiving coil channel k and the similarity of the frequency index γ f The weighted sum of .

7. The calibration method according to claim 6, characterized in that: The row S i The graph edge weights A with the surrounding neighbor matrix ij , is the similarity index γ s (S i ,S j ) in S i The sum of the similarity indexes between the rows of all low-resolution system matrices connected to it ∑γ s (S i ,S j ) in the proportion.

8. The calibration method according to claim 1, characterized in that: The convolution operation of the graph structure is performed through the multi-layer convolution layer of the graph convolution and image convolution hybrid model to extract the deep graph features of the target sample and obtain an updated low-resolution system matrix, including: The rows S in the low-resolution system matrix are transformed through multiple convolutional layers. i and its surrounding neighbor matrix rows S j Convolution, obtain the feature Conv(S i )、Conv(S j ); The self-loop coefficient λ is pre-set for different MPI data sets, and the graph edge weight A is combined ij For the feature Conv(S i )、Conv(S j ) is weighted summed and corrected by the ReLU function to obtain the updated low-resolution system matrix row S′ i , and sort out the updated low-resolution system matrix.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the multi-modal efficient system matrix super-resolution calibration method based on graph convolutional neural network as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the multi-modal efficient system matrix super-resolution calibration method based on graph convolutional neural network as described in any one of claims 1-8.

Images