Dynamic image motion correction method based on graph neural network

Abstract

The invention discloses a dynamic image motion correction method based on a graph neural network, and the method comprises the following steps: a), constructing a target region trajectory tracking model based on the graph neural network, and achieving the high-precision motion trajectory modeling; b) analyzing an image displacement mode in real time through a dynamic discrimination algorithm; and c) realizing adaptive image motion correction based on the target area track features. According to the method, the problem of complex motion trail modeling limitation caused by dependence on fixed template matching in a traditional method is innovatively solved, and the problem of spatial-temporal characteristic aliasing caused by unsteady state deformation of nervous tissues is effectively solved. According to the technical scheme, the dependence on a hardware synchronization signal acquisition module is eliminated, the resource configuration requirement of the edge computing equipment is remarkably reduced, and meanwhile, the multi-scale time sequence integration efficiency is improved. According to the method, dynamic imaging reconstruction of subcellular neural activities can be realized, and dynamic change details in a target neuron issuing process can be accurately restored.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of bio-imaging / medical image processing, in particular to a dynamic image motion correction method based on a graph neural network. BACKGROUND

[0002] The neuron activity detection technology based on fluorescence microscopic imaging is an important tool for neuroscience research. The voltage-sensitive fluorescent probe directly reflects the neural electrical activity through the membrane potential-dependent light intensity change, has the characteristics of non-invasive detection and subcellular level spatiotemporal resolution, and is suitable for in vivo animal neuron imaging. Compared with traditional electrophysiological recording, the traditional electrophysiological technology is limited by the spatial resolution limitation of invasive electrodes and the insufficient multi-point synchronous detection capability; compared with calcium imaging technology, the detection is more sensitive and the delay is lower, so it has important significance for brain science / neuroscience research.

[0003] However, in vivo imaging, the dynamic deformation caused by the spontaneous motion of the organism, the respiratory rhythm and the mechanical disturbance of the microscopic system, leads to the spatial drift and time series distortion of the fluorescence signal, which seriously affects the characterization accuracy of the fluorescence microscopic imaging on the rapid neural signal conduction. Traditional correction methods are mostly based on the rigid body motion assumption or a single deformation model, which has three technical limitations: first, the static displacement modeling is difficult to adapt to the non-steady dynamic characteristics of the in vivo sample; second, the multi-region motion correlation mechanism is not established, which leads to local correction and global structure mismatch; third, the difference in the characterization of heterogeneous disturbance sources affects the robustness in complex scenarios. The motion compensation based on the optical flow method in the prior art is easily disturbed by noise, and the rigid registration method based on feature matching cannot effectively handle non-uniform deformation, which are difficult to meet the high-precision dynamic imaging requirements, and most of them use a single motion compensation model, which cannot effectively separate the rigid / non-rigid composite deformation characteristics, resulting in time domain aliasing and spatial distortion when reconstructing high-frequency electrical signals, restricting the full play of the advantages of voltage imaging technology. How to realize the collaborative analysis and hierarchical correction of heterogeneous motion fields with higher efficiency, and then break through the limitations of traditional single model correction mechanism in terms of spatiotemporal fidelity, is an important problem to be solved, which is of great significance to brain science research such as neural circuit analysis. SUMMARY

[0004] In order to solve the technical bottleneck that the traditional dynamic correction technology in in vivo microscopic imaging cannot effectively handle non-steady dynamic deformation, rigid / non-rigid motion coupling and multi-scale motion interference. The present application provides a dynamic image correction method based on a graph neural network. The method realizes the decoupling analysis and collaborative compensation of heterogeneous motion patterns by establishing a dynamic graph structure representation model of neuron activity, significantly improves the spatiotemporal fidelity of fluorescence microscopic imaging on neural electrical activity, and provides high spatiotemporal fidelity for neural microcircuit dynamic analysis and whole brain scale functional connectivity research in the field of brain science.

[0005] To this end, the application adopts the technical scheme as follows: a dynamic image motion correction method based on a graph neural network, the specific steps of which are as follows: One. A dynamic image motion correction method based on a graph neural network Comprising the following steps: Step 1) Construct a target neuron trajectory tracking model fused with a graph neural network: perform accurate segmentation of a target region (a region where a target neuron is located) in a fluorescence microscopic image, and further build a graph neural network architecture that fuses topological feature vector extraction, spatial graph coding, synaptic weight dynamic coding, and functional subtype identification; Step 2) Dynamically determine the image displacement mode: realize displacement field modeling based on the motion trajectory tracking module of step 1), and establish a dynamic characterization of the target region motion trajectory; calculate the relative spatial offset of each frame of the target region from its average position, and determine whether the image is a rigid jitter image or a non-rigid jitter image according to the spatial offset; Step 3) Realize motion correction of the fluorescence microscopic image according to the target region trajectory: Step 3.1) For a rigid jitter image, average the trajectory points of the target region, and use the difference between the instantaneous positions of all target regions at the same time and the average points for reverse mapping to realize image correction; Step 3.2) For a non-rigid jitter image, implement a dual-domain correction strategy based on a graph neural network.

[0006] The step 1) is specifically: Step 1.1) Perform the following processing on all target neurons in all frames of the collected two-photon fluorescence microscopic imaging video: Calculate the threshold corresponding to the collected fluorescence microscopic imaging video using the following formula to obtain the corresponding threshold curve: Wherein, represents the accumulation of all pixel values in the marked region of the current neuron at the current frame number (time t ), represents the average value of ; Take the minimum value on the threshold curve as the segmentation threshold, and further segment all neurons in the two-photon fluorescence microscopic imaging video based on the obtained segmentation threshold to mark the contour of each neuron as the input of the graph neural network; Step 1.2) Based on the multi-level feature propagation mechanism and dynamic topology modeling capability of the graph neural network GNN, and using its heterogeneous motion field analysis advantage and spatial correlation modeling characteristics, a target neuron trajectory tracking model with dynamic topology reconstruction capability is constructed.

[0007] The step 2) is specifically: Step 2.1) modeling the displacement field of the target neuron based on the motion trajectory tracking module of step 1), and establishing the dynamic characterization of the motion trajectory of the target region; The target region is the segmented target neuron; Step 2.2) obtaining the average position matrix of the center of mass of each target region through spatial averaging operation, then calculating the relative spatial offset of the displacement vector of each target region in each frame and the average position, and discriminating whether the neuron image is a rigid jitter image or a non-rigid jitter image according to the spatial offset.

[0008] The method for discriminating rigid jitter or non-rigid deformation in step 2.2) is: When detecting that all target regions present spatial consistency translation under the same frame, it is determined as a rigid jitter image; if there is significant heterogeneity displacement mode between all target regions under the same frame, it is identified as a non-rigid jitter image.

[0009] The step 3.1) is specifically: First, the multi-scale target region segmentation extraction is implemented on the rigid jitter image, the topological feature matrix obtained by segmentation is input into the target neuron trajectory tracking model of step 2) for spatiotemporal feature coding, and then the motion trajectory parameters of each target region are solved; The average site information of each target region is obtained through spatial averaging operation, after establishing the reference coordinate system based on the site information, the distance matrix of the instantaneous coordinates of the target region in each frame and the average site is calculated, and the rigid displacement field quantization index is obtained; Based on the rigid displacement field quantization index, the reverse affine transformation is realized on each frame of image, and finally the image correction of rigid jitter is realized.

[0010] The step 3.2) is specifically: First, the adaptive segmentation algorithm is used to accurately define the target neuron region of the fluorescence microscopic image, the dynamic topological graph is constructed based on the target neuron trajectory tracking model of step 1) fusion graph neural network, and the nonlinear motion trajectory of each target neuron region is analyzed based on step 2.1); Then, after constructing the reference motion field based on the nonlinear motion trajectory set, the relative offset parameters of the displacement vector field of the segmented target neuron region and the reference field are calculated frame by frame; according to the relative offset parameters, the inverse affine transformation is implemented on each target region to realize feature correction; Further, for the background region, the affine registration matrix is constructed through global offset estimation, and the jitter correction of the non-feature region is completed by using linear interpolation algorithm, realizing the cooperative correction of non-rigid jitter image in the target region and the background.

[0011] The global offset of each frame background region is obtained by averaging the relative offset parameters of each target region under the current frame.

[0012] Two, a dynamic image motion correction device based on graph neural network comprising a memory and a processor; The memory is used for storing a computer program. The processor is used for implementing the above-mentioned dynamic image motion correction method based on graph neural network when executing the computer program.

[0013] Three, a computer readable storage medium The storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned dynamic image motion correction method based on graph neural network is implemented.

[0014] The beneficial effects of the present application are: The present application proposes an image dynamic correction method based on graph neural network for widely used fluorescence microscopic imaging, to solve the problems of spatial drift of fluorescence signal and time sequence distortion, and the decline of characterization accuracy of rapid neural signal conduction caused by dynamic deformation of biological spontaneous motion, respiratory rhythm and mechanical disturbance of microscopic system in fluorescence microscopic imaging. It can truly and accurately reflect the dynamic change process of neuron firing signal, and can reproduce the neural activity curve in voltage imaging data with higher time resolution, which has guiding significance for the application research and expansion of fluorescence microscopic imaging in the field of brain science. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is the flow chart of the dynamic image motion correction method based on graph neural network provided by the present application.

[0016] Figure 2 is the schematic diagram of dynamic discriminant image displacement mode provided by the present application.

[0017] Figure 3 is the dynamic image correction schematic diagram based on graph neural network provided by the present application. DETAILED DESCRIPTION

[0018] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0019] Step 1) Constructing a target neuron trajectory tracking model fused with graph neural network: the target region (the region where the target neuron is located) of the fluorescence microscopic image is precisely segmented, and a graph neural network architecture fused with "topological feature vector extraction-spatial graph coding-synaptic weight dynamic coding-functional subtype recognition" is further built.

[0020] Step 2) Dynamic discrimination of image displacement mode: Based on the motion trajectory tracking module constructed in step 1), the displacement field modeling is realized, and the dynamic characterization of the target region motion trajectory is established. Then the average position matrix of the center of mass of each target region is obtained through spatial averaging operation, and then the relative spatial offset of each frame target region and its average position is calculated, and whether it is rigid jitter or non-rigid deformation is judged according to the spatial offset.

[0021] Step 3) Image motion correction according to target region trajectory: According to step 2), for rigid jitter image, the target region displacement field is established, the distance matrix of the instantaneous coordinates of the target region in each frame and the average site is calculated, and the rigid displacement field quantitative index is obtained. Based on this, the inverse affine transformation of each frame image is realized, and the image correction of rigid jitter is realized; and for non-rigid image jitter, the reference motion field is constructed based on the motion trajectory set, and the relative offset parameters of the target region displacement vector field and the reference field are calculated frame by frame. According to the local deformation field, the inverse affine transformation of the target region is implemented to realize the feature correction. Further, for the background region, the affine registration matrix is constructed through the global offset estimation, and the jitter correction of the non-feature region is completed by using the bilinear interpolation algorithm, realizing the collaborative correction of non-rigid jitter in the target region and the background, so as to obtain the corrected fluorescence microscopic image Specific implementation method: The jittered fluorescence microscopic images collected are taken as the input of the motion correction model, and the average position matrix of the center of mass of each target region is obtained through spatial averaging operation, and then the relative spatial offset of the frame-by-frame displacement vector of the target region in each frame and the average position is calculated and judged to be rigid jitter or non-rigid deformation. If it is a rigid jitter image, the trajectory points of the average target region are obtained, and the difference between the instantaneous position of all target regions at the same time and the average site is used for inverse mapping to realize image correction. If it is non-rigid deformation, the relative offset parameters of the target region displacement vector field and the reference field are calculated frame by frame, and the non-rigid deformation correction of each target region is realized according to the relative offset parameters. Through systematic verification, the method proposed by us shows significant advantages in image rigid motion correction and non-rigid deformation. On the real data set of rigid jitter, the experimental results show that the method proposed by us significantly improves the correction accuracy while maintaining the integrity of the subcellular structure. Compared with the existing motion correction algorithms (such as NoRMCorre, Patchwarp), the peak signal-to-noise ratio is improved by 23% and 77% compared with NoRMCorre and Patchwarp, respectively. At the same time, on the simulated data of non-rigid deformation, the correlation coefficient between DeepMoCo algorithm and ground truth is significantly higher than that of NoRMCorre, Patchwarp and other algorithms.

Claims

1. A dynamic image motion correction method based on graph neural networks, characterized in that, Includes the following steps: Step 1) Construct a target neuron trajectory tracking model fused with graph neural network: Accurate segmentation of target regions is performed on fluorescence microscopy images, and a graph neural network architecture is further constructed; Step 2) Dynamically determine the image displacement mode: Based on the motion trajectory tracking module in Step 1), realize the displacement field modeling and establish the dynamic representation of the motion trajectory of the target area; Calculate the relative spatial offset between the target region and its average position in each frame, and determine whether the image is rigidly jittered or non-rigidly jittered based on the spatial offset. Step 3) Motion correction of the fluorescence micrograph based on the trajectory of the target region: Step 3.1) For rigid jitter images, image correction is achieved by averaging the trajectory positions of the target region and using the difference between the instantaneous positions of all target regions and the average position at the same time for inverse mapping. Step 3.2) For non-rigid jitter images, implement a dual-domain correction strategy based on graph neural networks.

2. The dynamic image motion correction method based on graph neural networks according to claim 1, characterized in that, Step 1) specifically refers to: Step 1.1) Perform the following processing on all target neurons in all frames of the acquired two-photon fluorescence microscopy video: The threshold corresponding to the acquired fluorescence microscopy imaging video is calculated using the following formula, thereby obtaining the corresponding threshold curve: in, This represents the sum of all pixel values ​​in the region labeled by the current neuron in the current frame. express The average value; The minimum value on the threshold curve is taken as the segmentation threshold. Based on the obtained segmentation threshold, all neurons in the two-photon fluorescence microscopy imaging video are automatically segmented, and the outline of each neuron is marked as the input of the graph neural network. Step 1.2) Based on the multi-level feature propagation mechanism and dynamic topology modeling capability of graph neural networks (GNNs), and taking advantage of their heterogeneous motion field analysis and spatial correlation modeling characteristics, a target neuron trajectory tracking model with dynamic topology reconstruction capability is constructed.

3. The dynamic image motion correction method based on graph neural networks according to claim 1, characterized in that, Step 2) specifically refers to: Step 2.1) Based on the motion trajectory tracking module in Step 1), the displacement field model of the target neuron is realized, and the dynamic representation of the motion trajectory of the target region is established; Step 2.2) Obtain the average position matrix of the centroid of each target region through spatial averaging operation, then calculate the relative spatial offset between the displacement vector of the target region and the average position in each frame, and determine whether the neuron image is a rigid jitter image or a non-rigid jitter image based on the spatial offset.

4. The dynamic image motion correction method based on graph neural networks according to claim 3, characterized in that, The method for distinguishing between rigid jitter and non-rigid deformation in step 2.2) is as follows: When a spatially consistent translation is detected across the entire target region within the same frame, it is identified as a rigid jitter image; if there are significant heterogeneous displacement patterns among the entire target region within the same frame, it is identified as a non-rigid jitter image.

5. The dynamic image motion correction method based on graph neural networks according to claim 1, characterized in that, Step 3.1) specifically refers to: First, multi-scale target region segmentation and extraction are performed on the rigid jitter image. The topological feature matrix obtained from the segmentation is then input into the target neuron trajectory tracking model in step 2) for spatiotemporal feature encoding, and then the motion trajectory parameters of each target region are calculated. The average position information of each target region is obtained by spatial averaging. After establishing a reference coordinate system based on the position information, the distance matrix between the instantaneous coordinates of the target region and the average position in each frame is calculated to obtain the rigid displacement field quantization index. Based on the rigid displacement field quantization index, an inverse affine transformation is performed on each frame of the image to ultimately achieve image correction of rigid jitter.

6. The dynamic image motion correction method based on graph neural networks according to claim 3, characterized in that, Step 3.2) specifically refers to: First, an adaptive segmentation algorithm is used to accurately define the target neuron region of the fluorescence microscopy image. Based on step 1), a dynamic topology map is constructed using the target neuron trajectory tracking model of the fused graph neural network. Then, based on step 2.1), the nonlinear motion trajectory of each target neuron region is analyzed. Subsequently, after constructing a reference motion field based on a set of nonlinear motion trajectories, the relative offset parameters between the displacement vector field of the segmented target neuron region and the reference field are calculated frame by frame; and feature correction is achieved by performing an inverse affine transformation on each target region according to the relative offset parameters. Furthermore, for the background region, an affine registration matrix is ​​constructed through global offset estimation, and a linear interpolation algorithm is used to complete the jitter correction of non-feature regions, thereby achieving the collaborative correction of non-rigid jitter images in the target region and the background.

7. A dynamic image motion correction device based on graph neural networks, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the dynamic image motion correction method based on graph neural networks as described in any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the dynamic image motion correction method based on a graph neural network as described in any one of claims 1 to 6.

Images