Single-particle diffusion quantification feature prediction method and device, electronic equipment and storage medium

Abstract

The application provides a single-particle diffusion quantitative characteristic prediction method and device, electronic equipment and storage medium, wherein the method comprises the following steps: obtaining a target input image corresponding to a target single particle; based on a pre-trained particle trajectory prediction model, a particle pseudo-trajectory image is predicted according to the target input image; and based on the particle pseudo-trajectory image, the diffusion quantitative characteristic of the target single particle is calculated; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of a single-particle motion blur image and a corresponding particle motion trajectory image. The method can quickly and accurately predict the diffusion coefficient of a large number of single particles in a high marker density environment, significantly reduce phototoxicity, and effectively distinguish the diffusion characteristics of different biological particles, thereby providing a powerful tool for the study of particle dynamics in diffusion systems and living cells.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of super-resolution imaging, and particularly relates to a single-particle diffusion quantitative feature prediction method and device, electronic equipment and storage medium. BACKGROUND

[0002] Traditional imaging methods are limited by the optical diffraction limit, and can only study the group molecular dynamic characteristics at a low spatiotemporal resolution. Single-molecule localization microscopy (SMLM) is a super-resolution imaging technology that won the Nobel Prize in Chemistry in 2014, and can image a single fluorescent particle or fluorescent biomolecule.

[0003] With the development of fluorescence microscopes and new labeling technologies, people use SMLM to perform real-time imaging, localization and tracking (single-particle tracking, SPT; single-molecule tracking, SMT; hereinafter referred to as SMT) of single particles or single biomolecules in extracellular diffusion systems and living cells. Analysis of the obtained molecular trajectories can obtain molecular diffusion coefficients, binding fractions, diffusion angle distributions (anisotropy), and other measurement indicators. Through these indicators, the biological activity of the labeled protein and the interaction strength with other molecules can be further inferred. This technological innovation is becoming a powerful tool for understanding the single-molecule dynamics and functions of biomolecules in the life processes of cells.

[0004] In the mainstream analysis of single-molecule dynamics in living cells that has been published so far, the general process is to perform single-molecule localization, connect the trajectories frame by frame, and then summarize and statistically analyze the length of the obtained molecular trajectories, and fit various physical models based on molecular diffusion to obtain parameters such as diffusion coefficients and binding percentages. After obtaining the original single-molecule fluorescence image, the single-molecule localization algorithm is first used to approximate the single-molecule fluorescence intensity distribution to the approximate function of the point spread function (PSF), i.e., the two-dimensional Gaussian function, to obtain the spatial coordinates corresponding to the maximum value of the function as the position of the molecule. Subsequently, statistical inference is used to connect the points closest in space and most likely to diffuse physically between two frames to form a trajectory.

[0005] To ensure the accuracy of trajectory linking, the output intensity of the photoactivating laser needs to be controlled. The more molecules present in each frame, the less accurate the algorithm will be in linking trajectories. To accurately obtain trajectories, low molecule density needs to be maintained throughout the imaging process to avoid false linking. Analysis of single molecule trajectories often relies on the length of the molecule trajectory, the longer the molecule trajectory, the more accurate the kinetic parameters extracted from it. However, in order to make subsequent kinetic analysis and model fitting statistically meaningful, it is necessary to obtain sufficient single molecule trajectory data in the experiment. Therefore, high frame rate imaging is required to capture a sufficient number of single molecules, during which the effects of phototoxicity and cell movement on the experiment cannot be ignored.

[0006] For the two-dimensional SMT imaging currently in common use, when the binding percentage of molecules is low and the diffusion coefficient is fast, it is difficult to obtain long trajectories under limited Z-axis resolution. On the other hand, most trajectory-based analyses require a large number of trajectories (usually at least one hundred thousand trajectories or more) to obtain statistically reliable results. A commonly used solution is to image many cells and pool the trajectories obtained from different cells together. Although the kinetic parameters obtained by statistically modeling a large number of single molecule trajectories are accurate, the results are averages and lack spatial information, which limits the ability to study single cell protein spatial dynamics using SMT.

[0007] Another limitation of trajectory-based analysis is the difficulty of measuring the dynamics of protein clusters. For example, RNA polymerase II (RNA Pol II, hereinafter referred to as Pol II) forms dynamic clusters in living cells with a radius close to or below the diffraction limit (the diffraction limit is about 200 nm). Since the spatial resolution of the current SMT using a highly inclined light sheet (HILO) mode total internal reflection fluorescence microscope is about 30 nm, it is essentially challenging to study the dynamic characteristics of clusters by sparsely labeling molecules within the clusters and tracking trajectories.

[0008] Currently, methods for characterizing the spatial distribution of molecular diffusion in living cells are still limited, including using trajectory average displacement to represent the speed of molecular motion at each point in space, but this method is affected by possible errors in the trajectory tracking process, including random blinking and photobleaching of fluorescent molecules leading to failed trajectory linking; and the density of molecules excited in each frame cannot be too high, otherwise it is easy to cause false linking. Therefore, this method requires long-term data collection, but long-term laser irradiation can cause the cell to move a certain distance, so this method is only suitable for imaging proteins that move slowly as a whole and do not change greatly in structure over time, such as chromatin histones.

[0009] In addition, there are also methods that directly use the tracked trajectories to analyze the spatial distribution of molecular diffusion, such as sptPALM. This method uses high-intensity laser excitation of molecules, uses a high numerical aperture objective lens (Olympus APO100XO-HR-SP, 1.65NA, while the numerical aperture NA value of the 100x lens commonly used is usually below 1.49) combined with total internal reflection illumination to track trajectories of cell membrane proteins in the light-tolerant COS7 cell line, and use the MSD-Δt fitting of each trajectory to obtain the diffusion coefficient to obtain the spatial distribution of membrane surface molecular diffusion.

[0010] On the one hand, this method relies on trajectory tracking, so there is a possibility of misconnection between molecules in each frame; on the other hand, this method relies on the fitting method of MSD-Δt, and this fitting requires a long molecular trajectory to obtain an accurate diffusion coefficient estimate. Since cell membrane proteins mainly diffuse in two dimensions, they are suitable for obtaining long trajectories (>10 frames); while in other environments such as the nucleus, molecules are three-dimensional diffusion, and the two-dimensional imaging method using HILO light illumination can only obtain a molecular trajectory length of 3-4 frames for stably bound molecules such as H2B, and the average molecular trajectory obtained for other faster diffusing molecules is even shorter, so there are difficulties in using the fitting method of MSD-Δt.

[0011] In addition, there is also a method of non-covalent binding of ATTO-647N dye molecule ligand modified with divalent nickel ion-trisnitrilotriacetic acid (Ni2+trisN-nitrilotriacetic acid, Ni-NTA) to membrane proteins with 6 histidine tags to achieve high-density labeling and tracking of membrane proteins, and finally using the median value of the trajectory displacement length in a 200nm space to represent the speed of molecular motion uPAINT (universal-points-accumulation-for-imaging-in-nanoscale-topography). This method improves the molecular imaging density compared to sptPALM, but reduces the accuracy of trajectory reconstruction, and in addition, this method can only be used to label membrane proteins.

[0012] In addition, there is also SMdM (Single-molecule displacement mapping) that arranges the sequence of two-frame stroboscopic exposure to distribute the 1ms stroboscopic exposure in the 1ms interval between two frames, which can further shorten the time interval between two frames and track molecular motion faster. This method tracks the trajectory displacement between two frames, and estimates the diffusion coefficient by fitting the distribution of all displacements within a fixed range (100x100nm).

[0013] This method is suitable for fast-moving molecules (diffusion coefficient higher than 10μm2 / s) in cells. However, it is found from simulation data that for transcriptional regulators with diffusion coefficient less than 10 μm 2 / s in the nucleus, the detection sensitivity of the motion speed of this part will be affected when considering the localization error of the imaged fluorescent molecules, because the average displacement of molecules with diffusion coefficient within 10 μm 2 / s is within 200 nm under a time interval of 1 ms, and when there is a 30 nm error for each molecule at the start of the trajectory, a total error range of 60 nm will have a great impact on the detection result.

[0014] Therefore, how to solve the problem that the traditional particle trajectory tracking technology has low prediction accuracy of diffusion quantification features and cannot capture the spatial distribution of particles in a high marker density environment is an important topic to be solved in the field of super-resolution imaging. SUMMARY

[0015] The present application provides a single particle diffusion quantification feature prediction method, device, electronic equipment and storage medium, which solves the defects of traditional single particle trajectory tracking technology in a high marker density environment, such as low prediction accuracy of diffusion quantification features and inability to capture the spatial distribution of particles, and can quickly and accurately predict the diffusion quantification features of a large number of single particles in a high marker density environment, and significantly reduce phototoxicity.

[0016] In one aspect, the present application provides a single particle diffusion quantification feature prediction method, comprising: obtaining a target input image corresponding to a target single particle; based on a pre-trained particle trajectory prediction model, predicting a particle pseudo-trajectory image according to the target input image; and calculating the diffusion quantification features of the target single particle based on the particle pseudo-trajectory image; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single particle motion blur images and their corresponding particle motion trajectory images.

[0017] Further, the obtaining of the target input image corresponding to the target single particle comprises: collecting an original single particle motion blur image of the target single particle; segmenting the original single particle motion blur image based on a pre-trained U-Net network to obtain a single particle signal mask; pre-localizing the single particle signal in the original single particle motion blur image to retain only a target single particle signal mask with one localization; filling the target single particle signal mask with the background and noise levels calculated by a median filter to obtain the target input image; wherein the U-Net network is obtained by training and optimizing a second training sample set composed of single particle motion blur images and their corresponding mask images.

[0018] Further, the diffusion quantitative feature of the target single particle includes a diffusion coefficient and a diffusion direction; accordingly, the diffusion quantitative feature of the target single particle is calculated based on the particle pseudo-trajectory image, including: determining a pseudo-trajectory area according to the particle pseudo-trajectory image; fitting the diffusion coefficient corresponding to the target single particle according to the quantitative relationship between the pseudo-trajectory area and the particle diffusion coefficient; fitting the diffusion direction corresponding to the target single particle according to the density space distribution of the particle pseudo-trajectory image.

[0019] Further, the training and optimization of the particle trajectory prediction model specifically includes: simulating a single particle motion blur image and obtaining a particle motion trajectory image corresponding to the single particle motion blur image to construct a first training sample set; taking the single particle motion blur image as the model input, taking the predicted pseudo-trajectory image as the model output, taking the difference between the predicted pseudo-trajectory image and the particle motion trajectory image as the training loss, and iteratively optimizing the particle trajectory prediction model to obtain a training-converged particle trajectory prediction model.

[0020] Further, the simulation of the single particle motion blur image includes: simulating a two-dimensional particle trajectory and superimposing a Gaussian function on each point on the two-dimensional particle trajectory to obtain a motion blur function; normalizing and pixelizing the motion blur function and introducing Gaussian white noise and Poisson shooting noise to obtain a single particle motion blur image under different signal-to-noise ratios and background levels.

[0021] Further, the fitting of the diffusion coefficient corresponding to the target single particle includes: in the case that the diffusion coefficient of the target single particle is greater than a set threshold, performing centroid calculation on the particle pseudo-trajectory image to obtain the positioning of the target single particle; in the case that the diffusion coefficient of the target single particle is less than or equal to a set threshold, performing ellipsoid Gaussian fitting on the target input image to obtain the positioning of the target single particle.

[0022] Further, after obtaining the positioning of the target single particle, it includes: gridding the positioning of the target single particle to obtain a first grid containing the particle positioning and a second grid not containing the particle positioning but adjacent to the particle positioning; generating a particle density probability map based on the positioning of the target single particle; based on the particle density probability map and the first grid, using a Gaussian weight sum to perform interpolation processing on the second grid to obtain the diffusion coefficient corresponding to the second grid; obtaining a diffusion coefficient matrix according to the diffusion coefficient corresponding to the first grid and the diffusion coefficient corresponding to the second grid; performing local smoothing processing on the diffusion coefficient matrix to obtain a motion property map and using an HSV color map for display to complete image rendering.

[0023] In a second aspect, the present application also provides a single-particle diffusion quantitative characteristic prediction device, comprising: a target input image acquisition module configured to acquire a target input image corresponding to a target single particle; a particle pseudo-trajectory image prediction module configured to predict a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model and the target input image; and a particle diffusion coefficient acquisition module configured to calculate a diffusion quantitative characteristic of the target single particle based on the particle pseudo-trajectory image; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single-particle motion blur images and corresponding particle motion trajectory images.

[0024] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the single-particle diffusion quantitative characteristic prediction method according to any one of the above aspects.

[0025] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the single-particle diffusion quantitative characteristic prediction method according to any one of the above aspects.

[0026] The single-particle diffusion quantitative characteristic prediction method provided by the present application comprises the following steps: acquiring a target input image corresponding to a target single particle, and predicting a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model and the target input image, and then calculating a diffusion quantitative characteristic of the target single particle based on the particle pseudo-trajectory image; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single-particle motion blur images and corresponding particle motion trajectory images. This method predicts the actual motion path of the target single particle from the target input image based on the particle trajectory prediction model, and calculates the diffusion quantitative characteristic of the target single particle based on the actual motion path, which can quickly and accurately predict the diffusion quantitative characteristics of a large number of single particles in a high-label-density environment, significantly reduces phototoxicity, and is suitable for live cell imaging for at least 10 minutes. The diffusion quantitative characteristic obtained by solving has a high dynamic range, effectively distinguishes the diffusion characteristics of different biological particles, and provides a powerful tool for the dynamic study of diffusion systems and particles in live cells. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without any creative effort.

[0028] Figure 1 is a flowchart of a single particle diffusion quantification feature prediction method provided by an embodiment of the present application.

[0029] Figure 2 is a structural diagram of a U-Net network provided by an embodiment of the present application.

[0030] Figure 3 is a second training sample set construction diagram of a single particle diffusion quantification feature prediction method provided by an embodiment of the present application.

[0031] Figure 4 is a first training sample set construction diagram provided by an embodiment of the present application.

[0032] Figure 5 is a training optimization diagram of a particle trajectory prediction model provided by an embodiment of the present application.

[0033] Figure 6 is a diffusion coefficient acquisition diagram of a target single particle provided by an embodiment of the present application.

[0034] Figure 7 is a fitting effect diagram of a linear relationship between a pseudo trajectory area and a particle diffusion coefficient provided by an embodiment of the present application.

[0035] Figure 8 is a positioning accuracy diagram of different particle positioning algorithms provided by an embodiment of the present application.

[0036] Figure 9 is a rendering diagram of particle positioning and particle diffusion coefficient provided by an embodiment of the present application.

[0037] Figure 10 is a whole flowchart of a single particle diffusion quantification feature prediction method provided by an embodiment of the present application.

[0038] Figure 11 is an inference effect diagram of a particle trajectory prediction model and a U-Net network provided by an embodiment of the present application.

[0039] Figure 12 is an image registration flowchart provided by an embodiment of the present application.

[0040] Figure 13 is a structure diagram of a single particle diffusion quantification feature prediction device provided by an embodiment of the present application.

[0041] Figure 14 is a physical structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0043] It is easy to understand that, in order to solve the problem that the conventional single-particle trajectory tracking technology cannot capture the spatial distribution of particles and the prediction accuracy of the diffusion coefficient is not high in a high marker density environment, the present application proposes a new single-particle diffusion quantitative feature prediction method, specifically, Figure 1 A flowchart of a single-particle (e.g., biomolecule) diffusion quantitative feature (e.g., diffusion coefficient) prediction method provided by an embodiment of the present application is shown.

[0044] As Figure 1 shown, the method comprises steps S110-S130, which will be described in detail below.

[0045] S110, obtaining a target input image corresponding to a target single particle.

[0046] It can be understood that, in order to make the single particle (e.g., single biomolecule) visible under a microscope, fluorescent labels need to be attached to the target single particle first. These fluorescent labels can emit light when irradiated by light of a specific wavelength, thereby making the particle visible. Common fluorescent labeling methods include antibody labeling, genetically engineered expression of fusion proteins with fluorescent proteins (such as GFP), etc.

[0047] Since the single particle (target single particle) is usually very weak, a high-sensitivity and high-resolution microscope system needs to be used. Common single-particle imaging microscope systems include but are not limited to photoactivated localization microscopy (PALM), stochastic optical reconstruction microscopy (STORM).

[0048] In order to observe the single particle (target single particle), it is necessary to accurately control the intensity and wavelength of the excitation light to avoid simultaneously exciting multiple adjacent fluorescent particles. In addition, a high-sensitivity detector such as an EMCCD camera or an APD (avalanche photodiode) needs to be used to capture extremely weak fluorescent signals.

[0049] After all the parameters are set, image acquisition begins. Long exposure or multiple exposures need to be accumulated to collect sufficient signals.

[0050] It should be noted that the collected images need to be processed to obtain relatively clear single particle images, i.e., the target input images described in the embodiment. The processing here includes but is not limited to background subtraction, signal enhancement, etc.

[0051] In this step, the target single particle can be a biomolecule (such as DNA, RNA, protein or other intracellular molecular structure, especially protein), and can also be a purified protein in a chemical solution, a chemically synthesized compound, a nanoparticle, etc., which is not specifically limited here.

[0052] The target input image is a single particle motion blur image / single particle fluorescence image after processing. The single particle motion blur image refers to the blur phenomenon in the image caused by the rapid movement of a single particle during single particle fluorescence imaging. This blur usually occurs when tracking a single particle in a living cell in real time, because the particles in the cell are often in a state of constant motion.

[0053] On the basis of obtaining the target input image corresponding to the target single particle in step S110, further, step S120 is executed.

[0054] S120, based on a pre-trained particle trajectory prediction model, a particle pseudo-trajectory image is predicted according to the target input image; wherein the particle trajectory prediction model is obtained by training and optimizing according to a first training sample set composed of a single particle motion blur image and its corresponding particle motion trajectory image.

[0055] The particle trajectory prediction model is used to recover or predict or reconstruct the motion trajectory of the particle (i.e., the particle pseudo-trajectory image) from the blurred single particle image (target input image), so as to more accurately track the behavior of the target single particle.

[0056] The particle trajectory prediction model is constructed based on a deep learning neural network, which can include a convolutional neural network, a recurrent neural network, or an architecture combining the advantages of both, which is not specifically limited here.

[0057] In one specific embodiment, the particle trajectory prediction model includes a convolutional feature extraction layer, a max-pooling layer, a bilinear upsampling layer, a normalization layer, and an activation function layer.

[0058] After determining the architecture of the particle trajectory prediction model, a first training sample set needs to be constructed to train and optimize the particle trajectory prediction model. Specifically, during the training and optimization, the single particle motion blur image in the first training sample set is used as the model input, the predicted pseudo trajectory image is used as the model input, and the difference between the predicted pseudo trajectory image and the particle motion trajectory image is used as the training loss. The particle trajectory prediction model is iteratively optimized, and thus a trained particle trajectory prediction model is obtained.

[0059] The single particle motion blur image can be real data obtained through experiments or data generated through simulation, which is not limited here.

[0060] The particle motion trajectory image is the real motion trajectory image of the biological particle, which can be obtained through high-precision imaging technology or manual annotation as the "true value" for model training.

[0061] After the particle trajectory prediction model is trained, the target input image obtained in step S110 is input into the particle trajectory prediction model, and the output particle pseudo trajectory image of the target single particle is obtained.

[0062] On the basis of predicting the particle pseudo trajectory image in step S120, step S130 is further performed.

[0063] S130, based on the particle pseudo trajectory image, the diffusion quantification feature of the target single particle is calculated.

[0064] Specifically, to simplify the problem, it can be assumed that the particle trajectory has less overlap, so that the area covered by the particle trajectory can be estimated as the trajectory area when the particle pseudo trajectory image is obtained. Then, according to the quantitative relationship between the pseudo trajectory area and the particle diffusion coefficient, the diffusion coefficient of the target single particle is fitted, and at the same time, according to the density spatial distribution of the particle pseudo trajectory image, the diffusion direction corresponding to the target single particle is fitted.

[0065] The quantitative relationship between the pseudo trajectory area and the particle diffusion coefficient is determined in advance, and when applied, only the pseudo trajectory area needs to be substituted into the equation corresponding to the linear relationship. The diffusion coefficient and the diffusion direction are both diffusion quantification features.

[0066] The diffusion coefficient of the target single particle is a physical quantity describing the random motion rate of the target single particle in the medium, which reflects the free diffusion ability of the target single particle under the action of no external force, denoted by D. The unit of diffusion coefficient is usually or , indicating the area covered by the particle diffusion per unit time.

[0067] It is worth mentioning that the single-particle diffusion quantification feature prediction method provided in the embodiment has a high dynamic range in the common diffusion coefficient range of biological large particles, and can effectively distinguish the diffusion characteristics of different particles, which provides a powerful tool for dynamic research of living cell particles.

[0068] In the embodiment, the diffusion quantification feature of the target single particle is obtained by acquiring a target input image corresponding to the target single particle, and predicting a particle pseudo-trajectory image from the target input image based on a pre-trained particle trajectory prediction model, and then calculating the diffusion quantification feature of the target single particle based on the particle pseudo-trajectory image. The particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single-particle motion blur images and their corresponding particle motion trajectory images. This method predicts the actual motion path of the target single particle from the target input image based on the particle trajectory prediction model, and calculates the diffusion quantification feature of the target single particle based on this, which can quickly and accurately predict the diffusion quantification feature of a large number of single particles in a high marker density environment, significantly reduce phototoxicity, and be suitable for at least 10 minutes of living cell imaging, and the diffusion coefficient obtained has a high dynamic range, effectively distinguishing the diffusion characteristics of different biological particles, and providing a powerful tool for dynamic research of living cell particles.

[0069] On the basis of the above-mentioned embodiments, further, the acquisition process of the target input image will be described in detail.

[0070] Acquiring the target input image corresponding to the target single particle includes: collecting an original single-particle motion blur image of the target single particle; segmenting the original single-particle motion blur image based on a pre-trained U-Net network to obtain a single-particle signal mask; pre-positioning the single-particle signal in the original single-particle motion blur image to retain a target single-particle signal mask containing only one positioned target single-particle signal; filling the target single-particle signal mask with the background and noise level calculated by a median filter to obtain the target input image; wherein the U-Net network is obtained by training and optimizing a second training sample set composed of single-particle motion blur images and their corresponding mask images.

[0071] It can be understood that, unlike typical positioning-based single-particle detection algorithms, after capturing the single-particle motion blur image, the pixels containing the single particle need to be segmented. The U-Net network is used in this embodiment to complete this task.

[0072] Figure 2 The structure of the U-Net network provided in the embodiment of the application is shown in the structural diagram. As shown in Figure 2 The U-Net network is composed of a contraction path (left side) and an expansion path (right side).

[0073] where the contracting path follows a typical architecture of a convolutional network, which includes the repeated application of a 3x3 convolution (unpadded convolution) twice, followed by a ReLU activation function and a 2x2 max-pooling operation with a stride of 2 for down-sampling. At each down-sampling step, the number of feature channels is doubled.

[0074] Each step in the expanding path includes an up-sampling of the feature map, followed by a 2x2 convolution ("up-convolution") that halves the number of feature channels, concatenation with the correspondingly cropped feature map from the contracting path, and two 3x3 convolutions each followed by a ReLU activation function. Cropping is necessary since each convolution loses boundary pixels.

[0075] At the last layer, a 1x1 convolution is used to map each 64-dimensional feature vector to the desired number of classes (here only one class, corresponding to the pixel range of a single-particle motion-blurred image). The U-Net network has a total of 23 convolutional layers.

[0076] Before practical application, a second training sample set is needed to train the optimized U-Net network. Each training sample in the constructed second training sample set includes a single-particle motion-blurred image and its corresponding mask image. Specifically, Figure 3 A second training sample set construction schematic diagram of the single-particle diffusion quantification feature prediction method provided by the embodiment of the application is shown.

[0077] As shown in Figure 3 , first, a simSPT algorithm is used to simulate a two-dimensional particle trajectory (2D particle trajectory), and a Gaussian function is superimposed on each point on the two-dimensional particle trajectory to obtain a motion modulus function Traj i,j .

[0078] Then, the motion blur function is normalized and pixelated to align with the actual camera pixel size (for example, 110 nanometers) to obtain a pixelated Traj i,j , followed by introducing Gaussian white noise and Poisson shot noise into the pixelated Traj i,j to generate single-particle motion-blurred images under different signal-to-noise ratios and background levels.

[0079] At the same time, a mask image is obtained by cutting more than 95% of the signal intensity area from the pixelated Traj i,j . Thus, according to the single-particle motion-blurred image and its corresponding mask image, the second training sample set can be constructed.

[0080] Then, the second training sample set is used to train the optimized U-Net network. Specifically, in the training, the single-particle motion blur image is taken as the model input, the predicted mask image is taken as the model input, and the difference between the predicted mask image and the (actual) mask image is taken as the training loss. The U-Net network is iteratively optimized, so as to obtain the trained U-Net network.

[0081] It is worth mentioning that, in the training, in order to increase the segmented particle density, the embodiment places four single-particle motion blur images in a 32x32 pixel wide image (pixel size 110 nm) and takes the weighted sum of the four single-particle motion blur images as the input of the U-Net network. The cross-entropy loss function is updated to force the U-Net network to learn the separation boundary between single-particle motion blurs.

[0082] Specifically, the weight map of the U-Net network can be calculated by the following formula (1).

[0083] .

[0084] In formula (1), is the weight map to balance the class frequency (i.e. the pixel number ratio between the instances with motion blur detection and the instances without motion blur detection), is the distance to the nearest motion blur boundary, is the distance to the second nearest motion blur boundary, and in the experiment, pixels and pixels.

[0085] Therefore, in the embodiment, the maximum detection density that the U-Net network can achieve is about 0.5 motion blur particle signal μm² / frame or 16.5 motion blur particle signals μm² / s (equivalent to placing 4 motion blurs in a 32x32 pixel area with a pixel size of 110 nm), which is comparable to the localization density of classical live cell PALM (Photoactivated Localization Microscopy) and more than 12 times higher than the localization density of fast flickering single-particle tracking.

[0086] In a specific embodiment, in order to train the U-Net network, 10320 pairs of single-particle motion blur images and mask images are simulated, with an exposure time of 30 milliseconds, covering 12 different diffusion coefficients D (ranging from 0.01 μm² / s to 10 μm² / s). Under each diffusion coefficient D, the single-particle motion blur image is randomly generated to simulate the actual imaging data, with a signal-to-noise ratio of 19-35 dB and a background level of 1000-3000. Dropout and random elastic deformation (rotation, translation and flipping) are included in the data augmentation to improve the generalization ability of the network training.

[0087] After the U-Net network is trained, it is used to segment the pixel containing a single particle to finally obtain the target input image.

[0088] Specifically, first, the original single particle motion blur image of the target single particle is collected, which can be collected by TIRF (Total Internal Reflection Fluorescence Microscopy) or HILO (Highly Inclined Light Optical Sheet).

[0089] Then, the obtained original single particle motion blur image is input into the trained U-Net network to generate a single particle signal mask covering the single particle signal.

[0090] Then, in order to remove the mask covering multiple single particle signals, ThunderSTORM is used to pre-position the single particle signal in the original single particle motion blur image, and only the single particle signal mask containing one positioning, i.e. the target single particle signal mask, is retained.

[0091] Next, a median filter is used to calculate the background and noise level of each pixel in the target single particle signal mask within 250 frames before and after, so that the target single particle signal mask can be filled with the calculated background and noise level, and the target input image can be obtained.

[0092] In this embodiment, by collecting the original single particle motion blur image of the target single particle, and based on the pre-trained U-Net network, the original single particle motion blur image is segmented to obtain a single particle signal mask, and then the single particle signal in the original single particle motion blur image is pre-positioned to retain a target single particle signal mask containing only one positioning, and the target single particle signal mask is filled with the background and noise level calculated by the median filter to obtain the target input image, so that the particle trajectory prediction model can predict a particle pseudo trajectory image with higher accuracy.

[0093] On the basis of the above embodiment, further, the training and optimization process of the particle trajectory prediction model will be described in detail.

[0094] It can be understood that before the particle trajectory prediction model is used to predict the particle pseudo trajectory image, the particle trajectory prediction model needs to be trained and optimized.

[0095] The training optimization particle trajectory prediction model specifically comprises: simulating a single particle motion blur image and obtaining a particle motion trajectory image corresponding to the single particle motion blur image to construct a first training sample set; using the single particle motion blur image as model input, using a predicted pseudo trajectory image as model output, using a difference between the predicted pseudo trajectory image and the particle motion trajectory image as training loss, iteratively optimizing the particle trajectory prediction model to obtain a trained convergent particle trajectory prediction model.

[0096] Figure 4 A construction schematic diagram of the first training sample set provided by the embodiment of the application is shown. Figure 4 As shown, first, a simSPT algorithm is used to simulate a two-dimensional particle trajectory (2D particle trajectory), and a Gaussian function is superimposed on each point on the two-dimensional particle trajectory to obtain a motion blur function Traj i,j .

[0097] Then, the motion blur function is normalized and pixelated to align with an actual camera pixel size (for example, 110 nanometers) to obtain a pixelated Traj i,j , and then Gaussian white noise and Poisson shooting noise are introduced into the pixelated Traj i,j to generate a single particle motion blur image under different signal-to-noise ratios and background levels. The single particle motion blur image has a size of 32x32 pixels.

[0098] In order to pair the particle motion trajectory with the single particle motion blur image for subsequent training, first, the Breitenbach straight line algorithm is applied to the simulated two-dimensional particle trajectory to generate a pixelated image with a width of 320x320 pixels, and then the pixelated image is convolved with a Gaussian filter with the same pixel width to generate a final trajectory image, i.e., a particle motion trajectory image.

[0099] Thus, according to the single particle motion blur image and the particle motion trajectory image, the first training sample set can be constructed.

[0100] Then, the particle trajectory prediction model is iteratively optimized using the first training sample set. Specifically, Figure 5 A training optimization schematic diagram of the particle trajectory prediction model provided by the embodiment of the application is shown.

[0101] According to Figure 5 , during training, first, the single particle motion blur image with noise (pixel size 110 nanometers) of 32x32 pixels is enlarged to 320x320 pixels (pixel size 11 nanometers) using nearest neighbor interpolation as an input image during training.

[0102] The input image is encoded by three convolutional feature extraction layers (each layer contains 3x3 convolutional kernels with output channels of 32, 64, 128, and 512, respectively), followed by batch normalization, ReLU activation function, and a max-pooling layer (2x2, stride 2). Subsequently, the image is subjected to three bilinear up-sampling layers (interpolation with 2x up-sampling) and convolutional feature extraction layers (each layer has 3x3 convolutional kernels with output channels of 128, 64, and 32, followed by batch normalization and ReLU activation function) to up-sample and decode the features layer by layer. Finally, a 1x1 convolutional layer without activation function is used to convert the 32-channel feature map into a single-channel density prediction image, so that the predicted pseudo-trajectory image from the single-particle motion blur image is obtained by the particle trajectory prediction model (image size 320x320, pixel size 11 nm).

[0103] Then, the difference between the predicted pseudo-trajectory image and the real trajectory image (i.e., the particle motion trajectory image in the first training sample set) is balanced by the sum of the mean absolute error loss function (L1 loss) and the mean square error loss function (MSE), and the weight parameters of the particle trajectory prediction model are continuously iteratively optimized.

[0104] In one specific embodiment, the exposure time is 30 milliseconds, covering a wide distribution of protein diffusion coefficients in living cells (a total of 27 diffusion coefficients, exponentially increasing from 0.001 pm / s to 31.62 pm / s). At each diffusion coefficient D, 3000 single-particle motion blur images are randomly generated to simulate actual imaging data, with a signal-to-noise ratio of 19-35 dB and a background level of 1000-3000. The total number of image pairs (single-particle motion blur image and particle motion trajectory image) in the training sample set is 81,000. During network training, the sum of the mean absolute error and the mean square error is used as the loss function to quantify and reduce the difference between the predicted pseudo-trajectory image and the particle motion trajectory image.

[0105] After training and optimizing the particle trajectory prediction model, in actual application, the target input image obtained is directly input into the particle trajectory prediction model, and the corresponding accurate particle pseudo-trajectory image can be predicted.

[0106] In the embodiment, the particle trajectory prediction model is trained and optimized, and then based on the particle trajectory prediction model, a particle pseudo trajectory image is predicted according to a target input image, and then a diffusion quantification feature of a target single particle is calculated based on the particle pseudo trajectory image. The particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of a single particle motion blur image and a corresponding particle motion trajectory image. The method can quickly and accurately predict the diffusion quantification features of a large number of single particles in a high marker density environment, and significantly reduces phototoxicity, so that it is suitable for live cell imaging for at least 10 minutes, and the diffusion coefficient obtained has a high dynamic range, effectively distinguishing the diffusion characteristics of different biological particles, and providing a powerful tool for live cell particle dynamic research.

[0107] On the basis of the above embodiment, further, the process of calculating the diffusion coefficient according to the particle pseudo trajectory image will be described in detail.

[0108] The diffusion quantification feature of the target single particle includes a diffusion coefficient and a diffusion direction. Correspondingly, the diffusion coefficient of the target single particle is calculated based on the particle pseudo trajectory image, including: determining a pseudo trajectory area according to the particle pseudo trajectory image; fitting the diffusion coefficient corresponding to the target single particle according to the quantitative relationship between the pseudo trajectory area and the diffusion coefficient of the particle; and fitting the diffusion direction corresponding to the target single particle according to the density spatial distribution of the particle pseudo trajectory image.

[0109] It can be understood that D is estimated by the particle pseudo trajectory image, assuming that the target single particle undergoes Brownian motion, and after a time interval Δτ, the particle is at a position The probability of finding the target single particle can be described by equation (2) as follows.

[0110] .

[0111] In equation (2), represents the displacement of the target single particle from the origin at time t, represents the probability density of the displacement, and D is the diffusion coefficient of the target single particle.

[0112] In single particle tracking, the trajectory displacement can be obtained, and equation (2) is fitted to extract the diffusion coefficient D. However, the particle pseudo trajectory image predicted by the particle trajectory prediction model is a spatial projection of the trajectory probability without time sequence. Therefore, the diffusion coefficient D is derived by the travel distance / trajectory distance of the target single particle instead of displacement.

[0113] The probability density of the jump distance r of the target single particle in a unit time can be obtained from equation (2) by integrating equation (2). For details, refer to equation (3) as follows.

[0114] .

[0115] The total distance of the particle trajectory in the entire exposure time T of the trajectory can be approximately equal to the sum of N discrete jump lengths with a time interval t. The probability density of each jump length is determined by equation (3). Assuming that each step is an independent and identically distributed event, since the diffusion coefficient D should be related to the square of the total distance R of the particle movement, the total expected R² can be written as equation (4) as follows.

[0116] .

[0117] After decomposing the total distance R of the particle movement into N consecutive jumps, equation (4) can be rewritten as equation (5) as follows.

[0118] .

[0119] Since , in combination with equation (3), we can get . In combination with , equation (5) is substituted to obtain equation (6) as follows.

[0120] .

[0121] In order to simplify the problem, the present embodiment assumes that the particle movement trajectory itself is rarely overlapped, and the particle pseudo-track image predicted by the particle trajectory prediction model can be regarded as a trajectory convolved with a narrow width w, which makes it possible to estimate the area covered by the particle pseudo-track image (pseudo-track area, referred to as PT area or pseudo-track area). For details, refer to equation (7) as follows.

[0122] .

[0123] In equation (7), represents the total trajectory distance of the target single particle.

[0124] Since w represents the uncertainty of the particle trajectory prediction model when predicting the particle trajectory (a constant after training), the present embodiment can obtain the quantitative relationship between the square of the pseudo-track area and the particle diffusion coefficient D by replacing R with equation (7) in equation (6). For details, refer to equation (8) as follows.

[0125] .

[0126] wherein .

[0127] Figure 6 A schematic diagram of obtaining the diffusion coefficient of the target single particle provided by the embodiment of the application is shown. Figure 6 As shown, when calculating the diffusion coefficient of the target single particle, first, the numerical range of the particle pseudo trajectory image is linearly adjusted to 0 to 1, and the pseudo trajectory area is defined as the pixel area exceeding 0.1, and then the diffusion coefficient corresponding to the target single particle can be fitted by equation (8).

[0128] In one specific embodiment, 1700 single particle motion blur images under high signal-to-noise ratio (35dB) are simulated, and the diffusion coefficient D ranges from 0.1 μm² / s to 31.62 μm² / s, Figure 7 A schematic diagram of the fitting effect of the quantitative relationship between the pseudo trajectory area and the particle diffusion coefficient provided by the embodiment of the application is shown. According to Figure 7 It can be known that by using equation (8) to fit the pseudo trajectory area and the diffusion coefficient D, an excellent determination coefficient (R 2 >0.9) can be obtained.

[0129] Therefore, equation (8) fitted under high signal-to-noise ratio can be used to convert the particle pseudo trajectory image predicted by the particle trajectory prediction model into the diffusion coefficient of the single particle motion blur image.

[0130] In this embodiment, the pseudo trajectory area is determined according to the particle pseudo trajectory image, and the diffusion coefficient corresponding to the target single particle is fitted according to the quantitative relationship between the pseudo trajectory area and the particle diffusion coefficient. This method predicts the actual motion path of the target single particle from the target input image through the particle trajectory prediction model, and calculates the diffusion coefficient of the target single particle based on this. It can quickly and accurately predict the diffusion coefficient of a large number of single particles under high marker density environment, and significantly reduces the phototoxicity, making it suitable for live cell imaging for at least 10 minutes, and the diffusion coefficient obtained by solving has a high dynamic range, effectively distinguishing the diffusion characteristics of different biological particles, and providing a powerful tool for live cell particle dynamic research.

[0131] On the basis of the above embodiment, further, the process of particle positioning will be described in detail.

[0132] The diffusion coefficient of the target single particle is calculated, and then includes: in the case that the diffusion coefficient of the target single particle is greater than a set threshold, performing centroid calculation on the particle pseudo trajectory image to obtain the positioning of the target single particle; in the case that the diffusion coefficient of the target single particle is less than or equal to the set threshold, performing ellipsoid Gaussian fitting on the target input image to obtain the positioning of the target single particle.

[0133] It can be understood that in the embodiment, a series of particle motion trajectories and corresponding single-particle motion blur images are first simulated, covering different diffusion coefficients D and signal-to-noise ratios, and specific reference can be made to Figure 8 , Figure 8 The positioning accuracy of different particle positioning algorithms provided by the embodiment of the application is shown. Compared with positioning particles using an elliptical Gaussian function, the barycenter algorithm used on the particle pseudo-trajectory image shows higher positioning accuracy (the positioning accuracy is evaluated by comparing the deviation of the particle positioning obtained by the two methods from the geometric center of the actual particle motion trajectory) when the diffusion coefficient D exceeds 1 μm² / s.

[0134] It is worth noting that for fast-diffusion particles with a diffusion coefficient D exceeding 10 μm² / s, the barycenter algorithm achieves an accuracy of 20 nanometers, while the Gaussian function fitting is 85 nanometers.

[0135] On the contrary, for slow-diffusion particles with a diffusion coefficient D lower than 1 μm² / s, the elliptical Gaussian function used to position particles performs better than the barycenter algorithm. Therefore, for downstream analysis, when the diffusion coefficient D is estimated to exceed 1 μm² / s, the embodiment uses the weighted barycenter of the particle pseudo-trajectory image to position the particle, and otherwise uses the elliptical Gaussian fitting on the preprocessed original motion blur image (target input image).

[0136] That is to say, the diffusion coefficient of the target single particle determines which way to use to obtain the positioning of the target single particle.

[0137] Therefore, in the embodiment, the diffusion coefficient of the target single particle is compared with a set threshold value, for the target single particle with a diffusion coefficient greater than the set threshold value, the barycenter of the particle pseudo-trajectory image predicted according to the particle trajectory prediction model is calculated, and the position of the barycenter is taken as the positioning of the target single particle; for the target single particle with a diffusion coefficient less than or equal to the set threshold value, the target input image thereof is subjected to elliptical Gaussian fitting, and the position obtained by the fitting is taken as the positioning of the target single particle.

[0138] The set threshold value can be set according to actual needs, and is not specifically limited here. For example, in a specific embodiment, the set threshold value is 1 .

[0139] In the embodiment, the centroid calculation is performed on the particle pseudo-trajectory image to obtain the localization of the target single particle when the diffusion coefficient of the target single particle is greater than a set threshold, and the ellipsoid Gaussian fitting is performed on the target input image to obtain the localization of the target single particle when the diffusion coefficient of the target single particle is less than or equal to the set threshold. The method can simultaneously capture the spatial localization and diffusion coefficient distribution of particles, can quickly and accurately predict the diffusion coefficients of a large number of single particles in a high marker density environment, significantly reduces the phototoxicity, and is suitable for live cell imaging for at least 10 minutes, and the obtained diffusion coefficient has a high dynamic range and can effectively distinguish the diffusion characteristics of different biological particles, thereby providing a powerful tool for dynamic research on live cell particles.

[0140] On the basis of the above-mentioned embodiments, further, the process of image rendering will be described in detail below.

[0141] The localization of the target single particle is obtained, and then includes: gridizing the localization of the target single particle to obtain a first grid containing the particle localization and a second grid adjacent to the particle localization but not containing the particle localization; generating a particle density probability map based on the localization of the target single particle; based on the particle density probability map and the first grid, using a sum of Gaussian weights to perform interpolation processing on the second grid to obtain a diffusion coefficient corresponding to the second grid; obtaining a diffusion coefficient matrix according to the diffusion coefficient corresponding to the first grid and the diffusion coefficient corresponding to the second grid; performing local smoothing processing on the diffusion coefficient matrix to obtain a mobility map, and displaying the mobility map using an HSV color map to complete image rendering.

[0142] According to the above embodiments, the localization and diffusion coefficient D of the target single particle can be obtained. In order to intuitively show the spatial distribution and correlation between the spatial organization and diffusion of particles, the present embodiment develops an image rendering algorithm, which can generate a particle density probability map (PALM), a diffusion coefficient map (Mobility map), and a particle diffusion density map (MPALM) that fuses the particle density and diffusion coefficient in the same data set.

[0143] Specifically, first, the positions of a total of M total particles are gridized and summed in a square pixel grid of a given size to obtain I count . The positions of the M total particles and the rendering pixel size are selected to achieve the required spatial resolution at the Nyquist sampling frequency.

[0144] Then, I count is convolved with a Gaussian kernel (σ x,y equal to the average localization error) to generate a PALM image I density , which is visualized using a hot red color map.

[0145] Next, M total The positioning of the particles is again grid-merged in a similar way to I count The average diffusion coefficient is calculated for each merged grid. For the second grid (D null ) that does not contain particle positions but is adjacent to the first grid (Draw) with particle positions, the second grid D null is interpolated using a sum of Gaussian weights, as shown in the following equation (9).

[0146] .

[0147] In equation (9), is the interpolated diffusion coefficient of the second grid, and are the pixel coordinates of the first grid with particle positions, and are the pixel coordinates of the second grid without particle positions but adjacent to the first grid. The resulting matrix (i.e., the diffusion coefficient matrix) after interpolation is denoted as I D . Subsequently, the diffusion coefficient matrix I D is locally smoothed to obtain the motility map I smoothed D , as shown in the following equation (10).

[0148] .

[0149] To fuse the information of particle position density and diffusion coefficient D, the embodiment selects an HSV color map to generate the MPALM image. The values of the diffusion coefficient map are mapped to the hue channel (Hue), the PALM image I density is mapped to the value channel (Value), and the saturation channel (Saturation) is set to 0 when the particle density is 0, and otherwise set to 1.

[0150] The image rendering result can be seen in Figure 9 , Figure 9 The rendering schematic diagram of the particle position and the particle diffusion coefficient provided by the embodiment of the application is shown. Figure 9 The particle density probability map (PALM), the diffusion coefficient map (Mobility map), and the particle diffusion density map (MPALM) are included.

[0151] In the embodiment, the particle density probability map, the diffusion coefficient map, and the particle diffusion density map are obtained by image rendering of the positioning and diffusion coefficient of the target single particle, so that the spatial distribution and correlation between the spatial organization and diffusion of the particles can be more intuitively represented.

[0152] In addition, taking the prediction of the diffusion coefficient and the spatial localization of a single particle as an example, Figure 10 A schematic diagram of the overall flow of the single-particle diffusion quantification feature prediction method provided by the embodiment of the application is shown.

[0153] As shown in Figure 10 , first, the original single-particle motion blur image of the target single particle (i.e., the single-particle motion blur signal in Figure 10 ) is collected, and then the original single-particle motion blur image is segmented by using a pre-trained U-Net network to obtain a single-particle signal mask.

[0154] Next, the single-particle signal in the original single-particle motion blur image is pre-localized by using ThunderSTORM, and a target single-particle signal mask containing only one localized target single particle is retained.

[0155] Further, the target single-particle signal mask is filled by using the background and noise levels calculated by the median filter to obtain a target input image.

[0156] Then, the target input image is input into a pre-trained particle trajectory prediction model to predict an output particle pseudo-trajectory image.

[0157] Then, the pseudo-trajectory area is calculated based on the particle pseudo-trajectory image, and the diffusion coefficient corresponding to the target single particle is fitted according to the quantitative relationship between the pseudo-trajectory area and the particle diffusion coefficient.

[0158] Finally, in the case where the diffusion coefficient of the target single particle is greater than a set threshold, the centroid of the particle pseudo-trajectory image is calculated to obtain the localization of the target single particle; in the case where the diffusion coefficient of the target single particle is less than or equal to the set threshold, the target input image is subjected to ellipsoid Gaussian fitting to obtain the localization of the target single particle.

[0159] Based on the above, the particle spatial distribution and the diffusion coefficient distribution of the target single particle can be captured. It should be noted that the above steps are described in detail in the above embodiment, and will not be described here.

[0160] It is worth mentioning that, for the single-particle diffusion quantification feature prediction method provided by the embodiment of the application, in order to verify the robustness of the method, a series of single-particle motion blur images generated by particle motion trajectories with different diffusion coefficients under different signal-to-noise ratios are calculated and simulated, and then the pseudo-trajectory areas obtained by the same particle motion trajectory under different signal-to-noise ratios are tested. The results show very small fluctuations, indicating that the particle trajectory prediction model is robust to noise.

[0161] Meanwhile, the robustness performance of the U-Net network segmentation under different signal-to-noise ratios is also tested, and it is found that the area of the single-particle signal mask generated by the same particle trajectory can also remain stable under different signal-to-noise ratios. Therefore, the particle trajectory prediction model and the U-Net network show strong robustness to different signal-to-noise ratios, and this characteristic helps to reduce errors caused by signal-to-noise ratio fluctuations in the imaging process, because these errors can lead to fluctuations in the estimated diffusion coefficient D.

[0162] The test results of the particle trajectory prediction model and the U-Net network can be specifically referred to Figure 11 , Figure 11 The inference effect schematic diagram of the particle trajectory prediction model and the U-Net network provided by the embodiment of the application is shown.

[0163] In some embodiments, the process of cell sample preparation and image data acquisition is also described in detail.

[0164] The target protein (i.e., the target single particle) is first labeled with a suitable fluorescent tag, which is connected to the N- or C-terminus of the target protein using HaloTag or SNAP-tag. Then the target protein is expressed in cells, and a high quantum yield fluorescent dye (such as PA-JF646, HM-SiR, etc.) modified with a tag protein ligand is used to incubate and stain the cells. Through total internal reflection fluorescence microscopy, in total internal reflection mode or high oblique light sheet mode, set appropriate exposure time (such as 30 ms) and shooting density (most of the adjacent particles are observed without overlap, or the particle density is less than 16.5 moving blur particle signals / μm² / s), and the moving blur image of the single particle is obtained by shooting.

[0165] In single-channel MPALM (particle diffusion density map) imaging, 2000 frames of MPALM images are continuously shot, and a whole image (wide-field image without single-particle image) is connected as a period. This period is repeated 5 to 20 times according to the survival ability of the cells and the particle photobleaching. During each MPALM frame, the excitation light (642 nanometers) continuously illuminates the excitation dye for 30 milliseconds of camera exposure time. When using a photoactivatable dye (such as PA-JF646), a 405 nanometer laser is pulsed for 0.5 milliseconds of camera dead time to activate the photoactivatable dye. For each large-scale image, the excitation laser (560 nanometers) and exposure time are adjusted to achieve the required signal-to-noise ratio.

[0166] In dual-channel MPALM-PALM imaging, 560 nm and 642 nm lasers are operated simultaneously for 30 ms of camera exposure time. A 405 nm laser is pulsed for 0.5 ms of camera dead time to activate photoconversion proteins, such as mEosEM, in the PALM channel. Two cameras are used simultaneously to detect the excitation light in both channels. In addition to this imaging mode, MPALM-MPALM dual-channel imaging can be adjusted as needed.

[0167] In some embodiments, the image registration process for dual-channel MPALM-PALM imaging is described in detail.

[0168] For dual-color imaging (e.g., dual-color MPALM-PALM), the images from one channel need to be registered to match the coordinates of the other channel due to the difference in camera pixel size between the two channels. See Figure 12 , Figure 12 An image registration flowchart is shown.

[0169] The target image is defined as the MPALM channel with a pixel size of 110 nm, and the moving image is the PALM channel with a pixel size of 160 nm. The task is to find a geometric transformation (including rotation, scaling, and translation) that, when applied to the moving image, results in an image with the same spatial coordinates as the target image. Prior to imaging each cell sample, 100 nm TetraSpeck fluorescent microspheres (Invitrogen, T7279) are imaged and used to estimate the transformation matrix for the two-channel image registration.

[0170] Initially, the transformation matrix is estimated using the phase correlation between the moving image and the target image. This estimation enables the conversion of the moving image to an initial registered image. Next, the microspheres are localized in the target image, the moving image, and the initial registered image using two-dimensional Gaussian fitting.

[0171] The microsphere localizations in the target image are paired with the microsphere localizations in the initial registered image. Then, the microsphere localizations in the initial registered image are inverted and paired with the moving image. This approach facilitates the pairing of microsphere localizations between the target image and the moving image using the localizations of the initial registered image as an intermediary guide.

[0172] Subsequently, the paired microsphere localizations are used as control points to update the estimate of the transformation matrix, which is then applied to align the localizations of the two channels. Using this approach, the relative localization distances of the same microspheres after image registration can be achieved with an average of 10 nm when aligning the two-channel fluorescent microsphere images.

[0173] In still other embodiments, sample drift correction is described in detail.

[0174] For single-color MPALM, the sample drift is estimated using the global images captured between the MPALM images (see above for details on acquisition of the first and second training sample sets). Specifically, the peak of the cross-correlation between global images is computed, and the deviation of the peak from the image center is used as an estimate of the sample drift. The sample drift is then interpolated throughout the MPALM image sequence using local linear regression, and the localization of each MPALM frame is corrected to the first frame.

[0175] In single-channel MPALM imaging, the sample drift is estimated by analyzing the global images captured between the MPALM frames (see the example of "Cell sample preparation and data acquisition" for details). Specifically, the peak of the cross-correlation between global images is computed, and the deviation of the peak from the image center is used as an estimate of the sample drift. Subsequently, the sample drift is interpolated throughout the MPALM image sequence by local linear regression. This corrects the localization of each MPALM frame to the first frame.

[0176] In dual-channel MPALM-PALM imaging, the PALM images are first aligned to the MPALM images (see the example of "Image registration" for details). The localization of the PALM channel is then divided into a suitable number of bins, typically 2000 frames per bin. The localization within each bin is then used to reconstruct individual PALM images. The peak of the cross-correlation between the reconstructed PALM images is computed, and the deviation of the peak from the image center is used as an estimate of the sample drift. Subsequently, the sample drift is interpolated throughout the PALM image sequence by local linear regression. Finally, the regression model is applied to correct the localization of each MPALM and PALM frame to align with the first frame.

[0177] Corresponding to the single particle diffusion quantification feature prediction method described in the above examples, the application also provides a single particle diffusion quantification feature prediction device, specifically, Figure 13 The structure schematic diagram of the single particle diffusion quantification feature prediction device provided by the embodiment of the application is shown.

[0178] As Figure 13 shown, the device comprises: a target input image acquisition module 1310, configured to acquire a target input image corresponding to a target single particle; a particle pseudo-trajectory image prediction module 1320, configured to predict a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model according to the target input image; and a particle diffusion coefficient acquisition module 1330, configured to calculate a diffusion quantification feature of the target single particle based on the particle pseudo-trajectory image; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of a single particle motion blur image and a particle motion trajectory image corresponding thereto.

[0179] In this embodiment, the target input image corresponding to a single target particle is acquired by the target input image acquisition module 1310. The particle pseudo-trajectory image prediction module 1320 predicts the particle pseudo-trajectory image based on the target input image using a pre-trained particle trajectory prediction model. Then, the particle diffusion quantization feature acquisition module 1330 calculates the diffusion quantization feature of the single target particle based on the particle pseudo-trajectory image. The particle trajectory prediction model is trained and optimized using a first training sample set consisting of a single particle motion blur image and its corresponding particle motion trajectory image. This device predicts the actual motion path of a single target particle from the target input image using the particle trajectory prediction model and calculates the diffusion quantization feature of the single target particle based on this. It can quickly and accurately predict the diffusion quantization feature of a large number of single particles in a high label density environment, and significantly reduces phototoxicity, making it suitable for live cell imaging for at least 10 minutes. Furthermore, the obtained diffusion quantization feature has a high dynamic range, effectively distinguishing the diffusion characteristics of different biological particles, providing a powerful tool for the dynamic study of live cell particles.

[0180] Figure 14 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 14 As shown, the electronic device may include: a processor 1410, a communications interface 1420, a memory 1430, and a communication bus 1440, wherein the processor 1410, the communications interface 1420, and the memory 1430 communicate with each other through the communication bus 1440. The processor 1410 can call logical instructions in the memory 1430 to execute a single-particle diffusion quantization feature prediction method, which includes: acquiring a target input image corresponding to a target single particle; predicting a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model according to the target input image; and calculating the diffusion quantization feature of the target single particle based on the particle pseudo-trajectory image; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set consisting of a single-particle motion blur image and its corresponding particle motion trajectory image.

[0181] Moreover, the logic instructions in the memory 1430 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0182] In another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the single particle diffusion quantification feature prediction method provided by the above-mentioned methods. The method comprises: obtaining a target input image corresponding to a target single particle; based on a pre-trained particle trajectory prediction model, predicting a particle pseudo-trajectory image according to the target input image; and based on the particle pseudo-trajectory image, calculating a diffusion quantification feature of the target single particle; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of a single particle motion blur image and a corresponding particle motion trajectory image.

[0183] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0184] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software and the necessary general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions essentially or the parts that make contributions to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0185] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A single particle diffusion quantification feature prediction method, characterized in that, The method comprises the following steps: obtaining a target input image corresponding to a target single particle, wherein the target input image is a processed single particle motion blur image; based on a pre-trained particle trajectory prediction model, a particle pseudo trajectory image is predicted according to the target input image; based on the particle pseudo trajectory image, the diffusion quantification feature of the target single particle is calculated; wherein the particle trajectory prediction model is constructed based on a deep learning neural network, and is obtained by training and optimizing a first training sample set composed of single particle motion blur images and their corresponding particle motion trajectory images; the diffusion quantification feature of the target single particle includes diffusion coefficient and diffusion direction; correspondingly, the diffusion quantification feature of the target single particle is calculated based on the particle pseudo trajectory image, which includes: determining the pseudo trajectory area according to the particle pseudo trajectory image; fitting the diffusion coefficient corresponding to the target single particle according to the quantitative relationship between the pseudo trajectory area and the particle diffusion coefficient; fitting the diffusion direction corresponding to the target single particle according to the density space distribution of the particle pseudo trajectory image; the calculation formula of the pseudo trajectory area is as follows: PT area = R * w; the quantitative relationship between the pseudo trajectory area and the particle diffusion coefficient is defined as follows: D = Const * (PT area) 2 ; where PT area represents a pseudo trajectory area, R represents a total trajectory distance of the target single particle, w represents a narrow width, D represents a diffusion coefficient of the target single particle, T represents an exposure time of the total trajectory of the target single particle, and t represents a time interval of a plurality of discrete jump lengths in the total trajectory distance.

2. The single particle diffusion quantification feature prediction method of claim 1, wherein, the target input image corresponding to the target single particle is obtained, which includes: collecting the original single particle motion blur image of the target single particle; segmenting the original single particle motion blur image based on a pre-trained U-Net network to obtain a single particle signal mask; pre-positioning the single particle signal in the original single particle motion blur image, and retaining only a target single particle signal mask containing a positioning; filling the target single particle signal mask with the background and noise level calculated by the median filter to obtain the target input image; wherein the U-Net network is obtained by training and optimizing a second training sample set composed of single particle motion blur images and their corresponding mask images.

3. The single particle diffusion quantification feature prediction method of claim 1, wherein, training and optimizing the particle trajectory prediction model, specifically including: simulating a single particle motion blur image and obtaining a particle motion trajectory image corresponding to the single particle motion blur image to construct a first training sample set; taking the single particle motion blur image as the model input, taking the predicted pseudo trajectory image as the model output, and taking the difference between the predicted pseudo trajectory image and the particle motion trajectory image as the training loss, the particle trajectory prediction model is iteratively optimized to obtain a training converged particle trajectory prediction model.

4. The single particle diffusion quantification feature prediction method of claim 3, wherein, the simulation of single particle motion blur image includes: simulating a two-dimensional particle trajectory and superimposing a Gaussian function on each point on the two-dimensional particle trajectory to obtain a motion blur function; normalizing and pixelizing the motion blur function, and introducing Gaussian white noise and Poisson shooting noise to obtain single particle motion blur images under different signal-to-noise ratios and background levels.

5. The single particle diffusive quantification feature prediction method of claim 1, wherein, fitting the diffusion coefficient corresponding to the target single particle, which includes: in the case that the diffusion coefficient of the target single particle is greater than a set threshold, the centroid of the particle pseudo trajectory image is calculated to obtain the positioning of the target single particle; In a case where the diffusion coefficient of the target single particle is less than or equal to a set threshold, ellipsoid Gaussian fitting is performed on the target input image to obtain the localization of the target single particle.

6. The single particle diffusive quantification feature prediction method of claim 5, wherein, The localization of the target single particle is obtained, and then includes: The localization of the target single particle is gridded to obtain a first grid containing particle localization and a second grid adjacent to the particle localization but not containing particle localization; Based on the localization of the target single particle, a particle density probability map is generated; Based on the particle density probability map and the first grid, the second grid is interpolated using a sum of Gaussian weights to obtain the diffusion coefficient corresponding to the second grid; According to the diffusion coefficient corresponding to the first grid and the diffusion coefficient corresponding to the second grid, a diffusion coefficient matrix is obtained; The diffusion coefficient matrix is locally smoothed to obtain a motion map, and the HSV color map is used for display to complete image rendering.

7. A single particle diffusion quantification feature prediction device applying the single particle diffusion quantification feature prediction method according to any one of claims 1 to 6, characterized by It includes: A target input image acquisition module is configured to acquire a target input image corresponding to a target single particle; A particle pseudo-trajectory image prediction module is configured to predict a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model and the target input image; A particle diffusion quantification feature acquisition module is configured to calculate the diffusion coefficient of the target single particle based on the particle pseudo-trajectory image. The particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single particle motion blur images and their corresponding particle motion trajectory images.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the single particle diffusion quantification feature prediction method of any one of claims 1-6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the single particle diffusion quantification feature prediction method of any one of claims 1-6.

Images