Single-particle diffusion quantization feature prediction method and apparatus, electronic device, and storage medium

Abstract

Provided in the present invention are a single-particle diffusion quantization feature prediction method and apparatus, an electronic device and a storage medium. The method comprises: acquiring a target input image corresponding to a target single particle; on the basis of a pre-trained particle trajectory prediction model and according to the target input image, predicting a particle pseudo-trajectory image; and, on the basis of the particle pseudo-trajectory image, calculating a diffusion quantization feature of the target single particle, the particle trajectory prediction model being obtained by means of training and optimization performed on the basis of a first training sample set constituted by a single particle motion blur image and a particle motion trajectory image corresponding to the single particle motion blur image. The method predicts an actual motion path of the target single particle from the target input image by means of the particle trajectory prediction model and, on this basis, calculates the diffusion quantification feature of the target single particle, thereby quickly and accurately predicting diffusion coefficients of a large number of single particles in a high marker density environment, and significantly reducing phototoxicity; and can further effectively distinguish between diffusion characteristics of different biological particles, providing a powerful tool for dynamics research on particles in diffusion systems and living cells.

Description

Single-particle diffusion quantification feature prediction method and device, electronic equipment and storage medium

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 202411353615.2, filed September 26, 2024, entitled “Single-particle diffusion quantification feature prediction method and device, electronic equipment and storage medium,” which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of super-resolution imaging, and particularly relates to a single-particle diffusion quantification feature prediction method and device, electronic equipment and storage medium. BACKGROUND

[0004] Traditional imaging methods are often limited to low spatiotemporal resolution for studying population molecular dynamic characteristics due to the limitation of optical diffraction limit. Single-molecule localization microscopy (SMLM) is a super-resolution imaging technology that won the Nobel Prize in Chemistry in 2014, which can image single fluorescent particles or fluorescent biomolecules.

[0005] With the development of fluorescence microscopy and new labeling technology, people use SMLM to perform real-time imaging, localization and tracking (single-particle tracking, SPT; single-molecule tracking, SMT; hereinafter referred to as SMT) of single particles or single biomolecules in extracellular diffusion systems and living cells. Analysis of the obtained molecular trajectories can obtain molecular diffusion coefficients, binding fractions, diffusion angle distributions (anisotropy), and other measurement indicators. Through these indicators, the biological activity of the labeled protein and the interaction strength with other molecules can be further inferred. This technological innovation is becoming a powerful tool for understanding the single-molecule dynamics and functions of biomolecules in the life processes of cells.

[0006] In the mainstream live-cell single-molecule kinetic analysis published so far, the general procedure is to locate single molecules, connect trajectories frame by frame, and then aggregate the lengths of the obtained molecular trajectories and fit various physical models based on molecular diffusion to obtain parameters such as diffusion coefficients and binding percentages. After obtaining the original single-molecule fluorescence image, the first step is to use a single-molecule localization algorithm to fit the single-molecule fluorescence intensity distribution to a two-dimensional Gaussian function, which is an approximate function of the point spread function (PSF), to obtain the spatial coordinates corresponding to the maximum value of the function as the position of the molecule. Subsequently, the closest and most physically diffused points between two frames are connected to form a trajectory using statistical inference.

[0007] In order to ensure the accuracy of trajectory connection, it is generally necessary to control the output intensity of the photoactivation laser. The more molecules that appear in each frame, the lower the accuracy of trajectory connection using the algorithm. In order to accurately obtain the trajectory, it is necessary to maintain a low molecule density during the entire SMT imaging process to avoid false connections. The analysis of single-molecule trajectories often relies on the length of the molecular trajectory, and the longer the molecular trajectory, the more accurate the kinetic parameters extracted therefrom. However, in order to make the subsequent kinetic analysis and model fitting statistically meaningful, it is necessary to obtain sufficient single-molecule trajectory data in the experiment. Therefore, high-frame-rate imaging is required to capture a sufficient number of single molecules, and the effects of phototoxicity and cell movement on the experiment cannot be ignored.

[0008] For the two-dimensional SMT imaging that is currently widely used, it is difficult to obtain long trajectories under limited Z-axis resolution when the binding percentage of the molecules is low and the diffusion coefficient is fast. On the other hand, most trajectory-based analyses require a large number of trajectories (usually at least one hundred thousand trajectories) to obtain statistically reliable results. A commonly used solution is to image many cells and pool the trajectories obtained from different cells together. Although the kinetic parameters obtained by statistically modeling a large number of single-molecule trajectories are accurate, the results are averages and lack spatial information, which limits the ability to use SMT to study single-cell protein spatial kinetics.

[0009] Another limitation of trajectory-based analysis is the difficulty of measuring the dynamics of protein clusters. For example, RNA polymerase II (RNA Pol II, hereinafter referred to as Pol II) forms dynamic clusters in live cells with a radius close to or below the diffraction limit (the diffraction limit is about 200 nm). Since the spatial resolution of SMT using a highly inclined light sheet (HILO) mode total internal reflection fluorescence microscope is about 30 nm, it is essentially challenging to study the dynamic characteristics of clusters by sparsely labeling molecules within the clusters and tracking trajectories.

[0010] Currently, the methods to characterize the spatial distribution of molecular diffusion in live cells are still limited, including the method that uses the average displacement of trajectories to represent the speed of molecular motion at each point in space, but this method is limited by the possible errors in the process of trajectory tracking, including the random blinking and photobleaching of fluorescent molecules leading to the failure of trajectory connection; and the density of molecules excited in each frame cannot be too high, otherwise it is easy to lead to false connection. Therefore, this method needs to collect long-time data, but long-time laser irradiation is easy to make the cell displace a certain distance, so this method is only suitable for shooting proteins that move slowly as a whole and the structure does not change greatly over time, such as chromatin histone.

[0011] There are also methods that directly use the tracked trajectories to analyze the spatial distribution of molecular diffusion, such as sptPALM. This method uses high-intensity laser to excite molecules, uses high numerical aperture objective (model Olympus APO100XO-HR-SP, 1.65NA, while the numerical aperture NA value of the 100x lens commonly used is usually below 1.49) combined with total internal reflection illumination, to track the trajectories of cell membrane proteins in the COS7 cell line which is resistant to phototoxicity, and to obtain the spatial distribution of membrane surface molecule diffusion by using the MSD-Δt fitting method of each trajectory to obtain the diffusion coefficient.

[0012] On the one hand, this method relies on trajectory tracking, so there is a possibility of misconnection between molecules in each frame; on the other hand, this method relies on the MSD-Δt fitting method, and this fitting requires longer molecular trajectories to obtain accurate diffusion coefficient estimates. Since cell membrane proteins mainly diffuse in two dimensions, they are suitable for obtaining long trajectories (>10 frames); while in other environments such as the nucleus, molecules are three-dimensional diffusion, and the two-dimensional imaging method using HILO light illumination can only obtain 3-4 frame molecular trajectory length for stably bound molecules such as H2B, and the average molecular trajectory length for other faster diffusing molecules is even shorter, so it is difficult to use the MSD-Δt fitting method.

[0013] In addition, there is a non-covalent binding of the ATTO-647N dye molecule ligand modified by divalent nickel ion-trisnitrilotriacetic acid (Ni2+trisN-nitrilotriacetic acid, Ni-NTA) and the membrane protein with 6 histidine tags to realize high-density labeling and tracking of membrane protein molecules, and finally using the median value of the trajectory displacement length in a 200 nm space to represent the speed of molecular motion uPAINT (universal-points-accumulation-for-imaging-in-nanoscale-topography). Although this method improves the molecular imaging density compared with sptPALM, it reduces the accuracy of trajectory reconstruction, and in addition, this method can only be used for labeling membrane proteins.

[0014] In addition, SMdM (Single-molecule displacement mapping) can further shorten the time interval between two frames by arranging the sequence of two frames of stroboscopic exposure, so that the stroboscopic exposure of 1 ms is distributed in the interval of 1 ms between two frames, so as to track the molecular motion faster. This method obtains the estimated diffusion coefficient by fitting the distribution of all displacements in a fixed range (100x100 nm) through the trajectory displacement between two frames.

[0015] This method has good measurement effect on the diffusion coefficient distribution of fast-moving molecules (diffusion coefficient higher than 10 μm 2 / s) such as fluorescent proteins without specific binding targets in cells. However, through simulation data, it is found that for transcriptional regulators in the nucleus with a diffusion coefficient less than 10 μm 2 / s, when considering the localization error of the imaged fluorescent molecules, the detection sensitivity of the motion speed of this part will be affected, because under the time interval of 1 ms, the average displacement of molecules with diffusion coefficient within 10 μm 2 / s is within 200 nm, and when there is a 30 nm error for each molecule at the beginning of the trajectory, a total error range of 60 nm will have a great influence on the detection result.

[0016] Therefore, how to solve the problem that the traditional particle trajectory tracking technology has low prediction accuracy of diffusion quantification characteristics in a high labeling density environment and cannot capture the spatial distribution of particles is an important topic to be solved in the field of super-resolution imaging. SUMMARY

[0017] The application provides a single-particle diffusion quantitative feature prediction method and device, electronic equipment and storage medium, to solve the defects that the traditional single-particle trajectory tracking technology has low prediction accuracy of diffusion quantitative features and cannot capture the spatial distribution of particles in a high marker density environment, and can quickly and accurately predict the diffusion quantitative features of a large number of single particles in a high marker density environment, and significantly reduce phototoxicity.

[0018] In one aspect, the application provides a single-particle diffusion quantitative feature prediction method, comprising: obtaining a target input image corresponding to a target single particle; based on a pre-trained particle trajectory prediction model, predicting a particle pseudo-trajectory image according to the target input image; and based on the particle pseudo-trajectory image, calculating the diffusion quantitative features of the target single particle; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single-particle motion blur images and their corresponding particle motion trajectory images.

[0019] Further, the obtaining of the target input image corresponding to the target single particle comprises: collecting an original single-particle motion blur image of the target single particle; segmenting the original single-particle motion blur image based on a pre-trained U-Net network to obtain a single-particle signal mask; pre-positioning the single-particle signal in the original single-particle motion blur image to retain a target single-particle signal mask containing only one positioned single particle; filling the target single-particle signal mask with the background and noise levels calculated by a median filter to obtain the target input image; wherein the U-Net network is obtained by training and optimizing a second training sample set composed of single-particle motion blur images and their corresponding mask images.

[0020] Further, the diffusion quantitative features of the target single particle include a diffusion coefficient and a diffusion direction; accordingly, the calculation of the diffusion quantitative features of the target single particle based on the particle pseudo-trajectory image comprises: determining a pseudo-trajectory area according to the particle pseudo-trajectory image; fitting the diffusion coefficient corresponding to the target single particle according to the quantitative relationship between the pseudo-trajectory area and the particle diffusion coefficient; and fitting the diffusion direction corresponding to the target single particle according to the density spatial distribution of the particle pseudo-trajectory image.

[0021] Further, the training and optimization of the particle trajectory prediction model specifically comprises: simulating a single-particle motion blur image and obtaining a particle motion trajectory image corresponding to the single-particle motion blur image to construct a first training sample set; taking the single-particle motion blur image as the model input, taking the predicted pseudo-trajectory image as the model output, and taking the difference between the predicted pseudo-trajectory image and the particle motion trajectory image as the training loss to iteratively optimize the particle trajectory prediction model to obtain a training-converged particle trajectory prediction model.

[0022] Further, the simulating the single-particle motion blur image comprises: simulating a two-dimensional particle trajectory, and superimposing a Gaussian function on each point on the two-dimensional particle trajectory to obtain a motion blur function; normalizing and pixelizing the motion blur function, and introducing Gaussian white noise and Poisson shot noise to obtain a single-particle motion blur image under different signal-to-noise ratios and background levels.

[0023] Further, the fitting the diffusion coefficient corresponding to the target single particle comprises: in a case where the diffusion coefficient of the target single particle is greater than a set threshold, performing centroid calculation on the particle pseudo-trajectory image to obtain the positioning of the target single particle; in a case where the diffusion coefficient of the target single particle is less than or equal to the set threshold, performing ellipsoid Gaussian fitting on the target input image to obtain the positioning of the target single particle.

[0024] Further, the obtaining the positioning of the target single particle comprises: gridizing the positioning of the target single particle to obtain a first grid containing the particle positioning and a second grid adjacent to the particle positioning and not containing the particle positioning; generating a particle density probability map based on the positioning of the target single particle; performing interpolation processing on the second grid based on the particle density probability map and the first grid using a sum of Gaussian weights to obtain a diffusion coefficient corresponding to the second grid; obtaining a diffusion coefficient matrix according to the diffusion coefficient corresponding to the first grid and the diffusion coefficient corresponding to the second grid; performing local smoothing processing on the diffusion coefficient matrix to obtain a motion property map, and displaying the motion property map using an HSV color map to complete image rendering.

[0025] In a second aspect, the present application further provides a single-particle diffusion quantification feature prediction device, comprising: a target input image acquisition module configured to acquire a target input image corresponding to a target single particle; a particle pseudo-trajectory image prediction module configured to predict a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model and the target input image; and a particle diffusion coefficient acquisition module configured to calculate a diffusion quantification feature of the target single particle based on the particle pseudo-trajectory image; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of a single-particle motion blur image and a corresponding particle motion trajectory image.

[0026] In a third aspect, the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the single-particle diffusion quantification feature prediction method according to any one of the above aspects when executing the computer program.

[0027] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the single particle diffusion quantification feature prediction method according to any one of the above aspects.

[0028] The single particle diffusion quantification feature prediction method provided by the present application comprises the following steps: obtaining a target input image corresponding to a target single particle; and based on a pre-trained particle trajectory prediction model, predicting a particle pseudo-trajectory image from the target input image, and then calculating a diffusion quantification feature of the target single particle based on the particle pseudo-trajectory image. The particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of a single particle motion blur image and a corresponding particle motion trajectory image. This method can quickly and accurately predict the diffusion quantification features of a large number of single particles in a high labeling density environment, and significantly reduces phototoxicity, making it suitable for live cell imaging for at least 10 minutes. The diffusion quantification features obtained by solving have a high dynamic range, effectively distinguishing the diffusion characteristics of different biological particles, and providing a powerful tool for the dynamic study of particles in diffusion systems and live cells. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0030] Fig. 1 is a flowchart of the single particle diffusion quantification feature prediction method provided by the present application.

[0031] Fig. 2 is a structural diagram of a U-Net network provided by the present application.

[0032] Fig. 3 is a diagram for constructing a second training sample set of the single particle diffusion quantification feature prediction method provided by the present application.

[0033] Fig. 4 is a diagram for constructing a first training sample set provided by the present application.

[0034] Fig. 5 is a diagram for training and optimizing a particle trajectory prediction model provided by the present application.

[0035] Fig. 6 is a diagram for obtaining a diffusion coefficient of a target single particle provided by the present application.

[0036] FIG. 7 is a schematic diagram of fitting effect of linear relationship between pseudo trajectory area and particle diffusion coefficient according to an embodiment of the present application.

[0037] FIG. 8 is a schematic diagram of positioning accuracy of different particle positioning algorithms according to an embodiment of the present application.

[0038] FIG. 9 is a schematic diagram of rendering of particle positioning and particle diffusion coefficient according to an embodiment of the present application.

[0039] FIG. 10 is a schematic diagram of overall flow of single particle diffusion quantification feature prediction method according to an embodiment of the present application.

[0040] FIG. 11 is a schematic diagram of inference effect of particle trajectory prediction model and U-Net network according to an embodiment of the present application.

[0041] FIG. 12 is a schematic diagram of image registration flow according to an embodiment of the present application.

[0042] FIG. 13 is a schematic diagram of structure of single particle diffusion quantification feature prediction device according to an embodiment of the present application.

[0043] FIG. 14 is a schematic diagram of physical structure of electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments but not all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0045] It is easy to understand that, in order to solve the problem that the diffusion coefficient prediction accuracy of the traditional single particle trajectory tracking technology is not high in a high marker density environment, and the spatial distribution of particles cannot be captured, the present application proposes a new single particle diffusion quantification feature prediction method. Specifically, FIG. 1 shows a schematic diagram of flow of single particle (such as biomolecule) diffusion quantification feature (such as diffusion coefficient) prediction method according to an embodiment of the present application.

[0046] As shown in FIG. 1, the method includes steps S110-S130, which will be described in detail below.

[0047] S110, obtaining a target input image corresponding to a target single particle.

[0048] It can be understood that, in order to make a single particle (for example, a single biomolecule) visible under a microscope, it is necessary to first attach fluorescent labels to the target single particle, which can emit light when irradiated by light of a specific wavelength, so that the particle becomes visible. Common fluorescent labeling methods include antibody labeling, genetically engineered expression of fusion proteins with fluorescent proteins (such as GFP), etc.

[0049] Since the single particle (target single particle) is usually very weak, it is necessary to use a high-sensitivity and high-resolution microscope system. Common single particle imaging microscope systems include, but are not limited to, photoactivated localization microscopy (PALM), stochastic optical reconstruction microscopy (STORM).

[0050] In order to observe the single particle (target single particle), it is necessary to precisely control the intensity and wavelength of the excitation light to avoid simultaneously exciting multiple adjacent fluorescent particles. In addition, it is also necessary to use a high-sensitivity detector, such as an EMCCD camera or an APD (avalanche photodiode), to capture extremely weak fluorescent signals.

[0051] After all parameters are set, image acquisition begins. Long exposure or multiple exposures are usually required to collect sufficient signals.

[0052] It should be noted that the acquired image needs to be processed to obtain a relatively clear single particle image, that is, the target input image described in the embodiment. The processing here includes, but is not limited to, background subtraction, signal enhancement, etc.

[0053] In this step, the target single particle can be a biomolecule (such as DNA, RNA, protein or other intracellular molecular structure, especially protein), and can also be a purified protein in a chemical solution, a chemically synthesized compound, a nanoparticle, etc., which is not limited here.

[0054] The target input image is a single particle motion blur image / single particle fluorescence image after processing. Single particle motion blur image refers to the blur phenomenon in the image caused by the rapid movement of a single particle during single particle fluorescence imaging. This blur usually occurs when tracking a single particle in a living cell in real time, because the particles in the cell are often in a state of constant motion.

[0055] On the basis of obtaining the target input image corresponding to the target single particle in step S110, further, step S120 is executed.

[0056] S120, based on the pre-trained particle trajectory prediction model, a particle pseudo-trajectory image is predicted according to the target input image; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single particle motion blur images and their corresponding particle motion trajectory images.

[0057] The particle trajectory prediction model is used to restore or predict or reconstruct the motion trajectory of the particle (i.e. the particle pseudo-trajectory image) from the blurred single particle image (target input image), so as to more accurately reflect the behavior of the target single particle.

[0058] The particle trajectory prediction model is constructed based on a deep learning neural network, which can include a convolutional neural network, a recurrent neural network, or an architecture combining the advantages of both, without specific limitation here.

[0059] In one specific embodiment, the particle trajectory prediction model includes a convolutional feature extraction layer, a max-pooling layer, a bilinear upsampling layer, a normalization layer, and an activation function layer.

[0060] After determining the architecture of the particle trajectory prediction model, a first training sample set needs to be constructed to train and optimize the particle trajectory prediction model. Specifically, in the training and optimization, the single particle motion blur image in the first training sample set is used as the model input, the predicted pseudo-trajectory image is used as the model input, and the difference between the predicted pseudo-trajectory image and the particle motion trajectory image is used as the training loss to iteratively optimize the particle trajectory prediction model, thereby obtaining the trained particle trajectory prediction model.

[0061] The single particle motion blur image can be real data obtained through experiments or data generated through simulation, without specific limitation here.

[0062] The particle motion trajectory image is the real motion trajectory image of the biological particle, which can be obtained through high-precision imaging technology or manual annotation as the "true value" for model training.

[0063] After the particle trajectory prediction model is trained, the target input image obtained in step S110 is input into the particle trajectory prediction model, and the output particle pseudo-trajectory image of the target single particle is obtained.

[0064] Based on the particle pseudo-trajectory image predicted in step S120, step S130 is further performed.

[0065] S130, based on the particle pseudo-trajectory image, a diffusion quantification feature of the target single particle is calculated.

[0066] Specifically, in order to simplify the problem, it can be assumed that there is less overlap in the particle trajectory, so that, in the case of obtaining a particle pseudo-trajectory image, the area covered by the particle trajectory can be estimated, that is, the trajectory area. Then, according to the quantitative relationship between the pseudo-trajectory area and the particle diffusion coefficient, the diffusion coefficient of the target single particle is fitted, and at the same time, according to the density space distribution of the particle pseudo-trajectory image, the diffusion direction corresponding to the target single particle can be fitted.

[0067] wherein the quantitative relationship between the pseudo-trajectory area and the particle diffusion coefficient is predetermined, and in application, only the pseudo-trajectory area needs to be substituted into the equation corresponding to the linear relationship. Both the diffusion coefficient and the diffusion direction are diffusion quantitative characteristics.

[0068] The diffusion coefficient of the target single particle is a physical quantity describing the random motion rate of the target single particle in the medium, which reflects the free diffusion ability of the target single particle under the action of no external force, and is denoted by D. The unit of the diffusion coefficient is usually m 2 / s or μm 2 / s, indicating the area covered by the particle diffusion per unit time.

[0069] It is worth mentioning that the single particle diffusion quantitative characteristic prediction method provided in the embodiment has a high dynamic range in the common diffusion coefficient range of biological large particles, and can effectively distinguish the diffusion characteristics of different particles, which provides a powerful tool for live cell particle dynamic research.

[0070] In the embodiment, the diffusion quantitative characteristics of the target single particle are calculated based on the target input image corresponding to the target single particle and the pre-trained particle trajectory prediction model. The particle trajectory prediction model is obtained by training and optimizing the first training sample set composed of single particle motion blur images and their corresponding particle motion trajectory images. The method predicts the actual motion path of the target single particle from the target input image through the particle trajectory prediction model, and calculates the diffusion quantitative characteristics of the target single particle based on the actual motion path. The method can quickly and accurately predict the diffusion quantitative characteristics of a large number of single particles in a high marker density environment, significantly reduces phototoxicity, and is suitable for live cell imaging for at least 10 minutes. The diffusion coefficient obtained by solving has a high dynamic range, effectively distinguishes the diffusion characteristics of different biological particles, and provides a powerful tool for live cell particle dynamic research.

[0071] On the basis of the above-mentioned embodiments, further, the acquisition process of the target input image will be described in detail.

[0072] The target input image corresponding to the target single particle is obtained, including: collecting a raw single particle motion blur image of the target single particle; segmenting the raw single particle motion blur image based on a pre-trained U-Net network to obtain a single particle signal mask; pre-positioning the single particle signal in the raw single particle motion blur image, and retaining a target single particle signal mask containing only one positioned single particle; filling the target single particle signal mask by using a background and noise level calculated by a median filter to obtain the target input image; wherein the U-Net network is obtained by training and optimizing a second training sample set composed of single particle motion blur images and corresponding mask images.

[0073] It can be understood that, unlike typical positioning-based single particle detection algorithms, after capturing the single particle motion blur image, the pixels containing the single particle need to be segmented. The embodiment uses a U-Net network to complete this task.

[0074] FIG. 2 shows a structural schematic diagram of the U-Net network provided by the embodiment of the application. As shown in FIG. 2, the U-Net network is composed of a contraction path (left side) and an expansion path (right side).

[0075] The contraction path follows the typical architecture of a convolutional network, which includes repeatedly applying a 3x3 convolution (non-padding convolution) twice, followed by a ReLU activation function and a 2x2 max-pooling operation with a step size of 2 for down-sampling. At each down-sampling step, the number of feature channels is doubled.

[0076] Each step in the expansion path includes up-sampling of the feature map, followed by a 2x2 convolution ("up-convolution") that halves the number of feature channels, concatenation with the corresponding cropped feature map from the contraction path, and two 3x3 convolutions each followed by a ReLU activation function. Since each convolution loses boundary pixels, cropping is necessary.

[0077] In the last layer, a 1x1 convolution is used to map each 64-dimensional feature vector to the required number of classes (only one class in this case, corresponding to the pixel range of the single particle motion blur image). The U-Net network has a total of 23 convolution layers.

[0078] Before actual application, a second training sample set needs to be constructed to train and optimize the U-Net network. Each training sample in the constructed second training sample set includes a single particle motion blur image and its corresponding mask image. Specifically, FIG. 3 shows a second training sample set construction schematic diagram of the single particle diffusion quantification feature prediction method provided by the embodiment of the application.

[0079] As shown in FIG. 3, first, a simSPT algorithm is used to simulate two-dimensional particle trajectories (2D particle trajectories), and a Gaussian function is superimposed on each point on the two-dimensional particle trajectories to obtain a motion blur function Traj i,j .

[0080] Then, the motion blur function is normalized and pixelated to align with the actual camera pixel size (e.g., 110 nanometers) to obtain a pixelated Traj i,j , and then Gaussian white noise and Poisson shot noise are introduced into the pixelated Traj i,j to generate single-particle motion blur images under different signal-to-noise ratios and background levels.

[0081] At the same time, a mask image is obtained by masking more than 95% of the signal intensity area from the pixelated Traj i,j . Thus, according to the single-particle motion blur image and its corresponding mask image, a second training sample set can be constructed.

[0082] Then, the second training sample set is used to train and optimize the U-Net network. Specifically, during training, the single-particle motion blur image is used as the model input, the predicted mask image is used as the model input, and the difference between the predicted mask image and the (actual) mask image is used as the training loss to iteratively optimize the U-Net network, thereby obtaining the trained U-Net network.

[0083] It is worth mentioning that, during training, in order to increase the segmented particle density, the present embodiment places four single-particle motion blur images in a 32x32 pixel wide image (pixel size 110 nanometers), and updates the cross-entropy loss function with a weight w(x) to force the U-Net network to learn the separation boundary between single-particle motion blurs.

[0084] Specifically, the weight map of the U-Net network can be calculated by the following formula (1).

[0085] In formula (1), w c is the weight map to balance the class frequency (i.e., the pixel number ratio between instances with motion blur detection and instances without motion blur detection), d1 is the distance to the nearest motion blur boundary, d2 is the distance to the second nearest motion blur boundary, and in the experiment, w0=10 pixels and σ=2 pixels are set.

[0086] Therefore, in the present embodiment, the maximum detection density that the U-Net network can achieve is about 0.5 motion blur particle signal μm 2 / frame or 16.5 motion blur particle signals μm 2 / s (equivalent to placing 4 motion blurs in a 32x32 pixel area with pixel size of 110 nm), which is comparable to the localization density of classical PALM (Photoactivated Localization Microscopy) and 12 times higher than that of fast-flashing single-particle tracking.

[0087] In one specific embodiment, to train the U-Net network, 10320 pairs of single-particle motion blur images and mask images are simulated, with an exposure time of 30 ms, covering 12 different diffusion coefficients D (ranging from 0.01 pm 2 / s to 10 pm 2 / s). At each diffusion coefficient D, the single-particle motion blur images are randomly generated to simulate the actual imaging data, with a signal-to-noise ratio between 19-35 dB and a background level between 1000-3000. Dropout and random elastic deformation (rotation, translation, and flipping) are included in data augmentation to improve the generalization ability of network training.

[0088] After the U-Net network is trained, it is used to segment the pixels containing single particles to finally obtain the target input image.

[0089] Specifically, first, the original single-particle motion blur image of the target single particle is collected, which can be collected by TIRF (Total Internal Reflection Fluorescence Microscopy) or HILO (Highly Inclined Light Optical Sheet).

[0090] Then, the obtained original single-particle motion blur image is input into the trained U-Net network to generate a single-particle signal mask covering the single-particle signal.

[0091] After that, in order to remove the mask covering multiple single-particle signals, ThunderSTORM is used to pre-localize the single-particle signal in the original single-particle motion blur image, and only the single-particle signal mask containing one localization, i.e. the target single-particle signal mask, is retained.

[0092] Next, a median filter is used to calculate the background and noise levels of each pixel in the target single-particle signal mask within the previous and next 250 frames, so that the target single-particle signal mask is filled with the calculated background and noise levels, and the target input image is obtained.

[0093] In the embodiment, the original single-particle motion blur image of the target single particle is collected, and the original single-particle motion blur image is segmented based on a pre-trained U-Net network to obtain a single-particle signal mask, and then the single-particle signal in the original single-particle motion blur image is pre-positioned, a target single-particle signal mask containing only one positioned target single-particle signal is reserved, and the target single-particle signal mask is filled by using the background and noise levels calculated by the median filter to obtain a target input image, so that the particle pseudo-trajectory image with higher accuracy can be predicted by the particle trajectory prediction model in the later stage.

[0094] On the basis of the above embodiment, further, the training and optimization process of the particle trajectory prediction model will be described in detail below.

[0095] It can be understood that before the particle pseudo-trajectory image is predicted by using the particle trajectory prediction model, the particle trajectory prediction model needs to be trained and optimized.

[0096] The training and optimization of the particle trajectory prediction model specifically includes: simulating a single-particle motion blur image and obtaining a particle motion trajectory image corresponding to the single-particle motion blur image to construct a first training sample set; taking the single-particle motion blur image as a model input, taking a predicted pseudo-trajectory image as a model output, taking a difference between the predicted pseudo-trajectory image and the particle motion trajectory image as a training loss, and iteratively optimizing the particle trajectory prediction model to obtain a training-converged particle trajectory prediction model.

[0097] FIG. 4 shows a construction schematic diagram of the first training sample set provided by the embodiment of the application. As shown in FIG. 4, first, a simSPT algorithm is used to simulate a two-dimensional particle trajectory (2D particle trajectory), and a Gaussian function is superimposed on each point on the two-dimensional particle trajectory to obtain a motion modulus function Traj i,j .

[0098] Then, the motion blur function is normalized and pixelated to align with the actual camera pixel size (for example, 110 nanometers) to obtain a pixelated Traj i,j , and then Gaussian white noise and Poisson shooting noise are introduced into the pixelated Traj i,j to generate a single-particle motion blur image under different signal-to-noise ratios and background levels. The single-particle motion blur image has a size of 32x32 pixels.

[0099] In order to pair the particle motion trajectory with the single-particle motion blur image for subsequent training, first, the Brezenham straight line algorithm is applied to the simulated two-dimensional particle trajectory to generate a pixelated image with a pixel width of 320x320, and then the pixelated image is convolved with a Gaussian filter with the same pixel width to generate the final trajectory image, i.e., the particle motion trajectory image.

[0100] Thus, according to the single-particle motion blur image and the particle motion trajectory image, the first training sample set can be constructed.

[0101] Then, the particle trajectory prediction model is iteratively optimized using the first training sample set. Specifically, FIG. 5 shows a training and optimization schematic diagram of the particle trajectory prediction model provided by the embodiment of the present application.

[0102] As can be seen from FIG. 5, during training, the single-particle motion blur image with noise of 32x32 pixels (pixel size 110 nm) is first enlarged to 320x320 pixels (pixel size 11 nm) using nearest neighbor interpolation as an input image during training.

[0103] The input image is encoded by three convolutional feature extraction layers (each convolutional feature extraction layer contains a 3x3 convolutional kernel, and the output channels are 32, 64, 128, and 512, respectively), followed by batch normalization, ReLU activation function, and maximum pooling layer (2x2, step 2). Subsequently, the three bilinear upsampling layers (2-fold upsampling interpolation) and the convolutional feature extraction layers (each layer has a 3x3 convolutional kernel, and the output channels are 128, 64, and 32, followed by batch normalization and ReLU activation function) are experienced to upsample and decode the features layer by layer. Finally, the 32-channel feature map is converted to a single-channel density prediction image using a 1x1 convolutional layer without activation function, so that the predicted pseudo-trajectory image (image size 320x320, pixel size 11 nm) predicted by the particle trajectory prediction model is obtained from the single-particle motion blur image.

[0104] Then, the difference between the predicted pseudo-trajectory image and the real trajectory image (i.e., the particle motion trajectory image in the first training sample set) is balanced by the sum of the mean absolute error loss function (L1 loss) and the mean square error loss function (MSE), and the weight parameters of the particle trajectory prediction model are iteratively optimized.

[0105] In a specific embodiment, the exposure time is 30 milliseconds, covering a wide distribution of protein diffusion coefficients in living cells (a total of 27 diffusion coefficients, from 0.001 pm 2 / s to 31.62 pm 2 / s). At each diffusion coefficient D, 3000 single-particle motion blur images are randomly generated to simulate actual imaging data, with a signal-to-noise ratio of 19-35 dB and a background level of 1000-3000. The total number of image pairs (single-particle motion blur image and particle motion trajectory image) in the training sample set is 81000. During network training, the sum of the mean absolute error and the mean square error is used as the loss function to quantify and reduce the difference between the predicted pseudo-trajectory image and the particle motion trajectory image.

[0106] After training the optimized particle trajectory prediction model, in actual application, the target input image obtained is directly input into the particle trajectory prediction model, and the corresponding accurate particle pseudo trajectory image can be predicted.

[0107] In the embodiment, the particle trajectory prediction model is trained and optimized by using the first training sample set composed of the single particle motion blur image and the corresponding particle motion trajectory image, and then the particle pseudo trajectory image is predicted based on the particle trajectory prediction model according to the target input image, and the diffusion quantitative feature of the target single particle is calculated based on the particle pseudo trajectory image. The method can quickly and accurately predict the diffusion quantitative features of a large number of single particles in a high marker density environment, and significantly reduces phototoxicity, so that it is suitable for live cell imaging for at least 10 minutes, and the diffusion coefficient obtained has a high dynamic range, effectively distinguishing the diffusion characteristics of different biological particles, and providing a powerful tool for live cell particle dynamic research.

[0108] On the basis of the above embodiment, further, the process of calculating the diffusion coefficient according to the particle pseudo trajectory image will be described in detail.

[0109] The diffusion quantitative feature of the target single particle includes the diffusion coefficient and the diffusion direction. Correspondingly, the diffusion coefficient of the target single particle is calculated based on the particle pseudo trajectory image, including: determining the pseudo trajectory area according to the particle pseudo trajectory image; fitting the diffusion coefficient corresponding to the target single particle according to the quantitative relationship between the pseudo trajectory area and the particle diffusion coefficient; and fitting the diffusion direction corresponding to the target single particle according to the density spatial distribution of the particle pseudo trajectory image.

[0110] It can be understood that D is estimated by the particle pseudo trajectory image, assuming that the target single particle undergoes Brownian motion, and after a time interval Δτ, the target single particle is at a position The probability of finding the target single particle can be described by equation (2).

[0111] In equation (2), represents the displacement of the target single particle from the origin at time t, represents the probability density of the displacement, and D is the diffusion coefficient of the target single particle.

[0112] In single-particle tracking, the displacement of the trajectory can be obtained and equation (2) can be fitted to extract the diffusion coefficient D. However, the particle pseudo-trajectory image predicted by the particle trajectory prediction model is a spatial projection of the trajectory probability without time sequence. Therefore, the diffusion coefficient D is derived by the travel distance / trajectory distance of the target single particle instead of the displacement.

[0113] By integrating equation (2), the probability density of the jump distance r of the target single particle in unit time can be derived from equation (2), which can be seen from equation (3) as follows.

[0114] The total distance of the particle trajectory in the exposure time T of the entire trajectory can be approximated as the sum of N discrete jump lengths with a time interval of t. The probability density of each jump length is determined by equation (3), assuming that each step is an independent identically distributed event. Since the diffusion coefficient D should be related to the square of the total distance R of the particle movement, the total expected R2 can be written as equation (4) as follows. E(R 2 )=Var(R)+E 2 (R) (4)。

[0115] After decomposing the total distance R of the particle movement into N consecutive jumps, equation (4) can be rewritten as equation (5) as follows. E(R 2 )=Var(r1+r2+…+r N )+E 2 (r1+r2+…+r N ) (5)。

[0116] Since Var(r)=4Dt-πDt can be obtained by combining equation (3). By substituting T=Nt into equation (5), equation (6) is obtained as follows.

[0117] To simplify the problem, the present embodiment assumes that the particle movement trajectory itself is rarely overlapped, and the particle pseudo-trajectory image predicted by the particle trajectory prediction model can be regarded as a trajectory convolved with a narrow width w, which makes it possible to estimate the area covered by the particle pseudo-trajectory image (pseudo-track area, referred to as PT area or pseudo-trajectory area), which can be seen from equation (7) as follows. PT area=R*w (7).

[0118] In equation (7), R represents the total trajectory distance of the target single particle.

[0119] Since w represents the uncertainty of the particle trajectory prediction model in predicting the particle trajectory (a constant after training), the embodiment can obtain a quantitative relationship between the square of the pseudo trajectory area and the particle diffusion coefficient D by replacing R with equation (7) in equation (6), and the specific equation (8) can be referred to as follows. D = Const * (PT area) 2 (8)。

[0120] wherein,

[0121] FIG. 6 shows a schematic diagram of obtaining the diffusion coefficient of a target single particle according to an embodiment of the present application. As shown in FIG. 6, when calculating the diffusion coefficient of the target single particle, first, the numerical range of the particle pseudo trajectory image is linearly adjusted to 0 to 1, and the pseudo trajectory area is defined as the pixel area exceeding 0.1, and then the diffusion coefficient corresponding to the target single particle can be fitted by equation (8).

[0122] In a specific embodiment, 1700 single particle motion blur images under high signal-to-noise ratio (35 dB) are simulated, and the diffusion coefficient D ranges from 0.1 μm 2 / s to 31.62 μm 2 / s, and FIG. 7 shows a schematic diagram of the fitting effect of the quantitative relationship between the pseudo trajectory area and the particle diffusion coefficient according to an embodiment of the present application. According to FIG. 7, by using equation (8) to fit the pseudo trajectory area and the diffusion coefficient D, an excellent determination coefficient (R 2 >0.9) can be obtained.

[0123] Therefore, equation (8) fitted under high signal-to-noise ratio can be used to convert the particle pseudo trajectory image predicted by the particle trajectory prediction model into the diffusion coefficient of the single particle motion blur image.

[0124] In the embodiment, the pseudo trajectory area is determined according to the particle pseudo trajectory image, and the diffusion coefficient corresponding to the target single particle is fitted according to the quantitative relationship between the pseudo trajectory area and the particle diffusion coefficient. This method predicts the actual motion path of the target single particle from the target input image by the particle trajectory prediction model, and calculates the diffusion coefficient of the target single particle based on this, which can quickly and accurately predict the diffusion coefficient of a large number of single particles under high marker density environment, and significantly reduces the phototoxicity, so that it is suitable for live cell imaging for at least 10 minutes, and the diffusion coefficient obtained by solving has a high dynamic range, effectively distinguishing the diffusion characteristics of different biological particles, and providing a powerful tool for live cell particle dynamic research.

[0125] On the basis of the above embodiment, further, the process of particle positioning will be described in detail.

[0126] The diffusion coefficient of the target single particle is calculated, and then the following steps are included: in the case that the diffusion coefficient of the target single particle is greater than a set threshold, performing centroid calculation on the particle pseudo-trajectory image to obtain the positioning of the target single particle; in the case that the diffusion coefficient of the target single particle is less than or equal to the set threshold, performing ellipsoid Gaussian fitting on the target input image to obtain the positioning of the target single particle.

[0127] It can be understood that in the embodiment, a series of particle motion trajectories and corresponding single particle motion blur images are first simulated, covering different diffusion coefficients D and signal-to-noise ratios, and specific reference can be made to FIG. 8, which shows a positioning accuracy diagram of different particle positioning algorithms provided by the embodiment of the application. Compared with positioning particles using an elliptical Gaussian function, when the diffusion coefficient D exceeds 1 μm 2 / s, the barycenter algorithm used on the particle pseudo-trajectory image shows higher positioning accuracy (the positioning accuracy is evaluated by comparing the deviation of the particle positioning obtained by the two methods from the geometric center of the actual particle motion trajectory).

[0128] It is worth noting that for fast-diffusion particles with a diffusion coefficient D exceeding 10 μm 2 / s, the barycenter algorithm achieves an accuracy of 20 nanometers, while the Gaussian function fitting is 85 nanometers.

[0129] On the contrary, for slow-diffusion particles with a diffusion coefficient D lower than 1 μm 2 / s, the elliptical Gaussian function positioning particles performs better than the barycenter algorithm. Therefore, for downstream analysis, when the estimated diffusion coefficient D exceeds 1 μm 2 / s, the embodiment uses the weighted barycenter of the particle pseudo-trajectory image to position the particles, otherwise, the ellipsoid Gaussian fitting on the preprocessed original motion blur image (target input image) is used.

[0130] That is to say, the diffusion coefficient of the target single particle determines which way to use to obtain the positioning of the target single particle.

[0131] Therefore, in the embodiment, the diffusion coefficient of the target single particle is compared with a set threshold, for the target single particle with a diffusion coefficient greater than the set threshold, the centroid of the particle pseudo-trajectory image predicted according to the particle trajectory prediction model is calculated, and the position of the centroid is taken as the positioning of the target single particle; for the target single particle with a diffusion coefficient less than or equal to the set threshold, ellipsoid Gaussian fitting is performed on the target input image of the target single particle, and the position obtained by the fitting is taken as the positioning of the target single particle.

[0132] The set threshold can be set according to actual needs, and is not specifically limited here. For example, in a specific embodiment, the set threshold is 1 μm 2 / s.

[0133] In the embodiment, the centroid calculation is performed on the particle pseudo-trajectory image to obtain the localization of the target single particle when the diffusion coefficient of the target single particle is greater than a set threshold, and the ellipsoid Gaussian fitting is performed on the target input image to obtain the localization of the target single particle when the diffusion coefficient of the target single particle is less than or equal to the set threshold. The method can simultaneously capture the spatial localization and diffusion coefficient distribution of particles, can quickly and accurately predict the diffusion coefficients of a large number of single particles in a high marker density environment, significantly reduces the phototoxicity, and is suitable for live cell imaging for at least 10 minutes, and the obtained diffusion coefficient has a high dynamic range and can effectively distinguish the diffusion characteristics of different biological particles, thereby providing a powerful tool for dynamic research on live cell particles.

[0134] On the basis of the above-mentioned embodiments, further, the process of image rendering will be described in detail below.

[0135] The localization of the target single particle is obtained, and then includes: gridizing the localization of the target single particle to obtain a first grid containing the particle localization and a second grid adjacent to the particle localization but not containing the particle localization; generating a particle density probability map based on the localization of the target single particle; based on the particle density probability map and the first grid, using a sum of Gaussian weights to perform interpolation processing on the second grid to obtain a diffusion coefficient corresponding to the second grid; obtaining a diffusion coefficient matrix according to the diffusion coefficient corresponding to the first grid and the diffusion coefficient corresponding to the second grid; performing local smoothing processing on the diffusion coefficient matrix to obtain a mobility map, and displaying the mobility map using an HSV color map to complete image rendering.

[0136] According to the above embodiments, the localization and diffusion coefficient D of the target single particle can be obtained. In order to intuitively show the spatial distribution and correlation between the spatial organization and diffusion of particles, the present embodiment develops an image rendering algorithm, which can generate a particle density probability map (PALM), a diffusion coefficient map (Mobility map), and a particle diffusion density map (MPALM) that fuses the particle density and diffusion coefficient in the same data set.

[0137] Specifically, first, the positions of a total of M total particles are gridized and summed in a square pixel grid of a given size to obtain I count . The positions of the M total particles and the rendering pixel size are selected to achieve the required spatial resolution at the Nyquist sampling frequency.

[0138] Then, I count is convolved with a Gaussian kernel (σ x,y equal to the average localization error) to generate a PALM image I density , which can be visualized using a hot red color map.

[0139] Immediately afterwards, M total The positioning of each particle is again similar to I. count The mesh is merged in a specific manner, and the average diffusion coefficient is calculated for each merged mesh. For a second mesh (D) that does not contain particle locations but is adjacent to the first mesh (Draw) that does contain particle locations... null ), using the sum of Gaussian weights on the second grid D null Interpolation is performed, and the specific interpolation formula can be found in equation (9) below.

[0140] In equation (9), D null(i,j) is the diffusion coefficient corresponding to the second grid obtained by interpolation, m and n are the pixel coordinates of the first grid with particle positioning, and i and j are the pixel coordinates of the second grid that is adjacent to the first grid but without positioning. The interpolated result matrix (i.e., the diffusion coefficient matrix) is denoted as I. D Subsequently, the diffusion coefficient matrix I was analyzed. D Local smoothing is performed to obtain motion map I. smoothed D The motion diagram can be found in equation (10) below.

[0141] To integrate information on particle localization density and diffusion coefficient D, this embodiment selects an HSV color map to generate the MPALM image. The values ​​of the diffusion coefficient map are mapped to the hue channel, and the PALM image I... density Mapped to the value channel, while the saturation channel is set to 0 when the particle density is zero, and 1 otherwise.

[0142] The image rendering results can be seen in Figure 9, which shows a rendering schematic diagram of particle localization and particle diffusion coefficient provided by an embodiment of the present invention. Figure 9 includes a particle density probability map (PALM), a diffusion coefficient map (Mobility map), and a particle diffusion density map (MPALM).

[0143] In this embodiment, by performing image rendering on the location and diffusion coefficient of the target single particle, a particle density probability map, a diffusion coefficient map, and a particle diffusion density map are obtained, which can more intuitively represent the spatial distribution and correlation between particle spatial organization and diffusivity.

[0144] In addition, taking the prediction of the diffusion coefficient and spatial location of a single particle as an example, Figure 10 shows a schematic diagram of the overall process of the single particle diffusion quantification feature prediction method provided in the embodiment of the present invention.

[0145] As shown in FIG. 10, first, the original single-particle motion blur image of the target single particle (i.e., the single-particle motion blur signal in FIG. 10) is collected, and then the original single-particle motion blur image is segmented by using the pre-trained U-Net network to obtain the single-particle signal mask.

[0146] Subsequently, the single-particle signal in the original single-particle motion blur image is pre-positioned by using ThunderSTORM, and a target single-particle signal mask containing only one positioned target single-particle signal is retained.

[0147] Further, the target single-particle signal mask is filled by using the background and noise levels calculated by the median filter to obtain a target input image.

[0148] Then, the target input image is input into the pre-trained particle trajectory prediction model to predict an output particle pseudo-trajectory image.

[0149] Then, the pseudo-trajectory area is calculated based on the particle pseudo-trajectory image, and the diffusion coefficient corresponding to the target single particle is fitted according to the quantitative relationship between the pseudo-trajectory area and the particle diffusion coefficient.

[0150] Finally, in the case where the diffusion coefficient of the target single particle is greater than a set threshold, the particle pseudo-trajectory image is subjected to centroid calculation to obtain the positioning of the target single particle; in the case where the diffusion coefficient of the target single particle is less than or equal to the set threshold, the target input image is subjected to ellipsoid Gaussian fitting to obtain the positioning of the target single particle.

[0151] Based on the above, the particle spatial distribution and the diffusion coefficient distribution of the target single particle can be captured. It should be further noted that the above steps are described in detail in the above embodiments, and will not be described here again.

[0152] It is worth mentioning that, in order to verify the robustness of the single-particle diffusion quantification feature prediction method provided by the embodiments of the present application, a series of single-particle motion blur images generated by particle motion trajectories with different diffusion coefficients under different signal-to-noise ratios are calculated and simulated, and then the pseudo-trajectory areas obtained by the same particle motion trajectory under different signal-to-noise ratios are tested. The results show very small fluctuations, indicating that the particle trajectory prediction model is robust to noise.

[0153] At the same time, the robustness performance of the U-Net network segmentation under different signal-to-noise ratios is also tested, and it is found that the area of the single-particle signal mask generated by the same particle motion trajectory can also remain stable under different signal-to-noise ratios. Therefore, the particle trajectory prediction model and the U-Net network exhibit strong robustness to different signal-to-noise ratios, and this characteristic helps to reduce the errors caused by the fluctuation of the signal-to-noise ratio in the imaging process, because these errors may lead to fluctuations in the estimated diffusion coefficient D.

[0154] The test results of the particle trajectory prediction model and the U-Net network can be specifically seen from FIG. 11, which shows an inference effect schematic diagram of the particle trajectory prediction model and the U-Net network provided by the embodiment of the application.

[0155] In some embodiments, the process of cell sample preparation and image data acquisition is also described in detail.

[0156] The target protein (i.e., the target single particle) is first labeled with a suitable fluorescent tag, which is connected to the N- or C-terminus of the target protein using HaloTag or SNAP-tag. Then the target protein is expressed in cells and incubated with the cells using a high quantum yield fluorescent dye (e.g., PA-JF646, HM-SiR, etc.) modified with a tag protein ligand. By total internal reflection fluorescence microscopy, in total internal reflection mode or high oblique light sheet mode, set the appropriate exposure time (e.g., 30 ms) and shooting density (macroscopically observe that most of the adjacent particles do not overlap, or the particle density is less than 16.5 moving blur particle signals / μm 2 / s), take a moving blur image of a single particle.

[0157] In single-channel MPALM (particle diffusion density map) imaging, 2000 frames of MPALM images are taken continuously, and a whole image (wide-field image of non-single particle image) is connected as a period. This period is repeated 5 to 20 times according to the survival ability of the cell and the particle photobleaching. During each MPALM frame, the excitation light (642 nm) continuously illuminates the excitation dye for 30 ms of camera exposure time. When using a photoactivatable dye (e.g., PA-JF646), a 405 nm laser is pulsed for 0.5 ms of camera dead time to activate the photoactivatable dye. For each large-scale image, the excitation laser (560 nm) and exposure time are adjusted to achieve the desired signal-to-noise ratio.

[0158] In dual-channel MPALM-PALM imaging, 560 nm and 642 nm lasers work simultaneously for 30 ms of camera exposure time. A 405 nm laser is pulsed for 0.5 ms of camera dead time to activate the photoconversion protein of the PALM channel, such as mEosEM. Two cameras are used simultaneously to detect the excitation light of the two channels. In addition to this shooting mode, MPALM-MPALM dual-channel imaging can also be adjusted according to requirements.

[0159] In some embodiments, the image registration process for dual-channel MPALM-PALM imaging is described in detail.

[0160] For dual-color imaging (e.g. dual-color MPALM-PALM), the images of one channel need to be registered to match the coordinates of the other channel due to the difference in camera pixel size between the two channels. See FIG. 12 for an illustration of the image registration procedure.

[0161] The target image is defined as the MPALM channel with pixel size of 110 nm, while the moving image is the PALM channel with pixel size of 160 nm. The task is to find a geometric transformation (including rotation, scaling and translation) that, when applied to the moving image, results in an image with the same spatial coordinates as the target image. Prior to imaging each cell sample, 100 nm TetraSpeck fluorescent microspheres (Invitrogen, T7279) were imaged and used to estimate the transformation matrix for the registration of the two channels.

[0162] Initially, the transformation matrix was estimated using the phase correlation between the moving image and the target image. This estimation enabled the conversion of the moving image into an initial registered image. Next, the microspheres were localized in the target image, the moving image and the initial registered image using two-dimensional Gaussian fitting.

[0163] The microsphere localizations in the target image were paired with the microsphere localizations in the initial registered image. Then, the microsphere localizations in the initial registered image were inverted and paired with the moving image. This approach facilitated the pairing of the microsphere localizations between the target image and the moving image using the localizations of the initial registered image as an intermediary guide.

[0164] Subsequently, the paired microsphere localizations were used as control points to update the estimate of the transformation matrix, which was then applied to align the localizations of the two channels. Using this approach, the relative localization distances of the same microspheres across the image registration were averaged to be 10 nm when aligning the two-channel fluorescent microsphere images.

[0165] In some other embodiments, the sample drift correction is described in detail.

[0166] For single-color MPALM, the sample drift was estimated using the global images captured between the MPALM images (see above for the acquisition of the first and second training sample sets). Specifically, the peak of the cross-correlation between the global images was calculated, and the deviation of the peak from the center of the image was used as the estimate of the sample drift. Then, the sample drift in the entire MPALM image sequence was interpolated using a local linear regression, and the localizations of each MPALM frame were corrected to the first frame.

[0167] In single-channel MPALM imaging, sample drift is estimated by analyzing the overall images captured between MPALM frames (see the example of "Cell sample preparation and data acquisition"). Specifically, the peak of cross-correlation between overall images is calculated, and the deviation of this peak from the image center is used as an estimate of sample drift. Subsequently, sample drift is interpolated across the entire MPALM image sequence by local linear regression. This corrects the localization of each MPALM frame to the first frame.

[0168] In dual-channel MPALM-PALM imaging, PALM images are first aligned to MPALM images (see the example of "Image registration"). Then, the localization of the PALM channel is divided into a suitable number of bins, typically 2000 frames per bin. The localization within each bin is then used to reconstruct individual PALM images. The peak of cross-correlation between reconstructed PALM images is calculated, and the deviation of this peak from the image center is used as an estimate of sample drift. Subsequently, sample drift is interpolated across the entire PALM image sequence by local linear regression. Finally, a regression model is applied to correct the localization of each MPALM and PALM frame to align with the first frame.

[0169] Corresponding to the single particle diffusion quantification feature prediction method described in the above embodiments, the application further provides a single particle diffusion quantification feature prediction device. Specifically, FIG. 13 shows a structural schematic diagram of a single particle diffusion quantification feature prediction device provided by an embodiment of the application.

[0170] As shown in FIG. 13, the device comprises: a target input image acquisition module 1310 configured to acquire a target input image corresponding to a target single particle; a particle pseudo-trajectory image prediction module 1320 configured to predict a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model and according to the target input image; and a particle diffusion coefficient acquisition module 1330 configured to calculate a diffusion quantification feature of the target single particle based on the particle pseudo-trajectory image; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of a single particle motion blur image and a particle motion trajectory image corresponding thereto.

[0171] In the embodiment, the target input image corresponding to the target single particle is acquired by a target input image acquisition module 1310, a particle pseudo-trajectory image prediction module 1320 predicts a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model according to the target input image, and then a particle diffusion quantitative feature acquisition module 1330 calculates the diffusion quantitative feature of the target single particle based on the particle pseudo-trajectory image. The particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single particle motion blur images and their corresponding particle motion trajectory images. The device predicts the actual motion path of the target single particle from the target input image through the particle trajectory prediction model, and calculates the diffusion quantitative feature of the target single particle based on this. It can quickly and accurately predict the diffusion quantitative features of a large number of single particles in a high marker density environment, and significantly reduces phototoxicity, making it suitable for live cell imaging for at least 10 minutes. The diffusion quantitative feature obtained by solving has a high dynamic range, effectively distinguishes the diffusion characteristics of different biological particles, and provides a powerful tool for live cell particle dynamic research.

[0172] FIG. 14 illustrates an example of an entity structure of an electronic device, as shown in FIG. 14, the electronic device can include a processor 1410, a communications interface 1420, a memory 1430, and a communications bus 1440, wherein the processor 1410, the communications interface 1420, and the memory 1430 complete mutual communication through the communications bus 1440. The processor 1410 can invoke the logical instructions in the memory 1430 to execute the single particle diffusion quantitative feature prediction method, which includes: acquiring a target input image corresponding to a target single particle; predicting a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model according to the target input image; and calculating the diffusion quantitative feature of the target single particle based on the particle pseudo-trajectory image; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single particle motion blur images and their corresponding particle motion trajectory images.

[0173] Moreover, the logic instructions in the memory 1430 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0174] In another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the single particle diffusion quantification feature prediction method provided by the above-mentioned methods, the method comprising: obtaining a target input image corresponding to a target single particle; based on a pre-trained particle trajectory prediction model, predicting a particle pseudo-trajectory image according to the target input image; and based on the particle pseudo-trajectory image, calculating a diffusion quantification feature of the target single particle; wherein the particle trajectory prediction model is obtained by training and optimizing according to a first training sample set composed of a single particle motion blur image and a corresponding particle motion trajectory image.

[0175] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0176] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software and the necessary general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions essentially or the parts that make contributions to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0177] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A single particle diffusion quantification feature prediction method, characterized in that, The method comprises: acquiring a target input image corresponding to a target single particle; based on a pre-trained particle trajectory prediction model, predicting a particle pseudo-trajectory image according to the target input image; based on the particle pseudo-trajectory image, calculating the diffusion quantification feature of the target single particle; wherein the particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single particle motion blur images and their corresponding particle motion trajectory images.

2. The single particle diffusion quantification feature prediction method of claim 1, wherein, The method of acquiring a target input image corresponding to a target single particle comprises: collecting an original single particle motion blur image of a target single particle; segmenting the original single particle motion blur image based on a pre-trained U-Net network to obtain a single particle signal mask; pre-positioning the single particle signal in the original single particle motion blur image, and retaining only a target single particle signal mask containing a single particle; filling the target single particle signal mask with the background and noise level calculated by a median filter to obtain the target input image; wherein the U-Net network is obtained by training and optimizing a second training sample set composed of single particle motion blur images and their corresponding mask images.

3. The single particle diffusion quantification feature prediction method of claim 1, wherein, The diffusion quantification feature of the target single particle includes a diffusion coefficient and a diffusion direction. Correspondingly, the method of calculating the diffusion quantification feature of the target single particle based on the particle pseudo-trajectory image comprises: determining a pseudo-trajectory area according to the particle pseudo-trajectory image; fitting the diffusion coefficient corresponding to the target single particle according to the quantitative relationship between the pseudo-trajectory area and the particle diffusion coefficient; fitting the diffusion direction corresponding to the target single particle according to the density spatial distribution of the particle pseudo-trajectory image.

4. The single particle diffusive quantification feature prediction method of claim 1, wherein, Training and optimizing the particle trajectory prediction model comprises: simulating a single particle motion blur image and acquiring a particle motion trajectory image corresponding to the single particle motion blur image to construct a first training sample set; taking the single particle motion blur image as the model input, taking the predicted pseudo-trajectory image as the model output, taking the difference between the predicted pseudo-trajectory image and the particle motion trajectory image as the training loss, and iteratively optimizing the particle trajectory prediction model to obtain a training-converged particle trajectory prediction model.

5. The single particle diffusion quantification feature prediction method of claim 4, wherein, The method of simulating a single particle motion blur image comprises: simulating a two-dimensional particle trajectory and superimposing a Gaussian function on each point on the two-dimensional particle trajectory to obtain a motion blur function; normalizing and pixelizing the motion blur function, and introducing Gaussian white noise and Poisson shooting noise to obtain a single particle motion blur image under different signal-to-noise ratios and background levels.

6. The single particle diffusive quantification feature prediction method of claim 3, wherein, The method of fitting the diffusion coefficient corresponding to the target single particle comprises: in the case that the diffusion coefficient of the target single particle is greater than a set threshold, performing centroid calculation on the particle pseudo-trajectory image to obtain the positioning of the target single particle; in the case that the diffusion coefficient of the target single particle is less than or equal to a set threshold, performing ellipsoid Gaussian fitting on the target input image to obtain the positioning of the target single particle.

7. The single particle diffusive quantification feature prediction method of claim 6, wherein, The method of obtaining the positioning of the target single particle comprises: Griding the positioning of the target single particle to obtain a first grid containing particle positioning and a second grid adjacent to the particle positioning but not containing particle positioning; Based on the positioning of the target single particle, a particle density probability map is generated; Based on the particle density probability map and the first grid, the second grid is interpolated using a Gaussian weight sum to obtain a diffusion coefficient corresponding to the second grid; According to the diffusion coefficient corresponding to the first grid and the diffusion coefficient corresponding to the second grid, a diffusion coefficient matrix is obtained; The diffusion coefficient matrix is locally smoothed to obtain a motion map, and the HSV color map is used for display to complete image rendering.

8. A single-particle diffusion quantification feature prediction device characterized by comprising: It comprises: A target input image acquisition module is configured to acquire a target input image corresponding to a target single particle; A particle pseudo-trajectory image prediction module is configured to predict a particle pseudo-trajectory image based on a pre-trained particle trajectory prediction model and the target input image; A particle diffusion quantification feature acquisition module is configured to calculate the diffusion coefficient of the target single particle based on the particle pseudo-trajectory image. The particle trajectory prediction model is obtained by training and optimizing a first training sample set composed of single particle motion blur images and their corresponding particle motion trajectory images.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the single particle diffusion quantification feature prediction method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the single particle diffusion quantification feature prediction method of any one of claims 1 to 7.

Images