Fluorescence microscopic imaging method, database, system and application method thereof

Abstract

The invention relates to the field of optical imaging and machine learning, and discloses a fluorescent molecule quantification and nanometer positioning method and system by means of combination of machine learning and fluorescent molecule imaging and application of the fluorescent molecule quantification and nanometer positioning method and system to quantitative nanometer resolution imaging of target molecules in biological medicine research.

Description

Technical Field

[0001] The present invention relates to the fields of optical imaging and artificial intelligence, and particularly to a method and system for molecular precise quantitative nano-positioning by combining artificial intelligence and fluorescence molecular imaging, and its application in quantitatively detecting specific molecules with nano-resolution in biomedical samples. Background Art

[0002] Precise molecular positioning and quantification of biomedical samples at the nanoscale or even molecular level are of great importance for both basic research and clinical applications. For the in-situ detection of various molecules in cells, currently, it is mainly achieved by specifically fluorescently labeling the samples and further imaging. Due to the existence of the optical diffraction limit, non-super-resolution fluorescence microscopes (including confocal microscopes, etc.) cannot resolve different molecules within the diffraction limit range, let alone determine their precise positions. Various super-resolution imaging techniques developed in the past two decades have broken through the optical diffraction limit and can resolve luminescent molecules or particles with distances less than the diffraction limit at different levels; these techniques include stimulated emission depletion microscopy (STED), structured illumination microscopy (SIM), and single molecule localization microscopy (SMLM), where SMLM includes stochastic optical reconstruction microscopy (STORM), photoactivated localization microscopy (PALM), and point accumulation for imaging in nanoscale topography (PAINT) that relies on the reversible binding of fluorescent molecules to targets for luminescence. Due to the limitations of their imaging principles, STED and SIM are difficult to precisely locate target molecules on the sample at the molecular level resolution and cannot determine the specific number of molecules on the target structure. Single molecule localization microscopy SMLM controls that at most only one molecule within the diffraction limit range emits light in each frame of imaging, and precisely locates its centroid (the centroid is the precise position of the luminescent molecule) by performing Gaussian fitting on its point spread function. The positioning accuracy is close to the molecular scale. Finally, the localizations of each frame are reconstructed into a super-resolution image, which can intuitively reflect the molecular distribution and organization mode in the imaged field of view; this technology won the 2014 Nobel Prize in Chemistry. However, due to the randomness of the number of times each fluorescent molecule emits light during the SMLM imaging process, the number of localizations in the finally reconstructed super-resolution image is non-linearly correlated with the number of molecules, making it difficult to precisely quantify the target sample; moreover, SMLM relies on high-intensity lasers to achieve sparse excitation, and high-intensity lasers may cause partial bleaching of some fluorescent molecules, hindering precise quantification; in addition, SMLM trades time for spatial resolution. To obtain a super-resolution localization distribution map of molecules in a certain field of view, thousands to tens of thousands of images need to be acquired for this field of view, and the corresponding imaging time usually takes from a few minutes to dozens of minutes, causing significant photo-bleaching damage to the sample, especially not suitable for imaging living cells; SMLM needs to rely on specific fluorescent molecules with blinking functions, and only a few fluorescent molecules are capable of performing such experiments. Summary of the Invention

[0003] In order to break through the current situation where existing imaging technologies cannot achieve both precise quantification and nanoscale resolution, the present invention aims to provide a method that can achieve both precise quantification and nanoscale localization, including a fluorescent molecule database, a machine learning method, an artificial intelligence model, a quantitative nanoscale resolution fluorescence microscopy method, and a sample detection method.

[0004] The present invention provides a first technical solution, aiming to provide a method that can achieve precise quantification. The specific solution is as follows: A fluorescent molecule database, including single-molecule spot images of individual fluorescent molecules emitting light.

[0005] Furthermore, an optimized solution of the first solution: The fluorescent molecule database further includes a composite spot image of multiple molecules emitting light formed by superimposing two or more single-molecule spot images.

[0006] Furthermore, an optimized solution of the first solution: The fluorescent molecule database, the composite spot image includes a fluorescent spot image formed by placing and superimposing two or more single-molecule spots with their molecular position distances not exceeding the diffraction limit, or / and also includes a fluorescent spot image generated by placing and superimposing two or more single-molecule spots with their molecular position distances exceeding the diffraction limit.

[0007] Furthermore, an optimized solution of the first solution: The fluorescent molecule database further includes the molecular position and number information corresponding to the single-molecule spot image, the molecular position and number information corresponding to the multiple single-molecule spot images of the composite spot image, and the molecular position and number information corresponding to the multiple fluorescent molecules of the composite spot image.

[0008] Furthermore, an optimized solution of the first solution: The single-molecule spot image in the fluorescent molecule database is a spot image after removing the background signal of the image.

[0009] The fluorescent molecule database includes a single-molecule spot data set, and the single-molecule spot images contained therein are spot images formed by individual fluorescent molecules emitting photons.

[0010] In biological research, fluorescence microscopy is one of the most commonly used research tools. Fluorescent molecules are used to label target molecules on cells for microscopic imaging. However, traditional fluorescence microscopy techniques (non-super resolution) are limited by the diffraction limit and cannot resolve two or more molecules within the diffraction limit range (a region with a diameter of two hundred to two hundred and fifty nanometers). The spots formed by such simultaneously emitting fluorescent molecules overlap with each other, and it is impossible to directly determine from the spot image how many molecules actually emit light to form the spot, nor can the nanoscale precision spatial positions of these molecules be determined.

[0011] Among the existing super-resolution imaging technologies, the lateral resolution of single-molecule localization imaging microscopy technology SMLM (including STORM or PALM) is about 20 nm, which is about 10 times higher than ordinary fluorescence imaging and 3-5 times higher than super-resolution imaging SIM. According to the research of this laboratory and literature reports, the inventors found that although SMLM is the only non-destructive microscopy technology that is close to the molecular size resolution so far, it has significant limitations in precise quantification, which is also a problem recognized by the super-resolution imaging industry. In fact, SMLM uses the random emission of a single fluorescent molecule within the diffraction limit at different moments of long-term imaging, so as to accurately locate the luminescent molecule through its Gaussian spot, so that it can break through the limitation of the diffraction limit through thousands of frames of imaging and single-molecule positioning and reconstruction, and obtain super-resolution structural images of target samples (such as cell nucleoli, microtubules, etc., which are larger than the diffraction limit). However, due to the random excitation and luminescence characteristics of the light-controlled fluorescent molecules that SMLM relies on to achieve single-molecule localization, the number of times each fluorescent molecule emits light and is located varies greatly under the same imaging conditions. As a result, when using SMLM to image biological structures, it is difficult for researchers to determine the precise number and position of molecules in each nanometer range based on the final nanolocalization super-resolution image. This limits the widespread application of SMLM in biomedical research.

[0012] In recent years, the rapid development of artificial intelligence (AI) has broken through many traditional hardware limitations. For example, networks based on deep Fourier channel attention such as DFCAN, its variant network DFGAN, and DL-SIM have demonstrated the ability to obtain STED or SIM-level super-resolution images from different conventional imaging modes. ANNA-PALM uses PALM images as training data and, through a generative adversarial network, can reconstruct high-quality super-resolution images with only one percent of the normal required number of frames without sacrificing spatial resolution. Single-frame super-resolution microscopy (SFSRM) based on the enhanced super-resolution generative adversarial network (ESRGAN) combines high temporal and spatial resolutions and can achieve a resolution comparable to STORM through single-frame imaging. Current machine learning methods have not effectively solved the problem of accurately quantifying the number of molecules and their localization information for the overlapping spots generated by the simultaneous emission of two or more molecules within the diffraction limit. The inventors conducted in-depth research and revealed the underlying reasons: on the one hand, existing networks are not designed specifically for the precise quantification and localization of molecules within the diffraction limit and lack pertinence; on the other hand, the training of existing networks relies on training data obtained from existing imaging techniques, such as SMLM data, but these data themselves suffer from insufficient or even complete lack of information on the specific number and precise position of molecules within the diffraction limit. This defect causes the trained networks to be able to extract the position information of molecules in images of dense spots but still unable to further distinguish the precise number and position of molecules within the diffraction limit.

[0013] The training data of deep learning networks currently applied to break through the diffraction limit can be roughly divided into two types: 1) The training data of machine learning networks are low-resolution wide-field images and their corresponding super-resolution images. Using an end-to-end image generation method, the low-resolution images are converted into high-resolution images. The super-resolution images obtained by super-resolution microscopy are used as the ground truth (GT) for model learning. For example, single-frame super-resolution microscopy (SFSRM) uses stochastic optical reconstruction microscopy (STORM) images as GT, ANNA-PALM uses photoactivated localization microscopy (PALM) images as GT, and DL-SIM uses SIM images as GT. This strategy enables these models to generate super-resolution images. However, the images used as GT have limitations. Existing super-resolution microscopy techniques cannot achieve precise quantification of molecules, and the GT does not contain information on the precise number and position of molecules. Therefore, the trained models are also restricted and cannot accurately provide the precise number and position of sub-diffraction-limit-distance molecules, resulting in their inability to precisely quantify and localize biomolecules in cell ultrastructures or complexes.

[0014] 2) By combining a randomly generated point spread function (PSF) with the SMLM imaging model to simulate the acquisition of wide-field spot images of the real imaging process, the generated images and the corresponding central positions of each PSF or the corresponding super-resolution images are used as the training data for the deep learning network. Such models are used to extract the position information of single molecules under dense spot conditions. For example, DECODE can output a set of coordinates with a confidence level higher than a specific threshold when inputting a low-resolution image, and Deep-STORM can output a super-resolution image when inputting a low-resolution image. However, in the case of two or more molecules forming sub-diffraction-limited spots with a distance of only nanoscale, the following problems still exist: it is difficult to accurately distinguish and locate the number of molecules, and the complex spatial distribution between molecules cannot be accurately analyzed.

[0015] In response, the inventor has proposed a new concept. On the one hand, it is necessary to design a deep learning model specifically for the precise quantification or / and positioning of molecules within the diffraction limit to overcome the deficiencies of existing models in this task. On the other hand, it is necessary to establish a high-quality dataset with clear annotations, which should include luminescent spot images with known numbers of molecules and their corresponding precise position annotation information. These two aspects complement each other and lay a technical foundation for achieving high-precision molecular quantification or / and positioning.

[0016] The success of machine learning depends on high-quality real data. Therefore, reasonably constructing a database with real-value annotations becomes the key breakthrough point. The new database not only needs to contain rich spot image samples, but also should provide accurate molecular numbers and position annotation information for each sample. On this basis, combined with a carefully designed deep learning model, it is possible to accurately predict the molecular number and position information within the diffraction limit, thereby improving the accuracy of quantification or / and positioning.

[0017] Directly obtaining sub-diffraction-limited spot images with precisely specified numbers of molecules through existing imaging techniques is an extremely challenging task, and it is even more difficult to further accurately determine the specific position information of these molecules. In addition, constructing a deep learning network with stable performance and strong generalization ability usually requires relying on a large amount of high-quality data. However, in actual experiments, it is not easy to obtain a sufficient number of spot image data with clear annotations, which significantly limits the application of deep learning technology in this field.

[0018] To address this challenge, the present invention proposes a strategy that combines single-molecule imaging experiments with simulated generation of multi-molecule luminescence data, providing a breakthrough solution for the training of the model. Through single-molecule imaging experiments, the spot images formed by the luminescence of individual fluorescent molecules and their high-precision positioning can be obtained as reliable experimental basic data. Further, by combining the single-molecule data directly from the experiment with the simulated multi-molecule data synthesized by superimposing the single-molecule experimental data, the scale of the data set and the diversity of spot morphology, quantity, and positioning information can be effectively expanded. Using these data as the input data for the deep learning model to train the deep learning model, not only can the learning ability of the model for spot features be improved, the accuracy of the model in molecular quantity recognition and position prediction be enhanced, but also the generalization ability of the model can be improved.

[0019] Taking the fluorescent molecule database of the present invention as the input data, for the deep learning model obtained by using any existing machine learning method, when measuring and / or positioning a molecular sample, an existing fluorescence microscopy imaging method is used to image the molecular sample to obtain an image, and combined with the above-obtained deep learning model, accurate quantification and / or positioning of the sample can be achieved.

[0020] This strategy fully combines the advantages of experiments and simulations. It relies on real experimental data to ensure the generalization ability of the model when it is subsequently applied to actual imaging experiments, and overcomes the spot images of multi-molecule luminescence within the diffraction limit and the corresponding molecular quantity or / and positioning training set and validation data set required for the training of the accurate quantification and / or positioning model by simulating the generation of multi-molecule data based on the experimental data. This provides important support for overcoming the bottleneck of insufficient data and improving the application effect of the deep learning model in sub-diffraction limit molecular quantification or / and positioning tasks. At the same time, according to different needs, different fluorescent molecules can be selected as the experimental basic data, and it can be more effectively applied to the research of specific molecules.

[0021] Based on a large number of single-molecule spot images obtained in the experiment, the present invention generates multi-molecule spot images by superposition, simulates the spot characteristics when multiple molecules emit light simultaneously, and accurately corresponds to the position information of each molecule.

[0022] Specifically, the present invention obtains a single fluorescent molecule spot image by imaging a single fluorescent molecule sample, and determines the molecular coordinates using the Gaussian fitting method. Based on these single-molecule images and their coordinates, a multi-molecule spot image is generated by superposition. Since the single-molecule images are from experiments and the quantity and position information are known, we obtain the corresponding molecular quantity or / and position coordinates while obtaining the generated synthetic spot image (i.e., the simulated multi-fluorescent molecule emission spot image). To improve the diversity and coverage of the data, the present invention simulates and restores the sub-diffraction limit distribution pattern of molecules in the experiment by randomly selecting molecules, setting the molecular distribution range, randomly choosing the molecular positions, etc. The single-molecule and multi-molecule spot images and their corresponding molecular quantities and / or centroid coordinates together constitute a fluorescent molecule database for training the model.

[0023] Taking the spot image and its corresponding molecular quantity and / or position coordinates as a pair of mapping relationships, an existing machine learning model is used to train a deep learning network. The deep learning network extracts image features, maps the feature information to the image category, and then divides the molecular quantity corresponding to each spot, or / and maps the feature information to the numerical value corresponding to the image, so as to accurately locate the centroid coordinates of each molecule. This method can calculate the accurate molecular quantity and / or position information from the low-resolution fluorescent molecule spot image, providing high-precision molecular quantification and / or positioning ability within the diffraction limit.

[0024] It should be noted here that the database refers to a collection including but not limited to the data information of the technical solution of the present invention, and is not necessarily a commercial database with an organized, managed, and user access mechanism. Of course, it can also be a commercial database including the data information collection of the technical solution of the present invention.

[0025] Before inputting the spot image into the deep learning model for training, it is necessary to preprocess the spot image through an image filter to improve the data quality and enhance the effect of model training. During the image acquisition process, the camera will inevitably be affected by various noises, such as thermal noise, readout noise, quantum noise, and ambient light interference. These noises will cause the intensity of each pixel collected to show an unstable phenomenon of rising and falling on the basis of the original signal. The existence of noise not only increases the randomness of the image signal but may also introduce distortion in the spot image. When the unfiltered image is directly input into the deep learning network for feature extraction, these noises may be learned by the network and become redundant features, thus affecting the weight distribution of the model and resulting in a decrease in the accuracy of the classification and regression results.

[0026] Through filtering processing, the influence of noise can be effectively weakened, especially high-frequency noise, thereby smoothing the image, reducing redundant information while maintaining its main features, reducing unnecessary noise components, making the image signal closer to the true molecular luminescence characteristics, and improving the image quality. The effects of filtering processing are mainly reflected in the following aspects: Firstly, by removing random noise, filtering can significantly improve the signal-to-noise ratio, make the intensity distribution of the light spot smoother, and enhance the overall quality of the image. Secondly, filtering processing helps to improve the accuracy of the deep learning model in the process of image feature extraction. Because after removing the noise, the input image features focus more on the target signal, thus reducing the model's dependence on redundant features. By reducing the influence of noise on the model weight distribution, filtering processing can improve the classification and regression accuracy of the model and enhance its performance in molecular quantification and / or localization tasks.

[0027] Therefore, filtering the light spot image before model training can effectively reduce noise interference, that is, obtain the light spot image after removing the background signal of the image, and provide higher-quality input data for the deep learning model, thereby improving the overall performance and application effect of the model. Of course, if the light spot image obtained without filtering processing and without removing the background signal of the image is used to form the fluorescence molecule database of the present invention, and the existing machine learning model is used to train the deep learning network, and then the existing fluorescence microscopy method is combined with the deep learning network, the purpose of obtaining accurate molecular quantity and / or position information can also be achieved.

[0028] The present invention also provides a second technical solution: a machine learning method, and the objects of learning include the information in any one of the above-mentioned fluorescence molecule databases.

[0029] After the information such as the light spot image, the number of molecules, and / or the molecular position in the above-mentioned fluorescence molecule database is completed, the features and knowledge of these information are learned by machine learning to form an artificial intelligence model system, and the construction strategy is as Figure 1 shown.

[0030] The present invention also provides a third technical solution: an artificial intelligence system obtained by using the above-mentioned machine learning method.

[0031] Furthermore, an optimized solution of the third solution: an artificial intelligence system, including a quantitative model and / or a localization model.

[0032] Furthermore, an optimized solution of the third solution: an artificial intelligence system, the output layer of the quantitative model includes a fully connected layer and an activation function for classification, and the output of the fully connected output layer includes the number of molecules corresponding to the light spot image; the localization model is an independent localization network trained separately for different numbers of molecules, and the output of its fully connected output layer includes the coordinate information of the molecular position corresponding to the light spot image.

[0033] The present inventor has proposed a new strategy for decomposing the task of obtaining molecular nano-distribution into two independent parts: quantification or / and localization, as Figure 2 shown. This method completes the task by a classification network and a localization network respectively, and trains independent localization networks for different numbers of molecules to better cope with the challenge of multi-molecule signal overlap within the diffraction limit to the localization accuracy. This idea stems from the characteristic differences of molecular spots at different numbers, and improves the pertinence and adaptability of the network through task refinement.

[0034] By making a fine-grained division of the key parameter of the number of molecules, it is avoided that the unified network has a decrease in accuracy due to multi-objective task interference. Specifically, the classification network first determines the number of molecules in each image, and then selects the corresponding localization network according to the classification result for precise localization. Each localization network focuses on processing the spot characteristics of a specific number of molecules, and specifically optimizes the learning parameters of the model. This modular design significantly reduces the complexity of model learning, while improving the localization accuracy and precision.

[0035] The process of obtaining the fluorescence molecular nano-distribution information can be decomposed into two main steps: 1) Quantity identification: Determine the number of fluorescence molecules through the spot image. 2) Determine the position: Further calculate the specific coordinates of each fluorescence molecule.

[0036] To achieve this goal, the tasks of identifying the number and position of fluorescence molecules are transformed into a two-step process of classification and regression localization. For this purpose, two types of deep learning networks are designed and trained respectively: 1) Classification network: Used to predict the number of fluorescence molecules in each image. The output of the classification network is used to select the appropriate regression localization network.

[0037] 2) Regression localization network: For the number prediction output by the classification network, use the corresponding regression localization network to accurately estimate the position information of the fluorescence molecules.

[0038] To achieve this, the deep neural network Vision-Transformer or ResNet is selected as the backbone network, and the existing backbone network is fine-tuned.

[0039] Classification network: The output layer consists of a fully connected layer and an activation function for classification, focusing on the prediction of the number of fluorescence molecules.

[0040] Regression localization network: For different numbers of fluorescence molecules, independent localization networks are trained respectively, and its fully connected output layer is designed to directly predict the coordinate information of the target molecules. This improved strategy dynamically adjusts the model output layer according to the number of spots, ensuring the accuracy of the classification and localization tasks.

[0041] The present invention also provides a fourth technical solution: a fluorescence microscopy imaging method, which includes using the above-mentioned machine learning method and artificial intelligence system.

[0042] Furthermore, an optimized solution of the fourth solution: a fluorescence microscopy imaging method, where the distance between target molecules is within the diffraction limit.

[0043] Non-super-resolution fluorescence imaging - artificial intelligence system together constitute the fluorescence microscopy imaging method in the present invention. Specifically, first, a low-resolution fluorescence image is obtained through non-super-resolution fluorescence imaging technology, and then this image is input into the quantitative model and / or localization model in the artificial intelligence system. The quantitative model is responsible for quantifying the number of molecules in the light spot, while the localization model is responsible for determining the specific position information of each molecule, thereby reconstructing the nano-distribution of each molecule on the basis of the low-resolution image and achieving the effect of super-resolution fluorescence imaging.

[0044] The specially designed quantitative model and localization model in this method are applicable to various sample environments, mainly for the case where the distance between molecules is within the diffraction limit range.

[0045] In the test of the synthetic data set formed by superimposing single-molecule experimental images, the accuracy rate of quantifying the number of fluorescent molecules on target molecules labeled with 1, 2, 3, or 4 fluorescent molecules by this system is above 83%; the accuracy rate of quantifying the number of fluorescent molecules on target molecules labeled with 1, 2, 3, 4, or 5 fluorescent molecules is above 70%; the accuracy rate of quantifying target molecules labeled with 1 or 2 fluorescent molecules is 98% or above.

[0046] In terms of localization performance, when the distribution distance of fluorescent molecules is less than 240 nm, for target molecules labeled with 2 fluorescent molecules, the average root mean square localization error is lower than 20 nm; for identifying the positions of target molecules labeled with 3 fluorescent molecules, the average root mean square localization error is lower than 30 nm. For identifying the positions of target molecules labeled with 4 fluorescent molecules, the average root mean square localization error is lower than 34 nm. For identifying the positions of target molecules labeled with 5 fluorescent molecules, the average root mean square localization error is lower than 37 nm, as Figure 4 shown.

[0047] Furthermore, an optimized solution of the fourth solution: a fluorescence microscopy imaging method, where the distance between target molecules is greater than 5 nm.

[0048] (It can be calculated according to the current experimental emission wavelength using the diffraction limit formula, and its value is close to 1 / 2 of the emission wavelength).

[0049] Since the sizes of biological macromolecules in cells are usually equal to or greater than 5 nm (for example, the size of the beta2-adrenergic receptor is about 5 nm, and the distance between two molecules is about 4-5 nm when forming a dimer), when preparing the training and validation data, we start with an intermolecular distance of 5 nm. As for larger distances, there are two possibilities: First, the target molecule is a larger-sized molecule, and the intermolecular distance in the aggregate it forms is greater than 5 nm. Second, the intermolecular distance between different aggregates of the same molecule is greater than 5 nm.

[0050] The present invention also provides a fifth technical solution: a fluorescence microscopy imaging system formed by using the above fluorescence microscopy imaging method.

[0051] The fluorescence microscopy imaging system of the present invention combines non-super-resolution fluorescence imaging and an artificial intelligence system based on a fluorescence molecule database, breaking through the diffraction limit of traditional fluorescence microscopy technology, and achieving accurate quantification of the number of molecules and precise positioning of their positions. The system can first obtain a single-frame low-resolution image, and then parse the corresponding high-precision molecular distribution information based on this image, overcoming the limitation that other AI methods cannot obtain the accurate number and precise position of corresponding molecules for images formed by multiple molecules emitting light simultaneously within the diffraction limit. At the same time, using this method not only greatly reduces the equipment cost but also improves the efficiency of the experiment. Through the independent design of the classification network and the positioning network, and by designing independent positioning networks for different numbers of molecules, interference during the processing of multi-target tasks by a unified network is avoided, thus significantly improving the positioning accuracy and quantitative accuracy. Especially for target molecules labeled with 1 to 4 fluorescent molecules, the quantitative accuracy rate and precision rate both exceed 83%, and the positioning error is less than 34 nm, demonstrating the high-precision performance of this method in quantifying and positioning fluorescent molecules within the diffraction limit range.

[0052] The present invention also provides a sixth technical solution: a sample detection method using the above fluorescence microscopy imaging system.

[0053] Due to the wide applicability of fluorescent molecule dyes, the fluorescence microscopy imaging system in the present invention can adapt to different types of samples. The fluorescence microscopy imaging system in the present invention is applicable to different samples. Combining with the method of removing the background signal of the image, this system has excellent adaptability and generalization ability. By using a fluorescence molecule database specific to a molecule for training, the artificial intelligence system can achieve accurate molecular quantification or / and positioning in different samples, experimental conditions, and imaging tasks, adapt to various biomedical research fields, and show broad application potential. By labeling fluorescent molecules onto the target molecules to be detected, quantitative analysis and positioning of the target molecules can be achieved.

[0054] Furthermore, an optimized solution of the sixth solution: a sample detection method, including a sample molecule quantitative detection method and / or a sample molecule positioning detection method.

[0055] The confusion matrix is an intuitive tool for evaluating the performance of classification models. By showing the correspondence between actual classes and predicted classes, it comprehensively reflects the classification results of the model, applies to binary classification and multi-classification scenarios, and is an important tool for analyzing the behavior of classifiers and improving models. The matrix consists of four core parts: True Positive (TP), representing the number of samples that are actually positive and are correctly predicted as positive; False Positive (FP), representing the number of samples that are actually negative but are incorrectly predicted as positive; True Negative (TN), representing the number of samples that are actually negative and are correctly predicted as negative; False Negative (FN), representing the number of samples that are actually positive but are incorrectly predicted as negative. The larger the diagonal elements (TP and TN), the better the classification performance of the model; the smaller the off-diagonal elements (FP and FN), the lower the misclassification rate of the model. Its confusion matrix is as Figure 5 shown, indicating that the overall performance of the classification model is excellent, with high accuracy and low misclassification rate; its accuracy, recall, and precision metrics are as Figure 6 shown.

[0056] In the test of the fluorescent dye sample using this fluorescence microscopy imaging system, the quantitative accuracy and precision of the target molecules labeled with 1 or 2 fluorescent molecules are above 89%.

[0057] The ROC curve (Receiver Operating Characteristic Curve) is an important tool for evaluating the performance of classification models. By showing the classification ability of the model at different decision thresholds, it intuitively demonstrates the trade-off between the sensitivity (True Positive Rate, TPR) and specificity (1 - False Positive Rate, FPR) of the model. The horizontal axis of the ROC curve is the false positive rate (FPR), representing the proportion of negative samples that are incorrectly predicted as positive; the vertical axis is the true positive rate (TPR), representing the proportion of positive samples that are correctly predicted. When the curve is closer to the upper left corner, it indicates that the model has achieved a better balance between high recall and low false alarm rate. The area under the curve (AUC, Area Under the Curve) quantitatively reflects the performance of the model. The closer the AUC value is to 1, the stronger the ability of the model to distinguish between positive and negative samples; when AUC is 0.5 (i.e., the dashed line in the figure), it means that the prediction of the model is equivalent to random guessing. As Figure 7As shown, in the test of the experimental sample data of 1 and 2 molecules, the AUC value of this quantitative model is 0.9555, close to the ideal value of 1, indicating that the model has excellent classification performance. This means that when weighing sensitivity and specificity, the model has a very high ability to distinguish between positive and negative samples and can accurately predict the target category in most cases. The AUC value reaching 0.9555 further proves the robustness and reliability of the model when dealing with the current dataset.

[0058] Furthermore, the optimization scheme of the sixth scheme: a sample detection method, including the acquisition of a sample molecular spot image, whose fluorescence excitation condition is molecular synchronous excitation, and the molecular synchronous excitation condition includes adding a 405 nm laser while irradiating with the excitation laser.

[0059] The present invention is based on the molecular synchronous excitation imaging method and aims to solve the problem of molecular quantitative accuracy caused by insufficient fluorescence molecule excitation efficiency. Through experimental verification, the inventors found that under most laser excitation experimental conditions, even when using a standard sample with a known number of fluorescent molecules, it is difficult to achieve synchronous luminescence of all molecules in the field of view within a short time (e.g., 20 ms). The main reason for this phenomenon is that the radiation process of fluorescent molecules is essentially a quantum random process. When a molecule absorbs energy, it will transition from the ground state to the excited state and then return to the ground state by emitting photons. However, this process is inherently random. After a fluorescent molecule is excited from the ground state to the excited state, there are multiple possibilities for it to return to the ground state: it can emit fluorescent photons through radiative transition, or enter the triplet state through intersystem crossing, or undergo a non-radiative relaxation process. The randomness of these processes stems from the probability of quantum transitions, the fluctuations in the molecular microenvironment, and the complexity of the energy level structure. In a conventional millisecond-level exposure time, a single fluorescent molecule can theoretically complete thousands of excitation-emission cycles, but due to the influence of quantum randomness, the actually detected fluorescence signals often show significant non-uniformity. Especially at the single-molecule level, even under the same excitation conditions, the luminescence behaviors of different fluorescent molecules will show great differences.

[0060] The randomness of the above process directly leads to the fact that in the same field of view, some labeled target molecules may not be detected because their fluorophores are in the "dark state", thus seriously affecting the quantitative accuracy of fluorescence imaging. Therefore, in order to improve the molecular quantitative accuracy of fluorescence microscopy, the present invention proposes a method for realizing synchronous excitation of fluorescent molecules by optimizing the excitation conditions, ensuring that as many molecules as possible emit light and are recorded in a single-frame microscopic image, thereby improving the accuracy of molecular detection.

[0061] In view of the above technical difficulties, the inventors of the present invention optimized the laser power for the adopted fluorescent molecules on the premise of using their known best excitation light band, and combined it with a 405 nm laser with an appropriate power intensity. The two lasers were used to simultaneously excite the target fluorescent molecules, so that all molecular luminescence could be recorded in the first and second short exposure times (exposure time is 20 ms) of synchronous imaging during laser irradiation. The laser excitation conditions were selected from the known best excitation band of the target fluorescent molecule to select an appropriate excitation wavelength. For example, a 641 nm laser was used as the basic excitation condition for the Cy5 fluorescent molecule, supplemented with a 405 nm excitation light, and the two lasers were simultaneously excited. Through a series of co-laser experiments with different combinations of 641 nm power and different 405 nm powers, the laser conditions with the highest molecular synchronous excitation efficiency were optimized: 641 nm of about 11 W / cm 2 and 1 to 2 W / cm 2 of 405 nm were used for co-excitation.

[0062] This method is not only applicable to general molecular quantification or / and localization tasks, but can also be extended to the quantification or / and localization research of plasma membrane proteins. The nano-distribution and quantification research of plasma membrane proteins in living cells is of great significance for disease research, drug development, and the development of biomedical engineering. The plasma membrane is the interface for the cell to exchange substances, energy, and signals with the external environment; many protein molecules on the plasma membrane are responsible for mediating material transport, intracellular signal transduction, and further regulating gene expression, thus playing a key role in important cellular events such as the occurrence of diseases (such as cancer), virus invasion, immune response, and drug response. However, since plasma membrane proteins are usually transmembrane proteins with strong hydrophobicity, it is difficult to separate and study them in vitro, and in situ research on living cells is highly needed; the abundance of these transmembrane proteins on the plasma membrane of living cells is extremely low, and they are often discretely distributed on the plasma membrane in the form of single molecules (monomers), two molecules, or several molecules aggregated (dimers or oligomers); moreover, different tissue forms (monomers, dimers, oligomers) of the same molecule on the plasma membrane of living cells result in different molecular functional activities. Due to the above unique biophysical properties, the research of plasma membrane proteins highly requires an imaging technology suitable for living cells, high sensitivity, precise quantification, and nanoscale resolution.

[0063] However, existing imaging methods (including single molecule localization microscopy, SMLM) are difficult to obtain quantitative nanoscale microscopic images of all target molecules sparsely distributed on living cells. The imaging system proposed in the present invention can not only perform quantitative nanoscale microscopy imaging, but also has a high time resolution of 20 ms, and can accurately obtain the nano-distribution and quantification information of plasma membrane proteins in living cells. Therefore, using this method to accurately quantify or / and localize plasma membrane proteins is expected to make breakthroughs in the fields of virus prevention and control, drug development, drug addiction control, and disease drug target mechanism research, and promote the progress and efficiency of biomedical research. Brief Description of the Drawings

[0064] Figure 1 It is a schematic diagram of the strategy for obtaining the number of molecules or / and localization in a single-frame image by combining fluorescence imaging and artificial intelligence; Figure 2 It is a schematic diagram of the quantitative model and localization model of the present invention; Figure 3 It is the test confusion matrix of the simulated dataset; Figure 4 It is the probability density distribution of the root mean square error of the simulated dataset; Figure 5 It is the test confusion matrix of the experimental sample data; Figure 6 It is the test result parameters of the experimental sample data; Figure 7 It is the test ROC curve of the experimental sample data. Detailed Description of the Invention

[0065] Example 1: Establishment of a Fluorescent Molecule Database In order to establish a fluorescent molecule database, the single-fluorescent molecule spot image is the basic experimental data. Therefore, first, the preparation of a single-molecule purity sample is required. In this example, a single Cy5-labeled target molecule (a single-stranded DNA with a length of about 19 nm in this example) purified by HPLC twice (molecular weight error less than 0.05%) is used, and the target molecule is diluted to 5 pM to make the target molecule in an extremely sparse state to ensure that the image obtained by imaging is a spot of single-molecule luminescence.

[0066] I. Preparation of a Single-Molecule Purity Sample 1.1 Sample Preparation 1.1.1 The fluorescent molecule selected in this example is Cyanine5 (abbreviated as Cy5) commonly used in molecular imaging methods. Of course, other fluorescent dyes can also be used. The target molecule labeled with a single Cy5 is diluted with PBS buffer; 1.2 Sealing the Slide 1.2.1 The existing slowfade imaging buffer and the diluted target molecule are thoroughly mixed in a ratio of 3:1; Take 0.1 - 3.0 uL (determined according to the size of the cover glass. For example, if the diameter of the cover glass is 12 mm, only about 0.1 ul needs to be added) of the liquid generated in step 1.2.1 and drop it on the glass slide, then cover the liquid drop with a cover glass and make the cover glass completely fit the glass slide.

[0067] 1.2.3 Seal the slide with nail polish.

[0068] II. Imaging of the Single-Molecule Purity Sample 2.1 Perform molecular synchronous fluorescence radiation imaging for each target field of view 2.1.1 Mount the slide obtained in 1.2 on the microscope stage and focus it; 2.1.2 Use an EMCCD camera for imaging, set the camera parameters, the exposure time is 20 ms, the number of frames captured by the camera is 3 - 1000, and the gain is 20 times; 2.1.3 Set the imaging field of view range in the pre - calibrated laser irradiation intensity uniform area, and set the size of the imaging field of view to 150×150 square pixels (the laser intensity in this field of view is nearly uniform); 2.1.4 Turn on the imaging mode of the camera.

[0069] 2.1.5 Open the automatic light shutter to make the already stably - turned - on laser irradiate the sample. The laser used is 11 mW / cm 2 641 nm laser, simultaneously paired with less than 1W / cm 2 405 nm laser, and use the two lasers to simultaneously excite the target fluorescent molecules; 2.1.6 Block the laser irradiation through the electric light switch and move the sample to a new field of view.

[0070] 2.2 Perform imaging for each field of view of the target sample 2.2.1 Repeat the process of 2.1.3 - 2.1.5 to obtain the image sequence of other fields of view of the target sample; 2.2.2 Obtain 2000 spot image sequences (raw data, i.e., raw field - of - view images) in the above - mentioned manner.

[0071] III. Data processing of single - molecule samples 3.1 Spot localization 3.1.1 For each frame of the 2000 spot image sequences obtained in 2.1, perform convolution using a band - pass filter (b - pass) to obtain the denoised full - field image; 3.1.2 Perform binarization on the image obtained in 3.1.1 with a threshold of 20 to obtain the rough localization of the molecules; 3.1.3 Based on the rough localization of each molecule obtained in 3.1.2, use the Gaussian fitting algorithm to accurately locate each molecule and calculate the centroid coordinates of each molecular spot.

[0072] 3.2 Calculate spot parameters 3.2.1 With the centroid coordinates of the spot as the center, intercept a 7×7 pixel area as the image of the spot; 3.2.2 The spot images obtained in 3.2.1 contain the images of individual fluorescent molecule spots, i.e., single-molecule spot images. However, they may also contain images that are not desired to be collected, including: images of two or more fluorescent molecules aggregated in a 7×7 pixel area, fluorescent molecules located at the edge of the field-of-view image, mispositioned noise signals, etc. Therefore, further screening needs to be carried out in the subsequent 4.1.

[0073] 3.2.3 Calculate the fitting error, spot intensity, ellipticity, and size of the spot Gaussian fitting: Obtain the fitting error by finding the error between the actual spot image and the ideal two-dimensional Gaussian function. Obtain the spot intensity by finding the sum of the pixels of the spot image in 3.2.1. The calculation of ellipticity and size is as follows, where σ x and σ y are the standard deviations of the fitted Gaussian function in the x or y dimension, respectively.

[0074] 3.3 Process all the field-of-view molecules frame by frame 3.3.1 Repeat the steps in 3.1 - 3.2 to process each frame of the sequence of all the spot images obtained in 2.2.2.

[0075] 3.3.2 In the above manner, approximately 160,000 single-molecule spot images, their centroid coordinates, and the parameters in 3.2.2 are obtained.

[0076] IV. Obtain the fluorescent molecule database 4.1 Screen all the spot images obtained in 3.2.1 4.1.1 To ensure that the subsequent processed spot images are single-molecule spot images, according to the statistical analysis of the parameters obtained in 3.2.3, determine the screening conditions and parameter value ranges in 4.1.2 to screen all the spot images; 4.1.2 In this embodiment, when a cylindrical lens for assisting three-dimensional imaging is added to the imaging optical path, the screening conditions include: fitting quality screening, screening out spots with a fitting error less than 0.01; ellipticity screening, with an ellipticity greater than 0.5 and less than 2; spot size screening, with a spot size greater than 2 and less than 4.3; spot position screening, spots whose centroids are not within the 7-pixel range at the edge of the image; spot distance screening, no two spots can exist simultaneously within a range of 1200 nm. (If there is no cylindrical lens in the imaging optical path, the conditions for ellipticity screening and spot size screening change: ellipticity is greater than 0.65 and less than 1.5, and spot size is greater than 1 and less than 6) 4.1.3 Eliminate the spot images whose corresponding parameters exceed the screening range in 4.1.2. The spot images obtained after screening are single-molecule spot images.

[0077] 4.2 Obtaining the single-molecule spot image after removing the background of the original field-of-view image and noise reduction 4.2.1 For the single-molecule spot image obtained in 3.3.3, expand it by 1 pixel in each of the four directions around the full-field image, adding a total of 32 pixels, and expand the single-molecule spot image from a 7×7 pixel image to a 9×9 pixel image. Sum the pixel values of the 32 expanded pixels and take the average as the background value of the original field-of-view image, which is used to remove the background of the original field-of-view image in the following steps. Subtract the background value from the single-molecule spot image to obtain the single-molecule spot image after removing the background of the original field-of-view image.

[0078] 4.2.2 The noise-reduced single-molecule spot image can be obtained by intercepting the noise-reduced original field-of-view image processed in 3.1.1 in the manner of 3.2.1.

[0079] 4.3 Integrating, packaging, and storing the single-molecule spot data 4.3.1 Through the screening in 4.1, approximately 150,000 single-molecule spot images are obtained (including the single-molecule spot images without further processing, the single-molecule spot images after removing the background of the original field-of-view image, and the noise-reduced single-molecule spot images). 4.3.2 Package and summarize the storage of each single-molecule spot image among the 150,000 (including the single-molecule spot images without further processing, the single-molecule spot images after removing the background of the original field-of-view image, and the noise-reduced single-molecule spot images), their corresponding centroid coordinates, and all parameter data in 3.2.2, that is, a fluorescence molecule database including single-molecule spot image data information is obtained.

[0080] 4.4 Data reading 4.4.1 Read the single-molecule spot image after removing the background of the original field-of-view image from the fluorescence molecule database and extract its corresponding luminescent molecule position information (centroid coordinates). In this embodiment, the pixel size of the original field-of-view image is 160 nm, and the spot image is intercepted according to 7×7 pixels.

[0081] 4.4.2 The 150,000 groups of fluorescence molecule images after removing the background of the original field-of-view image and their centroid coordinates read in 4.4.1 will be used to simulate the generation of multi-molecule spot images (formed by superimposing two or more single-molecule spot images, that is, synthetic spot images). Background removal processing is to prevent the doubling of the background when the spots are superimposed in the following steps.

[0082] 4.5 Generation of synthetic spot images 4.5.1 To establish a fluorescence molecule database that simulates the light spots generated by synchronous fluorescence radiation of target molecules labeled with 2, 3, 4, and 5 fluorescent molecules, it is formed by superimposing the single-molecule light spot images obtained by background removal processing in the data read using 4.4, simulating the light spots of target molecules labeled with 2, 3, 4, and 5 fluorescent molecules.

[0083] 4.5.2 Create a blank image of all zeros with a size of 2400×2400 square pixels with a pixel size of 1 nm; 4.5.3 Draw a circular area with a diameter of 240 nanometers at the center of the image obtained in 4.5.2 (by default, the distance between fluorescent molecules on the target molecule is less than 240 nm because most target molecules do not exceed this size); 4.5.4 Inside the circular area, randomly scatter points to generate 2 million coordinates (ensuring that the distance between any two coordinates does not exceed 240 nanometers); 4.5.5 Further, according to the type of simulated target molecule (2 or 3 or 4 or 5 fluorescent molecules), randomly extract the corresponding number of single-molecule light spot images and their centroid coordinates from the data read in 4.5, and randomly assign the corresponding number of coordinates among the 2 million coordinates generated in 4.5.4; 4.5.6 Sum and superimpose the single-molecule light spot images in such a way that the centroid of the single-molecule light spot is anchored to the specified coordinates; 4.5.7 To avoid the overinfluence of noise signal characteristics on the training of the model, in the next step, a band-pass filter will be applied to the light spot image for pixel-by-pixel convolution for noise reduction processing.

[0084] 4.5.8 According to the histogram of the background signal intensity distribution of the original image obtained statistically, expand the edge area of the light spot image by 7 pixels outward to reduce the influence of possible edge effects during the convolution process.

[0085] 4.5.9 When the types of simulated target molecules are 2, 3, 4, and 5 fluorescent molecules respectively, perform the processing of 4.5.1 - 4.5.8 to obtain the light spot images of synchronous fluorescence radiation imaging of target molecules labeled with 2 / 3 / 4 / 5 fluorescent molecules, that is, the synthesized light spot images.

[0086] 4.5.10 For the single-molecule light spot images, also adopt the same processing method as 4.5.1 - 4.5.8 above to obtain a target molecule light spot database including data information labeled with 1 fluorescent molecule.

[0087] 4.5.11 Cycle through steps 4.5.1 - 4.5.10 for 150,000 times to obtain the synchronous fluorescence radiation imaging spots of target molecules marked with 1 / 2 / 3 / 4 / 5 fluorescent molecules, each with 150,000, and their corresponding molecular position and number information. The corresponding position (centroid coordinates) of each molecule is represented as (x, y), and the corresponding number of molecules is represented as 1, 2, 3, 4, or 5, that is, a fluorescence molecule database including synthetic spot image data information is obtained.

[0088] Example 2: Establishment, experimental verification and correction of deep learning model I. Reading and data processing of fluorescence molecule database 1.1 Dataset 1.1.1 Data obtained through the steps of 4.5 in Example 1: The spot image, its corresponding number of molecules, and the centroid coordinates of each molecule are used as the dataset for training the deep learning model. Among them, the number of molecules is used as the label for training the deep learning classification model, and the centroid coordinates of each molecule are used as the label for training the deep learning regression model; 1.1.2 Randomly divide the dataset into a training set (80%), a validation set (10%), and a test set (10%).

[0089] 1.1.3 Convert the image to a Pytorch tensor and convert the data type to unsigned 16-bit integer.

[0090] II. Model construction and training configuration 2.1 Construction of classification model (quantitative model) 2.1.1 In this example, a convolutional neural network based on the ResNet-50 structure is used; 2.1.2 The number of input image channels is 1 (grayscale image). After passing through a convolutional layer conv1 with a kernel size of 3×3, the number of output channels is 64, and batch normalization (Batch Normalization) and ReLU activation function are connected after the output. This step maps the input features to a 64-dimensional feature space.

[0091] 2.1.3 Backbone structure (ResNet layer): The entire ResNet backbone consists of four stages, named conv2_x, conv3_x, conv4_x, and conv5_x here, corresponding to the classic ResNet structure, and the channel expansion multiple of the residual block is set to 4; 2.1.4 conv2_x: Keep the feature map size unchanged with a stride of 1 (stride = 1), continue to refine the number of channels from 64, and the output is also 64.

[0092] 2.1.5 conv3_x: Downsampling is achieved by setting stride = 2 at this stage, halving the size of the feature map and expanding the number of channels to 128 channels.

[0093] 2.1.6 conv4_x: Downsample again (stride = 2), and expand the number of channels to 256 channels.

[0094] 2.1.7 conv5_x: Downsample again (stride = 2), and expand the number of channels to 512 channels. This stage is the advanced stage of feature extraction.

[0095] 2.1.8 Global Average Pooling Layer: After completing all residual layers, the feature map is reduced to a spatial size of 1×1 through nn.AdaptiveAvgPool2d((1,1)). This compresses the spatial information into a global feature vector.

[0096] 2.1.9 Fully Connected Layer: The features after global average pooling (a vector of size 512 * 4) are input into a fully connected layer, mapping the features to the number of classes for the classification task. For this embodiment, the number of classes for the classification task is 5.

[0097] 2.1.10 Output Layer and Activation: Finally, the output of the fully connected layer is normalized through the softmax function to obtain the probabilities for each class.

[0098] 2.2 Import Libraries and Set Devices 2.2.1 Import the main libraries, including: torch, torch.nn, torchvision, numpy, and os, etc.; 2.2.2 Set the training device: Check whether the specified GPU (CUDA) is available, otherwise use the CPU; 2.3 Build the Dataset and DataLoader for the Training Set and Validation Set 2.3.1 Use torchvision.datasets.ImageFolder to load the image folder; 2.3.2 Use DataLoader to perform batch splitting on the Dataset and preload in parallel (num_workers); 2.3.3 The training set needs to be shuffled randomly with shuffle = True, and the validation set does not need to be shuffled.

[0099] 2.4 Training Configuration 2.4.1 Hyperparameter Setting: Set the batch size to 512, the initial learning rate (lr) to 1e-6, the weight decay to 1e-5, the maximum number of training epochs to 400, etc.; 2.4.2 Use the Adam optimizer and set weight_decay = 1e-5 to prevent overfitting.

[0100] 2.4.3 Use the learning rate scheduler CosineAnnealingLR based on cosine annealing to gradually decrease the learning rate in the form of a cosine function.

[0101] 2.5 Loss Function 2.5.1 For the classification of the number of fluorescent molecules, use cross-entropy (nn.CrossEntropyLoss) as the loss function. The loss function is as follows, where is the true value, is the network prediction value.

[0102] 2.5.2 For the regression of centroid coordinates, use mean squared error as the loss function. A great advantage of the MSE loss function is that it makes the mean of the final localization error smaller, as shown below: III. Model Training and Iteration Process 3.1 Classification Model Training 3.1.1 Enter the training of the i-th epoch (i <= 400); 3.1.2 Set the model to training mode; 3.1.3 Obtain batch data: Take out the images and target class labels from the data loader of the training set, which include the images of 1, 2, 3, 4, 5 molecular spots and the corresponding labels (1, 2, 3, 4, 5); 3.1.4 Move the data and labels to the specified GPU in 2.3.2 for calculation; 3.1.5 Input the images into the classification model in 2.1, and through forward propagation, obtain the output, which is a probability distribution representing the prediction probability of each class; 3.1.6 Use the loss function in 2.6.1 to calculate the loss between the prediction result and the true label, and record the loss; 3.1.7 Calculate the category corresponding to the maximum value in the probability distribution obtained in 3.1.4 as the predicted value of the model, compare it with the true label, count the number of correctly predicted samples, and divide by the total number of samples to obtain the accuracy; 3.1.8 Backpropagation: Calculate the gradient of the loss with respect to the model parameters through loss.backward(); 3.1.9 Save the model parameters trained in this round, and update the model parameters through the optimizer in 2.5.2 according to the gradient calculated in 3.2.8; 3.1.10 Take out the images from the data loader of the validation set, and use the model parameters trained in this round to predict the labels of the images; 3.1.11 Calculate the loss between the prediction result and the true label using the loss function in 2.6.1. If the loss is smaller than the loss of the model's previous performance on the validation set, record the model parameters trained in this round as the optimal parameters; 3.1.12 Enter the next round and repeat the process from 3.1.1 to 3.1.11 until the maximum number of training rounds set in 2.5.1 is reached 3.2 Regression model training 3.2.1 Train the corresponding regression model for different types (1 / 2 / 3 / 4 / 5 molecules) of light spots; 3.2.2 Enter the training of the i-th round (epoch) (i <= 400); 3.2.3 Set the model to training mode; 3.2.4 Obtain batch data: Take out the images and target coordinates from the data loader. For different types of regression models, the images are the images of 1 / 2 / 3 / 4 / 5 molecule light spots respectively, and the target coordinates are the coordinates corresponding to each molecule of the corresponding type; 3.2.5 Move the data and labels to the GPU specified in 2.3.2 for calculation; 3.2.6 Input the images into the regression model in 2.2, and obtain the output through forward propagation. The output is the coordinates of the molecules corresponding to each molecule; 3.2.7 Calculate the loss between the prediction result and the true coordinates using the loss function in 2.6.2, and record the loss; 3.2.8 Backpropagation: Calculate the gradient of the loss with respect to the model parameters through loss.backward(); 3.2.9 Save the model parameters trained in this round, and update the model parameters through the optimizer in 2.5.2 according to the gradient calculated in 3.2.8.

[0103] 3.2.10 Take out the images from the data loader of the validation set, and use the model parameters trained in this round to predict the labels of the images; 3.2.11 Use the loss function in 2.6.2 to calculate the loss between the prediction result and the true label. If the loss is smaller than the loss of the model's previous performance on the validation set, record the model parameters trained in this round as the optimal parameters; 3.2.12 Enter the next round, and repeat the process of 3.2.1 - 3.2.11 until the number of training rounds reaches the maximum number of training rounds set in 2.5.1.

[0104] IV. Model Evaluation and Testing and Correction with Experimental Data 4.1 Model Evaluation 4.1.1 Use the pre - divided test set in 1.1.2 to evaluate the model, and the test set has not been used for model training; 4.1.2 Take out the images from the test set, and use the optimal model parameters obtained from training to predict the labels or coordinates of the images; 4.1.3 For the classification task, use the optimal model parameters obtained from training in 3.1 to predict the labels of the images and obtain the number of molecules corresponding to the input pictures; 4.1.4 For the regression task, for different types (1 / 2 / 3 / 4 / 5 molecules) of light spots, use the optimal parameters of the corresponding type of model obtained from training in 3.2 to predict the coordinates of the images and obtain the centroid coordinate information of the molecules corresponding to the input pictures; 4.1.5 Further, compare the prediction information with the corresponding true information: For the classification model (predicting the number of molecules), calculate the model accuracy, recall, and precision metrics according to the formula. Taking binary classification as an example, the calculation formulas are as follows, where TP is true positive, FP is false positive, FN is false negative, and TN is true negative; The confusion matrix for the quantitative test of 1 - 5 molecule light spots is as Figure 3 shown. It can be seen from the confusion matrix that the model has high classification performance on multiple categories. Among them, the correct classification rates of categories 1 and 5 reach 90% and 75% respectively, showing excellent prediction ability. The correct classification rates of categories 2 and 3 are 73%, 60% and 53% respectively, and the overall performance is good.

[0105] 4.1.6 For the regression model (predictive centroid localization), the deviation degree between the predicted position and the true centroid coordinates is calculated according to the root mean square (RMS) formula.

[0106] 4.2 Experimental data testing 4.2.1 Since the multi-molecule light spots used in the training of the network models obtained in 3.1 and 3.2 are not directly from experiments but are formed by superimposing single-molecule light spots, these models need to be further verified and corrected with experimental data; 4.2.2 Synthesize DNA with a length within the diffraction limit (240 nm) by PCR method, and label 1, 2, 3, 4, or 5 fluorescent molecules on each DNA molecule; 4.2.3 Then, prepare a sample with a single DNA distribution state on the cover glass by the anti-aggregation dilution method described in 1.2 of Example 1; 4.2.4 Further obtain the full-field images of different types of samples (labeled with 1, 2, 3, 4, or 5 fluorescent molecules on each DNA molecule) by the molecular synchronous excitation imaging method described in 2.1 of Example 1.

[0107] 4.2.5 The full-field images are processed by the light spot localization method described in 3.1 of Example 1 and the method of 3.2.1 to obtain the corresponding light spot images of various samples. The original light spot images are processed by the method described in 4.2.1 of Example 1 to obtain the light spot images with the background of the original image removed; 4.2.6 The light spot images with the background of the original image removed are further subjected to the noise reduction processing described in 4.5.7 - 4.5.8 to obtain the light spot images after noise reduction and with the background of the original image removed; 4.2.7 Input these light spot images into the trained classification model and regression model in 3.1 and 3.2. The classification model predicts the number of molecules corresponding to each light spot image, and the regression model predicts the molecular position information corresponding to each light spot image.

[0108] 4.2.8 The number of molecules corresponding to these light spot pictures is known (i.e., the number of fluorescent molecules contained in each DNA molecule), and the centroid distance does not exceed the distance between two or more fluorescent molecules on the DNA during the design of the PCR reaction.

[0109] 4.2.9 Compare the predicted information obtained in 4.2.7 with the above-known data; 4.2.10 Calculate the accuracy, recall, and precision metrics using the formulas in 4.1.5, and generate the corresponding confusion matrix to show the classification performance of the model on different classes. The confusion matrix provides an intuitive understanding of the model's classification performance through visualization; 4.2.11 Optimize the model according to the experimental test results, including adjusting the hyperparameters in 2.5.1 and changing the network structures in 2.1 and 2.2.

[0110] 4.2.12 Improve the performance of the classification model in the fluorescence molecule image classification task and the performance of the regression model in the centroid localization task by continuously iterating this process.

[0111] Finally, as Figure 1 、 Figure 3 shown, according to the establishment strategy of the quantitative model and the localization model of the artificial intelligence system of the present invention, the artificial intelligence system of the embodiment of the present invention is obtained. The artificial intelligence system of this embodiment includes two parts: a quantitative model and a localization model. The quantitative model focuses on determining the number of molecules, and the localization model focuses on the molecular localization task. For different numbers of molecules, the corresponding independent localization networks are used to complete the localization respectively.

[0112] Deep-STORM is a deep learning model for solving overlapping spots. Deep-STORM is an end-to-end image network that can output super-resolution reconstructed images by inputting low-resolution wide-field images, thereby improving the resolution of single-molecule localization. We compared the performance of Deep-STORM and the model of the present invention in sub-diffraction-limited spot localization. We trained the Deep-STORM model based on the same training dataset and applied it to the test dataset. Different from the model of the present invention that directly predicts the positions of molecular spots, Deep-STORM represents the high-resolution spot features through pixels of different gray levels. For single-molecule spots, the prediction results of both methods are close to the true positions. However, for spots containing two molecules, Deep-STORM usually can only identify one gray peak, while the model of the present invention can accurately predict the coordinates of the two molecules.

[0113] To quantitatively compare these two methods, we uniformly processed the prediction results. Specifically, by plotting the spike images based on the prediction coordinates of the present invention and using the same Gaussian kernel as that for generating the heat map by Deep-STORM, a heat map consistent with Deep-STORM was generated. The purpose of such an approach was to maintain the same form as the output of Deep-STORM. Then, we calculated the structural similarity index (SSIM) between the heat map generated by each method and the true heat map. The closer the SSIM value is to 1, the better the reconstruction quality of the image. Among 10,000 test images, for the test results of the present invention, about 45% of the SSIM values were concentrated in the range of 0.996 - 1, and about 55% of the SSIM values were distributed in the range of 0.985 - 0.996. For the test results of Deep-STORM, about 60% of the SSIM values were distributed in the range of 0.985 - 0.997, and about 40% of the SSIM values were distributed in the range of 0.975 - 0.985. The present invention showed higher stability and consistency, while Deep-STORM showed greater fluctuations and its SSIM values had a wider distribution. The results indicated that the SSIM results of the model of the present invention were significantly better than those of Deep-STORM, indicating that its prediction results were closer to the true distribution. This also further demonstrated that the present invention had significant advantages in using a discrete network to complete the spot localization within the diffraction limit.

[0114] Example 3: Method for Target Molecule Counting and Localization In this embodiment, 1 - 5 fluorescent molecules are labeled on the same target molecule to be detected, or each molecule in a molecular complex formed by the binding of 1 - 5 target molecules is labeled with a fluorescent molecule (such as Cy5). For a single molecule or molecular complex to be detected, the specific number and position of the fluorescent molecules therein are unknown (the distance between adjacent molecules is less than the diffraction limit, usually not exceeding 20 nm). The fluorescence microscope of the present invention is used for molecular counting and localization detection.

[0115] I. Preparation of Target Molecule Samples Labeled with Single Fluorescent Molecules and Molecular Imaging 1.1 For target molecules or molecular aggregates that need to be quantitatively analyzed and localized, a pure sample including a single fluorescently labeled molecule should be prepared using steps 1.1 - 1.4 in Example 1. 1.2 Perform molecular synchronous fluorescence radiation imaging on the sample obtained in 1.1, and use steps 2.1 - 2.2 in Example 1 to obtain 2000 original field-of-view sequences. II. Image Data Processing and Fluorescent Molecule Database of Single-Molecule Spots of Target Fluorescent Molecules 2.1 Process the obtained original field-of-view sequence according to the steps 3.1 - 3.3 in Example 1 to obtain approximately 160,000 spot images, their centroid coordinates, and a series of parameters (fitting error of spot Gaussian fitting, spot intensity, ellipticity, size); 2.2 To obtain the single-molecule spot images of the target fluorescent molecules, screen all the spot images obtained in 2.1 in the manner of 4.1.1 - 4.1.3 in Example 1. The spot images obtained after screening are the single-molecule spot images of the target fluorescent molecules; 2.3 For the above single-molecule spot images, obtain the single-molecule spot images of the target fluorescent molecules with the background of the original field-of-view image removed and noise reduced in the manner of 4.2 in Example 1; 2.4 Package and summarize for storage the obtained approximately 150,000 single-molecule spot images (including single-molecule spot images without further processing, single-molecule spot images with the background of the original field-of-view image removed, and single-molecule spot images with noise reduced), their corresponding centroid coordinates, and all the parameter data in 2.1, that is, obtain the fluorescence molecule database of the single-molecule spot images of the target fluorescent molecules.

[0116] III. Simulate the multi-molecule spot images of the target fluorescent molecules and obtain the deep learning dataset 3.1 Obtain the spots of the synchronous fluorescence radiation imaging of the target molecules labeled with 1 / 2 / 3 / 4 / 5 fluorescent molecules and their corresponding molecular position and number information for 150,000 in the manner of 4.4 - 4.5 in Example 1. The corresponding position (centroid coordinates) of each molecule is expressed as (x, y), and the corresponding number of molecules is expressed as 1, 2, 3, 4, or 5, that is, obtain the fluorescence molecule database of the synthetic spot images of the target fluorescent molecules.

[0117] 3.2 Obtain the deep learning dataset of the target fluorescent molecules: training set (80%), validation set (10%), and test set (10%) by following the steps of 1.1 in Example 2.

[0118] IV. Build and train the model and evaluate the model 4.1 In this example, use a convolutional neural network based on the ResNet-50 structure to complete the configuration of building and training the model in the manner of 2.1 - 2.6 in Example 2; 4.2 Input the training set and the validation set into the network, and follow the steps of 3.1 - 3.2 in Example 2 to train a five-classification model for the target aggregating 1, 2, 3, 4, or 5 fluorescent molecules, and a regression model for localizing the aggregation of 1, 2, 3, 4, or 5 fluorescent molecules respectively; 4.3 Evaluate the performance of the classification model and the regression model of the target fluorescent molecules by following the steps of 4.1 in Example 2.

[0119] V. Preparation of Fluorescent Molecule Samples to be Detected and Molecular Synchronous Fluorescence Radiation Imaging 5.1 Samples with a single molecule distribution state to be detected on the cover glass are prepared by following the steps 1.1 - 1.4 in Example 1. The target molecules to be detected in the samples are labeled with 1, 2, 3, 4, or 5 fluorescent molecules, or the target molecule aggregates to be detected contain 1, 2, 3, 4, or 5 target molecules (each molecule is labeled with one fluorescent molecule); 5.2 Further, 10 original field - of - view image sequences of the samples are obtained by using the molecular synchronous excitation imaging method described in 2.1 of Example 1.

[0120] 5.3 The full - field images are processed by the methods in 4.2.5 - 4.2.6 of Example 2 to obtain the spot images after noise reduction and removal of the background spots of the original images; VI. Determination of the Number and Position of Target Fluorescent Molecules 6.1 The spot images obtained in 5.3 are input into the trained classification model in 4.2, and the classification model predicts the number of aggregated target fluorescent molecules (1 / 2 / 3 / 4 / 5) corresponding to the spot images; 6.2 The spot images are input into the corresponding regression model according to their predicted numbers to predict the molecular position information corresponding to each spot image.

[0121] 6.3 Performing 6.1 - 6.2 on all the obtained spot images completes the quantification and localization of the molecules.

[0122] In this example, the spot images to be tested input into the model are those not seen by the model, and the number of molecules is unknown to the model. Finally, through the quantification model and the localization model, the quantified number and nano - distribution of the target fluorescent molecules on each molecule are obtained.

[0123] Example 4: In - situ Super - resolution Localization and Precise Quantification of Target Molecules on the Plasma Membrane The sample used is BXPC - 3 cells. Of course, it can also be Hela, Escell, Panc - 1, huvec, Hpne, PSN1, 293T cells. However, due to the differences in the autofluorescence of the plasma membrane during imaging of different cells, the imaging signal - to - noise ratio will vary. For the case where the autofluorescence of the cell is too strong resulting in a low signal - to - noise ratio, molecular quenching methods such as the reduction quenching method with sodium borohydride (NaBH4), Sudan Black B staining quenching method, glycine (Glycine) quenching method, etc. in the existing technology can be used to reduce the autofluorescence of the plasma membrane.

[0124] I. Background Fluorescence Removal Treatment 1.1 Prepare 1 mg / ml sodium borohydride (18.2) with 18.2 Ω deionized water, and treat each coverslip with sodium borohydride for 7 min; 1.2 Aspirate sodium borohydride and wash three times with deionized water; 1.3 Wash with PBS for 30 min; 1.4 Repeat the previous three steps twice; II. Sample Preparation 2.1 Gradient dilute the insulin target molecule coupled with cy5 molecules using PBS buffer, and add a blank control group and a group of Cy5-labeled secondary antibodies with the same concentration as a non-specific control group; 2.2 Use the diluted insulin-cy5 and Cy5-secondary antibody with a concentration of 1.69×10⁻ 9 M to bind at room temperature in the dark for 10 min; 2.3 Aspirate the dye, wash 3 times, and wash with PBS for 2 + 5 + 5 + 5 min; III. Sealing the Coverslip Take 1 μl of the existing slowfade imaging buffer and the diluted target molecule, mix well at a ratio of 3:1, drop it on the glass slide, cover it with a coverslip, and seal it with nail polish.

[0125] IV. Perform molecular synchronous excitation imaging on the cell samples with fluorescence labeled on the membrane structure 4.1 During imaging, first adjust the focus so that the focal plane is at the cytoplasmic membrane level marked with fluorescent molecules.

[0126] 4.2 Imaging and excitation are carried out simultaneously. Only 1 - 2 frames are taken for each field of view, and the exposure time for each frame is 8 - 20 ms. Excitation and imaging stop simultaneously to avoid over-irradiation and damage to the sample, and obtain wide-field fluorescence microscopic images (150×150) of each field of view.

[0127] V. Obtain molecular spot images and perform image processing 5.1 Process the obtained wide-field fluorescence microscopic images according to the method in 3.1 of Example 1 to obtain fluorescence molecular spot images (7×7) on the cytoplasmic membrane and record the coordinates of each spot in the corresponding image.

[0128] 5.2 Obtain fluorescence molecular spot images after removing the background of the original field-of-view image according to the method in 4.2.1 of Example 1.

[0129] 5.3 Perform noise reduction processing on the fluorescence molecular spot images after removing the background of the original field-of-view image according to the methods in 4.5.7 - 4.5.8 of Example 1 to obtain noise-reduced fluorescence molecular spot images after removing the background of the original field-of-view image.

[0130] VI. Quantification and / or localization of molecules 6.1 Input the fluorescence molecular spot images obtained in 5.3 into the quantitative model established for cy5 molecules in Example 2, and the model outputs the predicted number of molecules corresponding to each spot image.

[0131] 6.2 According to the spot images and their corresponding predicted number of molecules, input the spot images into the localization model established for cy5 molecules in Example 2, and the model outputs the positions of each molecule corresponding to each spot image.

[0132] VII. Image super-resolution reconstruction 7.1 Obtain the number of molecules and the positions of each molecule corresponding to each spot image through 6.1 - 6.2, and restore them to the wide-field fluorescence microscopy image according to the coordinates of the spots in the corresponding image.

[0133] 7.2 Reconstruct to obtain the number of target molecules and the super-resolution localization distribution map at different positions within the field of view of the plasma membrane imaging.

[0134] Thus, it can be seen that by using the database and artificial intelligence system of the present invention, in-situ super-resolution localization and accurate quantification of target molecules on the cytoplasmic membrane can be achieved.

[0135] The target fluorescent molecule mentioned in all the above embodiments is Cy5. When using the fluorescent molecule database, it can be the fluorescent molecule database established in Example 1, and the quantitative model and localization model established in Example 2 can be used. Of course, if Cy5 is replaced with other fluorescent molecules, a fluorescent molecule database can also be established for this fluorescent molecule according to the method in Example 1, and other methods can also be implemented with reference to the above embodiments.

[0136] The above are only the embodiments of the present invention, and common general knowledge such as specific technologies and characteristics known in the art are not described in detail here. It should be pointed out that for those skilled in the art, without departing from the technical solution of the present invention, several modifications and improvements can be made. These should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicality of the patent. The protection scope required by this application should be based on the content of its claims, and the specific implementation manners described in the specification can be used to interpret the content of the claims.

Claims

1. A fluorescent molecule database, characterized in that: Single-molecule spot images including the emission of a single fluorescent molecule.

2. The fluorescent molecule database according to claim 1, characterized in that: It also includes a synthetic spot image of multi-molecule luminescence formed by superimposing two or more single-molecule spot images.

3. The fluorescent molecule database according to claim 2, characterized in that: The synthetic spot image includes placing two or more single-molecule spots at a distance not exceeding the diffraction limit between their corresponding luminescent molecular positions and superimposing a synthetic fluorescent spot image, or / also includes placing two or more single-molecule spots at a distance exceeding the diffraction limit between their corresponding luminescent molecular positions and superimposing a synthetic fluorescent spot image.

4. The fluorescent molecule database according to claim 2, characterized in that: It also includes the luminescent molecule position and number information corresponding to the single-molecule spot image, the luminescent molecule position and number information corresponding to multiple single-molecule spot images of the synthetic spot image, and the molecular position and number information corresponding to multiple fluorescent molecules of the synthetic spot image.

5. The fluorescent molecule database according to claim 1, characterized in that: The single-molecule spot image is the spot image after removing the image background signal.

6. A machine learning method, characterized in that: The object of learning includes the information in any one of the fluorescent molecule databases of claims 1 to 5.

7. An artificial intelligence system obtained using the machine learning method of claim 6.

8. The artificial intelligence system according to claim 7, characterized in that: Includes quantitative models and / or positioning models.

9. The artificial intelligence system according to claim 8, characterized in that: The output layer of the quantitative model includes a fully connected layer and an activation function for classification, and the output of the fully connected output layer includes the number of molecules corresponding to the spot image; the localization model trains independent localization networks for different numbers of molecules, and the output of its fully connected output layer includes the coordinate information of the position of the luminescent molecules corresponding to the spot image.

10. A fluorescence microscopy method, characterized in that: The method comprises using the machine learning method of claim 6 and / or the artificial intelligence system of any one of claims 7 to 9.

11. The fluorescence microscopy imaging method according to claim 10, characterized in that: The distance between target molecules can be within the diffraction limit or beyond the diffraction limit.

12. The fluorescence microscopy imaging method according to claim 11, characterized in that: The distance between the target molecules is greater than 5 nm.

13. A fluorescence microscopy system formed by the fluorescence microscopy method according to any one of claims 10 to 12.

14. A sample detection method using the fluorescence microscopy imaging system of claim 13.

15. The sample detection method according to claim 14, characterized in that: It includes a sample molecule quantitative detection method and / or a sample molecule positioning detection method.

16. The sample detection method according to claim 14, characterized in that: Including the acquisition of sample molecular spot images, the fluorescence imaging conditions can be synchrotron radiation, and the synchrotron radiation conditions include excitation laser of matching wavelength band and simultaneous addition of 405nm laser.

Images