Training method of single-molecule spectral feature recognition model and spectral feature recognition method

Abstract

A training method for a spectral feature recognition model of a single molecule and a spectral feature recognition method, wherein the training method comprises: using a simulated data set to train the spectral feature recognition model to obtain a preliminarily trained spectral feature recognition model, the simulated data set comprising a plurality of simulated spectral information sample images and corresponding spectral centroid labels; obtaining a real data set, the real data set comprising a plurality of real spectral information sample images and corresponding spectral centroid pseudo labels; using the simulated data set as source domain data and the real data set as target domain data, performing domain adaptation training on the preliminarily trained spectral feature recognition model to obtain a trained spectral feature recognition model. The present invention performs spectral feature recognition of a single molecule based on a machine learning model, and the spectral feature recognition model obtained by training using the training method of the present application is used to perform spectral feature recognition of a single molecule, which can effectively improve recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of spectral analysis, and in particular to a training method for a single-molecule spectral feature recognition model and a spectral feature recognition method. Background Art

[0002] Observing dynamic processes in biological systems is of great significance for understanding the internal mechanisms of life activities. Many biological dynamic processes occur on the millisecond time scale and nanometer spatial scale, and are mostly affected by the cell microenvironment parameters. Fluorescence spectroscopy is a key tool for studying the properties of biomolecules. It can detect environmental changes such as pH, polarity, viscosity and molecular arrangement, reveal the environmental parameters of cells, and make it possible to study processes such as the dynamics of biomolecular interactions at the molecular level.

[0003] Fluorescence spectrum is the spectrum of fluorescence emitted by fluorescent dye molecules after absorbing photons of excitation light of a specific wavelength. The fluorescence spectrum of fluorescent dye molecules can indicate changes in environmental parameters. For example, by analyzing the characteristics of the fluorescence spectrum, such as the spectral centroid, changes in the cell microenvironment can be monitored. Since imaging needs to be performed at the nanoscale, each acquisition can only obtain the spectral distribution information of a single fluorescent dye molecule. The current methods of measuring spectra can be mainly divided into two methods: fluorescence spectrophotometer-based and multi-spectral super-resolution imaging technology based on dispersion prisms. Traditional spectrophotometer-based methods usually measure the average effect of a group of molecules, which is not ideal for single molecule analysis in cells. Multi-spectral super-resolution imaging technology based on dispersion prisms can accurately observe and measure single molecules in complex biological environments. However, due to the weak fluorescence signal and the influence of factors such as background fluorescence, the signal-to-noise ratio is poor. In this case, the traditional method of identifying fluorescence spectrum characteristics has low accuracy. Summary of the invention

[0004] The main technical problem solved by the present invention is that the signal-to-noise ratio of a single molecule is low, resulting in low recognition accuracy of its spectral centroid.

[0005] According to the first aspect, in one embodiment, a training method for a single-molecule spectral feature recognition model is provided, wherein the spectral feature recognition model is used to identify the spectral centroid of the single molecule, wherein the spectral centroid is the wavelength corresponding to the maximum intensity peak in the spectrum of the single molecule, and the training method comprises:

[0006] Acquire a simulated data set, wherein the simulated data set includes a plurality of simulated spectral information sample images and corresponding spectral centroid labels, wherein the simulated spectral information sample images are simulated single-molecule spectral information images;

[0007] Inputting the simulated spectral information sample image into a spectral feature recognition model to identify the spectral centroid, and obtaining a first spectral centroid prediction result;

[0008] Training the spectral feature recognition model according to a label prediction loss function to obtain a preliminarily trained spectral feature recognition model, wherein the label prediction loss function is constructed according to the first spectral centroid prediction result and the corresponding spectral centroid label;

[0009] Acquire a real data set, wherein the real data set includes a plurality of real spectral information sample images and corresponding spectral centroid pseudo labels, wherein the real spectral information sample images are spectral information images of single molecules collected in an actual environment;

[0010] The simulated data set is used as source domain data, and the real data set is used as target domain data, and domain adaptation training is performed on the initially trained spectral feature recognition model to obtain a trained spectral feature recognition model.

[0011] In some embodiments, the label prediction loss function is:

[0012] ,

[0013] in N s is the number of simulated spectral information sample images in the simulated data set, y s,i represents the first i The spectral centroid label corresponding to the simulated spectral information sample image, represents the first i The first spectral centroid prediction result of the simulated spectral information sample image.

[0014] In some embodiments, the spectral feature recognition model includes an image feature extraction network and a regression prediction network, and the step of inputting the simulated spectral information sample image into the spectral feature recognition model to identify the spectral centroid to obtain a first spectral centroid prediction result includes:

[0015] Performing feature extraction on the simulated spectrum information sample image through the image feature extraction network to obtain a first feature vector;

[0016] The first feature vector is input into the regression prediction network to perform regression prediction of the spectral centroid to obtain the first spectral centroid prediction result.

[0017] In some embodiments, the image feature extraction network includes a linear projection layer and a Transformer encoder, wherein the Transformer encoder is an encoder part of the Transformer model, and the Transformer encoder can receive a data frame including N p+1 input vector sequence, and output includes N p +1 output vector sequence of output vectors;

[0018] The step of extracting features from the simulated spectral information sample image through the image feature extraction network to obtain a first feature vector includes:

[0019] The simulated spectrum information sample image is divided into N p image blocks; among them, N p is an integer not less than 2;

[0020] The N p The image blocks are input into the linear projection layer for linear transformation to obtain the N p The embedding vector of each image patch;

[0021] respectively configuring a position encoding vector for each of the image blocks and the preset category embedding vector, adding the embedding vector of each image block to the corresponding position encoding vector to obtain a representation vector of the image block, and adding the category embedding vector to the corresponding position encoding vector to obtain a category representation vector;

[0022] The category representation vector is used as the first input vector of the input vector sequence, and the N p The representation vector of each image block is used as the back of the input vector sequence. N p An input vector is input into the Transformer encoder, and the first feature vector is obtained from the output vector sequence output by the Transformer encoder, wherein the first feature vector is the first output vector of the output vector sequence.

[0023] In some embodiments, the regression prediction network is a multi-layer perceptron.

[0024] In some embodiments, domain adaptation training is performed on the initially trained spectral feature recognition model to obtain a trained spectral feature recognition model, including:

[0025] Mixing the simulated data set and the real data set to obtain a mixed data set;

[0026] Reading a spectral information sample image from the mixed data set and inputting it into the preliminarily trained spectral feature recognition model to identify the spectral centroid, thereby obtaining a second spectral centroid prediction result;

[0027] Adding a gradient reversal layer to the initially trained spectral feature recognition model, wherein the gradient reversal layer is located between the image feature extraction network and the regression prediction network, and is used to act as an identity function during the forward pass, so that the data passes without modification, and during the reverse pass, the gradient is multiplied by a negative constant and then passed backward;

[0028] Reading spectral information sample images from the mixed data set and inputting the initially trained spectral feature recognition model with a gradient reversal layer to identify the spectral centroid, and obtaining adversarial prediction results;

[0029] The initially trained spectral feature recognition model is trained according to a domain adaptation loss function to obtain a trained spectral feature recognition model; wherein the domain adaptation loss function is constructed according to the second spectral centroid prediction result, the adversarial prediction result, the spectral centroid label corresponding to the simulated spectral information sample image, and the spectral centroid pseudo label corresponding to the real spectral information sample image.

[0030] In some embodiments, the domain adaptation loss function is composed of a regression loss function of a source domain, a regression loss function of a target domain, and a regression edge difference inconsistency loss function;

[0031] The regression loss function of the source domain is constructed according to the second spectral centroid prediction result of the simulated spectral information sample image and the corresponding spectral centroid label, with the purpose of reducing the difference between the second spectral centroid prediction result of the simulated spectral information sample image and the corresponding spectral centroid label;

[0032] The regression loss function of the target domain is constructed according to the second spectral centroid prediction result of the real spectral information sample image and the corresponding spectral centroid pseudo-label, with the purpose of reducing the difference between the second spectral centroid prediction result of the real spectral information sample image and the corresponding spectral centroid pseudo-label;

[0033] The regression edge difference inconsistency loss function is constructed based on the second spectral centroid prediction result and adversarial prediction result of the simulated spectral information sample image, and the second spectral centroid prediction result and adversarial prediction result of the real spectral information sample image, with the aim of reducing the distribution difference between the second spectral centroid prediction result of the simulated spectral information sample image and the second spectral centroid prediction result of the real spectral information sample image.

[0034] In some embodiments, the domain adaptation loss function is:

[0035] ,

[0036] in L s represents the regression loss function of the source domain, and

[0037] ,

[0038] L t represents the regression loss function of the target domain, and

[0039] ,

[0040] D γ represents the regression edge difference inconsistency loss function, and

[0041] ,

[0042] in, N s is the number of simulated spectral information sample images in the simulated data set, y s,i represents the first i The spectral centroid label corresponding to the simulated spectral information sample image, represents the first i The second spectral centroid prediction result of the simulated spectral information sample image, represents the first i Adversarial prediction results of Zhang’s simulated spectral information sample images. N t is the number of real spectral information sample images in the real dataset, y t,j represents the first j The spectral centroid pseudo label corresponding to the real spectral information sample image, represents the first j The second spectral centroid prediction result of the real spectral information sample image, represents the first j Adversarial prediction results of a sample image with real spectral information.

[0043] In some embodiments, the spectral centroid pseudo-label is determined by: collecting the spectrum of a fluorescent ball multiple times in the same environment as the environment in which the single molecule is located when the real spectral information sample image is collected, and calculating the average spectrum of the multiple collected spectra, and obtaining the wavelength corresponding to the maximum intensity peak of the average spectrum as the spectral centroid pseudo-label.

[0044] According to the second aspect, an embodiment provides a method for identifying spectral features of a single molecule, comprising:

[0045] Obtain spectral information images of single molecules;

[0046] The spectral information image is input into the spectral feature recognition model trained by the above training method to identify the spectral centroid, so as to obtain the spectral centroid of the single molecule.

[0047] In some embodiments, the spectral information image of the single molecule is obtained by splicing the light intensity image of the single molecule and the dispersion spectrum image of the single molecule, wherein the light intensity image of the single molecule and the dispersion spectrum image of the single molecule are respectively formed by two paths of light emitted by the single molecule after being excited, one path of light is directly imaged on the image sensor to form the light intensity image, and the other path of light is dispersed by a dispersion prism and then imaged on the image sensor to form the dispersion spectrum image.

[0048] According to the third aspect, an embodiment provides a computer-readable storage medium, on which a program is stored, and the program can be executed by a processor to implement the above-mentioned training method or spectral feature recognition method.

[0049] According to the training method, spectral feature recognition method and computer-readable storage medium of the single-molecule spectral feature recognition model of the above-mentioned embodiment, the spectral feature recognition model is trained to recognize the spectral features of the single molecule. During the training process, the spectral feature recognition model is first trained with a simulated data set so that the model has the ability to recognize the spectral centroid. Then, the simulated data set is used as the source domain data and the real data set is used as the target domain data to perform domain adaptation training on the preliminarily trained spectral feature recognition model to obtain a trained spectral feature recognition model, so that the spectral feature recognition model can adapt to the spectral feature recognition task of the target domain, so that the spectral feature recognition model can also have good performance when deployed in an actual environment. The present invention performs spectral feature recognition of single molecules based on a machine learning model. By using the spectral feature recognition model obtained by training the training method of the present application to perform spectral feature recognition of single molecules, the recognition accuracy can be effectively improved; and in the case where the real spectral centroid label of the spectral information image of the single molecule collected in the actual environment cannot be obtained, a spectral feature recognition model with excellent performance can also be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a method for training a single-molecule spectral feature recognition model in some embodiments;

[0051] Figure 2 A schematic diagram of the structure of a single-molecule spectral feature recognition model in some embodiments;

[0052] Figure 3 It is a schematic structural diagram of a single molecule spectral feature recognition model in some other embodiments;

[0053] Figure 4 A flowchart of extracting features from a simulated spectral information sample image through an image feature extraction network to obtain a first feature vector in some embodiments;

[0054] Figure 5 A flowchart of performing domain adaptation training on a preliminarily trained spectral feature recognition model to obtain a trained spectral feature recognition model in some embodiments;

[0055] Figure 6 A schematic diagram of constructing a regression loss function of a source domain, a regression loss function of a target domain, and an RMDD loss function in some embodiments;

[0056] Figure 7 A flowchart of a method for identifying spectral features of a single molecule in some embodiments;

[0057] Figure 8 In some embodiments, the emission light of a single molecule is divided into two paths of light to form a light path diagram of a light intensity image and a dispersion spectrum image respectively;

[0058] Fig. 9 Schematic diagram of the relationship between the intensity image and the dispersion spectrum image;

[0059] Fig.10 This is a result diagram of spectral feature recognition of fluorescent beads using the spectral feature recognition method of the present application. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0061] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0062] The serial numbers assigned to the components herein, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0063] In some embodiments of the present application, a spectral feature recognition model is provided, the purpose of which is to be able to identify spectral features, such as the degree of redshift / blueshift, from the spectral curve of a noisy single molecule, thereby realizing real-time monitoring of changes in the molecular microenvironment. Specifically, the spectral feature recognition model is used to input the spectral information image of a single molecule, predict, identify and output the spectral centroid of the spectral information image. Among them, the spectral centroid is the wavelength corresponding to the maximum intensity peak in the spectrum of a single molecule, and the spectral feature recognition model is composed of a neural network, such as a convolutional neural network, a multi-layer perceptron MLP (MultilayerPerceptron), etc.

[0064] A single molecule may refer to a single fluorescent dye molecule, and the fluorescent dye molecule is preferably an environmentally sensitive dye molecule, such as a Nile red lipid dye molecule, a seau-647 dye molecule, etc. A spectral information image of a single molecule is an image including spectral information of the single molecule, such as a dispersion spectral image, and a spectral information image can be obtained by imaging the single molecule using an optical imaging device (such as a microscope, etc.) with spectral imaging capability.

[0065] When training the above-mentioned spectral feature recognition model, a labeled simulated data set is used for training so that the model learns the ability to recognize the spectral centroid. However, the simulated data set (source domain) cannot include all the situations encountered in actual experiments. Therefore, deploying the model in the actual environment (target domain) often causes performance degradation. In order to address this problem, the present application proposes a training method for the above-mentioned single-molecule spectral feature recognition model, which performs model enhancement based on domain adaptation so that it can adapt to the spectral feature recognition task of the target domain. In some embodiments, a gradient reversal layer (GRL) strategy is utilized to add a gradient reversal layer to the model. This layer strategically modifies the direction of the gradient flow during the reverse transfer, thereby encouraging the model to learn cross-domain invariant features; during the forward transfer, it acts as an identity function, allowing data to pass without change.

[0066] Please refer to Figure 1 The training method in some embodiments of the present application includes steps 100 to 500, which are described in detail below.

[0067] Step 100: Obtain a simulated data set.

[0068] The simulated data set includes multiple simulated spectral information sample images and corresponding spectral centroid labels, and the simulated spectral information sample images are simulated spectral information images of single molecules. Specifically, the spectral information image of a single molecule can be simulated according to parameters such as the point spread function of an optical imaging device used for imaging, and the simulated data set can be artificially produced.

[0069] Step 200: Inputting the simulated spectrum information sample image into the spectrum feature recognition model to identify the spectrum centroid, and obtaining a first spectrum centroid prediction result.

[0070] Step 300: Train the spectral feature recognition model according to the label prediction loss function to obtain a preliminarily trained spectral feature recognition model.

[0071] The label prediction loss function is constructed based on the first spectral centroid prediction result and the corresponding spectral centroid label, and is used to measure the difference between the first spectral centroid prediction result and the corresponding spectral centroid label, with the purpose of enabling the model to learn to accurately predict the spectral centroid. In some embodiments, the label prediction loss function is:

[0072] ,

[0073] in N s is the number of simulated spectral information sample images in the simulated data set, y s,i Indicates the first i The spectral centroid label corresponding to the simulated spectral information sample image, Indicates the first i The first spectral centroid prediction result of the simulated spectral information sample image.

[0074] Step 400: Obtain a real data set.

[0075] The real data set includes multiple real spectral information sample images and corresponding spectral centroid pseudo labels. The real spectral information sample images are spectral information images of single molecules collected in the actual environment, such as fluorescent dye molecules in the actual cell environment. Spectral imaging is performed on them, and the spectral information images collected are the real spectral information sample images.

[0076] It is understandable that it is difficult to know the exact spectral centroid of the spectral information image of a single molecule collected in an actual environment. Therefore, the real spectral information sample image does not have a real spectral centroid label. In order to facilitate subsequent training, a spectral centroid pseudo label can be added to it artificially. Since the spectrum of the fluorescent sphere is unchanged, in some embodiments, the spectral centroid of the fluorescent sphere under the same conditions can be used as the spectral centroid pseudo label. Specifically, the spectral centroid pseudo label is determined in the following way: for the fluorescent sphere in the same environment as the single molecule when the real spectral information sample image is collected, its spectrum is collected multiple times, and the average spectrum of the multiple collected spectra is calculated, and the wavelength corresponding to the maximum intensity peak of the average spectrum is obtained as the spectral centroid pseudo label.

[0077] Step 500: Using the simulated data set as source domain data and the real data set as target domain data, domain adaptation training is performed on the initially trained spectral feature recognition model to obtain a trained spectral feature recognition model.

[0078] After domain adaptation training, the spectral centroid recognition capability of the spectral feature recognition model in the source domain can be transferred to the target domain. The trained spectral feature recognition model can be used to recognize the spectral features of single molecules in the target domain. This method of using a machine learning model for spectral feature recognition can effectively improve the recognition accuracy.

[0079] The training method of the above embodiment sets corresponding spectral centroid pseudo labels for real spectral information sample images, first trains the spectral feature recognition model on a simulated data set, and then performs domain adaptation training using the simulated data set as source domain data and the real data set as target domain data. Even when it is impossible to obtain the real spectral centroid labels of the spectral information images of single molecules collected in an actual environment, a spectral feature recognition model with excellent performance can be trained.

[0080] Please refer to Figure 2 In some embodiments, the spectral feature recognition model includes an image feature extraction network and a regression prediction network. Step 200 specifically includes: extracting features of the simulated spectral information sample image through the image feature extraction network to obtain a first feature vector; inputting the first feature vector into the regression prediction network to perform regression prediction of the spectral centroid to obtain a first spectral centroid prediction result.

[0081] Please refer to Figure 3 In some embodiments, the image feature extraction network includes a linear projection layer and a Transformer encoder, wherein the Transformer encoder is the encoder part of the Transformer model, and the Transformer encoder can receive the image feature extraction network including N p +1 input vector sequence, and output includesN p +1 output vector sequence of output vectors, where N p is an integer not less than 2, Figure 3 China-Israel N p =80 as an example, it does not mean that only 80 can be taken.

[0082] like Figure 3 As shown in the figure, the Transformer encoder consists of 6 encoding modules connected in sequence. Each encoding module includes a multi-head attention layer, two superposition and regularization layers and a forward propagation network. The input features are first processed by the multi-head attention layer, superimposed and regularized with the original input features to obtain the regularized attention features. After that, the regularized attention features are input into the forward propagation network for processing, superimposed and regularized with the original regularized attention features to obtain the output of the encoding module of this layer.

[0083] like Figure 3 As shown, the input spectral information image is first divided into N p image blocks, N p The image blocks are input into the linear projection layer for linear transformation and then N p The embedding vectors of each image block are used to represent the feature information of the image block. The Transformer encoder requires that a position encoding vector be configured for each image block to capture the relative position relationship between the image blocks. The position encoding vector has the same dimension as the embedding vector of the image block. After adding the position encoding vector to the embedding vector of the image block, the representation vector of the image block is obtained and input into the Transformer encoder. In order to predict the spectral centroid of the spectral information image, this embodiment further adds a category embedding vector cls token. The category embedding vector also has a position encoding vector with the same dimension. The category embedding vector is added to the corresponding position encoding vector to obtain a category representation vector, which is used as the first input vector of the input vector sequence of the Transformer encoder and is input into the Transformer encoder together with the representation vector of the image block. Finally, the first output vector of the output vector sequence is used as the first feature vector for predicting the spectral centroid.

[0084] The category embedding vector and each position encoding vector are learnable parameters, whose initial values ​​are set randomly, and they are also updated during the training process of the model.

[0085] Please refer to Figure 4 In this embodiment, the image feature extraction network is used to extract features from the simulated spectrum information sample image to obtain a first feature vector, which includes the following steps:

[0086] Step 201: Divide the simulated spectrum information sample image into N p image blocks;

[0087] Step 202: N p The image blocks are input into the linear projection layer for linear transformation and then N p The embedding vector of each image patch;

[0088] Step 203: configuring a position coding vector for each image block and a preset category embedding vector respectively, adding the embedding vector of each image block to the corresponding position coding vector to obtain a representation vector of the image block, and adding the category embedding vector to the corresponding position coding vector to obtain a category representation vector;

[0089] Step 204: Use the category representation vector as the first input vector of the input vector sequence, and N p The representation vector of each image block is used as the input vector sequence. N p An input vector is input into the Transformer encoder, and the first feature vector is obtained from the output vector sequence output by the Transformer encoder, where the first feature vector is the first output vector of the output vector sequence.

[0090] This embodiment uses the Transformer encoder to extract image features. Due to the multi-head attention mechanism and position encoding characteristics in the Transformer encoder, the positional relationship between different pixels can be well learned, which is conducive to improving recognition accuracy.

[0091] For the regression prediction network, a multilayer perceptron is used in some embodiments.

[0092] Please refer to Figure 5 In some embodiments, step 500 includes steps 501 to 505, which are described in detail below.

[0093] Step 501: Mix the simulated data set and the real data set to obtain a mixed data set.

[0094] Specifically, the simulated data set and the real data set may be randomly mixed so that the simulated spectral information sample images and the real spectral information sample images are randomly distributed in the mixed data set.

[0095] Step 502: Read the spectral information sample image from the mixed data set and input it into the initially trained spectral feature recognition model to identify the spectral centroid to obtain a second spectral centroid prediction result. The spectral information sample image read here may be a simulated spectral information sample image or a real spectral information sample image.

[0096] like Figure 6 As shown in the figure, when the image feature extraction network is a Transformer encoder, the output vector sequence obtained after the spectral information sample image is input includes multiple output vectors f 1. f 2…… f cls ,in f cls is the first eigenvector, f cls Input the regression prediction network to obtain the second spectrum centroid prediction result.

[0097] Step 503: Add a gradient reversal layer to the initially trained spectral feature recognition model. The gradient reversal layer is located between the image feature extraction network and the regression prediction network. It is used to act as an identity function during the forward pass, allowing the data to pass without change. During the backward pass, the gradient is multiplied by a negative constant (- λ ) and then pass it backwards.

[0098] Step 504: Read the spectral information sample image from the mixed data set and input it into the spectral feature recognition model preliminarily trained with the gradient inversion layer to identify the spectral centroid and obtain the adversarial prediction result. Similarly, the spectral information sample image read here may be a simulated spectral information sample image or a real spectral information sample image.

[0099] like Figure 6 As shown in the figure, a gradient reversal layer is added between the Transformer encoder and the regression prediction network. After the spectral information sample image is input, the first eigenvector is obtained. f cls , the first eigenvector f cls After passing the gradient reversal layer, it is input into the regression prediction network to obtain the adversarial prediction result.

[0100] Step 505: Train the initially trained spectral feature recognition model according to the domain adaptation loss function to obtain a trained spectral feature recognition model.

[0101] In order to solve the problem that the simulated data set (source domain) cannot include all situations encountered in the actual environment, this embodiment provides a new training method. First, a gradient reversal layer is added to the spectral feature recognition model to obtain adversarial prediction results. During training, the model parameters are updated by the back propagation method. During the back propagation, the gradient is multiplied by a negative constant (- λ ) and then pass it backward. This operation effectively reverses the gradient flow direction of the adversarial prediction, prompting the model to learn cross-domain invariant features. Its mathematical expression is:

[0102] ,

[0103] in θ represents the parameters of the spectral feature recognition model, represents the gradient of the domain adaptation loss function with respect to the parameters of the spectral feature recognition model, Represents the gradient after the gradient reversal layer.

[0104] Secondly, a domain adaptation loss function is constructed according to the second spectral centroid prediction result, the adversarial prediction result, the spectral centroid label corresponding to the simulated spectral information sample image, and the spectral centroid pseudo label corresponding to the real spectral information sample image.

[0105] In order to better achieve domain adaptation of the model, this application proposes a regression margin disparity discrepancy (RMDD) loss function, which aims to evaluate and reduce the distribution difference between the spectral centroid prediction results of the source domain and the spectral centroid prediction results of the target domain in the domain adaptation task. In short, the RMDD loss function measures the distribution difference between the source domain and the target domain, with special attention to the difference in the context of margin loss. Figure 6 As shown, the RMDD loss function is constructed according to the second spectrum centroid prediction result and the adversarial prediction result. Specifically, the regression edge difference inconsistency loss function is constructed according to the second spectrum centroid prediction result and the adversarial prediction result of the simulated spectrum information sample image, the second spectrum centroid prediction result and the adversarial prediction result of the real spectrum information sample image, and the purpose is to reduce the distribution difference between the second spectrum centroid prediction result of the simulated spectrum information sample image and the second spectrum centroid prediction result of the real spectrum information sample image. In some embodiments, the RMDD loss function is:

[0106] ,

[0107] in, N s is the number of simulated spectral information sample images in the simulated data set, Indicates the first iThe second spectral centroid prediction result of the simulated spectral information sample image, Indicates the first i Adversarial prediction results of Zhang’s simulated spectral information sample images. N t is the number of sample images with real spectral information in the real dataset, Indicates the first j The second spectral centroid prediction result of the real spectral information sample image, Indicates the first j Adversarial prediction results of sample images with real spectral information. m is a preset constant and m >0.

[0108] in, The part is used to measure the difference between the actual prediction in the source domain and its adversarial prediction, The part is used to measure the difference between the actual prediction in the target domain and its adversarial prediction, m Used to scale the differences in the source domain to control its impact on the overall domain adaptation process.

[0109] The complete domain adaptation loss function consists of the regression loss function of the source domain, the regression loss function of the target domain, and the above RMDD loss function. Figure 6 As shown, the regression loss function of the source domain is constructed according to the second spectral centroid prediction result of the simulated spectral information sample image and the corresponding spectral centroid label, with the purpose of reducing the difference between the second spectral centroid prediction result of the simulated spectral information sample image and the corresponding spectral centroid label. In some embodiments, the regression loss function of the source domain is:

[0110] ,

[0111] in, N s is the number of simulated spectral information sample images in the simulated data set, Indicates the first i The second spectral centroid prediction result of the simulated spectral information sample image.

[0112] Similarly, the regression loss function of the target domain is constructed based on the second spectral centroid prediction result of the real spectral information sample image and the corresponding spectral centroid pseudo-label, with the purpose of reducing the difference between the second spectral centroid prediction result of the real spectral information sample image and the corresponding spectral centroid pseudo-label. In some embodiments, the regression loss function of the target domain is:

[0113] ,

[0114] in,y t,j Indicates the first j The spectral centroid pseudo label corresponding to the sample image of the real spectral information.

[0115] In some embodiments, the domain adaptation loss function is a weighted sum of the regression loss function of the source domain, the regression loss function of the target domain, and the RMDD loss function, which can be specifically:

[0116] .

[0117] Based on the above-mentioned spectral feature recognition model and its training method, this application provides a single-molecule spectral feature recognition method, please refer to Figure 7 In some embodiments, the method comprises the following steps:

[0118] Step 600: Acquire a spectrum information image of a single molecule;

[0119] Step 700: Input the spectral information image into the spectral feature recognition model trained by the training method of any of the above embodiments to identify the spectral centroid and obtain the spectral centroid of a single molecule.

[0120] In some embodiments, a spectral information image of a single molecule is obtained by splicing a light intensity image of the single molecule and a dispersion spectrum image of the single molecule, wherein the light intensity image of the single molecule and the dispersion spectrum image of the single molecule are respectively formed by two paths of light emitted by the single molecule after being excited, one path of light is directly imaged on an image sensor to form a light intensity image, and the other path of light is dispersed by a dispersion prism and then imaged on the image sensor to form a dispersion spectrum image.

[0121] That is to say, the spectral information image obtained in step 600, the real spectral information sample image in step 400 and the simulated spectral information sample image in step 100 are all obtained by splicing the light intensity image and the dispersion spectrum image. The difference is that the light intensity image and the dispersion spectrum image spliced ​​into the simulated spectral information sample image are not collected in an actual environment, but are obtained by simulating the emission light emitted after a single molecule is excited and divided into two paths of light and imaging them on an image sensor respectively, one of which is simulated as directly imaging on the image sensor to obtain a simulated light intensity image, and the other is simulated as imaging on the image sensor after being dispersed by a dispersion prism to obtain a simulated dispersion spectrum image.

[0122] Here, the same image sensor is used to image the two paths of light. In some embodiments, in the optical imaging device, Figure 8The optical path setting shown enables the image sensor to simultaneously acquire the intensity image and dispersion spectrum image of a single molecule. The emission light emitted by a single molecule after being excited is divided into two light paths by a beam splitter, one of which enters the image sensor with the help of a reflector (without dispersion) to directly form an intensity image, and the other light is dispersed by a dispersion prism and then enters the image sensor to form a dispersion spectrum image.

[0123] Please refer to Fig. 9 In the dispersion spectrum image, different pixel positions or different columns along the horizontal axis of the image correspond to different wavelengths. In theory, the corresponding wavelength can be determined by the position of the maximum intensity peak in the dispersion spectrum image. However, in the actual environment, single molecules will move, and their dispersed light will be imaged at different positions on the image sensor. It is impossible to consider each pixel position as fixedly corresponding to a wavelength to determine the wavelength corresponding to the maximum intensity peak. In this embodiment, the light intensity image is introduced as the position reference image. Since the light intensity image and the dispersion spectrum image are imaged by the same molecule and the same image sensor, when the single molecule moves, the relative positions of the light intensity image and the dispersion spectrum image imaged on the image sensor are relatively fixed. The light intensity peak point in the light intensity image can be used as the position reference point, and the wavelength corresponding to the maximum intensity peak can be determined by judging the pixel distance between the maximum intensity peak in the dispersion spectrum image and the position reference point.

[0124] In some embodiments, the spectral information image can be cropped to a fixed size and then input into the spectral feature recognition model, such as a size of 16×80. If the spectral information image is composed of an intensity image and a dispersion spectrum image, the intensity image and the dispersion spectrum image can be cropped to a fixed size and then spliced. For example, the intensity image is cropped to a size of 16×16, and the dispersion spectrum image is cropped to a size of 16×64, and then spliced ​​into a spectral information image of 16×80.

[0125] The applicant used fluorescent beads to experiment with the spectral feature recognition method of the present application. A fluorescent bead was tracked for nearly 20 seconds and its spectral centroid was obtained multiple times (for example, once every 1.5 ms). The results are as follows: Fig.10 As shown, μ V , σ V are respectively the mean and variance of the spectral centroid measured by the spectral feature recognition method of the present application, μ norm , σ norm They are respectively the mean and variance of the spectral centroid measured by the traditional normal fitting method. By comparing the variances of the two, it can be seen that the spectral feature recognition method of the present application has higher accuracy.

[0126] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0127] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A training method for a single-molecule spectral feature recognition model, characterized in that: The spectral feature recognition model is used to identify the spectral centroid of a single molecule, where the spectral centroid is the wavelength corresponding to the maximum intensity peak in the spectrum of the single molecule. The training method includes: Acquire a simulated data set, wherein the simulated data set includes a plurality of simulated spectral information sample images and corresponding spectral centroid labels, wherein the simulated spectral information sample images are simulated single-molecule spectral information images; Inputting the simulated spectral information sample image into a spectral feature recognition model to identify the spectral centroid, and obtaining a first spectral centroid prediction result; Training the spectral feature recognition model according to a label prediction loss function to obtain a preliminarily trained spectral feature recognition model, wherein the label prediction loss function is constructed according to the first spectral centroid prediction result and the corresponding spectral centroid label; Acquire a real data set, wherein the real data set includes a plurality of real spectral information sample images and corresponding spectral centroid pseudo labels, wherein the real spectral information sample images are spectral information images of single molecules collected in an actual environment; Using the simulated data set as source domain data and the real data set as target domain data, performing domain adaptation training on the preliminarily trained spectral feature recognition model to obtain a trained spectral feature recognition model; The spectral feature recognition model includes an image feature extraction network and a regression prediction network. The image feature extraction network includes a linear projection layer and a Transformer encoder. The Transformer encoder is the encoder part of the Transformer model. The Transformer encoder can receive the image feature extraction network and the regression prediction network. N p +1 input vector sequence, and output includes N p +1 output vector sequence of output vectors; The step of inputting the simulated spectral information sample image into a spectral feature recognition model to identify the spectral centroid to obtain a first spectral centroid prediction result includes: The simulated spectrum information sample image is divided into N p image blocks; among them, N p is an integer not less than 2; The N p The image blocks are input into the linear projection layer for linear transformation to obtain the N p The embedding vector of each image patch; respectively configuring a position encoding vector for each of the image blocks and the preset category embedding vector, adding the embedding vector of each image block to the corresponding position encoding vector to obtain a representation vector of the image block, and adding the category embedding vector to the corresponding position encoding vector to obtain a category representation vector; The category representation vector is used as the first input vector of the input vector sequence, and the N p The representation vector of each image block is used as the back of the input vector sequence. N p An input vector is input into the Transformer encoder, and a first feature vector is obtained from an output vector sequence output by the Transformer encoder, wherein the first feature vector is the first output vector of the output vector sequence; The first feature vector is input into the regression prediction network to perform regression prediction of the spectral centroid to obtain the first spectral centroid prediction result.

2. The training method according to claim 1, characterized in that: The label prediction loss function is: ， in N s is the number of simulated spectral information sample images in the simulated data set, y s,i represents the first i The spectral centroid label corresponding to the simulated spectral information sample image, represents the first i The first spectral centroid prediction result of the simulated spectral information sample image.

3. The training method according to claim 1, characterized in that: The regression prediction network is a multi-layer perceptron.

4. The training method according to claim 1, characterized in that: Performing domain adaptation training on the initially trained spectral feature recognition model to obtain a trained spectral feature recognition model includes: Mixing the simulated data set and the real data set to obtain a mixed data set; Reading a spectral information sample image from the mixed data set and inputting it into the preliminarily trained spectral feature recognition model to identify the spectral centroid, thereby obtaining a second spectral centroid prediction result; Adding a gradient reversal layer to the initially trained spectral feature recognition model, wherein the gradient reversal layer is located between the image feature extraction network and the regression prediction network, and is used to act as an identity function during the forward pass, so that the data passes without modification, and during the reverse pass, the gradient is multiplied by a negative constant and then passed backward; Reading spectral information sample images from the mixed data set and inputting the initially trained spectral feature recognition model with a gradient reversal layer to identify the spectral centroid, and obtaining adversarial prediction results; The initially trained spectral feature recognition model is trained according to a domain adaptation loss function to obtain a trained spectral feature recognition model; wherein the domain adaptation loss function is constructed according to the second spectral centroid prediction result, the adversarial prediction result, the spectral centroid label corresponding to the simulated spectral information sample image, and the spectral centroid pseudo label corresponding to the real spectral information sample image.

5. The training method according to claim 4, characterized in that: The domain adaptation loss function is composed of a regression loss function of the source domain, a regression loss function of the target domain, and a regression edge difference inconsistency loss function; The regression loss function of the source domain is constructed according to the second spectral centroid prediction result of the simulated spectral information sample image and the corresponding spectral centroid label, with the purpose of reducing the difference between the second spectral centroid prediction result of the simulated spectral information sample image and the corresponding spectral centroid label; The regression loss function of the target domain is constructed according to the second spectral centroid prediction result of the real spectral information sample image and the corresponding spectral centroid pseudo-label, with the purpose of reducing the difference between the second spectral centroid prediction result of the real spectral information sample image and the corresponding spectral centroid pseudo-label; The regression edge difference inconsistency loss function is constructed based on the second spectral centroid prediction result and adversarial prediction result of the simulated spectral information sample image, and the second spectral centroid prediction result and adversarial prediction result of the real spectral information sample image, with the aim of reducing the distribution difference between the second spectral centroid prediction result of the simulated spectral information sample image and the second spectral centroid prediction result of the real spectral information sample image.

6. The training method according to claim 5, characterized in that: The domain adaptation loss function is: ， in L s represents the regression loss function of the source domain, and ， L t represents the regression loss function of the target domain, and ， D γ represents the regression edge difference inconsistency loss function, and ， in, N s is the number of simulated spectral information sample images in the simulated data set, y s,i represents the first i The spectral centroid label corresponding to the simulated spectral information sample image, represents the first i The second spectral centroid prediction result of the simulated spectral information sample image, represents the first i Adversarial prediction results of Zhang’s simulated spectral information sample images. N t is the number of real spectral information sample images in the real dataset, y t,j represents the first j The spectral centroid pseudo label corresponding to the real spectral information sample image, represents the first j The second spectral centroid prediction result of the real spectral information sample image, represents the first j Adversarial prediction results of sample images with real spectral information. m is a preset constant and m >0.

7. The training method according to any one of claims 1 to 6, characterized in that: The spectral centroid pseudo-label is determined in the following manner: the spectrum of the fluorescent ball is collected multiple times in the same environment as the environment in which the single molecule is located when the real spectral information sample image is collected, and the average spectrum of the multiple collected spectra is calculated, and the wavelength corresponding to the maximum intensity peak of the average spectrum is obtained as the spectral centroid pseudo-label.

8. A method for identifying spectral characteristics of a single molecule, characterized in that: include: Obtain spectral information images of single molecules; The spectral information image is input into a spectral feature recognition model trained by the training method according to any one of claims 1 to 7 to identify the spectral centroid, thereby obtaining the spectral centroid of the single molecule.

9. The spectral feature recognition method according to claim 8, characterized in that: The spectral information image of the single molecule is obtained by splicing the light intensity image of the single molecule and the dispersion spectrum image of the single molecule, wherein the light intensity image of the single molecule and the dispersion spectrum image of the single molecule are respectively formed by two paths of light emitted by the single molecule after being excited, one path of light is directly imaged on the image sensor to form the light intensity image, and the other path of light is dispersed by a dispersion prism and then imaged on the image sensor to form the dispersion spectrum image.

10. A computer-readable storage medium, characterized in that: The medium stores a program, which can be executed by a processor to implement the training method according to any one of claims 1 to 7 or the spectral feature recognition method according to any one of claims 8 to 9.

Images