Retina organoid early differentiation prediction method and system based on multi-modal imaging

Abstract

The invention discloses a multi-modal imaging-based retinal organ early differentiation prediction method. The method comprises the following steps: receiving a bright field microscopic image and an OCT three-dimensional image of a sample collected at specified sampling time; the sample images are preprocessed, and the sampling time is normalized; and inputting the preprocessed sample image and the normalized sampling time into the time coding neural network differentiation prediction model to obtain a final differentiation prediction result. According to the method, accurate differentiation prediction can be realized at multiple time points in the directional differentiation process of the human embryonic stem cells to the retinal organs.

Description

Technical Field

[0001] The present invention belongs to the technical field of retinal organoid culture, and in particular relates to a retinal organoid early differentiation prediction technology based on multimodal imaging. Background Art

[0002] Currently, effective treatments are urgently needed for blindness caused by retinal degenerative diseases such as macular degeneration and inherited retinal dystrophies. Retinal organoids, cultured from pluripotent stem cells, have attracted widespread attention as important resources for disease models, drug screening, and cell transplantation. However, during in vitro culture, due to subtle differences in experimental procedures and random fluctuations in gene expression, the differentiation of retinal organoids exhibits significant inter-individual heterogeneity and randomness, including developmental delays, morphological abnormalities, and cell death. This not only affects the stability of large-scale batch production but also restricts their reliability in basic research and clinical applications. Therefore, it is necessary to assess the differentiation and predict the fate of individual samples in the early stages of differentiation.

[0003] Existing evaluation methods primarily rely on brightfield imaging or fluorescent staining, which often require prolonged culture or invasive labeling. This, combined with reliance on expert experience, significantly limits early prediction and real-time quality control. Existing research has utilized optical imaging combined with deep learning techniques, but these studies focus on retinal organoids cultured from mouse stem cells, lacking clinical relevance and physiological significance. Furthermore, these studies lack multidimensional and multimodal characterization of the samples, and the proposed models lack the ability to process data from multiple time points, significantly limiting their practical application.

[0004] Therefore, there is currently a lack of multi-temporal, non-invasive, and reliable differentiation prediction methods for retinal organoids cultured from human stem cells in the early differentiation stages. Summary of the Invention

[0005] In light of this, the present invention provides a multimodal imaging-based method for predicting early differentiation of retinal organoids. By acquiring brightfield microscopic images and three-dimensional optical coherence tomography (OCT) images of biological samples, combined with an innovatively constructed time-coded neural network differentiation prediction model, accurate differentiation prediction can be achieved at multiple time points during the directed differentiation of human embryonic stem cells into retinal organoids. Furthermore, the present invention also discloses a multimodal imaging-based retinal organoid early differentiation prediction system, electronic equipment, and computer-readable storage medium for implementing the above method.

[0006] The first aspect of the present invention discloses a method for predicting early differentiation of retinal organoids based on multimodal imaging, comprising:

[0007] Receiving a bright field microscopic image and an optical coherence tomography (OCT) three-dimensional image of a sample collected at a specified sampling time, wherein the sample is a biological sample in the process of directed differentiation of a human embryonic stem cell suspension into a retinal organoid;

[0008] Preprocessing the bright field microscopic image and the OCT three-dimensional image respectively and normalizing the sampling time;

[0009] Inputting the preprocessed bright field microscopic image and OCT three-dimensional image and the normalized sampling time into a time coding neural network differentiation prediction model trained by a multimodal neural network to obtain a final differentiation prediction result, wherein the final differentiation prediction result includes differentiation success and differentiation failure;

[0010] In the time-coding neural network differentiation prediction model, feature vectors are extracted from bright-field microscopic images and three-dimensional OCT images by their respective corresponding encoders. The feature vectors are added to the encoded features of the sampling time and then input into the corresponding classifier. The classification results output by each classifier are weightedly added to obtain the final differentiation prediction result.

[0011] Optionally, the multimodal neural network is constructed as follows:

[0012] z=λ1·g BF [f BF (x BF )+f time (T)]+λ2·g OCT [f OCT (x OCT )+f time (T)]

[0013] Where z is the classification score, which is used to represent the final differentiation prediction result. The classification score is binarized by threshold to obtain the final differentiation prediction result; λ1 and λ2 are modality fusion weights; x BF and x OCT Represent the input bright field microscopy image and 3D OCT image respectively; T after represents the normalized sampling time; f BF Encoder representing bright field microscopy image; f OCT The encoder representing the 3D OCT image, f time represents the time encoder; g BF represents a classifier for bright field microscopy images; g OCT Representation of a classifier for 3D OCT images.

[0014] Optionally, the time encoder f time is a multi-layer perceptron; the encoder f BFThe backbone network adopts the ResNet101 model of the convolutional neural network architecture; the encoder f OCT The backbone network adopts the channel separation convolutional network CSN50 model; the classifier g BF and g OCT Both are multi-layer perceptrons.

[0015] Optionally, the training process of the multimodal neural network includes the following steps: collecting bright field microscopic images and OCT three-dimensional images of the sample multiple times in sequence within a specified sampling time range; binary labeling of whether the sample is finally successfully differentiated into retinal organoids based on empirical standards of neural retinal characteristics to obtain training labels; preprocessing the sample images collected within the specified sampling time range and normalizing the sampling time; the preprocessed sample images and training labels constitute a training data set, each group of paired training data in the data set includes corresponding three-dimensional OCT images and bright field microscopic images, the sampling time of the group of images, and the training labels corresponding to the group of images; the training data set is used to train the multimodal neural network, and the required time-coded neural network differentiation prediction model is obtained.

[0016] Optionally, the designated sampling time range includes from the 6th day of culture to the 30th day of culture.

[0017] Optionally, during the training of the multimodal neural network, cross entropy loss is used as a loss function for supervised training;

[0018] The loss function expression is as follows:

[0019] L=―[z·log(z label )+(1―z)·log(1―z label )]

[0020] Where z is the classification score output by the network (z∈(0,1)), z label is the true label (z label ∈{0,1});

[0021] During the training process, the Adam optimizer is used to optimize the multimodal neural network. The optimization is terminated when the preset conditions are met to obtain the required time-coded neural network differentiation prediction model.

[0022] Optionally, the bright field microscopic image and the OCT three-dimensional image are preprocessed separately, specifically including: cropping and scaling the bright field microscopic image to a first set size, and then performing image enhancement processing; cropping the OCT three-dimensional image to a second set size, and then uniformly downsampling along a specified dimension.

[0023] A second aspect of the present invention discloses a retinal organoid early differentiation prediction system based on multimodal imaging, comprising an optical coherence tomography component and a bright field microscope component;

[0024] The optical coherence tomography assembly includes a scanning galvanometer, an OCT objective lens, a beam splitter, a dispersion compensating plate, an iris, a reflector, a collimator, an optical fiber, and an OCT host; broadband low-coherence near-infrared light is derived from the OCT host through an optical fiber, collimated by a collimator, and then reaches the beam splitter; half of the incident light is deflected after passing through the beam splitter, and then passes through the dispersion compensating plate and the iris in sequence, and is reflected by the reflector and returned along the original path, and the obtained reflected light is the reference light; the other half of the incident light passes through the beam splitter and is reflected twice by the scanning galvanometer before entering the OCT objective lens and focusing on the biological sample, and the light reflected and scattered by the biological sample is collected again by the OCT objective lens and returned along the original path, and the obtained returned light is the sample light; the sample light and the reference light interfere with each other, and the interference light then returns to the OCT host through the collimator and the optical fiber to be detected by the spectrometer therein, and the spectral signal obtained by the detection is processed to obtain three-dimensional OCT image data representing the distribution of local reflectivity and scattering characteristics of the biological sample;

[0025] The brightfield microscope assembly includes a microscope objective, a tube lens, a low-pass filter and a single-channel camera; a biological sample illuminated by visible light is placed at the front focal plane of the microscope objective, the sample surface is first imaged by the microscope objective to an infinite distance behind the microscope objective, and then the tube lens is used to project the infinite image plane of the microscope objective onto the single-channel camera at the rear focal plane of the tube lens to obtain brightfield microscopic image data of the biological sample.

[0026] The third aspect of the present invention discloses an electronic device comprising a processor and a memory; the processor is used to call a computer program stored in the memory to execute the method for predicting early differentiation of retinal organoids based on multimodal imaging as described in the first aspect of the present invention or any one of its optional solutions.

[0027] The fourth aspect of the present invention discloses a computer-readable storage medium for storing program code, wherein the program code is used to execute the method for predicting early differentiation of retinal organoids based on multimodal imaging as described in the first aspect of the present invention or any one of its optional solutions.

[0028] The present invention has the following beneficial effects:

[0029] (1) The present invention cultivates retinal organoids based on human embryonic stem cells. During the directed differentiation of organoids, thanks to the introduction of three-dimensional multimodal information obtained through three-dimensional OCT images, combined with neural networks, it is possible to capture subtle features that are difficult for humans to distinguish and use them to predict the differentiation fate of the sample, thus accurately predicting the differentiation outcome in the early stages of differentiation. Experimental verification has shown that at the earliest sampling time (day 6 of culture), the classification accuracy has reached 0.815, the sensitivity has reached 0.855, and the specificity is 0.807. At the same time point, the sensitivity of manual discrimination is only 0.091.

[0030] (2) The present invention not only processes image data from a single time point but also predicts image data sampled at multiple time points throughout the differentiation cycle, significantly improving its adaptability to multiple time points. Experimental validation demonstrated an accuracy of 0.872, a specificity of 0.868, and a sensitivity of 0.893 for a comprehensive dataset encompassing multiple time points, broadening the model's application scope.

[0031] (3) The present invention can well predict and characterize the differentiation process of human embryonic stem cells with a long culture time into retinal organoids at an early stage, which enables the production process of retinal organoids cultured from human stem cells to obtain more reliable non-invasive, high-throughput quality control means, thereby promoting the high-quality standardized production of retinal organoids with high clinical relevance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of the multimodal optical imaging module, where the dark red light path represents the near-infrared light path of the OCT component, and the gray light path represents the visible light path of the bright-field microscope component.

[0033] Figure labels: 1: scanning galvanometer; 2: OCT objective lens; 3: spectrometer; 4: dispersion compensation plate; 5: variable aperture; 6: reflector; 7: collimator; 8: optical fiber; 9: OCT host; 10: computer; 11: living cell imaging environment control box; 12: automated dual-axis translation platform; 13: biological sample; 14: microscope objective lens; 15: tube lens; 16: low-pass filter; 17: single-channel camera.

[0034] Figure 2 This is a schematic diagram of a temporal coding neural network, where the ends of the arrows and the intersection of the data streams represent the addition of feature vectors.

[0035] Figure 3 This is a comparison chart of the image data acquisition and annotation process and the human embryonic stem cell culture retinal organoid process.

[0036] Figure 4This is a schematic diagram of data annotation on day 30, including bright-field microscopic images of successfully differentiated retinal organoids and two typical differentiation failure samples. Scale bar: 500 μm. DETAILED DESCRIPTION

[0037] The technical solutions of the present invention will be described in more detail below with reference to specific embodiments and the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0038] The sample in the present invention refers to a biological sample in the process of directed differentiation of a human embryonic stem cell suspension into a retinal organoid, which can also be referred to as a "biological sample" for short.

[0039] The present invention discloses a method for predicting early differentiation of retinal organoids based on multimodal imaging, which mainly includes the following steps:

[0040] Step 1: Receive a bright field microscopic image and an OCT three-dimensional image of a sample collected at a specified sampling time, wherein the sample is a biological sample in the process of directed differentiation of a human embryonic stem cell suspension into a retinal organoid;

[0041] Step 2: Preprocessing the bright field microscopic image and the OCT three-dimensional image respectively and normalizing the sampling time;

[0042] Step 3: Inputting the preprocessed bright field microscopic image and OCT three-dimensional image and the normalized sampling time into the time encoding neural network differentiation prediction model trained by the multimodal neural network to obtain the final differentiation prediction result, which includes differentiation success and differentiation failure;

[0043] Step 4: In the time-coding neural network differentiation prediction model, feature vectors are extracted from bright-field microscopic images and three-dimensional OCT images by their respective corresponding encoders. The feature vectors are added to the encoded features of the sampling time and then input into the corresponding classifier. The classification results output by each classifier are weighted and added to obtain the final differentiation prediction result.

[0044] Furthermore, embodiments of the present invention also provide a retinal organoid early differentiation prediction system that can be used to implement the above-mentioned prediction method. The retinal organoid early differentiation prediction system mainly consists of a multimodal optical imaging module and a time-coded neural network differentiation prediction model. The multimodal optical imaging module serves as the hardware foundation of the system, and the time-coded neural network differentiation prediction model serves as the software component of the system, which can be stored in the computer 10 or in a server or platform connected thereto. A detailed description is as follows:

[0045] The multimodal optical imaging module mainly includes an optical coherence tomography (OCT) component, a bright field microscope component, and a high-throughput automatic scanning stage component.

[0046] The OCT assembly uses commercial spectral domain OCT (SD-OCT), equipped with a 900nm superluminescent diode and an OCT objective lens with a numerical aperture of 0.075, achieving an axial resolution of approximately 3μm and a lateral resolution of 8μm. Specifically, the OCT assembly includes a scanning galvanometer 1, an OCT objective lens 2, a beam splitter 3, a dispersion compensator 4, an iris 5, a reflector 6, a collimator 7, an optical fiber 8, and an OCT main unit 9.

[0047] Broadband, low-coherence near-infrared light is guided from the OCT mainframe 9 via optical fiber 8, collimated by collimator 7, and finally reaches the beamsplitter 3. Half of the incident light is redirected by the beamsplitter 3, then passes through the dispersion compensator 4 and the variable aperture 5, before being reflected by the reflector 6 and returning along the original path. This reflected light is called the reference light. The other half of the incident light passes through the beamsplitter 3, reflects twice by the scanning galvanometer 1, and then enters the OCT objective lens 2. It is focused onto a biological sample 13 placed in a live cell imaging environmental control box 11 on an automated biaxial translation stage 12. Light reflected and scattered by the biological sample 13 is collected by the OCT objective lens 2 and returned along the original path. This returned light is called the sample light. The sample light and the reference light interfere with each other, and the interference light then returns to the OCT mainframe 9 through collimator 7 and optical fiber 8, where it is detected by the spectrometer. The resulting spectral signal is processed by computer 10 to produce an OCT signal representing the local reflectivity and scattering characteristics of the biological sample, i.e., the three-dimensional OCT image of the biological sample required for the time-coded neural network differentiation prediction model.

[0048] The brightfield microscope assembly is an inverted microscope with an infinity-corrected imaging system. Specifically, the brightfield microscope assembly primarily includes a microscope objective 14, a tube lens 15, a low-pass filter 16, and a single-channel camera 17. The numerical aperture of the microscope objective 14 is 0.13, the focal length of the tube lens 15 is 200 mm, and the cutoff wavelength of the low-pass filter 16 is 750 nm.

[0049] A biological sample 13 illuminated by visible light is placed at the front focal plane of a microscope objective 14. The sample surface is first imaged by the microscope objective 14 at an infinite distance behind the microscope objective 14. A tube lens 15 is then used to project the infinite image plane of the microscope objective 14 onto a single-channel camera 17 located at the rear focal plane of the tube lens 15, forming a brightfield microscopic image of the biological sample. A low-pass filter 16 is placed between the tube lens 15 and the single-channel camera 17 to shield the near-infrared light from the OCT component. The brightfield microscope assembly is capable of two-dimensional projection imaging with a lateral resolution of approximately 3μm, producing the brightfield microscopic image of the biological sample required for the time-coded neural network differentiation prediction model.

[0050] It is worth noting that the brightfield microscopy image reflects the overall morphology of the sample, such as the edge morphology, area, and shape. Visual inspection of brightfield microscopy images is the main means for retinal organoid researchers to evaluate the quality of retinal organoid culture. Therefore, this modal image can provide two-dimensional morphological features for the neural network. The OCT image is a three-dimensional point cloud reconstruction of the local reflectivity distribution of the sample. While characterizing the three-dimensional morphological characteristics of the sample, it also reveals the local reflectivity distribution inside the sample, providing three-dimensional information to supplement the two-dimensional brightfield image. In the present invention, the brightfield microscope component and the OCT component complement each other in modality and dimension, jointly realizing multimodal, multi-dimensional, and non-invasive retinal organoid imaging.

[0051] Furthermore, to achieve high-throughput multi-sample image acquisition, the present invention utilizes an automated dual-axis translation platform 12 to carry a 96-well plate used for biological culture, and utilizes a control program to scan samples at different positions on the plate. During imaging, the 96-well plate is placed within a live cell imaging environmental control box 11 to ensure optimal temperature, humidity, and carbon dioxide concentration for the biological samples. The OCT objective lens 2 is coaxially aligned with the microscope objective lens 14, with the automated dual-axis translation platform 12, which houses the biological sample 13, positioned between the two lenses.

[0052] Brightfield microscopic images are common single-channel two-dimensional images, while OCT data are large-scale three-dimensional data. The two need to be reasonably integrated. To this end, the present invention also innovatively designs a time-coding neural network differentiation prediction model to integrate the two modal data of three-dimensional OCT images and brightfield microscopic images to match the data acquisition input of the multimodal optical imaging module, and introduces a time coding module to ensure the network's adaptability to data at each time point.

[0053] Combine Figure 2 As shown in the time-coding neural network differentiation prediction model, the bright field microscopic image and the three-dimensional OCT image are respectively passed through their corresponding encoders to obtain feature vectors, which are then added to the encoded features of the sampling time and input into the corresponding classifier. The classification results of the two imaging modalities are weightedly added to obtain the overall final prediction score used to characterize the final differentiation prediction result.

[0054] The multimodal neural network can be specifically described as:

[0055] z=λ1·z BF +λ2·z OCT

[0056] =λ1·g BF [f BF (x BF )+f time (T after )]+λ2·g OCT [f OCT (x OCT )+f time (T after )]

[0057] In the formula, the variable z BF With z OCT represents the output scores of the bright field microscopy image data stream and the 3D OCT data stream respectively; λ1 and λ2 are modality fusion weights, which are pre-set known parameters; z represents the final prediction score, i.e., the classification score. The classification score is binarized by the threshold to obtain the final differentiation prediction result (i.e., differentiation success and differentiation failure); x BF and x OCT Represent the input bright field microscopy image and 3D OCT image respectively; T after represents the normalized sampling time, that is, the sampling time T is normalized to the interval [0,1]; f represents the encoder, and g represents the classifier.

[0058] The normalization formula is as follows:

[0059]

[0060] Where T represents the sampling time, T∈[6,9,12,…,30].

[0061] In the multimodal neural network, the encoder f is responsible for extracting the spatial features of the image and encoding the temporal information into a high-dimensional feature space; the classifier g is specifically a five-layer multi-layer perceptron classifier, which is mainly used to predict the success or failure of differentiation. In the encoder f, the encoder f of the bright field microscopy image is BF The backbone network adopts the ResNet101 model of the convolutional neural network architecture, and the encoder f OCT The backbone network adopts the channel separation convolutional network CSN50 model, and the time encoder f timeA four-layer multilayer perceptron (MLP) is used. The MLP is a simple, nonlinear feedforward neural network suitable for processing low-dimensional temporal data with high computational efficiency. The ResNet101 model, with its deep residual connection structure, effectively extracts image features and is a widely used and validated convolutional neural network (CNN) architecture in the image processing field. The CSN50 model, a channel-separated CNN, significantly reduces computational complexity compared to a fully connected 3D-CNN, making it ideal for processing complex three-dimensional data such as OCT data and capable of efficiently extracting key features from 3D structures.

[0062] Classifier g includes classifier g for bright field microscopy images BF and g of 3D OCT images OCT , both are five-layer multilayer perceptron classifiers. It should be noted that the multilayer perceptron is used in this application because it has the ability to fully express high-dimensional features and nonlinear mapping modeling, and at the same time has a simple structure and low computational overhead, making it suitable as a classifier.

[0063] It is understandable that in order to obtain the usable weight parameters of the above-mentioned time-coding neural network differentiation prediction model (corresponding to the "weights" of each node of the neural network), four steps are usually required: biological sample image data collection, data annotation, training data preparation, and model training. The details are as follows:

[0064] Image data acquisition: The target is biological samples during the directed differentiation of human embryonic stem cell suspensions into retinal organoids in 96-well plates. The acquisition period is from day 6 to day 30 of culture, with a sampling interval of every three days for a total of nine times, namely day 6, day 9, day 12, day 15, day 18, day 21, day 24, day 27, and day 30. Each sample (well) will undergo a complete nine image acquisitions, with each sampling period sequentially involving OCT 3D scanning and brightfield microscopy image acquisition.

[0065] Understandably, the 30th day is the time point that biologists have conventionally used to observe bright-field microscopic images with the naked eye and use experience to judge the differentiation status. In fact, the characteristics of successful differentiation may be present on any of the days before or after. Therefore, the culture time, sampling interval, sampling time range, etc. can also be adjusted according to actual conditions.

[0066] like Figure 3As shown in the figure, the dataset was collected over 27 days, from the time when the sample morphology changed from embryoid body to neural retina, with images collected every 3 days for a total of 9 times. The time origin (day 0) is when the human embryonic stem cell suspension is seeded into a 96-well plate. From day 0 to day 6, the stem cells proliferate, aggregate into clusters, and initially develop into embryoid bodies. From day 6 to day 18, they are induced to form the neural epithelium and develop towards the neural ectoderm. From day 18 to day 30, the neural epithelium develops into the optic vesicle and ultimately the neural retina, which is considered to have the characteristics of a retinal organoid. After day 30, the neural retina further differentiates into a variety of functional cells, ultimately developing into mature retinal organoids.

[0067] Data annotation: Based on the bright-field microscopic images of the samples collected on day 30, combined with empirical criteria for whether the biological samples in each well have successfully differentiated into retinal organoids, a binary annotation is performed to obtain training labels.

[0068] like Figure 4 As shown in the figure, samples with clear neural retinal features are labeled as successfully differentiated, while samples without clear neural retinal features are labeled as failed differentiation. Neural retinal features refer to bright, sharp, continuous edges on the field microscopy image, with a clear clear layer between the edge and the center. The training labels reflect the ultimate differentiation fate of the biological sample.

[0069] Training data generation: Brightfield microscopy images and 3D OCT images at all nine time points were preprocessed. The preprocessed images and training labels constituted a training dataset, which included multiple sets of paired training data. Each set of paired training data included a 3D OCT image, a brightfield microscopy image, the sampling time (T∈[6,9,12,…,30]) for each image, and the corresponding training label.

[0070] Image preprocessing mainly includes image cropping, resizing, image enhancement, downsampling and other operations. Among them, image cropping is mainly to remove invalid information in non-sample areas; resizing can reduce computational overhead; image enhancement can use histogram equalization, contrast-limited histogram normalization and other technologies to improve image contrast.

[0071] Model training: The multimodal neural network is trained using training data to obtain the required time-encoded neural network differentiation prediction model.

[0072] During the training process, cross entropy loss is used as the loss function for supervised training. The loss function expression is as follows:

[0073] L=―[z·log(z label )+(1―z)·log(1―zlabel )]

[0074] Where z is the classification score output by the network (z∈(0,1)), z label is the true label (z label {0,1}).

[0075] The neural network is optimized using the Adam (Adaptive Moment Estimation) optimizer, focusing on the weight parameters of each node in the neural network structure. Optimization is terminated after 200 rounds of iterative training, resulting in the desired time-coded neural network differentiation prediction model. Training can also be terminated when the loss function converges, a specific setting that can be set as needed.

[0076] Based on the trained time-coding neural network differentiation prediction model, a multimodal optical imaging module was used to capture brightfield microscopy and 3D optical coherence tomography (OCT) images on day X (X∈[6,30)) of human embryonic stem cell retinal organoid culture. The pre-processed brightfield microscopy and 3D OCT images, along with the sampling time T, were then input into the time-coding neural network differentiation prediction model to determine whether the sample successfully differentiated. When X≤18, this time point is much earlier than traditional methods can determine successful differentiation, enabling prediction of the sample's differentiation fate early in the differentiation process.

[0077] The main purpose of this invention is to predict the differentiation fate of a single suspension cultured cell cluster at multiple time points between day 6 and day 30 during the culture process of human embryonic stem cells into retinal organoids. The specific implementation steps are as follows:

[0078] (1) The temperature of the living cell imaging environment control box 11 was adjusted to be stable at 37° C., and a gas premixed with 5% carbon dioxide and 95% air was introduced.

[0079] (2) On the Xth day of culture (X∈[6,30)), the 96-well plate containing the sample is placed in a live cell imaging environment control box 11 fixed on an automated dual-axis translation platform 12, and the position of the target sample is moved to the co-optical axis position of the OCT objective lens 2 and the microscope objective lens 14 using the automated dual-axis translation platform 12.

[0080] (3) Finely adjust the vertical position of the microscope objective lens 14 of the bright field microscope so that the cell cluster sample is within the depth of field of the focal plane of the objective lens, and record the image through the single-channel camera 17 to obtain a bright field microscopic image.

[0081] (4) Fine-tune the overall vertical height of the OCT assembly so that the cell cluster sample is within the depth of field of the focal plane of the OCT objective lens 2. The OCT assembly scans and images through the scanning galvanometer 1 to obtain a three-dimensional OCT image.

[0082] (5) The pixel size of the obtained single-channel bright field microscopy image is 1920*1200. The part containing the sample signal is cropped to obtain a 1200*1200 sub-image and scaled to 512*512; then contrast-limited adaptive histogram normalization is applied to enhance the image, with a contrast limit of 2 and a grid size of 10*10; the enhanced image is input into an image segmentation network (e.g., U-Net network, SegmentAnythingModel network, etc.) or the ROI area containing the sample body is manually segmented, and the grayscale value of the remaining background area is set to 0.

[0083] (6) The pixel size of the obtained OCT three-dimensional image is 400*400*376. The part containing the sample signal is cropped to obtain a sub-image of 200*200*356; the sub-image is uniformly downsampled along the first dimension to obtain a downsampled three-dimensional image of 11*200*356.

[0084] (7) The sampling time X is normalized to [0,1], and the normalization formula is:

[0085] (8) Input the data obtained from steps (5), (6), and (7) into the trained temporal coding neural network differentiation prediction model, with modal fusion weights λ1 = 0.7, λ2 = 0.3, and calculate the classification score. For example, the classification score calculated for a certain sample is 0.92.

[0086] (9) The classification scores are binarized by thresholding to obtain the final differentiation prediction result. The classification threshold is set to 0.8. Samples with a classification score greater than or equal to 0.8 are considered to be able to differentiate successfully, otherwise they are considered to fail differentiation. For example, the sample classification score in step (8) is 0.92. Since 0.92 is greater than 0.8, it is considered to be able to differentiate successfully.

[0087] Based on the above process, the neural network model was tested on 3060 paired data sets consisting of 340 samples (285 of which were successfully differentiated) at 9 time points. The overall accuracy, specificity and sensitivity of the neural network model were 0.872, 0.868 and 0.893.

[0088] As can be understood, as a differentiation prediction evaluation metric, sensitivity is defined as the ratio of correctly predicted differentiation failure samples to the actual number of differentiation failure samples, indicating the model's ability to correctly predict positive samples (differentiation failures); specificity is defined as the ratio of correctly predicted differentiation success samples to the actual number of successful differentiation samples, indicating the model's ability to correctly predict negative samples (differentiation successes). High sensitivity means the model can effectively predict samples that will fail differentiation, reducing missed detections; high specificity means the model can effectively exclude samples that will successfully differentiate, reducing false positives.

[0089] The classification accuracy of the sample data of the present invention on the 6th day of culture has reached 0.815, the sensitivity has reached 0.855, and the specificity has reached 0.807. The specific performance parameters at the 9 time points are shown in Table 1:

[0090] Table 1

[0091]

[0092] Furthermore, an embodiment of the present invention also discloses an electronic device, which mainly includes a processor and a memory. The processor is mainly used to call the computer program stored in the memory and execute the retinal organoid early differentiation prediction method based on multimodal imaging provided in the above embodiment.

[0093] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the method for predicting early differentiation of retinal organoids based on multimodal imaging provided by the present invention.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the program code in the electronic device and computer-readable storage medium described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0095] Finally, it should be noted that although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and not restrictive. Under the guidance of this specification, those skilled in the art can also make many forms without departing from the scope of protection of the claims of the present invention, and all of these forms are protected by the present invention.

Claims

1. A method for predicting early differentiation of retinal organoids based on multimodal imaging, characterized in that: include: Receiving a bright field microscopic image and an optical coherence tomography (OCT) three-dimensional image of a sample collected at a specified sampling time, wherein the sample is a biological sample in the process of directed differentiation of a human embryonic stem cell suspension into a retinal organoid; Preprocessing the bright field microscopic image and the OCT three-dimensional image respectively and normalizing the sampling time; Inputting the preprocessed bright field microscopic image and OCT three-dimensional image and the normalized sampling time into a time coding neural network differentiation prediction model trained by a multimodal neural network to obtain a final differentiation prediction result, wherein the final differentiation prediction result includes differentiation success and differentiation failure; In the time-coding neural network differentiation prediction model, feature vectors are extracted from bright-field microscopic images and three-dimensional OCT images by their respective corresponding encoders. The feature vectors are added to the encoded features of the sampling time and then input into the corresponding classifier. The classification results output by each classifier are weightedly added to obtain the final differentiation prediction result.

2. The method for predicting early differentiation of retinal organoids according to claim 1, wherein: The multimodal neural network is constructed as follows: z=λ1·g BF [f BF (x BF )+f time (T)]+λ2·g OCT [f OCT (x COT )+f time (T)] Where z is the classification score, which is used to represent the final differentiation prediction result. The classification score is binarized by threshold to obtain the final differentiation prediction result. λ1 and λ2 are modality fusion weights; x BF and x OCT Represent the input bright field microscopy image and 3D OCT image respectively; T after represents the normalized sampling time; f BF Encoder representing bright field microscopy image; f OCT The encoder representing the 3D OCT image, f time represents the time encoder; g BF represents a classifier for bright field microscopy images; g OCT Representation of a classifier for 3D OCT images.

3. The method for predicting early differentiation of retinal organoids according to claim 2, wherein: The time encoder f time is a multi-layer perceptron; the encoder f BF The backbone network adopts the ResNet101 model of the convolutional neural network architecture; the encoder f OCT The backbone network adopts the channel separation convolutional network CSN50 model; the classifier g BF and g OCT Both are multi-layer perceptrons.

4. The method for predicting early differentiation of retinal organoids according to any one of claims 1 to 3, wherein: The training process of the multimodal neural network includes the following steps: Bright field microscopic images and OCT 3D images of the sample are acquired multiple times sequentially within a specified sampling time range; Based on the empirical criteria of neural retinal characteristics, the samples are binary labeled to determine whether they have successfully differentiated into retinal organoids to obtain training labels; Preprocessing the sample images collected within a specified sampling time range and normalizing the sampling time; The preprocessed sample images and training labels constitute a training data set. Each set of paired training data in the data set includes the corresponding three-dimensional OCT images and bright-field microscopy images, the sampling time of the set of images, and the training labels corresponding to the set of images. The training data set is used to train the multimodal neural network and obtain the required time-coded neural network differentiation prediction model.

5. The method for predicting early differentiation of retinal organoids according to claim 4, wherein: The designated sampling time range includes from the 6th day of culture to the 30th day of culture.

6. The method for predicting early differentiation of retinal organoids according to any one of claims 1 to 3, wherein: During the training of the multimodal neural network, cross entropy loss is used as the loss function for supervised training; The loss function expression is as follows: L=―[z·log(z label )+(1―z)·log(1―z label )] Where z is the classification score output by the network (z∈(0,1)), z label is the true label (z label ∈{0,1}); During the training process, the Adam optimizer is used to optimize the multimodal neural network. The optimization is terminated when the preset conditions are met to obtain the required time-coded neural network differentiation prediction model.

7. The method for predicting early differentiation of retinal organoids according to any one of claims 1 to 3, wherein: The bright field microscopic image and the OCT three-dimensional image are preprocessed separately, specifically including: cropping and scaling the bright field microscopic image to a first set size, and then performing image enhancement processing; The OCT 3D image is cropped to a second set size and then uniformly downsampled along the specified dimension.

8. A retinal organoid early differentiation prediction system based on multimodal imaging, characterized by: Includes optical coherence tomography components and bright field microscope components; The optical coherence tomography assembly includes a scanning galvanometer, an OCT objective lens, a beam splitter, a dispersion compensation plate, an iris, a reflected mirror, a collimator, an optical fiber and an OCT host; Broadband low-coherence near-infrared light is guided out of the OCT mainframe through an optical fiber, collimated by a collimator, and then reaches the beam splitter. Half of the incident light is redirected after passing through the beam splitter, and then passes through a dispersion compensator and a variable aperture in sequence before being reflected by a reflector and returning along the original path. The resulting reflected light is the reference light. The other half of the incident light passes through the beam splitter and is reflected twice by a scanning galvanometer before entering the OCT objective lens and focusing on the biological sample. The light reflected and scattered by the biological sample is collected again by the OCT objective lens and returned along the original path. The resulting returned light is the sample light. The sample light and the reference light interfere with each other, and the interference light then passes through the collimator and optical fiber back to the OCT mainframe to be detected by the spectrometer therein. The detected spectral signal is processed to obtain three-dimensional OCT image data representing the distribution of the local reflectivity and scattering characteristics of the biological sample. The bright field microscope assembly includes a microscope objective, a tube lens, a low-pass filter and a single-channel camera; A biological sample illuminated by visible light is placed at the front focal plane of the microscope objective. The sample surface is first imaged by the microscope objective to an infinite distance behind the microscope objective. Then, a tube lens is used to project the infinite image plane of the microscope objective onto a single-channel camera at the rear focal plane of the tube lens to obtain bright-field microscopic image data of the biological sample.

9. An electronic device, characterized in that: It comprises a processor and a memory; the processor is used to call the computer program stored in the memory to execute the method for predicting early differentiation of retinal organoids based on multimodal imaging according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the method for predicting early differentiation of retinal organoids based on multimodal imaging according to any one of claims 1 to 7.

Images