Cell imaging method and apparatus based on deep learning

Abstract

This invention relates to the field of deep learning, disclosing a deep learning-based cell imaging method and apparatus. The method includes: reconstructing a fundus image using a trained pre-processor neural network model to obtain a reconstructed fundus image; registering the reconstructed image and the cell imaging image to obtain a registered reconstructed image and a registered cell imaging image; fusing the registered reconstructed image and the registered cell imaging image to obtain a fused image; using the reconstructed image as input data for a pre-trained post-processor neural network model, and using the cell imaging image and the fused image as target data for the pre-trained post-processor neural network model, thereby training the pre-trained post-processor neural network model using the input data and target data; and performing cell imaging on a current patient's fundus image using the trained pre-processor neural network model and the trained post-processor neural network model to obtain cell imaging results. This invention can meet clinical needs through cell imaging.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and more particularly to a cell imaging method and apparatus based on deep learning. Background Technology

[0002] During the development of glaucoma, ganglion cells are lost every year. Even in subjects without eye disease, a small fraction of these cells die each year as part of normal aging (0.19–0.72% / year). Therefore, it is clinically necessary to differentiate between ganglion cell loss caused by disease and ganglion cell loss caused by aging as early as possible, and to intervene with appropriate treatment to minimize cell death.

[0003] Quantitative characterization of retinal ganglion cells (GCs) at the cellular level is an important biomarker that can improve the diagnosis and monitoring of neurodegenerative diseases such as glaucoma, Parkinson's disease, and Alzheimer's disease. However, due to limited resolution, individual GCs cannot be visualized by commonly used ophthalmic imaging systems. Even with techniques such as optical coherence tomography (OCT), the evaluation of individual GCs is limited to total slice thickness analysis. Adaptive optical OCT (AO-OCT) can perform in vivo imaging of individual retinal GCs, but its acquisition area is small, and the process is time-consuming, making it difficult to apply clinically. Therefore, there is an urgent need for a cell imaging protocol that can meet clinical needs. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a deep learning-based cell imaging method and apparatus that can meet clinical needs through cell imaging.

[0005] In a first aspect, the present invention provides a cell imaging method based on deep learning, comprising:

[0006] Collect fundus images of historical patients and collect cell imaging maps of the eye corresponding to the fundus images;

[0007] The fundus image is reconstructed using a trained pre-trained neural network model to obtain a reconstructed image of the fundus image;

[0008] The reconstructed image and the cell imaging image are registered to obtain a registered reconstructed image and a registered cell imaging image. The registered reconstructed image and the registered cell imaging image are then fused to obtain a fused image.

[0009] The reconstructed image is used as input data for a pre-trained post-neural network model, and the cell imaging image and the fused image are used as target data for the pre-trained post-neural network model. The pre-trained post-neural network model is trained using the input data and the target data to obtain a trained post-neural network model.

[0010] The trained pre-neural network model and the trained post-neural network model are used to perform cell imaging on the current patient's fundus image to obtain cell imaging results.

[0011] In one possible implementation of the first aspect, before performing image reconstruction on the fundus image using a trained pre-neural network model to obtain the reconstructed image of the fundus image, the method further includes:

[0012] Acquire training fundus images, and use the generative network in the pre-trained pre-neighborhood model to reconstruct the resolution of the training fundus images to obtain pseudo-high resolution images;

[0013] In the discriminant network of the pre-trained pre-neural network model, a feature map of the pseudo-high resolution image is constructed to obtain a pseudo feature map, and a feature map of the true high resolution image corresponding to the pseudo-high resolution image is constructed to obtain a true feature map.

[0014] Based on the pseudo-feature map and the real feature map, the loss value of the generator network is calculated using the following formula to obtain the generator network loss:

[0015]

[0016] Where H(p,q) represents the generator network loss, p(x) i ) represents the true feature map, q(x) i ) represents the pseudo-feature map, i represents the feature index in the feature map, and x represents the pseudo-feature map. i Represents the features in the feature map;

[0017] Based on the pseudo-feature map and the real feature map, the loss value of the discriminant network is calculated using the following formula to obtain the discriminant network loss:

[0018] loss=∣yf|+H(p,q)

[0019] Where loss represents the loss of the discriminative network, H(p,q) represents the loss of the generative network, y represents the true feature map, and f represents the pseudo feature map;

[0020] When both the generation network loss and the discrimination network loss are greater than the preset loss, the network weights of the pre-trained pre-neural network model are updated to obtain the pre-neural network model with updated network weights.

[0021] When the generation network loss and discrimination network loss of the pre-neighborhood model after updating the network weights are both no greater than the preset loss, the trained pre-neighborhood model is obtained.

[0022] In one possible implementation of the first aspect, the step of reconstructing the resolution of the training fundus image using the generative network in the pre-trained pre-trained neural network model to obtain a pseudo-high-resolution image includes:

[0023] The shallow features of the training fundus image are extracted using the first convolutional layer in the generative network to obtain a shallow feature map;

[0024] The shallow feature map is corrected by using the residual layer in the generator network to obtain a corrected feature map.

[0025] The deep features of the corrected feature map are extracted using the residual blocks in the generator network to obtain the deep feature map;

[0026] The deep feature map is processed by feature convolution using the second convolutional layer in the generative network to obtain the second convolutional map.

[0027] The corrected feature map and the second convolutional map are superimposed to obtain the superimposed feature map;

[0028] The resolution of the superimposed feature map is reconstructed to obtain the pseudo-high-resolution image.

[0029] In one possible implementation of the first aspect, constructing a feature map of the pseudo-high-resolution image in the discriminative network of the pre-trained pre-neural network model to obtain the pseudo-feature map includes:

[0030] Feature extraction is performed on the pseudo-high-resolution image to obtain a high-resolution feature map;

[0031] The high-resolution feature map is dimensionally adjusted to obtain a dimensionally adjusted feature map;

[0032] The dimension-adjusted feature map is linearly mapped to obtain a linearly mapped feature map.

[0033] Calculate the classification probability of the linear mapping feature map, and use the image constructed from the classification probabilities as the pseudo feature map.

[0034] In one possible implementation of the first aspect, fusing the registered reconstructed image with the registered cell imaging map to obtain a fused image includes:

[0035] Image registration is performed between multiple registered cell imaging maps in the registered cell imaging map to obtain a secondary registered cell imaging map;

[0036] Multiple secondary registration cell imaging maps in the secondary registration cell imaging map are fused to obtain a fused cell imaging map;

[0037] The fused cell imaging image is obtained by fusing the fused cell imaging image with the registered and reconstructed image.

[0038] In one possible implementation of the first aspect, fusing the fused cell imaging map with the registered reconstructed image to obtain the fused image includes:

[0039] Multi-scale wavelet decomposition is performed on the fused cell imaging image and the registered and reconstructed image respectively to obtain the low-frequency and high-frequency components of the fused cell imaging image and the low-frequency and high-frequency components of the registered and reconstructed image.

[0040] The low-frequency components of the fused cell imaging image and the low-frequency components of the registered and reconstructed image are fused together to obtain a low-frequency fused image;

[0041] The high-frequency components of the fused cell imaging image and the high-frequency components of the registered and reconstructed image are fused together to obtain a high-frequency fused image;

[0042] The low-frequency fused image and the high-frequency fused image are fused to obtain the fused image.

[0043] In one possible implementation of the first aspect, the step of training the pre-trained post-neural network model using the input data and the target data to obtain a trained post-neural network model includes:

[0044] Construct the diffusion steps of the target data and its Gaussian noise image;

[0045] Based on the number of diffusion steps, the target data is forward diffused to obtain a forward diffusion map;

[0046] The input data is superimposed with the forward diffusion map to obtain an input-diffusion overlay map;

[0047] Based on the number of diffusion steps, the input-diffusion overlay image is subjected to image denoising processing to obtain a denoised image;

[0048] The mean squared loss between the denoised image and the Gaussian noise image is calculated using the following formula:

[0049]

[0050] Where MSE represents the mean squared loss, y j This represents the Gaussian noise image. The image is described as being denoised, where j represents the index of a pixel in the image and N represents the total number of pixels in the image.

[0051] The pre-trained post-neural network model is trained using the mean squared loss to obtain a trained post-neural network model.

[0052] In one possible implementation of the first aspect, constructing the diffusion steps of the target data and its Gaussian noise image includes:

[0053] Select the diffusion steps from the preset number of training iterations;

[0054] The diffusion steps are converted into feature vectors; a Gaussian noise image of the feature vectors is constructed.

[0055] In one possible implementation of the first aspect, the step of performing image denoising processing on the input-diffusion overlay image based on the number of diffusion steps to obtain a denoised image includes:

[0056] Obtain the feature vector and pre-trained post-neural network model corresponding to the number of diffusion steps;

[0057] The feature vector and the input data in the input-diffusion overlay are used as the supervision vector of the pre-trained post-neural network model;

[0058] Based on the supervision vector, the pre-trained post-neural network model is used to perform image denoising on the input-diffusion overlay image to obtain the denoised image.

[0059] Secondly, the present invention provides a cell imaging device based on deep learning, the device comprising:

[0060] The image acquisition module is used to acquire fundus images of historical patients and to acquire cell imaging maps of the eye corresponding to the fundus images;

[0061] The image reconstruction module is used to reconstruct the fundus image using a trained pre-trained neural network model to obtain a reconstructed image of the fundus image.

[0062] An image fusion module is used to register the reconstructed image and the cell imaging image to obtain a registered reconstructed image and a registered cell imaging image, and to fuse the registered reconstructed image and the registered cell imaging image to obtain a fused image;

[0063] The model training module is used to take the reconstructed image as input data for a pre-trained post-neural network model, and take the cell imaging image and the fused image as target data for the pre-trained post-neural network model, so as to train the pre-trained post-neural network model with the input data and the target data to obtain a trained post-neural network model.

[0064] The cell imaging module is used to perform cell imaging on the current patient's fundus image using the trained pre-neural network model and the trained post-neural network model to obtain cell imaging results.

[0065] Compared with existing technologies, the technical principles and beneficial effects of this solution are as follows:

[0066] This invention first acquires fundus images from historical patients to train a neural network model using a corresponding cell-containing fundus image, enabling the model to meet clinical needs. Further, this invention acquires a cell imaging map of the eye corresponding to the fundus image, using this image as output data in the neural network model. Further, this invention uses a trained pre-processor neural network model to reconstruct the fundus image, converting a low-resolution image into a high-resolution image consistent with the cell imaging map. Further, this invention fuses the registered reconstructed image with the registered cell imaging map, using the fused image as the expected output data of the reconstructed image in the neural network model. By training the neural network model, it can output a corresponding OCT image with added cellular structures based on the input reconstructed image. Further, this invention trains a pre-trained post-processor neural network model to output a corresponding cell imaging map and an OCT image with added cellular structures based on the input fundus image, thus enabling its application in clinical medicine. Therefore, the cell imaging method and device based on deep learning proposed in this embodiment of the invention can meet clinical needs through cell imaging. Attached Figure Description

[0067] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a schematic flowchart of a cell imaging method based on deep learning, provided in an embodiment of the present invention.

[0070] Figure 2 As shown in one embodiment of the present invention Figure 1 A schematic diagram of the structure of a generative network for a deep learning-based cell imaging method is provided.

[0071] Figure 3 As shown in one embodiment of the present invention Figure 1 A schematic diagram of the discriminant network structure for a deep learning-based cell imaging method is provided.

[0072] Figure 4 This is a schematic diagram of a deep learning-based cell imaging device according to an embodiment of the present invention.

[0073] Figure 5 This is a schematic diagram of the internal structure of an electronic device that implements a deep learning-based cell imaging method according to an embodiment of the present invention. Detailed Implementation

[0074] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0075] This invention provides a deep learning-based cell imaging method. The execution entity of this deep learning-based cell imaging method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this invention: a server, a terminal, etc. In other words, the deep learning-based cell imaging method can be executed by software or hardware installed on a terminal device or a server device. The software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0076] See Figure 1The diagram shown is a flowchart illustrating a deep learning-based cell imaging method according to an embodiment of the present invention. Figure 1 The deep learning-based cell imaging methods described in the document include:

[0077] S1. Collect fundus images of historical patients and collect cell imaging images of the eye corresponding to the fundus images.

[0078] This invention collects fundus images from historical patients to train a neural network model using the fundus image containing cells corresponding to the generated fundus image, so that the neural network model can meet clinical needs.

[0079] The fundus image refers to an OCT fundus image acquired using optical coherence tomography (OCT) technology.

[0080] Furthermore, in this embodiment of the invention, a cell imaging map of the eye corresponding to the fundus image is acquired, which is then used as the output data of the fundus image in the neural network model.

[0081] The cell imaging map refers to the cell imaging map of the region corresponding to the fundus image acquired using AO-OCT; AO-OCT refers to adaptive optical OCT technology (AO-OCT).

[0082] S2. The fundus image is reconstructed using the trained pre-neural network model to obtain the reconstructed image of the fundus image.

[0083] In this embodiment of the invention, a pre-trained neural network model is used to reconstruct the fundus image, so as to reconstruct the low-resolution fundus image into a high-resolution image consistent with the cell imaging image.

[0084] The reconstructed image refers to a high-resolution fundus image.

[0085] In one embodiment of the present invention, before reconstructing the fundus image using a pre-trained pre-neural network model to obtain the reconstructed fundus image, the method further includes: acquiring a training fundus image; reconstructing the resolution of the training fundus image using the generative network in the pre-trained pre-neural network model to obtain a pseudo-high-resolution image; constructing a feature map of the pseudo-high-resolution image in the discriminative network of the pre-trained pre-neural network model to obtain a pseudo-feature map, and constructing a feature map of the true high-resolution image corresponding to the pseudo-high-resolution image to obtain a true feature map; and calculating the loss value of the generative network based on the pseudo-feature map and the true feature map using the following formula to obtain the generative network loss:

[0086]

[0087] Where H(p,q) represents the generator network loss, p(x) i ) represents the true feature map, q(x) i ) represents the pseudo-feature map, i represents the feature index in the feature map, and x represents the pseudo-feature map. i Represents the features in the feature map;

[0088] Based on the pseudo-feature map and the real feature map, the loss value of the discriminant network is calculated using the following formula to obtain the discriminant network loss:

[0089] loss=∣yf|+H(p,q)

[0090] Where loss represents the loss of the discriminative network, H(p,q) represents the loss of the generative network, y represents the true feature map, and f represents the pseudo feature map;

[0091] When both the generation network loss and the discrimination network loss are greater than the preset loss, the network weights of the pre-trained pre-neural network model are updated to obtain the pre-neural network model with updated network weights; when both the generation network loss and the discrimination network loss of the pre-neural network model with updated network weights are not greater than the preset loss, the trained pre-neural network model is obtained.

[0092] Among them, the cross-entropy loss function has the good property that "the weights are updated quickly when the error is large and the weights are updated slowly when the error is small"; the absolute value loss function is used for datasets that are not significantly affected by outliers.

[0093] In another embodiment of the present invention, the step of using the generative network in the pre-trained pre-trained neural network model to reconstruct the resolution of the training fundus image to obtain a pseudo-high-resolution image includes: extracting shallow features of the training fundus image using a first convolutional layer in the generative network to obtain a shallow feature map; performing parameter correction on the shallow feature map using a residual layer in the generative network to obtain a corrected feature map; extracting deep features of the corrected feature map using a residual block in the generative network to obtain a deep feature map; performing feature convolution processing on the deep feature map using a second convolutional layer in the generative network to obtain a second convolutional map; superimposing the corrected feature map and the second convolutional map to obtain a superimposed feature map; and reconstructing the resolution of the superimposed feature map to obtain the pseudo-high-resolution image.

[0094] The first convolutional layer refers to a convolutional layer with 64 channels and a kernel size of 9×9; the residual layer refers to a linear unit (PReLU) used for parameter correction; the residual block includes 8 residual blocks, each residual block includes two convolutional layers, a residual layer sandwiched between the two convolutional layers, and a summation sign of the last layer; the second convolutional layer refers to a convolutional layer with 64 channels and 3 convolutional layers with a kernel size of 3×3.

[0095] Optionally, the process of extracting deep features from the corrected feature map using the residual blocks in the generator network to obtain the deep feature map is as follows: In each residual block, after the feature map is output through two convolutional layers and a residual layer sandwiched between the two convolutional layers, the output feature map needs to be superimposed with the input feature map input to this residual block, and the superposition result is then used as the input feature map of the next layer's residual block.

[0096] Optionally, the process of reconstructing the resolution of the superimposed feature map to obtain the pseudo-high-resolution image is implemented through a third convolutional layer, which refers to a convolutional layer with 1 channel and a kernel size of 9×9.

[0097] See Figure 2 As shown, this is one embodiment of the present invention. Figure 1 A schematic diagram of the generative network structure for a deep learning-based cell imaging method is provided. Figure 2 In this context, G network represents a generator network.

[0098] In another embodiment of the present invention, the step of constructing a feature map of the pseudo-high-resolution image in the discriminant network of the pre-trained pre-neural network model to obtain a pseudo-feature map includes: extracting features from the pseudo-high-resolution image to obtain a high-resolution feature map; adjusting the dimensions of the high-resolution feature map to obtain a dimension-adjusted feature map; performing a linear mapping on the dimension-adjusted feature map to obtain a linearly mapped feature map; calculating the classification probability of the linearly mapped feature map; and using the image composed of the classification probabilities as the pseudo-feature map.

[0099] Optionally, the process of extracting features from the pseudo-high-resolution image to obtain a high-resolution feature map is as follows: the pseudo-high-resolution image is extracted using the convolutional layer, residual layer, and normalization layer in the discriminant network.

[0100] Optionally, the process of adjusting the dimensions of the high-resolution feature map to obtain a dimension-adjusted feature map is as follows: the high-resolution feature map is dimension-adjusted through a dense layer in the discriminant network.

[0101] Optionally, the process of linearly mapping the dimension-adjusted feature map to obtain a linearly mapped feature map is as follows: the dimension-adjusted feature map is linearly mapped using the residual layer (LeakyRelu) in the discriminant network.

[0102] Optionally, the process of calculating the classification probability of the linear mapping feature map is implemented through the activation function (sigmond) in the discriminant network.

[0103] See Figure 3 As shown, this is one embodiment of the present invention. Figure 1 A schematic diagram of the discriminant network structure for a deep learning-based cell imaging method is provided. Figure 3 In this context, D network represents the discriminant network.

[0104] In another embodiment of the present invention, the principle of constructing the feature map of the true high-resolution image corresponding to the pseudo high-resolution image to obtain the true feature map is similar to the principle of constructing the feature map of the pseudo high-resolution image to obtain the pseudo feature map in the discriminant network of the pre-trained pre-neural network model, and will not be elaborated further here.

[0105] Optionally, the process of updating the network weights of the pre-trained neural network model when both the loss of the generator network and the loss of the discriminator network are greater than a preset loss, to obtain the pre-trained neural network model with updated network weights, is as follows: During the training process, one of the generator network and the discriminator network needs to be fixed, while the network state of the other network is updated. This process is repeated alternately. Throughout the training process, both networks strive to optimize their own networks, thus forming a competitive confrontation, until both sides reach a dynamic equilibrium. At this point, the data trained by the generator network is almost identical to the distribution of real data, and the discriminator network can no longer distinguish between true and false data. The number of iterations is in the range of 1,000 to 10,000.

[0106] Optionally, the process of reconstructing the fundus image using the trained pre-neural network model to obtain the reconstructed fundus image is achieved by using the generative network in the trained pre-neural network model.

[0107] S3. Perform image registration on the reconstructed image and the cell imaging image to obtain a registered reconstructed image and a registered cell imaging image, and fuse the registered reconstructed image and the registered cell imaging image to obtain a fused image.

[0108] Optionally, the process of image registration between the reconstructed image and the cell imaging image to obtain a registered reconstructed image and a registered cell imaging image is as follows: extract the feature domains of the reconstructed image and the cell imaging image respectively; based on the feature point distance between the feature domains of the reconstructed image and the cell imaging image, find feature points in the feature domain of the reconstructed image that match the feature domain of the cell imaging image to obtain mutually matching feature point pairs; and use homography transformation to perform rotation, translation, and other operations on the feature domain of the reconstructed image so that the coordinate system of the feature domain of the reconstructed image can be kept consistent with the coordinate system of the feature domain of the cell imaging image.

[0109] Furthermore, in this embodiment of the invention, the registered reconstructed image and the registered cell imaging image are fused together to use the fused image as the expected output data of the reconstructed image in the neural network model. By training the neural network model, the neural network model can output a corresponding OCT image with added cell structures based on the input reconstructed image.

[0110] The fused image refers to an OCT fundus image with cell imaging.

[0111] In one embodiment of the present invention, fusing the registered reconstructed image and the registered cell imaging image to obtain a fused image includes: performing image registration between multiple registered cell imaging images in the registered cell imaging image to obtain a secondary registered cell imaging image; performing image fusion on multiple secondary registered cell imaging images in the secondary registered cell imaging image to obtain a fused cell imaging image; and fusing the fused cell imaging image and the registered reconstructed image to obtain the fused image.

[0112] Optionally, the process of performing image registration between multiple registered cell imaging images in the registered cell imaging image to obtain a secondary registered cell imaging image is as follows: select one of the registered cell imaging images as a reference frame, and perform image registration on the other images. The registration method can be a rigid or non-rigid method, such as template matching, phase cross-correlation, affine, BSpline and other registration methods.

[0113] Optionally, the process of fusing multiple secondary registration cell imaging maps in the secondary registration cell imaging map to obtain a fused cell imaging map is as follows: multi-frame image fusion is performed on the registered image, which can be performed by direct averaging or bilateral weighting.

[0114] In another embodiment of the present invention, fusing the fused cell imaging image and the registered and reconstructed image to obtain the fused image includes: performing multi-scale wavelet decomposition on the fused cell imaging image and the registered and reconstructed image respectively to obtain low-frequency and high-frequency components of the fused cell imaging image and the registered and reconstructed image; performing component fusion on the low-frequency components of the fused cell imaging image and the registered and reconstructed image to obtain a low-frequency fused image; performing component fusion on the high-frequency components of the fused cell imaging image and the registered and reconstructed image to obtain a high-frequency fused image; and fusing the low-frequency fused image and the high-frequency fused image to obtain the fused image.

[0115] Optionally, the process of performing component fusion of the low-frequency components of the fused cell imaging image and the low-frequency components of the registered and reconstructed image to obtain a low-frequency fused image is achieved through low-frequency fusion weighting; the process of performing component fusion of the high-frequency components of the fused cell imaging image and the high-frequency components of the registered and reconstructed image to obtain a high-frequency fused image is achieved through high-frequency fusion weighting.

[0116] S4. The reconstructed image is used as input data for a pre-trained post-neural network model, and the cell imaging image and the fused image are used as target data for the pre-trained post-neural network model. The pre-trained post-neural network model is trained using the input data and the target data to obtain a trained post-neural network model.

[0117] This invention provides an embodiment of the invention that trains a pre-trained post-neural network model to enable the post-neural network model to output a corresponding cell imaging map and an OCT image with added cell structures based on the input fundus image, thereby applying it to clinical medicine.

[0118] In one embodiment of the present invention, the step of training the pre-trained post-neural network model using the input data and the target data to obtain a trained post-neural network model includes: constructing the diffusion step count and its Gaussian noise image of the target data; performing forward diffusion on the target data based on the diffusion step count to obtain a forward diffusion map; overlaying the input data and the forward diffusion map to obtain an input-diffusion overlay map; performing image denoising processing on the input-diffusion overlay map based on the diffusion step count to obtain a denoised image; and calculating the mean squared loss between the denoised image and the Gaussian noise image using the following formula:

[0119]

[0120] Where MSE represents the mean squared loss, y j This represents the Gaussian noise image. The image is described as being denoised, where j represents the index of a pixel in the image and N represents the total number of pixels in the image.

[0121] The pre-trained post-neural network model is trained using the mean squared loss to obtain a trained post-neural network model.

[0122] The mean squared loss refers to the mean squared error.

[0123] In another embodiment of the present invention, constructing the diffusion steps and Gaussian noise image of the target data includes: selecting the diffusion steps from a preset number of training iterations; converting the diffusion steps into a feature vector; and constructing a Gaussian noise image of the feature vector.

[0124] Optionally, the process of selecting the diffusion step number from the preset training number is as follows: during training, the step number n is randomly selected from 1 to T (T is 100 or more), indicating that n iterations of training are required; the process of converting the diffusion step number into a feature vector is as follows: the diffusion step number n is converted into a feature vector t through a linear layer; the process of constructing a Gaussian noise image of the feature vector is as follows: a Gaussian noise image e of a feature vector t is randomly generated.

[0125] In another embodiment of the present invention, the step of performing image denoising processing on the input-diffusion overlay image based on the diffusion step number to obtain a denoised image includes: obtaining a feature vector corresponding to the diffusion step number and a pre-trained post-neural network model; using the feature vector and the input data in the input-diffusion overlay image as the supervision vector of the pre-trained post-neural network model; and performing image denoising processing on the input-diffusion overlay image using the pre-trained post-neural network model based on the supervision vector to obtain the denoised image.

[0126] Optionally, the t feature vector and the input data in the input-diffusion overlay are added to the encoder network as supervision vectors for each layer of the encoder in the post-neural network model, and the encoder-decoder structure of the post-neural network model is used to perform image denoising on the input-diffusion overlay to obtain a denoised image.

[0127] Optionally, the process of training the pre-trained post-neural network model using the mean squared loss to obtain the trained post-neural network model is as follows: updating the weights of the post-neural network through backward gradient propagation.

[0128] S5. Using the trained pre-neural network model and the trained post-neural network model, perform cell imaging on the current patient's fundus image to obtain cell imaging results.

[0129] The cell imaging results include an AO-OCT cell imaging map corresponding to the current patient's fundus image and an OCT image with added cell structures.

[0130] As can be seen, this embodiment of the invention first collects fundus images from historical patients to train a neural network model using the fundus image containing cells corresponding to the generated fundus image, enabling the neural network model to meet clinical needs. Further, this embodiment collects a cell imaging map of the eye corresponding to the fundus image, using the cell imaging map as the output data of the fundus image in the neural network model. Further, this embodiment uses a trained pre-neural network model to reconstruct the fundus image, reconstructing the low-resolution fundus image into a high-resolution image consistent with the cell imaging map. Further, this embodiment fuses the registered reconstructed image and the registered cell imaging map, using the fused image as the expected output data of the reconstructed image in the neural network model. By training the neural network model, it can output a corresponding OCT image with added cellular structures based on the input reconstructed image. Further, this embodiment trains the pre-trained post-neural network model to enable it to output a corresponding cell imaging map and an OCT image with added cellular structures based on the input fundus image, thereby applying it to clinical medicine. Therefore, the deep learning-based cell imaging method proposed in this embodiment of the invention can meet clinical needs through cell imaging.

[0131] like Figure 4 The diagram shown is a functional block diagram of the cell imaging device based on deep learning according to the present invention.

[0132] The deep learning-based cell imaging device 400 described in this invention can be installed in an electronic device. Depending on the functions implemented, the deep learning-based cell imaging device may include an image acquisition module 401, an image reconstruction module 402, an image fusion module 403, a model training module 404, and a cell imaging module 405. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0133] In this embodiment of the invention, the functions of each module / unit are as follows:

[0134] The image acquisition module 401 is used to acquire fundus images of historical patients and to acquire cell imaging maps of the eye corresponding to the fundus images;

[0135] The image reconstruction module 402 is used to reconstruct the fundus image using a trained pre-trained neural network model to obtain a reconstructed image of the fundus image.

[0136] The image fusion module 403 is used to perform image registration between the reconstructed image and the cell imaging image to obtain a registered reconstructed image and a registered cell imaging image, and to fuse the registered reconstructed image and the registered cell imaging image to obtain a fused image;

[0137] The model training module 404 is used to take the reconstructed image as input data for a pre-trained post-neural network model, and take the cell imaging image and the fused image as target data for the pre-trained post-neural network model, so as to train the pre-trained post-neural network model using the input data and the target data to obtain a trained post-neural network model.

[0138] The cell imaging module 405 is used to perform cell imaging on the current patient's fundus image using the trained pre-neural network model and the trained post-neural network model to obtain cell imaging results.

[0139] In detail, the modules in the deep learning-based cell imaging device 400 described in this embodiment of the invention employ the same methods as described above during use. Figures 1 to 3 The deep learning-based cell imaging method described herein uses the same techniques and can produce the same technical effects, so it will not be elaborated here.

[0140] like Figure 5 The diagram shown is a schematic representation of the electronic device that implements the deep learning-based cell imaging method of the present invention.

[0141] The electronic device may include a processor 50, a memory 51, a communication bus 52, and a communication interface 53, and may also include a computer program stored in the memory 51 and executable on the processor 50, such as a deep learning-based cell imaging program.

[0142] In some embodiments, the processor 50 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 50 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 51 (e.g., executing deep learning-based cell imaging programs) and calls data stored in the memory 51 to perform various functions of the electronic device and process data.

[0143] The memory 51 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 51 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 51 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 51 can include both internal and external storage units of the electronic device. The memory 51 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a database configuration connection program, but also to temporarily store data that has been output or will be output.

[0144] The communication bus 52 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 51 and at least one processor 50, etc.

[0145] The communication interface 53 is used for communication between the electronic device 5 and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0146] Figure 5 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 5 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0147] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 50 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0148] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in terms of the scope of the patent invention.

[0149] The database configuration connection program stored in the memory 51 of the electronic device is a combination of multiple computer programs, which, when run in the processor 50, can achieve the following:

[0150] Collect fundus images of historical patients and collect cell imaging maps of the eye corresponding to the fundus images;

[0151] The fundus image is reconstructed using a trained pre-trained neural network model to obtain a reconstructed image of the fundus image;

[0152] The reconstructed image and the cell imaging image are registered to obtain a registered reconstructed image and a registered cell imaging image. The registered reconstructed image and the registered cell imaging image are then fused to obtain a fused image.

[0153] The reconstructed image is used as input data for a pre-trained post-neural network model, and the cell imaging image and the fused image are used as target data for the pre-trained post-neural network model. The pre-trained post-neural network model is trained using the input data and the target data to obtain a trained post-neural network model.

[0154] The trained pre-neural network model and the trained post-neural network model are used to perform cell imaging on the current patient's fundus image to obtain cell imaging results.

[0155] Specifically, the specific implementation method of the above-mentioned computer program by the processor 50 can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0156] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0157] The present invention also provides a storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0158] Collect fundus images of historical patients and collect cell imaging maps of the eye corresponding to the fundus images;

[0159] The fundus image is reconstructed using a trained pre-trained neural network model to obtain a reconstructed image of the fundus image;

[0160] The reconstructed image and the cell imaging image are registered to obtain a registered reconstructed image and a registered cell imaging image. The registered reconstructed image and the registered cell imaging image are then fused to obtain a fused image.

[0161] The reconstructed image is used as input data for a pre-trained post-neural network model, and the cell imaging image and the fused image are used as target data for the pre-trained post-neural network model. The pre-trained post-neural network model is trained using the input data and the target data to obtain a trained post-neural network model.

[0162] The trained pre-neural network model and the trained post-neural network model are used to perform cell imaging on the current patient's fundus image to obtain cell imaging results.

[0163] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0164] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0166] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0167] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0168] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0169] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A cell imaging method based on deep learning, characterized in that, The method includes: Collect fundus images of historical patients and collect cell imaging maps of the eye corresponding to the fundus images; The fundus image is reconstructed using a trained pre-trained neural network model to obtain a reconstructed image of the fundus image; The reconstructed image and the cell imaging image are registered to obtain a registered reconstructed image and a registered cell imaging image. The registered reconstructed image and the registered cell imaging image are then fused to obtain a fused image. The reconstructed image is used as input data for a pre-trained post-neural network model, and the cell imaging image and the fused image are used as target data for the pre-trained post-neural network model. The pre-trained post-neural network model is trained using the input data and the target data to obtain a trained post-neural network model. The trained pre-neural network model and the trained post-neural network model are used to perform cell imaging on the current patient's fundus image to obtain cell imaging results; Before reconstructing the fundus image using the trained pre-trained neural network model to obtain the reconstructed fundus image, the method further includes: Acquire training fundus images, and use the generative network in the pre-trained pre-neighborhood model to reconstruct the resolution of the training fundus images to obtain pseudo-high resolution images; In the discriminant network of the pre-trained pre-neural network model, a feature map of the pseudo-high resolution image is constructed to obtain a pseudo feature map, and a feature map of the true high resolution image corresponding to the pseudo-high resolution image is constructed to obtain a true feature map. Based on the pseudo-feature map and the real feature map, the loss value of the generator network is calculated using the following formula to obtain the generator network loss: in, This represents the loss of the generating network. This represents the true feature map. Let i represent the pseudo-feature map, where i represents the index of the feature in the feature map. Represents the features in the feature map; Based on the pseudo-feature map and the real feature map, the loss value of the discriminant network is calculated using the following formula to obtain the discriminant network loss: in, This indicates the determination of network loss. This represents the loss of the generating network. This represents the true feature map. This represents the pseudo-feature map; When both the generation network loss and the discrimination network loss are greater than the preset loss, the network weights of the pre-trained pre-neural network model are updated to obtain the pre-neural network model with updated network weights. When the generation network loss and the discrimination network loss of the pre-neighborhood model after updating the network weights are both no greater than the preset loss, the trained pre-neighborhood model is obtained. The step of training the pre-trained post-neural network model using the input data and the target data to obtain a trained post-neural network model includes: Construct the diffusion steps of the target data and its Gaussian noise image; Based on the number of diffusion steps, the target data is forward diffused to obtain a forward diffusion map; The input data is superimposed with the forward diffusion map to obtain an input-diffusion overlay map; Based on the number of diffusion steps, the input-diffusion overlay image is subjected to image denoising processing to obtain a denoised image; The mean squared loss between the denoised image and the Gaussian noise image is calculated using the following formula: in, Indicates the average squared loss. This represents the Gaussian noise image. This refers to the denoised image. N represents the index of a pixel in the image, and N represents the total number of pixels in the image. The pre-trained post-neural network model is trained using the mean squared loss to obtain a trained post-neural network model.

2. The method according to claim 1, characterized in that, The step of using the generative network in the pre-trained pre-trained neural network model to reconstruct the resolution of the training fundus image to obtain a pseudo-high-resolution image includes: The shallow features of the training fundus image are extracted using the first convolutional layer in the generative network to obtain a shallow feature map; The shallow feature map is corrected by using the residual layer in the generator network to obtain a corrected feature map. The deep features of the corrected feature map are extracted using the residual blocks in the generator network to obtain the deep feature map; The deep feature map is processed by feature convolution using the second convolutional layer in the generative network to obtain the second convolutional map. The corrected feature map and the second convolutional map are superimposed to obtain the superimposed feature map; The resolution of the superimposed feature map is reconstructed to obtain the pseudo-high-resolution image.

3. The method according to claim 1, characterized in that, In the discriminant network of the pre-trained pre-trained neural network model, the feature map of the pseudo-high-resolution image is constructed to obtain the pseudo-feature map, including: Feature extraction is performed on the pseudo-high-resolution image to obtain a high-resolution feature map; The high-resolution feature map is dimensionally adjusted to obtain a dimensionally adjusted feature map; The dimension-adjusted feature map is linearly mapped to obtain a linearly mapped feature map. Calculate the classification probability of the linear mapping feature map, and use the image constructed from the classification probabilities as the pseudo feature map.

4. The method according to claim 1, characterized in that, The fusion of the registered reconstructed image and the registered cell imaging image to obtain the fused image includes: Image registration is performed between multiple registered cell imaging maps in the registered cell imaging map to obtain a secondary registered cell imaging map; Multiple secondary registration cell imaging maps in the secondary registration cell imaging map are fused to obtain a fused cell imaging map; The fused cell imaging image is obtained by fusing the fused cell imaging image with the registered and reconstructed image.

5. The method according to claim 4, characterized in that, The fusion of the fused cell imaging map and the registered reconstructed image to obtain the fused image includes: Multi-scale wavelet decomposition is performed on the fused cell imaging image and the registered and reconstructed image respectively to obtain the low-frequency and high-frequency components of the fused cell imaging image and the low-frequency and high-frequency components of the registered and reconstructed image. The low-frequency components of the fused cell imaging image and the low-frequency components of the registered and reconstructed image are fused together to obtain a low-frequency fused image; The high-frequency components of the fused cell imaging image and the high-frequency components of the registered and reconstructed image are fused together to obtain a high-frequency fused image; The low-frequency fused image and the high-frequency fused image are fused to obtain the fused image.

6. The method according to claim 1, characterized in that, The number of diffusion steps and the Gaussian noise image used to construct the target data include: Select the diffusion steps from the preset number of training iterations; The diffusion steps are converted into feature vectors; a Gaussian noise image of the feature vectors is constructed.

7. The method according to claim 1, characterized in that, The step of performing image denoising processing on the input-diffusion overlay image based on the number of diffusion steps to obtain a denoised image includes: Obtain the feature vector and pre-trained post-neural network model corresponding to the number of diffusion steps; The feature vector and the input data in the input-diffusion overlay are used as the supervision vector of the pre-trained post-neural network model; Based on the supervision vector, the pre-trained post-neural network model is used to perform image denoising on the input-diffusion overlay image to obtain the denoised image.

8. A cell imaging device based on deep learning, characterized in that, The device includes: The image acquisition module is used to acquire fundus images of historical patients and to acquire cell imaging maps of the eye corresponding to the fundus images; The image reconstruction module is used to reconstruct the fundus image using a trained pre-trained neural network model to obtain a reconstructed image of the fundus image. An image fusion module is used to register the reconstructed image and the cell imaging image to obtain a registered reconstructed image and a registered cell imaging image, and to fuse the registered reconstructed image and the registered cell imaging image to obtain a fused image; The model training module is used to take the reconstructed image as input data for a pre-trained post-neural network model, and take the cell imaging image and the fused image as target data for the pre-trained post-neural network model, so as to train the pre-trained post-neural network model with the input data and the target data to obtain a trained post-neural network model. The cell imaging module is used to perform cell imaging on the current patient's fundus image using the trained pre-neural network model and the trained post-neural network model to obtain cell imaging results; Before reconstructing the fundus image using the trained pre-trained neural network model to obtain the reconstructed fundus image, the method further includes: Acquire training fundus images, and use the generative network in the pre-trained pre-neighborhood model to reconstruct the resolution of the training fundus images to obtain pseudo-high resolution images; In the discriminant network of the pre-trained pre-neural network model, a feature map of the pseudo-high resolution image is constructed to obtain a pseudo feature map, and a feature map of the true high resolution image corresponding to the pseudo-high resolution image is constructed to obtain a true feature map. Based on the pseudo-feature map and the real feature map, the loss value of the generator network is calculated using the following formula to obtain the generator network loss: in, This represents the loss of the generating network. This represents the true feature map. Let i represent the pseudo-feature map, where i represents the index of the feature in the feature map. Represents the features in the feature map; Based on the pseudo-feature map and the real feature map, the loss value of the discriminant network is calculated using the following formula to obtain the discriminant network loss: in, This indicates the determination of network loss. This represents the loss of the generating network. This represents the true feature map. This represents the pseudo-feature map; When both the generation network loss and the discrimination network loss are greater than the preset loss, the network weights of the pre-trained pre-neural network model are updated to obtain the pre-neural network model with updated network weights. When the generation network loss and the discrimination network loss of the pre-neighborhood model after updating the network weights are both no greater than the preset loss, the trained pre-neighborhood model is obtained. The step of training the pre-trained post-neural network model using the input data and the target data to obtain a trained post-neural network model includes: Construct the diffusion steps of the target data and its Gaussian noise image; Based on the number of diffusion steps, the target data is forward diffused to obtain a forward diffusion map; The input data is superimposed with the forward diffusion map to obtain an input-diffusion overlay map; Based on the number of diffusion steps, the input-diffusion overlay image is subjected to image denoising processing to obtain a denoised image; The mean squared loss between the denoised image and the Gaussian noise image is calculated using the following formula: in, Indicates the average squared loss. This represents the Gaussian noise image. This refers to the denoised image. N represents the index of a pixel in the image, and N represents the total number of pixels in the image. The pre-trained post-neural network model is trained using the mean squared loss to obtain a trained post-neural network model.

Images