Fluid three-dimensional dynamic imaging method and imaging device based on OPT and neural network

Abstract

The present invention provides a method for three-dimensional dynamic imaging of fluid based on multi-channel OPT and neural network, comprising the following steps: using laser to provide illumination, emitting laser to biological samples, the biological samples will scatter light and form sample images, continuously collecting sample images of n angles of the biological samples, and obtaining a number of original data graphs; using neural networks to train the collected sample images with spatiotemporal characteristics to obtain a network model; using the network model to predict the original image; obtaining a two-dimensional sample feature distribution map, and using OPT technology to reconstruct three-dimensional images of these two-dimensional sample feature distribution maps to obtain dynamic three-dimensional images. The present invention also provides an imaging device. The present invention improves the accuracy of image feature recognition, has good imaging quality, and can quickly obtain three-dimensional dynamic images.

Description

Technical Field

[0001] The present invention relates to the technical field of optical imaging of near-transparent flow samples, and in particular to a three-dimensional dynamic imaging method and an imaging device of fluid based on OPT and neural network. Background Art

[0002] Optical Projection Tomography (OPT) is a new three-dimensional optical imaging technology that can achieve in vivo three-dimensional imaging of small animals with a detection depth of 10mm. Optical projection tomography (OPT) is similar to the principle of medical CT. It is a combination of optical imaging technology and CT technology. It is widely used in the field of medical tomography due to its stability and high resolution. The imaging principle of this method is: set one of the measured target and the camera to be fixed relative to the other, the camera takes images of the object at different angles, and then obtains the projection data of the image with the assistance of a CMOS camera. Through Radon transform, the projection of a two-dimensional image in the direction of the ray at a specified angle is line-integrated to obtain the projection of the image in this direction, that is, to describe the process of X-ray scanning the object to form a CT image. If you want to obtain more sufficient projection data, you need to rotate more times. Then, the line integral value and integral path of the known function are used to find the function, that is, the process of image reconstruction. Since the inverse Radon transform is the inverse Fourier transform of the two-dimensional image, the calculation amount is large and time-consuming, so the back projection method can be used to achieve it. The filtered back-projection method first performs a one-dimensional Fourier transform on all the collected projection signals, then filters all the projection signals in the frequency domain, and then performs a one-dimensional inverse Fourier transform on the filtered signals to restore them to the spatial domain. Finally, the filtered projection signals are back-projected, and the three-dimensional shape of the object can be obtained by preprocessing and superimposing the back-projection data at various angles through a computer. The filtered back-projection algorithm uses a method of filtering first and then back-projecting reconstruction, which converts two two-dimensional Fourier transforms into two one-dimensional Fourier transforms, greatly improving the calculation speed. However, the premise of filtered back-projection is that the data to be projected is ideal parallel data. The OPT two-dimensional projection data that has been corrected for bright field and dark field is closest to the ideal parallel projection data. Therefore, the corrected data is the data that is ultimately used for three-dimensional reconstruction.

[0003] Image imaging processing usually uses deep learning for image segmentation, that is, extracting image features and imaging image features.

[0004] Deep learning methods have achieved remarkable results in the field of image processing. The U-net network is an improvement and expansion based on the Fully Convolutional Network (FCN). Its network structure is completely symmetrical. By integrating the features of context and details, it improves the segmentation accuracy to a certain extent. U-net is suitable for segmentation problems of various biomedical images and is suitable for processing two-dimensional images.

[0005] There are many methods for three-dimensional blood flow imaging based on OPT technology that have been proposed, but they all have certain shortcomings. For example, Chinese patent CN108020509A, although it is suitable for biological imaging, improves image resolution and expands the depth of field of optical projection tomography, the fusion process of multiple depth-of-field images is complex and time-consuming, and the imaging speed is slow, and it is only suitable for imaging high-thickness samples. Chinese patent CN108937909A, although it can perform three-dimensional imaging of blood flow velocity, the technology cannot fully extract image features, and does not perform three-dimensional imaging of blood vessels. Chinese patent CN109859146A, uses an improved U-net neural network to improve the segmentation effect of the segmentation model and optimize image quality, but it only processes the spatial features of the two-dimensional image without considering the spatiotemporal features, and the U-net neural network image training requires a large number of original images, which is time-consuming. Summary of the invention

[0006] In order to solve the problems existing in the prior art, the present invention aims to provide a three-dimensional dynamic imaging method of fluid based on OPT and neural network.

[0007] Another object of the present invention is to provide an imaging device.

[0008] A method for three-dimensional dynamic imaging of fluid based on OPT and neural network, characterized by comprising the following steps:

[0009] Laser is used to provide illumination, and the laser is emitted to the biological sample, and the biological sample scatters light and forms a sample image, and the sample images of the biological sample at n angles are continuously collected to obtain a number of original data graphs;

[0010] The collected sample images with spatiotemporal characteristics are trained using a neural network to obtain a network model;

[0011] Predicting the original image through the network model;

[0012] A two-dimensional sample characteristic distribution map is obtained, and the two-dimensional sample characteristic distribution map is reconstructed into a three-dimensional image using the OPT technology to obtain a dynamic three-dimensional image.

[0013] Preferably, the method of using laser to provide illumination and continuously collecting images of n angles of the biological sample to obtain a plurality of raw data graphs comprises the following steps:

[0014] Parallel light with uniform illumination is irradiated onto the sample; wherein the sample is placed in a glass tube filled with matching liquid, and the glass tube is placed in a glass pool filled with the same matching liquid as that in the glass tube; laser is irradiated onto the sample surface, and the biological sample scatters light and forms a sample image, and the sample image is a speckle pattern;

[0015] Rotate the sample by an angle; collect T images of the sample at each rotation angle, until all T x*y-sized sample images of n angles of the sample are collected.

[0016] Preferably, the sample contains fluorescent material or is fluorescently labeled, and laser is irradiated to the biological sample. Light scattered from the biological sample and incident light pass through a filter, the incident light irradiating the biological sample is filtered out, and the fluorescence excited by the sample is retained, that is, the sample image formed is a fluorescence image, and the fluorescence image is collected.

[0017] Preferably, the laser includes lasers of various wavelengths.

[0018] Preferably, the filter is a bandpass filter with a central wavelength of 510 nm, a bandpass filter with a central wavelength of 560 nm, or a long-wave pass filter above 640 nm.

[0019] Preferably, the method of training the collected sample images using a neural network to obtain a network model comprises the following steps:

[0020] Label making: take an original image of size x*y at a certain angle to mark the target position, assign the gray value of the target position to 1, and assign the rest to 0, to get a label of the original image; then copy this label T-1 copies along the T axis, to get T labels of size x*y, then perform pixel-by-pixel cross-section operation on these T images along the Y axis, to get y cross-section images of size x*T, and the label cross-section image becomes the training sample of the neural network, which contains information in the X-axis and Y-axis directions, as well as information in the T axis direction;

[0021] Image training: Take T new raw data images collected, perform the same cross-section operation on them along the T axis, and obtain y images of size x*T; then compare the y labeled cross-section images with the corresponding y raw data cross-section images. Figure 1 Putting it into the U-net neural network is equivalent to putting the x*T-sized graph into the neural network along the T-axis direction for training. The network is trained by the characteristics of the time and space directions to allow the machine to learn the model.

[0022] Preferably, predicting the original image by using the network model comprises the following steps:

[0023] Take T new original data and perform the same cross-sectional operation to convert them into y original cross-sectional images of size x*T, and then put them into the trained network model for prediction, and finally get the predicted image.

[0024] Preferably, the two-dimensional sample feature distribution map is obtained, and three-dimensional image reconstruction is performed on these two-dimensional sample feature distribution maps to obtain a dynamic three-dimensional image, including the following steps: restoring the predicted y prediction maps of size x*T into T prediction maps of size x*y, and averaging the predicted probability values ​​of the target position along the T direction, and finally obtaining a probability distribution map of size x*y; performing three-dimensional imaging processing on the target angiography images at different angles through Amira software, and then obtaining a three-dimensional target image.

[0025] A device for three-dimensional dynamic imaging of fluid based on multi-channel OPT and neural network, comprising: a multi-wavelength laser, a collimation system, a motor, an optical displacement platform, a CMOS camera, a computer, a telecentric lens and a filter turntable; the multi-wavelength laser, the collimation system, the filter turntable, the telecentric lens and the CMOS camera are arranged along the optical path;

[0026] The collimation system comprises a first lens, a baffle and a second lens arranged along the optical path, the baffle is provided with a small hole, and the first lens, the baffle and the second lens are arranged according to the optical path; the light is focused by the first lens of the collimation system, so that the focus of the light passes through the small hole of the baffle and then exits through the second lens;

[0027] The filter turntable comprises a turntable body, and four holes are arranged on the turntable body, one of which is not used to place any filter, and the other three holes are used to place a bandpass filter with a central wavelength of 510nm, a bandpass filter with a central wavelength of 560nm, and a long-wave pass filter with a wavelength of more than 640nm respectively;

[0028] The optical displacement platform is provided with a station for placing samples, and the optical displacement platform is driven to rotate by a motor.

[0029] A three-dimensional dynamic imaging method comprises the following steps:

[0030] A1. Use multi-wavelength laser to provide illumination, emit multi-wavelength laser to biological samples, and continuously collect images of biological samples at n angles;

[0031] Among them, the high coherence of laser is used to generate speckle phenomenon when laser irradiates scattering particles to form images; laser is used to excite fluorescent substances in samples to produce fluorescence to form images;

[0032] Among them, the laser is irradiated onto the surface of the sample, the camera receives the scattered signal of the sample, the acquisition card collects and stores the data, and then obtains the characteristic information of the sample; the light is irradiated onto the area to be tested of the biological sample and then reflected to the CMOS camera; the CMOS camera is fixed, and each time the optical displacement platform rotates an angle, the CMOS camera takes T images of the biological sample at the corresponding angles, and each time an image of an angle is stored, the computer determines whether the images of n angles have been collected. If not, the optical displacement platform rotates to the next angle and continues to collect until all angles are collected; if all angles have been collected, it is determined whether the four holes of the filter turntable have been rotated; if the four holes have not been rotated, the filter turntable is rotated to the next hole; if the four holes have been rotated, it is determined whether the three wavelengths have been collected; if the three wavelengths of laser have not been collected, lasers of different wavelengths are replaced and collected again; if the three wavelengths of laser have been collected, the system is shut down;

[0033] The CMOS camera converts the received light signal into an electrical signal, and then transmits the collected original image to the computer through the acquisition card;

[0034] A2. Train the collected images using a neural network to obtain a network model;

[0035] A3. Predicting the original image through the network model;

[0036] A4. Obtain two-dimensional sample characteristic distribution maps, and perform three-dimensional image reconstruction on these two-dimensional sample characteristic distribution maps.

[0037] To achieve the above object, the present invention adopts the following technical solutions:

[0038] The beneficial effects of the present invention are:

[0039] 1. The neural network method is introduced to fully identify and extract the spatiotemporal features, which improves the accuracy of image feature recognition, has good imaging quality, and can quickly obtain three-dimensional images. 2. It is not necessary to perform bright field and dark field correction on the data to obtain a more ideal multi-angle angiography image, which reduces the time of data processing; the angiography image reconstructed by OPT can be quickly predicted by the network model and can be obtained using a small number of original data, which effectively solves the problem of requiring a large amount of data when using OPT to reconstruct images in three dimensions, and has the advantages of fast processing speed and high temporal resolution. 3. It can realize fluorescence three-dimensional dynamic imaging and fluorescence concentration measurement. 4. For experiments with blood flow or blood flow models as samples, it suppresses the imaging of the tube wall, reduces artifacts, and makes the sample features clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention is further described in detail below based on the accompanying drawings and embodiments.

[0041] Figure 1 is a schematic diagram of the device described in the present invention;

[0042] Figure 2 It is a data collection flow chart;

[0043] Figure 3 This is the U-net neural network architecture diagram;

[0044] Figure 4 It is a data processing flow chart.

[0045] in:

[0046] 1---Multi-wavelength laser; 2----Collimation system; 3----Motor; 4----Optical displacement platform; 5----CMOS camera; 6----Computer; 7----Matching fluid (in a glass pool); 8----Sample (placed in a glass tube containing matching fluid); 9----Telecentric lens; 10-Filter turntable. DETAILED DESCRIPTION

[0047] In order to make the technical problems solved by the present invention, the technical solutions adopted and the technical effects achieved clearer, the technical solutions of the embodiments of the present invention are further described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0048] The spatiotemporal direction described in the present invention specifically refers to the time direction and the space direction. The spatiotemporal features described in the present invention are information in the time direction and the space direction, wherein the time direction corresponds to the T-axis direction, and the space direction corresponds to the X-axis and Y-axis directions. Features in the space direction refer to that in the captured image, only the features of the sample at this one shooting time point can be obtained; features in the time direction refer to that T images are continuously shot at the same position and the same angle, and during this continuous period of time, the sample shot is constantly changing (for example, blood is constantly flowing). At this time, the T images are connected together to obtain the changes that occurred in the sample during the time of collecting T images, thereby obtaining features in the time direction.

[0049] See also Figure 1 and Figure 2 As shown, the method for three-dimensional dynamic imaging of fluid based on multi-channel OPT and neural network of the present invention comprises the following steps:

[0050] S1. Using laser to provide illumination, emitting laser to the biological sample, the biological sample will scatter light and form a sample image, and continuously collecting sample images of n angles of the biological sample to obtain a number of original data graphs;

[0051] Specifically: parallel light with uniform illumination is irradiated onto the sample; the sample is placed in a glass tube filled with matching liquid, and the glass tube is placed in a glass pool filled with the same matching liquid as that in the glass tube; laser is irradiated onto the sample surface, and the biological sample scatters light and forms a sample image, which is a speckle pattern; the angle of the sample is rotated; each time an angle is rotated, T images of the sample at that angle are collected until T x*y size sample images of n angles of the sample are collected.

[0052] In another case, the sample contains fluorescent substances or is fluorescently labeled. The laser irradiates the biological sample. The light scattered by the biological sample and the incident light pass through a filter, filtering out the incident light irradiating the biological sample and retaining the fluorescence excited by the sample. That is, the sample image formed is a fluorescent image, and the fluorescent image is collected.

[0053] Wherein, the filter is a bandpass filter with a central wavelength of 510nm, a bandpass filter with a central wavelength of 560nm, or a long-wave pass filter above 640nm.

[0054] Among them, different fluorescence images are obtained by using incident light of different excitation wavelengths. The use of lasers of different wavelengths is, on the one hand, to study the speckle three-dimensional images and fluorescence three-dimensional images obtained by different samples under lasers of different wavelengths; on the other hand, it is possible to observe the three-dimensional imaging effect of the same sample under lasers of different wavelengths. In addition, the use of lasers of different wavelengths allows for more diverse sample selections. According to the speckle pattern obtained when the laser is irradiated onto the sample surface, the characteristics of the sample at the corresponding angle can be obtained;

[0055] By irradiating the sample with laser to excite fluorescence, fluorescence imaging of the sample can be performed; by measuring the intensity of the excitation light of the substance at different wavelengths, the fluorescence concentration at the corresponding wavelength can be obtained, that is, the fluorescence concentration of the sample can be measured.

[0056] S2. The collected sample images with spatiotemporal characteristics are trained using a neural network to obtain a network model; the details are as follows:

[0057] Label making: take an original image of size x*y at a certain angle to mark the target position, assign the gray value of the target position to 1, and assign the rest to 0, that is, get a label of the original image; then copy this label T-1 pieces along the T axis direction, that is, get T pieces of x*y labels, then perform pixel-by-pixel cross-section operation on these T pieces along the Y axis direction, and get y pieces of x*T cross-section images. The label cross-section image becomes the training sample of the neural network, and the label cross-section image contains the features in the X-axis and Y-axis directions, as well as the features in the T axis direction;

[0058] Image training: Take T new raw data images collected, perform the same cross-section operation on them along the Y axis, and obtain y images of size x*T; then compare the y labeled cross-section images with the corresponding y raw data cross-section images. Figure 1 Putting it into the U-net neural network is equivalent to putting the x*T-sized graph into the neural network for training along the T-axis direction, realizing the use of data with spatiotemporal characteristics to train the network and let the machine learn the model.

[0059] S3, predicting the original image through the network model; the details are as follows:

[0060] Take T new original data and perform the same cross-sectional operation to convert them into y original cross-sectional images of size x*T, and then put them into the trained network model for prediction, and finally get the predicted image.

[0061] S4, obtain a two-dimensional sample characteristic distribution map, and use the OPT technology to reconstruct a three-dimensional image of the two-dimensional sample characteristic distribution map to obtain a dynamic three-dimensional image. The details are as follows:

[0062] The two-dimensional sample feature distribution map is obtained, and three-dimensional image reconstruction is performed on these two-dimensional sample feature distribution maps, including the following steps: restoring the predicted y prediction maps of x*T size into T prediction maps of x*y size, and taking the average value of the predicted probability value of the target position along the T direction, and finally obtaining a probability distribution map of x*y size; performing three-dimensional imaging processing on the target angiography images at different angles through Amira software, and then obtaining a three-dimensional target image.

[0063] Through the above steps, it can be known that in this method, laser is irradiated onto the sample surface, the camera receives the scattered signal of the sample, converts the received light signal into an electrical signal, the acquisition card collects and stores data, and then obtains the characteristic information of the sample, and then processes the data and reconstructs the three-dimensional image of the sample; if the sample image is a speckle pattern, it is reconstructed into a three-dimensional image in the form of speckle of the biological sample; if the sample image is a fluorescent image, the reconstructed fluorescent three-dimensional image of the biological sample has a fluorescent distribution effect and presents different fluorescence intensities, and based on this and subsequent processing, the fluorescence distribution and fluorescence concentration can be detected.

[0064] The imaging device of the present invention comprises: a multi-wavelength laser, a collimation system, a motor, an optical displacement platform, a CMOS camera, a computer, a telecentric lens and a filter turntable; the multi-wavelength laser, the collimation system, the filter turntable, the telecentric lens and the CMOS camera are arranged along the optical path. The collimation system comprises a first lens, a baffle and a second lens arranged along the optical path, a small hole is arranged on the baffle, and the first lens, the baffle and the second lens are arranged according to the optical path; the light is focused by the first lens of the collimation system, so that the focus of the light passes through the small hole of the baffle and then exits through the second lens;

[0065] The filter turntable includes a turntable body, on which four holes are arranged, one of which is not used to place any filter, and the other three holes are respectively used to place a bandpass filter with a central wavelength of 510nm, a bandpass filter with a central wavelength of 560nm, and a long-wave pass filter above 640nm. A station for placing samples is arranged on the optical displacement platform, and the optical displacement platform is driven to rotate by a motor.

[0066] According to the method for three-dimensional dynamic imaging of fluid based on multi-channel OPT and neural network described in the present invention, the specific operation of step S1 is as follows: using the high coherence of laser, the laser irradiates the scattering particles to generate speckle phenomenon to achieve imaging; using the laser to excite the fluorescent substance in the sample to generate fluorescence to achieve imaging, and analyzing the intensity of the fluorescence to achieve fluorescence concentration measurement;

[0067] The laser irradiates the sample surface, the camera receives the scattered signal of the sample, and the acquisition card collects and stores the data, thereby obtaining the characteristic information of the sample; the light irradiates the area to be tested of the biological sample and then reflects to the CMOS camera; the CMOS camera is fixed, and each time the optical displacement platform rotates an angle, the CMOS camera takes T images of the biological sample at the corresponding angles. Each time an image of an angle is stored, the computer determines whether the images of n angles have been collected. If not, the optical displacement platform rotates to the next angle and continues to collect until all angles are collected; if all angles have been collected, it is determined whether the four holes of the filter turntable have been rotated. If the four holes have not been rotated, the filter turntable is rotated to the next hole; if the four holes have been rotated, it is determined whether the three wavelengths have been collected; if the three wavelengths of laser have not been collected, lasers of different wavelengths are replaced and collected again; if the three wavelengths of laser have been collected, the system is shut down;

[0068] The CMOS camera converts the received light signal into an electrical signal, and then transmits the collected original image to the computer through the acquisition card.

[0069] Taking a blood sample as a specific embodiment of the present invention, the following scheme is included:

[0070] A: Imaging device

[0071] The experimental device includes: multi-wavelength laser [1], collimation system [2], motor [3], optical displacement platform [4], CMOS camera [5], computer [6], matching liquid [7], sample [8], telecentric lens [9], filter turntable

[10] . Figure 1 As shown: a multi-wavelength laser (three wavelengths are provided: 488nm, 532nm, 631nm) is used as the light source, and a certain wavelength laser is controlled to be emitted. The light is focused by the first lens of the collimation system, so that the focus of the light passes through the small hole of the baffle and then exits through the second lens. Since the distance from the second lens to the baffle is exactly the focal length of the second lens, the emitted light passes through the second lens and becomes parallel light with uniform illumination to irradiate the sample.

[0072] The sample is connected to the optical displacement platform and driven to rotate by a motor. The sample is placed in a glass tube filled with matching liquid, and the glass tube is placed in a glass pool filled with the same matching liquid as the glass tube, in order to match the refractive index.

[0073] The motor is controlled by a computer, thereby controlling the angle of rotation of the sample.

[0074] Each time the sample rotates by an angle, T images of the sample at that angle are collected until T images of the sample with a size of x*y at n angles of the sample are collected. The speckle pattern will be obtained when the laser is irradiated on the sample surface, and the information of the sample at the corresponding angle can be obtained;

[0075] Fluorescence imaging of the sample can be performed by irradiating the sample with laser to excite fluorescence;

[0076] By measuring the intensity of the excitation light of the fluorescent substance under different wavelengths, the fluorescence concentration under the corresponding wavelength is obtained, that is, the fluorescence concentration of the sample is measured.

[0077] The light scattered by the sample (wavelength of 488nm, 532nm or 631nm) can be filtered by rotating the filter turntable, that is, filtering out the incident light irradiating the sample and retaining the fluorescence excited by the sample. We use three wavelengths of incident light, and each time we only inject one wavelength of light, and then excite the fluorescence, and filter it through the filter of a specific wavelength range in the filter turntable. Then replace the laser with another incident wavelength and choose another filter accordingly.

[0078] The sample itself needs to contain fluorescent substances or be fluorescently labeled to realize fluorescence imaging and concentration measurement. For example, in biological samples, the blood in the blood vessels is fluorescently labeled, while other substances other than blood are not labeled. The final emitted light is fluorescence or incident light that has not been filtered (wavelength is 488nm, 532nm or 631nm).

[0079] The filter turntable has four holes. By rotating different holes, different wavelengths of excitation light can be obtained. The first circular hole does not have a filter (it is a through hole), and the incident light can pass directly, so ordinary dynamic speckle three-dimensional imaging of the sample can be performed. The remaining three circular holes are respectively placed with bandpass filters with central wavelengths of 510nm and 560nm and long-wave pass filters above 640nm, which are used to filter out light of 500-520nm, light of 550-570nm, and light of wavelengths above 640nm, that is, to filter out the incident light and only retain the fluorescence emitted by the sample under stimulation, so that the fluorescence distribution and concentration of the sample can be imaged.

[0080] Therefore, in addition to dynamic speckle imaging, the device can also perform three-dimensional fluorescence imaging and concentration measurement. In addition, the three-dimensional imaging effect of dynamic speckle and fluorescence signals of samples under different wavelengths can be studied. Finally, the signal of the sample is collected by a CMOS camera and then transmitted to a computer. The CMOS camera is placed on an optical displacement platform, and the depth of field of the imaging can be improved to a certain extent by connecting the CMOS camera with a telecentric lens.

[0081] B: Collection of original data

[0082] like Figure 2 As shown:

[0083] First, the acquisition system is initialized, including: light source device, detector, external trigger system and rotating motor. Then, appropriate acquisition parameters are set, including: detector parameters, rotation parameters and acquisition card parameters.

[0084] By utilizing the high coherence of the laser, the laser will produce a speckle phenomenon when irradiating the scattering particles to form an image (the principle of speckle imaging). The laser is used to excite the fluorescent substance in the sample to produce fluorescence to form an image, and the fluorescence concentration is measured by analyzing the intensity of the fluorescence (the principle of fluorescence imaging).

[0085] The laser irradiates the sample surface, the camera receives the scattered signal of the sample, and the acquisition card collects and stores the data, thereby obtaining the characteristic information of the sample. The light irradiates the area of ​​the biological sample to be tested and then reflects to the CMOS camera. The CMOS camera is fixed. Every time the motor rotates an angle, the CMOS camera takes T images of the biological sample at the corresponding angle. After storing an image of an angle, the computer determines whether the images of n angles have been collected. If not, the motor rotates to the next angle and continues to collect until all angles are collected;

[0086] If all angles have been collected, determine whether the four holes of the filter turntable have been selected. If the four holes have not been rotated, rotate the filter turntable to the next hole; if the four holes have been rotated, determine whether the three wavelengths have been collected. If the three wavelengths of laser have not been collected, replace the laser with a different wavelength and collect again; if the three wavelengths of laser have been collected, shut down the system.

[0087] The camera converts the received light signal into an electrical signal, and then transmits the collected original image to the computer through the acquisition card, thereby saving the sample information, and then processing the data and reconstructing the sample image in three dimensions. The rectangle represents the process, and the diamond represents the judgment.

[0088] C: U-net neural network architecture

[0089] U-net is an improvement based on the fully convolutional neural network. By using data enhancement, it can train some relatively small sample data. The U-net structure has no connection layer, which greatly reduces the parameters that need to be trained and enables the network structure to run more efficiently. Due to the special U-shaped structure, all the information in the image can be well preserved. U-net can basically perform convolution operations on images of any shape and size.

[0090] like Figure 3As shown in the figure: The entire neural network is mainly composed of two parts: the compression path (on the left side of the dotted line) and the expansion path (on the right side of the dotted line). The input of the network is an n×n (in the figure, the input image size is assumed to be 572×572) image with mirrored edges (that is, adding a symmetrical edge to the image to increase the image resolution). On the compression path, every two 3×3 convolutional layers are followed by a 2×2 maximum pooling layer, and each convolutional layer is followed by a ReLu activation function to downsample the original image. In addition, each downsampling will increase the number of channels and double the number of feature maps.

[0091] In the expansion path, each step has a 2×2 convolution layer and two 3×3 convolution layers, and the ReLu activation function is also used for upsampling, multiplying the size of the feature map by 2 and halving the number (except for the last layer of the network). At the same time, each upsampling step will be fused with the same scale as the number of channels corresponding to the feature extraction part, that is, the feature map from the corresponding compression path that has been cropped to maintain the same shape is added and then fused.

[0092] The last layer of the network is a 1×1 convolution layer, which can convert the feature vector of 64 channels into the number of required classification results. Since zero padding is applied to each convolution layer, the width and height of the output image are consistent with the width and height of the input image. Since this is a binary classification process, two feature maps will be output in the end.

[0093] The compression path is mainly used to extract the context information of the image, while the expansion path is used to accurately locate the part of the image that needs to be segmented. The local pixel features extracted on the compression path are combined with the new feature map during the upsampling process, that is, the accuracy of the prediction of the pixel category is improved through information integration, so as to retain the important feature information of the downsampling process to the greatest extent and then accurately locate it.

[0094] Among them, the gray / white box represents the feature map; the purple arrow represents the 3×3 convolution layer, which is used for feature extraction; the gray arrow represents the skip connection, which is used for feature fusion, where the copy in copy and crop is the feature map of the channels corresponding to the compression path and the expansion path, and the crop is to cut the image so that the length and width of the two are consistent; the orange arrow represents the maximum pooling, which is used to reduce the dimension; the white arrow represents upsampling, which is used to restore the dimension; the green arrow represents the 1×1 convolution, which is used for the result output.

[0095] D: Data Processing and Contrast Imaging

[0096] like Figure 4As shown in the figure, it includes three parts: image training (dashed border on the left), image prediction (dotted box in the middle) and OPT angiography (dashed box on the right). The specific processing flow is as follows:

[0097] (1) U-net network training and prediction

[0098] ① Label production: Take an original image of size x*y at a certain angle to mark the position of the blood vessels, assign the gray value of the blood vessel position to 1, and assign the rest to 0, that is, get a label of the original image. Then copy this label T-1 pieces along the T axis direction, that is, get T pieces of x*y labels, and then perform pixel-by-pixel cross-section operations on these T pictures along the Y axis direction to get y pieces of x*T cross-section pictures. The labeled cross-section pictures become the training samples of the neural network; the labeled cross-section pictures contain information in the X-axis and Y-axis directions, as well as information in the T axis direction.

[0099] ② Image training: Take T new raw data images collected, perform the same cross-section operation on them along the Y axis, and obtain y images of size x*T. Then compare the y labeled cross-section images with the corresponding y raw data cross-section images. Figure 1 Putting it into the U-net neural network is equivalent to putting the x*T-sized graph into the neural network for training along the T-axis direction, which realizes the training of the network by extracting the features in the time and space directions, allowing the machine to learn the model.

[0100] ③ Image prediction: After training is completed, take T new raw data and perform the same cross-sectional operation to convert them into y raw cross-sectional images of size x*T, and then put them into the trained network model for prediction, and finally get the predicted image. In particular, this process does not require labeling.

[0101] (2) Data processing and OPT imaging

[0102] The predicted y x*T size prediction maps are restored to T x*y size prediction maps, and the predicted probability values ​​of the blood flow position are averaged along the T direction, and finally a x*y size probability distribution map is obtained, which is the blood flow angiography image.

[0103] Finally, the blood flow angiography images at different angles were processed into three-dimensional images using Amira software to obtain three-dimensional blood flow images.

[0104] The method of three-dimensional dynamic imaging of fluid based on multi-channel fluorescence OPT and neural network described in the present invention achieves:

[0105] 1. The neural network method is introduced to fully identify and extract the spatiotemporal features, which improves the accuracy of image feature recognition, has good imaging quality, and can quickly obtain three-dimensional images. By taking cross-section operations in the time direction of T original images, the feature information within a period of time is obtained, that is, the two-dimensional spatial feature image is converted into a two-dimensional spatiotemporal feature image, and the U-net neural network is used to extract the features of the two-dimensional image in the spatiotemporal direction.

[0106] 2. Ideal multi-angle angiography images can be obtained without bright field and dark field correction of the data, which reduces the time of data processing; the angiography images reconstructed by OPT can be quickly predicted by the network model and obtained using a small number of original data, which effectively solves the problem of requiring a large amount of data when using OPT to reconstruct images in three dimensions, and has the advantages of fast processing speed and high temporal resolution. The U-net neural network only needs to predict a small number of original images to obtain blood flow angiography images with high feature resolution.

[0107] 3. It can realize fluorescence three-dimensional dynamic imaging and fluorescence concentration measurement. Using multi-wavelength laser as the light source, by controlling the wavelength of the incident light of the sample, the intensity of the sample excitation light under different wavelengths can be collected, and the sample data can be continuously collected over a period of time to obtain the dynamic changes of the sample, thereby realizing the fluorescence OPT dynamic imaging of the sample and the measurement of fluorescence distribution and concentration.

[0108] 4. For experiments with blood flow or blood flow models as samples, the imaging of the tube wall is suppressed, artifacts are reduced, and the sample features are clearer. When the laser irradiates the sample, the tube wall will also be imaged due to the influence of scattered particles. When the U-net neural network is trained on the image, only the blood flow features are extracted from the original image, thus removing the features of the tube wall. The resulting predicted image does not have the image of the tube wall, that is, the imaging of the tube wall is suppressed.

[0109] In the description herein, the reference terms "an embodiment", "example", etc., mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example.

[0110] In addition, it should be understood that although this specification is described according to implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0111] The technical principle of the present invention is described above in conjunction with specific embodiments. These descriptions are only for explaining the principle of the present invention and cannot be interpreted as limiting the scope of protection of the present invention in any way. Based on the explanations herein, those skilled in the art can associate other specific implementations of the present invention without paying creative labor, and these methods will fall within the scope of protection of the present invention.

Claims

1. A method for three-dimensional dynamic imaging of fluid based on OPT and neural network, characterized in that: The following steps are involved: Laser is used to provide illumination, and the laser is emitted to the biological sample, and the biological sample scatters light and forms a sample image, and the sample images of the biological sample at n angles are continuously collected to obtain a number of original data graphs; The collected sample images with spatiotemporal characteristics are trained using a neural network to obtain a network model; Predicting the original image through the network model; Obtaining a two-dimensional sample characteristic distribution map, reconstructing the two-dimensional sample characteristic distribution map into a three-dimensional image using the OPT technology to obtain a dynamic three-dimensional image; The laser is used to provide illumination, and images of the biological sample are continuously collected at n angles to obtain a number of raw data graphs. The following steps are involved: Parallel light with uniform illumination is irradiated onto the sample; wherein the sample is placed in a glass tube filled with matching liquid, and the glass tube is placed in a glass pool filled with the same matching liquid as that in the glass tube; laser is irradiated onto the sample surface, and the biological sample scatters light and forms a sample image, and the sample image is a speckle pattern; Rotate the sample by an angle; collect T images of the sample at each angle, until all T images of the sample with a size of x*y at n angles are collected; The sample has a fluorescent substance or is fluorescently labeled, and the laser irradiates the biological sample. The light scattered by the biological sample and the incident light pass through a filter, and the incident light irradiating the biological sample is filtered out, and the fluorescence excited by the sample is retained, that is, the sample image formed is a fluorescent image, and the fluorescent image is collected; The laser includes lasers of various wavelengths; The filter is a bandpass filter with a central wavelength of 510nm, a bandpass filter with a central wavelength of 560nm, or a long-wave pass filter above 640nm; The method of training the collected sample images using a neural network to obtain a network model comprises the following steps: Label making: Take an original image of size x*y at a certain angle to mark the target position, assign the grayscale value of the target position to 1, and assign the rest to 0, to get a label of the original image; then copy this label T-1 copies along the T axis, to get T labels of size x*y, then perform pixel-by-pixel cross-section operation on these T images along the Y axis, to get y cross-section images of size x*T, and the label cross-section image becomes the training sample of the neural network, which contains information in the X-axis and Y-axis directions, as well as information in the T axis direction; Image training: Take T new raw data images collected, perform the same cross-section operation on them along the T axis, and obtain y images of size x*T; then put the y labeled cross-section images and the corresponding y raw data cross-section images into the U-net neural network, which is equivalent to putting the x*T images along the T axis into the neural network for training. The network is trained by the characteristics of the time and space directions to allow the machine to learn the model.

2. The method for three-dimensional dynamic imaging of fluid based on OPT and neural network according to claim 1, characterized in that: Predicting the original image through the network model includes the following steps: Take T new original data and perform the same cross-sectional operation to convert them into y original cross-sectional images of size x*T, and then put them into the trained network model for prediction, and finally get the predicted image.

3. According to the method of claim 1, the method of three-dimensional dynamic imaging of fluid based on OPT and neural network, wherein the two-dimensional sample characteristic distribution map is obtained, and the three-dimensional image reconstruction is performed on the two-dimensional sample characteristic distribution map to obtain a dynamic three-dimensional image, comprising the following steps: The predicted y prediction images of size x*T are restored to T prediction images of size x*y, and the predicted probability values ​​of the target position are averaged along the T direction to finally obtain a probability distribution map of size x*y; the target angiography images at different angles are processed into three-dimensional imaging through Amira software to obtain a three-dimensional target image.

Images