Image generation method, super-resolution detection method and super-resolution detection system

Abstract

The application provides a deep learning-based image generation method, comprising: receiving at least one first fringe image, the first fringe image being an image of a sample to be measured collected when fringe structured light irradiates the sample to be measured; extracting deep information and shallow information of the at least one first fringe image based on a preset deep learning semantic segmentation model, fusing the deep information and the shallow information to obtain semantic information; and generating a plurality of second fringe images according to the semantic information, the at least one first fringe image and the plurality of second fringe images being used to obtain biological information of the sample to be measured. The application also provides a deep learning-based super-resolution detection method and a deep learning-based super-resolution detection system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biochemical information detection, and in particular to an image generation method based on deep learning, a super-resolution detection method and a super-resolution detection system. BACKGROUND

[0002] Gene sequencing technology refers to a technology for analyzing the sequence of four bases on DNA. So far, it has been widely applied to many research fields of life science and medicine, including various genomics, etiology of complex diseases, prenatal diagnosis, individualized drug treatment, etc. The basic method of gene sequencing is to make the four bases carry corresponding fluorescent groups by biochemical methods. The fluorescent groups emit different wavelengths of fluorescence after being excited by different wavelengths of laser, and the base types are identified by the fluorescence, thereby realizing sequencing.

[0003] Structured illumination microscopy (SIM) technology is a typical wide-field imaging technology, which is suitable for high-resolution fluorescence microscopic imaging systems of gene sequencers. The optical hardware system of the SIM technology is mainly divided based on the devices used, typical devices such as gratings (Grating-SIM), spatial light modulators (SLM-SIM), digital micromirror devices (DMD-SIM), etc. A digital micromirror device (Digital Micromirror Device, DMD) is a micro-optical machine system (Micro-electrical-mechanical system, MEMS) composed of a micro-mirror array with a high-reflective aluminum film on the surface. A single micro-mirror is called a pixel unit (Pixel). The pixel unit has two states: ON and OFF. The ON state corresponds to the high-reflective state, and the OFF state corresponds to the non-reflective state. The ON and OFF states are realized by controlling the rotation angle of the mirror (the most common system is a deflection angle of ±12°). DMD-SIM uses an electric control method to realize the projection of stripes in X / Y directions. DMD realizes high-speed direction switching and stripe phase shift by switching micro-mirrors.

[0004] Under the traditional DMD-SIM mode, a large number of stripe images need to be collected, and the image collection is time-consuming. Although super-resolution detection is realized, it is not conducive to improving the detection rate. Therefore, how to further speed up the detection rate on the basis of realizing super-resolution is a technical problem to be solved in the field. SUMMARY

[0005] In one aspect, the present application provides an image generation method based on deep learning, comprising:

[0006] receiving at least one first fringe image, the first fringe image being an image of a sample under test captured when fringe structured light illuminates the sample under test;

[0007] extracting deep information and shallow information of the at least one first fringe image based on a preset deep learning semantic segmentation model, fusing the deep information and the shallow information to obtain semantic information;

[0008] generating a plurality of second fringe images according to the semantic information, the at least one first fringe image and the plurality of second fringe images being used to obtain biological information of the sample under test.

[0009] Another aspect of the present application provides a deep learning-based super-resolution detection method, comprising:

[0010] receiving at least one first fringe image, the first fringe image being an image of a sample under test captured when fringe structured light illuminates the sample under test;

[0011] extracting deep information and shallow information of the at least one first fringe image based on a preset deep learning semantic segmentation model, fusing the deep information and the shallow information to obtain semantic information;

[0012] generating a plurality of second fringe images according to the semantic information;

[0013] performing super-resolution reconstruction according to the at least one first fringe image and the plurality of second fringe images to obtain biological information of the sample under test.

[0014] Another aspect of the present application provides a deep learning-based super-resolution detection system, comprising:

[0015] a light source module configured to emit light source light;

[0016] a light modulator configured to modulate the light source light into fringe structured light, the fringe structured light being capable of being guided to a sample under test to make the sample under test emit detection light;

[0017] an image acquisition module configured to acquire at least one first fringe image according to the detection light; and

[0018] a controller electrically connected to the light modulator and the image acquisition module, configured to adjust a direction and / or a phase of the fringe structured light, and configured to generate a plurality of second fringe images based on the at least one first fringe image according to the deep learning-based image generation method, and perform super-resolution reconstruction according to the at least one first fringe image and the plurality of second fringe images to obtain biological information of the sample under test.

[0019] The above deep learning-based image generation method, super-resolution detection method and super-resolution detection system are used to obtain biological information of a to-be-detected sample, and a plurality of fringe images of the to-be-detected sample need to be obtained under the condition that different fringe structured lights irradiate the to-be-detected sample. Since a part of the plurality of fringe images can be actually collected as a first fringe image, and the remaining required images can be generated as a second fringe image based on a deep learning method, the number of collected fringe images is reduced, thereby facilitating shortening of the image collection time and further facilitating improvement of the detection speed. In addition, due to the significant reduction in the number of collected images, reagents are fully saved, thereby further reducing the detection cost. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The structure schematic diagram of the super-resolution detection system, the to-be-detected sample and the sequencing chip provided in the embodiment of the present application.

[0021] Figure 2 For Figure 1 The optical path structure schematic diagram of the super-resolution detection system in the embodiment.

[0022] Figure 3 For Figure 2 The plane structure schematic diagram of the light modulator in the embodiment.

[0023] Figure 4 For Figure 2 The deflection state schematic diagram of the micro-mirror in the embodiment.

[0024] Figure 5A The structure schematic diagram of the light spot formed by a fringe structured light on the to-be-detected sample.

[0025] Figure 5B The structure schematic diagram of the light spot formed by another fringe structured light on the to-be-detected sample.

[0026] Figure 5C The structure schematic diagram of the light spot formed by another fringe structured light on the to-be-detected sample.

[0027] Figure 5D The structure schematic diagram of the light spot formed by another fringe structured light on the to-be-detected sample.

[0028] Figure 5E The structure schematic diagram of the light spot formed by another fringe structured light on the to-be-detected sample.

[0029] Figure 5F The structure schematic diagram of the light spot formed by another fringe structured light on the to-be-detected sample.

[0030] Figure 6 The flowchart of the deep learning-based image generation method provided in the embodiment.

[0031] Figure 7 A structural schematic diagram of a deep learning semantic segmentation model U-Net provided for the embodiment.

[0032] Figure 8 A flowchart of a deep learning-based super-resolution detection method provided for the embodiment.

[0033] Figure 9 Another flowchart of a deep learning-based super-resolution detection method provided for the embodiment.

[0034] Figure 10A A schematic diagram of a wide field spectrum provided for the embodiment.

[0035] Figure 10B A schematic diagram of a spectrum obtained in a single direction provided for the embodiment.

[0036] Figure 10C A schematic diagram of a spectrum obtained in multiple directions provided for the embodiment.

[0037] Figure 11A A schematic diagram of an image of a sample to be measured collected when a sample to be measured is irradiated with a stripe structured light provided for the embodiment.

[0038] Figure 11B A schematic diagram of a spectrum of an image of a sample to be measured collected when a sample to be measured is irradiated with a stripe structured light provided for the embodiment.

[0039] Figure 12 A super-resolution reconstruction effect comparison provided for the embodiment.

[0040] Main element symbol explanation

[0041] Super-resolution detection system 10

[0042] Light source module 11

[0043] Laser 111

[0044] Mirror 112

[0045] Dichroic mirror 113

[0046] Light modulation module 12

[0047] Light modulator 121

[0048] Micro-mirror 1211

[0049] Total internal reflection mirror 122

[0050] Image acquisition module 13

[0051] Controller 14

[0052] Sample to be measured 20

[0053] Encoder 31

[0054] Decoder 32

[0055] Bridge 33

[0056] Steps S11, S12, S13, S21, S22, S23, S24

[0057] The following detailed description will further explain the present application in conjunction with the above-mentioned figures. DETAILED DESCRIPTION

[0058] Referring to Figure 1 The super-resolution detection system 10 of the embodiment can be used to detect biological information of the sample 20 to be tested. The sample 20 to be tested can be a nucleic acid sample (DNA or RNA), a protein, or a cell, etc. In the embodiment, the sample 20 to be tested is a nucleic acid sample, and the biological information can be base sequence information of the sample 20 to be tested.

[0059] During the working process of the super-resolution detection system 10, the reference light is emitted to the sample 20 to be tested. The relative motion between the sample 20 to be tested and the super-resolution detection system 10 can be generated by moving the sample 20 to be tested. The relative motion between the sample 20 to be tested and the super-resolution detection system 10 can be generated by moving the sample 20 to be tested. The reference light can be projected onto different regions of the sample 20 to be tested by setting the relative motion between the sample 20 to be tested and the super-resolution detection system 10. This process can also be referred to as "scanning". Since the field of view of the reference light on the sample 20 to be tested cannot completely cover the sample 20 to be tested, the relative motion between the sample 20 to be tested and the super-resolution detection system 10 can be generated by moving the sample 20 to be tested, so that the super-resolution detection system 10 can scan the entire sample 20 to be tested.

[0060] In the embodiment, different bases on the sample 20 to be tested are labeled by different fluorescent substances. When the reference light irradiates the sample 20 to be tested, different fluorescent substances are excited to generate fluorescent light of different wavelengths as detection light. The super-resolution detection system 10 is used to obtain the biological information of the sample 20 to be tested according to the detection light.

[0061] Referring to Figure 2 , Figure 2 The solid arrows in the figure represent the propagation direction of the laser, and the dashed arrows represent the propagation direction of the fluorescent light. The super-resolution detection system 10 includes a light source module 11, a light modulation module 12, an image acquisition module 13, and a controller 14.

[0062] The light source module 11 includes two lasers 111, which are configured to emit laser beams of different wavelengths. For example, one of the two lasers 111 is configured to emit a red laser beam, and the other is configured to emit a green laser beam. In other embodiments, the light source module 11 includes a different number of lasers, and each laser is configured to emit a laser beam of a different wavelength. The number of lasers can depend on the types of fluorescent substances on the sample 20 to be measured.

[0063] The light source module 11 further includes a light combining component configured to combine the first light and the second light to form a light source light. In this embodiment, the light combining component includes a mirror 112 and a dichroic mirror 113. The laser beams emitted by the two lasers 111 are combined by the dichroic mirror 113 as a light source light. The light source module 11 further includes a lens group configured to expand the light source light to meet the requirements of the field of view in the subsequent optical path.

[0064] Please refer to Figure 2 , the light modulation module 12 includes a total internal reflection mirror 122 and a light modulator 121. The total internal reflection mirror 122 is configured to receive the expanded light source light and project it to the light modulator 121. The light modulator 121 is configured to modulate the received light to generate a structured light with a fringe pattern. The total internal reflection mirror 122 is further configured to guide the structured light with the fringe pattern to the image acquisition module 13.

[0065] Please refer to Figure 3 , the light modulator 121 is a digital micromirror device (DMD). The light modulator 121 includes a plurality of micromirrors 1211 arranged on the same plane. The plurality of micromirrors 1211 are arranged as a micromirror array including a plurality of rows and a plurality of columns. In this embodiment, each micromirror 1211 is substantially rectangular, and the micromirror array on the light modulator 121 is substantially rectangular.

[0066] Please refer to Figure 4 , each micromirror 1211 can be deflected within a certain angle range. In this embodiment, each micromirror 1211 can be deflected in two opposite directions about an axis, and the maximum deflection angles in the two opposite directions are the same, which are defined as α and -α, respectively. During the operation of the super-resolution detection system 10, the controller 14 controls the state of each micromirror 1211 to be the deflection angle α or the deflection angle -α, respectively. The state of each micromirror 1211 is defined as "ON" when the deflection angle is α, and "OFF" when the deflection angle is -α. By adjusting the state of each micromirror 1211 in the light modulator 121, the form of the structured light emitted by the light modulator 121 can be adjusted.

[0067] In this embodiment, the structured light emitted by the light modulator 121 is fringe structured light. By adjusting the state of each micro-mirror 1211 in the light modulator 121, the fringe direction and phase of the fringe structured light emitted by the light modulator 121 can be adjusted. The fringe structured light can be projected onto the sample 20 to be measured. When the fringe structured light is projected onto the sample 20 to be measured, a light spot including a plurality of parallel fringes is formed on the surface of the sample 20 to be measured.

[0068] In the working process of the super-resolution detection system 10, a plurality of detection cycles are used. In each detection cycle, the image acquisition module 13 is used to acquire at least one image of the sample 20 to be measured. The image of the sample 20 to be measured acquired by the super-resolution detection system 10 is defined as a first fringe image. In each detection cycle, the image acquisition module 13 is used to acquire at least one first fringe image.

[0069] Please refer to Figure 2 , the controller 14 is electrically connected to the light source module 11, the light modulation module 12 and the image acquisition module 13 respectively. The controller 14 is used to control the light source module 11 to emit light, and is used to control the deflection state of each micro-mirror 1211 in the light modulator 121, so as to adjust the fringe direction and phase of the fringe structured light projected onto the sample 20 to be measured. The controller 14 is also used to receive at least one first fringe image acquired by the image acquisition module 13, to perform data processing on the at least one first fringe image to generate a plurality of second fringe images, and to obtain biological information of the sample 20 to be measured according to the at least one first fringe image and the plurality of second fringe images.

[0070] In this embodiment, six images are required to obtain the biological information of the sample 20 to be measured. In one detection cycle, the image acquisition module 13 acquires two first fringe images, and the controller 14 is used to generate four second fringe images based on a deep learning algorithm according to the two first fringe images.

[0071] It is defined that Figure 5A the fringe structured light shown is fringe structured light with the fringe extension direction being the X direction, and it is defined that Figure 5D the fringe structured light shown is fringe structured light with the fringe extension direction being the Y direction, and the X direction is perpendicular to the Y direction. The two first fringe images are respectively Figure 5A and Figure 5D the images shown are acquired when the fringe structured light irradiates the sample 20 to be measured, and the two first fringe images are respectively represented as X1 and Y1. The four second fringe images are respectively represented as X2, X3, Y2 and Y3. Please refer to Figure 5B and Figure 5C X2 and X3 are respectively images that can be acquired when fringe structured light with different phases from the fringe structured light in Figure 5A irradiates the sample 20 to be measured along the X direction. Please refer toFigure 5E and Figure 5F Y2 and Y3 are images that can be captured when the sample 20 is irradiated with the fringe structured light different from the fringe structured light in Y direction. Figure 5D

[0072] The phase relationship of the fringe images X1 / X2 / X3 / Y1 / Y2 / Y3 is shown in Table 1:

[0073] Table 1

[0074]

[0075] It should be noted that the second fringe images X2, X3, Y2 and Y3 are not the images actually captured by the image capturing module 13, but the images generated by the controller 14, which are used to simulate the images that can be captured by the image capturing module 13 when the sample 20 is irradiated with the fringe structured light.

[0076] In an embodiment, the image capturing module 13 captures one first fringe image, and the controller 14 generates the remaining five second fringe images based on the deep learning method. That is, the image capturing module 13 captures any one of X1 / X2 / X3 / Y1 / Y2 / Y3 as a first fringe image, and the controller 14 generates the remaining five as five second fringe images based on the deep learning method.

[0077] In another embodiment, the image capturing module 13 captures three first fringe images, and the controller 14 generates the remaining three second fringe images based on the deep learning method. That is, the image capturing module 13 captures any three of X1 / X2 / X3 / Y1 / Y2 / Y3 as three first fringe images, and the controller 14 generates the remaining three as three second fringe images based on the deep learning method.

[0078] In another embodiment, the biological information of the sample 20 can be obtained without six fringe images, for example, only three fringe images.

[0079] The method of generating the second fringe image based on the deep learning method by the controller 14 is described below:

[0080] Referring to Figure 6 The embodiment also provides a deep learning-based image generation method applied to the controller 14, which comprises:

[0081] Step S11, receiving at least one first fringe image, the first fringe image being an image of the sample captured when the sample is irradiated with the fringe structured light;

[0082] ​Step S12, based on the preset deep learning semantic segmentation model, deep information and shallow information of the at least one first fringe image are extracted, the deep information and the shallow information are fused to obtain semantic information; in the convolutional neural network, the shallow information contains image contour and other information, and the deep information is more abstract detailed information which is deeper in image level and not easily discovered by human eyes compared with the shallow information;

[0083] Step S13, according to the semantic information, a plurality of second fringe images are generated, and the at least one first fringe image and the plurality of second fringe images are used to obtain biological information of the sample to be tested.

[0084] In step S11, two first fringe images are received, and the two first fringe images received are two first fringe images X1 and Y1 (see the foregoing) photographed by the image acquisition module 13.

[0085] In step S12, the specific type of the deep learning semantic segmentation model is not limited, and the deep learning semantic segmentation model can be FCN, U-Net, Deeplab or HRNet, but is not limited thereto. In the embodiment, the deep learning semantic segmentation model is U-Net.

[0086] Please refer to Figure 7 In the embodiment, the deep learning semantic segmentation model U-Net adopts an Encoder-Decoder (encoding-decoding) structure. In the embodiment, the U-Net model includes an input layer, an encoder (Encoder) 31 connected with the input layer, a decoder (Decoder) 32 connected with an output end of the encoder 31, and an output layer connected with an output end of the decoder 32, and preferably, the encoder 31 and the decoder 32 are connected through a bridge (Bridge) 33.

[0087] In the embodiment, the first time of the input layer is 320x320, so the size of each first fringe image input into the U-Net model is 320x320.

[0088] In the embodiment, the encoder 31 uses the first 3 layers of the backbone network based on Resnet50: conv1, conv2_x, and conv3_x, as shown in Table 2:

[0089] Table 2

[0090]

[0091] Please continue to refer to Figure 7The encoder 31 is composed of multiple layers of convolutional layers, including three times of down-sampling: 320→ 160→ 80→ 40. The bridge includes two layers of convolutional layers (conv), which are beneficial to increase the depth so as to learn deeper features of the input first fringe image and introduce more nonlinearity.

[0092] The decoder 32 part includes convolutional transposed layers (conv-transpose) and concatenate layers. The convolutional transposed layers can obtain parameters through deep learning training or bilinear interpolation. The convolutional transposed layers are used to fuse the image features at different depths of the encoder 31 part with the up-sampled image features. As shown in Figure 7 the embodiment, the decoder 32 part includes three concatenate layers, which perform three times of fusion respectively. In the embodiment, the last time of fusion is performed with the features in the original input first fringe image. The fusion includes the first fusion of the input end of the encoder with the decoder, the second fusion of the first down-sampled encoder with the decoder, and the last fusion of the first up-sampled decoder with the encoder.

[0093] The deep learning semantic segmentation model adopts a mean square error (MSE) to generate a Loss target function, which can be simply and effectively applied to weight updating. The optimizer (Optimizer) can adopt RMSprop, Adam, etc., and meanwhile, learning rate adjustment, early stopping, etc. training strategies are added, so that the model achieves good effects in training speed and effect.

[0094] The embodiment also provides a deep learning-based super-resolution detection method, which is applied to the super-resolution detection system 10. Please refer to Figure 8 The deep learning-based super-resolution detection method includes:

[0095] Step S21, at least one first fringe image is collected;

[0096] Step S22, deep information and shallow information of the at least one first fringe image are extracted based on a preset deep learning semantic segmentation model, and the deep information and the shallow information are fused to obtain semantic information;

[0097] Step S23, multiple second fringe images are generated according to the semantic information;

[0098] Step S24, biological information of the sample to be tested is obtained according to the at least one first fringe image and the multiple second fringe images.

[0099] The at least one first fringe image and the plurality of second fringe images constitute a source image, the source image corresponding to at least two different directions, and each of the directions having at least three source images.

[0100] Please refer to Figure 8 and Figure 9 In step S21, two first fringe images are collected, and the two first fringe images are captured by the image collection module 13 under the control of the controller 14.

[0101] Steps S22 and S23 are the same as steps S12 and S13 in the image generation method described above.

[0102] In step S24, the controller 14 is configured to perform super-resolution reconstruction on the two first fringe images and the four second fringe images, so as to obtain the biological information of the sample 20.

[0103] The super-resolution reconstruction process is described below.

[0104] In the super-resolution detection system 10, laser irradiation excites fluorescence on the sample 20 labeled with a fluorescent substance, and the image relationship is as follows:

[0105]

[0106] wherein r=(x, y) represents a two-dimensional spatial position vector, H(r) represents the PSF of the system, D(r) is the image distribution function, S(r) is the fluorescence concentration, I(r) is the excitation light intensity, represents convolution. For the super-resolution detection system 10, the laser intensity satisfies the cosine distribution, i.e.

[0107]

[0108] wherein I0 is the average intensity of the fringe structured light, m is the modulation degree, p is the illumination fringe frequency, is the fringe phase. Substituting equation (1) gives:

[0109]

[0110] At this time, D(r) is the collected image intensity distribution function after illumination by the fringe structured light. In order to further analyze the influence of the fringe structured light illumination on the spectrum, equation (3) is converted into the frequency domain by Fourier transform, and the spectral expression is:

[0111]

[0112] wherein is obtained by Fourier transform of S(r), The Fourier transform of H(r) is OTF, and according to the space translation property of delta function, formula (4) is simplified as:

[0113]

[0114] The analysis of formula (5) is easy to obtain, and the image function is In addition to the base frequency Two other frequencies And Thus, the spatial spectrum that the detector can collect is expanded to k∈[-k0-p, k0+p]. Thus, the image spectrum after the illumination of the stripe structure light contains high-frequency information, and the spatial frequency of the stripe illumination structure light determines the range of the expanded information. However, the expanded spectral information is currently at the wrong position. In order to obtain a super-resolution image, the expanded spectrum needs to be repositioned to the correct position.

[0115] The analysis of formula (5) shows that the collected spectral information contains three unknown spectral components. In order to obtain a mathematically accurate solution, at least three independent equations need to be constructed, and the modulation parameter of the initial phase is suitable for constructing independent equations. The stripe structure light with different initial phases is illuminated on the sample 20 to be measured, and the spectral information collected is:

[0116]

[0117] In this application, the initial phase is set to Substituting formula (6) and simplifying can obtain:

[0118]

[0119] After separating the three spectral components, the containing the high-frequency components of the sample is shifted to the correct position, that is, each is multiplied by the shift factor e i2π ,e -i0πpr The three spectra after correct shifting are superimposed (as shown in Figure 10B ), and finally an inverse Fourier transform is performed to convert to the spatial domain, that is, the diffraction limit in one dimension (X direction) is broken through to obtain a super-resolution image.

[0120] The above formula (1)-(7) is repeated, and the stripe structure light is projected on the sample 20 to be measured in the Y direction, so as to expand the spatial spectrum in the Y direction, thereby obtaining a two-dimensional expanded spectrum (as shown in Figure 10C ), and a nearly isotropic two-dimensional super-resolution image is obtained through two-dimensional inverse Fourier transform.

[0121] Thus, a super-resolution image of the sample 20 to be tested can be obtained through the super-resolution reconstruction process, and the controller 14 can obtain the biological information of the sample 20 to be tested (in this embodiment, the biological information is the base sequence of the sample 20 to be tested) based on the super-resolution image. The specific process of obtaining the biological information of the sample 20 to be tested based on the super-resolution image will not be described in detail in this application.

[0122] In this embodiment, the aforementioned stripe image is mainly evaluated in terms of phase and super-resolution (the following evaluation methods are all based on acquiring two first stripe images and generating four second stripe images):

[0123] First aspect: Phase evaluation for striped structured light.

[0124] In the super-resolution detection system 10 based on fringe structured light illumination, the first fringe image of the sample 20 to be tested (e.g., ...) is acquired. Figure 11A As shown), its spectrum is obtained by two-dimensional Fourier transform (as shown). Figure 11B (As shown).

[0125] As mentioned earlier, to obtain a two-dimensional super-resolution image, high-density fringes need to be projected in both the X and Y directions. Here, we assume the fringes are oriented as follows: Clearly, the fringe structured light in equation (2) is determined by the fringe direction θ and the fringe phase. The expression is: Equation (2) can be rewritten as:

[0126]

[0127] Where, p θ = (p·cosθ, p·sinθ) is the frequency vector of the illumination fringe. The corresponding fringe structured light illumination image is also rewritten from equation (3):

[0128]

[0129] Where N(r) represents the noise information, The spectrum is represented as

[0130] To evaluate the phase, the illumination fringe frequency p must first be evaluated. θ .

[0131] To eliminate OTF The impact, especially Multiply The conjugate variable, i.e.: Will Perform spectrum shift p θ After that, I received Therefore, calculation The correlation coefficient C1 between and θ is the illumination fringe frequency.

[0132]

[0133] Next, the fringe phase is evaluated.

[0134] The illumination fringe frequency p obtained by formula (10) θ and an arbitrary fringe phase is used as the initial estimate of the real fringe phase A two-dimensional cosine function is constructed in the spatial domain:

[0135]

[0136] In the spatial domain, the correlation coefficient C2 between and is calculated, and is gradually optimized by iteration When |C2| reaches its maximum value, the real fringe phase is gradually approached.

[0137] Using the above method, the fringe phase is evaluated in two cases: 1) six fringe images X1 / Y1 / X2 / X3 / Y2 / Y3 are actually collected by the image collection module 13; 2) two fringe images X1 / Y1 (i.e. the two first fringe images described in this embodiment) are actually collected by the image collection module 13, and four fringe images X2 / X3 / Y2 / Y3 (i.e. the four second fringe images described in this embodiment) are generated by deep learning.

[0138] The phase evaluation results (see Table 3) show that the fringe phase obtained by actually collecting two first fringe images by the image collection module 13 and generating four second fringe images by deep learning is basically consistent with the fringe phase obtained by actually collecting six fringe images by the image collection module 13, and the maximum fringe phase difference is 10.19 ° . Select as the maximum threshold of the fringe phase difference; the fringe phase generated by deep learning meets the requirements.

[0139] Table 3

[0140]

[0141]

[0142] Second aspect: super-resolution evaluation of fringe images.

[0143] The classical frequency domain super-resolution reconstruction method using formulas (1)-(7) is used to reconstruct the fringe images in two cases: 1) as shown in the figure. Figure 11A As shown, six stripe images X1 / Y1 / X2 / X3 / Y2 / Y3 are actually acquired by the image acquisition module 13; 2) as Figure 11B As shown, the image acquisition module 13 actually acquires two stripe images X1 / Y1 (that is, the two first stripe images in this embodiment), and deep learning generates four stripe images X2 / X3 / Y2 / Y3 (that is, the four second stripe images in this embodiment).

[0144] Figure 12 The first column shows the image of the sample 20 to be tested. The wide-field image here is obtained by weighted averaging of six stripe images. Figure 12 The second column shows a magnified view of a portion of the image, revealing that the super-resolution image has higher contrast and higher resolution. Figure 12 The third column is the spectrogram, which shows that the spectrum corresponding to the super-resolution image is expanded compared to the spectrum of the wide-field image. Figure 12 The fourth column shows the normalized cutoff frequency f of the spectrum calculated using decorrelation analysis. cutoff The spatial resolution is: ps is the pixel size, and the cutoff frequency f is visible. cutoff The higher the value, the higher the spatial resolution. Calculate the cutoff frequency f for each of the four spectrograms sequentially. cutoff The values ​​are: 0.858, 1.280, 0.852, and 1.254. Here, we make two comparisons:

[0145] Firstly, in the two cases, the resolutions of the wide-field image are 0.858 and 0.852 respectively, and the difference between them is as follows: The resolution difference is negligible, indicating that the wide-field image obtained by the deep learning stripe generation method is highly consistent with the wide-field image obtained by the method of actually acquiring six stripe images. Therefore, the deep learning method can replace the method of actually acquiring six stripe images.

[0146] Secondly, in the two cases, the resolutions of the super-resolution images are 1.280 and 1.254, respectively, and the difference between them is as follows: The resolution difference is negligible, indicating that the super-resolution image obtained by the deep learning stripe generation method is highly consistent with the super-resolution image obtained by the method of actually acquiring six stripe images. The deep learning method can replace the method of actually acquiring six stripe images.

[0147] The image generation method based on deep learning, the super-resolution detection method and the super-resolution detection system 10 provided by the embodiment are used to acquire biological information of a sample 20. A plurality of fringe images of the sample 20 need to be acquired respectively under the condition that different fringe structures are used to irradiate the sample 20. The embodiment can realize that a part of the plurality of fringe images is actually collected as a first fringe image, and the rest of the required images are generated as second fringe images based on a deep learning method. The number of collected fringe images is reduced, so as to shorten the image collection time, and further improve the detection speed. In addition, due to the significant reduction in the number of collected images and the significant shortening of the image collection time, the reagent can be fully saved, and the detection cost is further reduced.

[0148] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present application, and are not used as a limitation of the present application. Any appropriate changes and variations made to the above embodiments within the spirit and principle of the present application fall within the scope of the present application.

Claims

1. A deep learning-based image generation method, characterized by, The method comprises the following steps: receiving at least one first fringe image, the first fringe image being an image of a sample to be measured collected when the sample to be measured is irradiated by fringe structured light; extracting deep information and shallow information of the at least one first fringe image based on a preset deep learning semantic segmentation model, and fusing the deep information and the shallow information to obtain semantic information; generating a plurality of second fringe images according to the semantic information, the fringe direction in the at least one first fringe image being the same as the fringe direction in the plurality of second fringe images, and the fringe phase in the at least one first fringe image and the plurality of second fringe images being different from each other, the at least one first fringe image and the plurality of second fringe images being used to obtain biological information of the sample to be measured after super-resolution reconstruction.

2. The deep learning-based image generation method of claim 1, wherein, The preset deep learning semantic segmentation model comprises an input layer, an encoder connected with the input layer, a decoder connected with the output end of the encoder, and an output layer connected with the output end of the decoder, and the method comprises three fusions of the encoder and the decoder.

3. The deep learning-based image generation method of claim 2, wherein, The semantic segmentation model is a U-Net model, the encoder comprises an encoder based on a Resnet50 network, and the encoder is composed of a plurality of convolutional layers.

4. The deep learning-based image generation method of claim 2, wherein, The encoder performs three times of downsampling, and the decoder performs corresponding upsampling. 5.The deep learning based image generation method of claim 2, wherein, The fusion comprises a first fusion of the input end of the encoder and the decoder, a second fusion of the encoder after the first downsampling and the decoder, and a last fusion of the encoder after the first upsampling of the decoder. 6.The deep learning based image generation method of claim 1, wherein, The at least one first fringe image and the plurality of second fringe images constitute source images, the source images correspond to at least two different directions of the fringe structured light, and each direction has at least three source images.

7. The deep learning-based image generation method of claim 6, wherein, The step of receiving at least one first fringe image is to receive two first fringe images. The step of generating a plurality of second fringe images is to generate four second fringe images.

8. A deep learning-based super-resolution detection method, characterized in that, The method comprises the following steps: collecting at least one first fringe image, the first fringe image being an image of a sample to be measured collected when the sample to be measured is irradiated by fringe structured light; extracting deep information and shallow information of the at least one first fringe image based on a deep learning semantic segmentation model in the deep learning-based image generation method according to any one of claims 1-7, and fusing the deep information and the shallow information to obtain semantic information; generating a plurality of second fringe images according to the semantic information, the fringe direction in the at least one first fringe image being the same as the fringe direction in the plurality of second fringe images, and the fringe phase in the at least one first fringe image and the plurality of second fringe images being different from each other; performing super-resolution reconstruction according to the at least one first fringe image and the plurality of second fringe images to obtain biological information of the sample to be measured.

9. A deep learning based super-resolution detection system, characterized by, The method comprises the following steps: a light source module for emitting light source light; a light modulator for modulating the light source light into fringe structured light, the fringe structured light being capable of being guided to a sample to be measured to make the sample to be measured emit detection light; an image acquisition module configured to acquire at least one first fringe image according to the detection light; and a controller electrically connected to the light modulator and the image acquisition module, configured to adjust the direction and / or phase of the fringe structured light, and configured to generate a plurality of second fringe images based on the at least one first fringe image according to the deep learning-based image generation method of any one of claims 1-7, the fringe direction in the at least one first fringe image being the same as the fringe direction in the plurality of second fringe images, and the fringe phase in the at least one first fringe image and the plurality of second fringe images being different from each other, and perform super-resolution reconstruction according to the at least one first fringe image and the plurality of second fringe images to obtain the biological information of the sample to be detected.

10. The deep learning-based super resolution detection system of claim 9, wherein, the controller is configured to adjust the fringe direction of the fringe structured light; the image acquisition module is configured to acquire two first fringe images, the two first fringe images being images obtained when the sample to be detected is irradiated by fringe structured light with different fringe directions; the controller is configured to generate four second fringe images based on the two first fringe images. the controller is configured to adjust the fringe direction of the fringe structured light; the image acquisition module is configured to acquire two first fringe images, the two first fringe images being images obtained when the sample to be detected is irradiated by fringe structured light with different fringe directions; the controller is configured to generate four second fringe images based on the two first fringe images.

Images