Cyclic neural network assisted wavefront sensing method and device based on holographic projection

Abstract

Embodiments of the present application provide a holographic projection-based recurrent neural network auxiliary wavefront sensing method and device, electronic equipment, computer readable storage medium and computer program product, which are related to the fields of microscopic imaging and artificial intelligence. The method comprises: acquiring a first fluorescence image; then determining a region of interest of the first fluorescence image; and then performing wavefront detection processing on the region of interest of the first fluorescence image based on a recurrent neural network to obtain target wavefront information corresponding to the region of interest of the first fluorescence image, wherein the target wavefront information comprises target wavefront aberration. The embodiments of the present application improve the efficiency of wavefront detection.

Description

Holographic projection-based recurrent neural network-assisted wavefront sensing method and device Technical Field

[0001] This application relates to the fields of microscopic imaging and artificial intelligence technology. Specifically, this application relates to a method, device, electronic device, computer-readable storage medium, and computer program product based on a holographic projection-assisted recurrent neural network for wavefront sensing. Background Technology

[0002] With the rapid development of microscopic imaging technology, it has begun to be applied in various fields, such as the application of multiphoton fluorescence microscopy in the life sciences. Multiphoton fluorescence microscopy, as an important imaging method in the life sciences, has gradually attracted widespread attention due to its unique illumination and detection methods. Multiphoton microscopy achieves specific labeling of substances by activating fluorophores through the multiphoton nonlinear absorption effect of fluorophores. Due to its nonlinear effect, fluorophore activation only occurs at the focal point, resulting in: 1) excellent optical tomography. Its operating wavelength is close to an integer multiple of the single-photon excitation wavelength of the fluorophore, with common wavelengths being 780nm, 920nm, and 1030nm. Within this wavelength range, 2) biological tissues and other samples experience minimal absorption and scattering, ensuring good penetration of the probe light and effectively detecting thick samples. Simultaneously, lasers capable of exciting nonlinear absorption are typically femtosecond or picosecond pulsed lasers with high peak energy and low average power, resulting in: 3) less thermal damage to the sample. The above three points constitute the main advantages of multiphoton fluorescence microscopy over confocal microscopy. However, when faced with imaging of extremely thick samples and samples with non-uniform refractive index distribution, there is still a problem of low imaging sharpness. It is necessary to introduce adaptive optics for wavefront detection in order to correct the fluorescence image of the imaging system.

[0003] In related technologies, wavefront detection is required for the entire acquired fluorescence image, which results in low efficiency of wavefront detection. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for wavefront sensing based on holographic projection and recurrent neural network-assisted wavefront sensing, which solves the technical problem of low efficiency in wavefront detection.

[0005] According to one aspect of the embodiments of this application, a recurrent neural network-assisted wavefront sensing method based on holographic projection is provided, comprising:

[0006] Acquire the first fluorescence image;

[0007] Determine the region of interest in the first fluorescence image;

[0008] A recurrent neural network is used to perform wavefront detection processing on the region of interest of the first fluorescence image to obtain the target wavefront information corresponding to the region of interest of the first fluorescence image. The target wavefront information includes the target wavefront aberration.

[0009] In one possible implementation, determining the region of interest in the first fluorescence image includes:

[0010] In response to a selection operation on the display interface displaying the first fluorescence image, a region of interest in the first fluorescence image is determined; and / or,

[0011] The region of interest in the first fluorescence image is determined based on the region in the first fluorescence image that meets the preset conditions of interest.

[0012] In one possible implementation, the conditions of interest include at least one of the following:

[0013] The target proportion corresponding to the region is greater than the preset proportion threshold;

[0014] The target percentage for a given region shall not be lower than the target percentage for other regions.

[0015] The target proportion is the ratio of the number of connected target pixels in the region to the total number of pixels in the region, and the pixel value of the target pixel is greater than a set intensity threshold.

[0016] In one possible implementation, the first fluorescence image is generated based on at least two probe wavefronts;

[0017] Wavefront detection processing is performed on the region of interest (ROI) of the first fluorescence image based on a recurrent neural network to obtain the target wavefront information corresponding to the ROI of the first fluorescence image, including:

[0018] A first fluorescence intensity sequence is obtained, which includes at least two first fluorescence intensities. The at least two first fluorescence intensities are obtained by applying at least two probe wavefronts to each pixel of interest in the region of interest of the first fluorescence image. The at least two first fluorescence intensities correspond one-to-one with at least two probe wavefronts. The first fluorescence intensity is related to the fluorescence intensity of each pixel of interest when the same probe wavefront is applied to the region of interest of the first fluorescence image.

[0019] The target wavefront information corresponding to the region of interest in the first fluorescence image is determined based on a recurrent neural network and the first fluorescence intensity sequence.

[0020] In one possible implementation, the target wavefront information corresponding to the region of interest in the first fluorescence image is determined based on a recurrent neural network and a first fluorescence intensity sequence, including:

[0021] The first fluorescence intensity sequence is input into the trained wavefront detection model to obtain the target wavefront aberration output by the wavefront detection model.

[0022] The training methods for wavefront detection models include:

[0023] At least two training data pairs are acquired, each training data pair including wavefront aberration and a second fluorescence intensity sequence. The second fluorescence intensity sequence includes at least two second fluorescence intensities. The at least two second fluorescence intensities are obtained by applying at least two probe wavefronts to each pixel of interest in the region of interest of the second fluorescence image. The at least two second fluorescence intensities correspond one-to-one with at least two probe wavefronts. The second fluorescence intensity is related to the fluorescence intensity of each pixel of interest when the same probe wavefront is applied to the region of interest of the second fluorescence image. The second fluorescence image is generated based on at least two probe wavefronts.

[0024] A recurrent neural network is trained using at least two training data pairs to obtain a wavefront detection model.

[0025] In one possible implementation, the first fluorescence intensity is linearly positively correlated with the sum of the fluorescence intensities of the pixels of interest in the region of interest of the first fluorescence image;

[0026] The second fluorescence intensity is linearly positively correlated with the sum of the fluorescence intensities of the pixels of interest in the region of interest of the second fluorescence image.

[0027] In one possible implementation, the first and second fluorescence images are generated based on a pre-defined number of terms of at least two probe wavefronts and Zelnik polynomials.

[0028] According to another aspect of the embodiments of this application, a recurrent neural network-assisted wavefront sensing device based on holographic projection is provided, comprising:

[0029] The image acquisition module is used to acquire the first fluorescence image;

[0030] The region of interest determination module is used to determine the region of interest in the first fluorescence image;

[0031] The wavefront detection module is used to perform wavefront detection processing on the region of interest of the first fluorescence image based on a recurrent neural network to obtain the target wavefront information corresponding to the region of interest of the first fluorescence image, including the target wavefront aberration.

[0032] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of any of the above aspects.

[0033] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps of the methods of any of the above aspects.

[0034] According to another aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the methods described in any of the above aspects.

[0035] The beneficial effects of the technical solution provided in this application embodiment are as follows: by determining the region of interest of the first fluorescence image and performing wavefront detection processing on the region of interest of the first fluorescence image, since the region for wavefront detection processing is the region of interest of the first fluorescence image, this embodiment can perform wavefront detection on the region of interest. Therefore, this application embodiment can solve the technical problem in the related technology that it is necessary to perform wavefront detection on the entire fluorescence image that has been completely acquired, resulting in low efficiency of wavefront detection, thereby achieving the technical effect of improving the efficiency of wavefront detection. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0037] Figure 1 is a schematic diagram of a complete fluorescence image provided in an embodiment of this application;

[0038] Figure 2 is a schematic diagram of the system architecture provided in an embodiment of this application;

[0039] Figure 3 is a flowchart illustrating a holographic projection-based recurrent neural network-assisted wavefront sensing method provided in an embodiment of this application.

[0040] Figures 4a-4b illustrate a 3D spatial scanning scheme using DMD holographic projection mode provided in an embodiment of this application.

[0041] Figure 5 is a schematic diagram illustrating the working principle of wavefront prediction based on a recurrent neural network according to an embodiment of this application.

[0042] Figure 6 is a schematic diagram of the structure of a holographic projection-based recurrent neural network-assisted wavefront sensing device provided in an embodiment of this application;

[0043] Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0045] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” indicates implementation as “A,” or implementation as “A,” or implementation as “A and B.”

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0047] First, let's introduce and explain several terms used in this application:

[0048] Holography is a technique that uses optical media or computers to record and reproduce the positional information of three-dimensional objects through the principles of interference and diffraction. In the reconstruction phase, the hologram is illuminated with the same incident light beam as in the generation phase, thus obtaining a holographically encoded three-dimensional image. Holography can be used for optical storage and reproduction, as well as for information processing.

[0049] Wavefront: A wavefront is the surface formed by the particles that have just begun to displace at a given moment as a wave propagates through a medium. It represents the spatial position where the wave energy has reached at that moment, and it is in motion. The wavefront is orthogonal to the ray, but in anisotropic media such as birefringent crystals, the ray is generally not perpendicular to the wavefront. Therefore, using rays or wavefronts to study waves is equivalent. Based on the shape of the wavefront, waves can generally be classified into spherical waves, plane waves, cylindrical waves, etc. The wavefront is the surface formed by points in phase during the propagation of a light wave. A light wave is a transverse wave with its vibration surface perpendicular to the direction of propagation; the wavefront is a plane formed by electromagnetic vibrations. The wavefront of the entire light wave is formed by points in phase within different wavefronts. The propagation of a light wave is actually a process of continuous wavefront reproduction. Point sources form spherical wavefronts, while parallel sources form plane wavefronts.

[0050] Wavefront aberration: Wavefront aberration is defined as the deviation between the actual wavefront and the ideal, unbiased wavefront. It is the distance between the waveform formed by a spherical wave emitted from a point source after passing through an optical system and the ideal spherical wave. The meaning of wavefront aberration can be expressed by Zernike polynomials or geometric aberrations such as spherical aberration and coma. Ideal lenses have surfaces whose curvature varies with the prescription diopter, and the curved portion is calculated to correct various aberrations and distortions of individual lenses.

[0051] Zernike polynomials: Aberrations in optical systems are typically described using power series expansions. Because the Zernike polynomials and the aberration polynomials observed in optical detection have the same form, they are often used to describe wavefront characteristics. They can be described as functions of radial and angular coordinates, and are divided into odd and even classes. Their mathematical form is as follows:

[0052]

[0053] Through the above decomposition, any wavefront can be represented as a number of orthogonal wavefront terms, meaning that each wavefront can be uniquely determined by a set of definite coefficients. Therefore, the goal of finding systematic aberrations for correction can be transformed into the process of finding Zelnik coefficients that can correctly represent the wavefront.

[0054] The region of interest (ROI) refers to the area in the image being processed that needs to be delineated using shapes such as rectangles, circles, ellipses, or irregular polygons in machine vision and image processing.

[0055] In related techniques, wavefront detection is required on the entire acquired fluorescence image. However, in actual fluorescence holograms, only a portion of the area may be useful.

[0056] Figure 1 is a schematic diagram of a complete fluorescence image provided in an embodiment of this application. The complete fluorescence image shown in Figure 1 includes sparsely distributed structures such as axons and dendrites, as well as cellular regions. In Figure 1, the focus is primarily on the cellular regions.

[0057] Although only a portion of the image may be of interest, the relevant techniques still perform wavefront detection on the entire fluorescence image, resulting in low efficiency of wavefront detection.

[0058] Therefore, the holographic projection-based recurrent neural network-assisted wavefront sensing method, device, electronic device, computer-readable storage medium, and computer program product provided in this application aim to solve the above-mentioned technical problem of low efficiency in existing wavefront detection.

[0059] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0060] Figure 2 is a schematic diagram of the system architecture provided in an embodiment of this application. The system architecture shown in Figure 2 includes a microscope device and electronic equipment. The microscope device includes an optical path structure and a microscope. The microscope device is used to form an image, while the electronic equipment acquires a first fluorescence image, which covers the field of view of the holographic fluorescence microscope. The electronic equipment determines the region of interest (ROI) of the first fluorescence image and performs wavefront detection processing on the ROI of the first fluorescence image based on a recurrent neural network to obtain the target wavefront information corresponding to the ROI of the first fluorescence image. The target wavefront information includes target wavefront aberrations.

[0061] Figure 2 shows the optical structure of a sensorless adaptive optics laser scanning fluorescence (AO-TPE) microscope system based on a digital micromirror device (DMD). The laser source is a Ti:sapphire femtosecond laser. Since the DMD simultaneously functions as a programmable binary mask and a blazed grating that introduces negative dispersion, a grating is placed in the optical path to pre-compensate for dispersion in the laser beam. Beam expanders L1 and L2 are placed between the grating and the DMD to match different dispersion angles. After the DMD, the dispersion-free laser beam is guided through L3, L4, and L5 to the infinity correction objective (OL1). A spatial filter is placed on the back focal plane of L3 to spatially suppress all diffracted beams except for the first-order diffraction of the hologram. The fluorescence emitted by the sample is transmitted through a dichroic mirror (DM) to a photomultiplier tube for generating the fluorescence image.

[0062] To simultaneously achieve modal wavefront correction and 3D random scanning via a DMD, Zernike polynomials describing wavefront distortion are superimposed onto the designed scanning wavefront. The synthesized wavefront is then converted into a binary pattern using a binary holographic scheme, and finally loaded into the DMD to manipulate the incident dispersive laser beam.

[0063] Optionally, the electronic devices in this embodiment include, but are not limited to, user terminals or servers. Servers include, but are not limited to, physical servers or cloud servers, and may also be server clusters. The aforementioned user terminals include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, in-vehicle terminals, wearable electronic devices, AR / VR devices, etc.

[0064] Please refer to Figure 3, which is a flowchart illustrating a holographic projection-based recurrent neural network-assisted wavefront sensing method provided in this embodiment. This embodiment uses an application of the holographic projection-based recurrent neural network-assisted wavefront sensing method in an electronic device as an example. As shown in Figure 3, the holographic projection-based recurrent neural network-assisted wavefront sensing method includes:

[0065] S310, Acquire the first fluorescence image.

[0066] The first fluorescence image can be a pre-generated fluorescence image. In this embodiment, optionally, the first fluorescence image can cover the field of view of the holographic fluorescence microscope. Specifically, the first fluorescence image covering the field of view of the holographic fluorescence microscope can be understood as each scanning point in the microscope's field of view having at least one corresponding pixel in the first fluorescence image. Optionally, the first fluorescence image in this embodiment can be an image marked with fluorescence, which is not limited here. The first fluorescence image in this embodiment can be a holographic image, which can be a binary holographic image, such as a Lee hologram. A Lee hologram refers to a binary holographic image generated using the Lee holographic generation scheme.

[0067] Please refer to Figures 4a-4b, which illustrate a 3D spatial scanning scheme using a DMD holographic projection mode provided in this application embodiment. As shown in Figure 4a, axial scanning can be achieved by controlling the collimation of the laser beam entering the objective lens, where converging or diverging beams will respectively move the focal point toward or away from the objective lens; the dashed line represents the original focal plane of the collimated beam. This concept can be achieved by using a DMD to shape the wavefront of the input beam into a spherical wavefront; and the radius of the spherical wavefront directly determines the position of the focal point. In other words, the DMD can act as either a convex or concave mirror, as shown in Figure 4b. The phase of the spherical wavefront can be expressed as:

[0068]

[0069] The binary holographic image generated by the Lee holographic generation scheme can generate a binary hologram of any user-defined DMD spherical wavefront:

[0070]

[0071] R(x,y)=x·sin(α)+y·cos(α)

[0072] Where R(x, y) is the term controlling the lateral scanning of the focus, and φ(x, y) is the term controlling the axial movement of the focus. p(x, y) represents the actively added item (the controlled object in this scheme). The meanings and functions of other parameters are as follows: λ represents the wavelength of the system, f represents the equivalent focal length of the control wavefront, q (0≤q≤1 / 2) is a constant that controls the fringe width, T is a constant that controls the distance of the generated hologram order, α represents the parameter that controls the hologram angle, and k represents an integer.

[0073] In this embodiment, after the first fluorescence image is generated, it can be sent to an electronic device for further processing.

[0074] S320. Determine the region of interest in the first fluorescence image.

[0075] In this embodiment, the region of interest (ROI) can be an area that requires further observation. It should be noted that the ROI in the first fluorescence image can be one or more regions, depending on the actual situation, and is not limited here. As shown in Figure 1, the ROI in the first fluorescence image can be a cellular region, etc., and is not limited here. Optionally, in this embodiment, the ROI can cover all or part of the field of view of the holographic fluorescence microscope, depending on the actual situation, and is not limited here.

[0076] Generally, strong fluorescence intensity indicates a dense aggregation of fluorescent molecules, meaning that the experimental object to be observed is located there, and this area is designated as the region of interest.

[0077] S330. Based on a recurrent neural network, wavefront detection processing is performed on the region of interest of the first fluorescence image to obtain the target wavefront information corresponding to the region of interest of the first fluorescence image. The target wavefront information includes the target wavefront aberration.

[0078] Recurrent Neural Networks (RNNs) are a type of recursive neural network that takes sequence data as input, recurses along the sequence's direction, and connects all nodes (recurrent units) in a chain-like manner. RNNs possess memory, parameter sharing, and Turing completeness, giving them an advantage in learning the nonlinear features of sequences. RNNs are used in Natural Language Processing (NLP), such as in speech recognition, language modeling, and machine translation, and are also used for various time series forecasting applications. RNNs constructed using Convolutional Neural Networks (CNNs) can handle computer vision problems involving sequential inputs.

[0079] The region of interest (ROI) is a region within the first fluorescence image. The size of the ROI is less than or equal to the size of the region within the first fluorescence image. In some cases, the ROI may be smaller than the size of the region within the first fluorescence image.

[0080] The technical solution of this embodiment determines the region of interest (ROI) of a first fluorescence image and performs wavefront detection processing on the ROI based on a recurrent neural network. Since the region undergoing wavefront detection processing is the ROI of the first fluorescence image, this embodiment can perform wavefront detection on the ROI. Therefore, this embodiment solves the technical problem in related technologies where wavefront detection is required on the entire acquired fluorescence image, resulting in low wavefront detection efficiency, thereby improving the efficiency of wavefront detection. Furthermore, since the wavefront detection processing is performed on a specific ROI, targeted correction can be performed, thereby improving the wavefront correction effect and enhancing the microscopic imaging effect. Additionally, if there are at least two ROIs, each ROI can undergo targeted wavefront detection, thereby improving the flexibility and accuracy of wavefront detection.

[0081] In one possible implementation, determining the region of interest in the first fluorescence image includes:

[0082] In response to a selection operation on the display interface displaying the first fluorescence image, the region of interest of the first fluorescence image is determined.

[0083] In this embodiment, a first fluorescence image can be displayed on a display interface first, and the region of interest of the first fluorescence image can be determined based on the user's selection operation on the display interface.

[0084] Optionally, the region of interest can be a preset area centered on the location selected by the user in the first fluorescence image.

[0085] The technical solution of this embodiment can determine the region of interest according to the user's needs, which can improve the user experience and the flexibility of determining the region of interest.

[0086] In another possible implementation, determining the region of interest in the first fluorescence image includes:

[0087] The region of interest in the first fluorescence image is determined based on the region in the first fluorescence image that meets the preset conditions of interest.

[0088] The conditions of interest can be predetermined and used to determine the region of interest in the first fluorescence image. In this embodiment, any region in the first fluorescence image can be judged to determine whether the conditions of interest are met. If they are met, the region is designated as the region of interest. Optionally, the size of the region can be set as needed, for example, 50% of the first fluorescence image can be used as the region to determine whether the conditions of interest are met; this is not limited here.

[0089] Optionally, the conditions of interest include at least one of the following:

[0090] The target proportion corresponding to the region is greater than the preset proportion threshold;

[0091] The target percentage for a given region shall not be lower than the target percentage for other regions.

[0092] The target proportion is the ratio of the number of connected target pixels in the region to the total number of pixels in the region, and the pixel value of the target pixel is greater than a set intensity threshold.

[0093] Generally, the region of interest (ROI) is a region with high brightness. For example, in a first fluorescence image including cells, the ROI is the cellular region of a nerve cell, which is also a region with high brightness. In this embodiment, the ROI is a region where fluorescent dots are relatively concentrated. Therefore, the ROI can be determined based on the region where fluorescent dots are concentrated in the first fluorescence image. Here, pixel values ​​can be brightness values, and intensity thresholds can be brightness thresholds. Target pixels can be understood as brighter dots. In this embodiment, optionally, if the ratio of the number of connected target pixels in a region to the total number of pixels in that region is greater than a preset percentage threshold, then there are more fluorescent dots in that region, and that region is the ROI. It is understood that the preset percentage threshold can be set as needed, for example, to 50%, and is not limited here. Optionally, if the ratio of the number of connected target pixels in a region to the total number of pixels in that region is not lower than the ratio of the number of connected target pixels in other regions to the total number of pixels in that region, then the proportion of target pixels in that region is the highest, and there are more fluorescent dots in that region, and that region is the ROI. Optionally, the region of interest can be determined by a combination of the above conditions. For example, if the target proportion of a region is greater than a preset proportion threshold and the target proportion of a region is not lower than the target proportion of other regions, then the region is considered a region of interest.

[0094] Understandably, determining whether a region is a region of interest by checking whether the target percentage of the region is greater than a preset percentage threshold, and whether the target percentage of the region is the largest among all regions, can improve the accuracy of region of interest determination.

[0095] The technical solution of this embodiment improves the efficiency of determining the region of interest by setting interest conditions and determining whether each region in the first fluorescence image meets the interest conditions.

[0096] In another possible implementation, determining the region of interest in the first fluorescence image includes:

[0097] In response to a selection operation on the display interface displaying the first fluorescence image, the region of interest of the first fluorescence image is determined;

[0098] The region of interest in the first fluorescence image is determined based on the region in the first fluorescence image that meets the preset conditions of interest.

[0099] In this embodiment, the user can first select a certain area on the display interface, and then determine whether the surrounding areas of the selected area meet the conditions of interest, using the selected area as the center area.

[0100] Optionally, one pixel in the peripheral area and one pixel in the central area can be the same pixel.

[0101] The technical solution of this embodiment improves the recognition efficiency of determining the region of interest by selecting a certain area on the display interface and then determining whether the surrounding areas of the selected area meet the conditions of interest.

[0102] The following embodiments, based on any of the above embodiments, further illustrate how to perform wavefront detection processing on the region of interest of the first fluorescence image.

[0103] In one possible implementation, the first fluorescence image is generated based on at least two probe wavefronts.

[0104] Accordingly, wavefront detection processing is performed on the region of interest (ROI) of the first fluorescence image based on a recurrent neural network to obtain the target wavefront information corresponding to the ROI of the first fluorescence image, including:

[0105] A first fluorescence intensity sequence is obtained, which includes at least two first fluorescence intensities. The at least two first fluorescence intensities are obtained by applying at least two probe wavefronts to each pixel of interest in the region of interest of the first fluorescence image. The at least two first fluorescence intensities correspond one-to-one with at least two probe wavefronts. The first fluorescence intensity is related to the fluorescence intensity of each pixel of interest when the same probe wavefront is applied to the region of interest of the first fluorescence image.

[0106] The target wavefront information corresponding to the region of interest in the first fluorescence image is determined based on a recurrent neural network and the first fluorescence intensity sequence.

[0107] In this embodiment, optionally, a probe wavefront Wp1 can be applied to each pixel of interest within the region of interest (ROI) of the first fluorescence image; then, the fluorescence intensity of each pixel of interest is recorded when the probe wavefront Wp1 is applied to the ROI of the first fluorescence image; and finally, the first fluorescence intensity corresponding to the probe wavefront Wp1 is determined based on the fluorescence intensity of each pixel of interest. Then, a probe wavefront Wp2 can be applied to each pixel of interest within the ROI of the first fluorescence image; then, the fluorescence intensity of each pixel of interest is recorded when the probe wavefront Wp2 is applied to the ROI of the first fluorescence image; and finally, the first fluorescence intensity corresponding to the probe wavefront Wp2 is determined based on the fluorescence intensity of each pixel of interest. This process is repeated, with each probe wavefront having a corresponding first fluorescence intensity. Thus, at least two corresponding first fluorescence intensities can be obtained based on at least two probe wavefronts, thereby obtaining a first fluorescence intensity sequence.

[0108] Specifically, in practical applications, since the wavefront phase of light is usually a complex curved surface, it can be effectively optically realized by selecting an orthogonal basis to decompose it into a sum of mutually orthogonal functions. In holographic fluorescence microscopy, Zernike coefficients are a widely used set of orthogonal bases that can effectively simulate wavefront types of different orders, achieving good fitting results. Therefore, by adding the equivalent Zernike coefficients to the wavefront, the phase distribution function of the light wavefront is generated, and a Lee hologram is used to transform the phase distribution into a corresponding binary hologram. When the laser point loading the probe wavefront acts on the fluorescent molecule, the fluorescence intensity is correlated with the incident light intensity (the intensity and the loaded wavefront have a nonlinear relationship). Therefore, the fluorescence intensity is collected as the basis for determining the target wavefront aberration.

[0109] In one possible implementation, the target wavefront information corresponding to the region of interest in the first fluorescence image can be determined based on the first fluorescence intensity sequence and a specific functional relationship.

[0110] In another possible implementation, the target wavefront information corresponding to the region of interest in the first fluorescence image is determined based on a recurrent neural network and a first fluorescence intensity sequence, including:

[0111] The first fluorescence intensity sequence is input into the trained wavefront detection model to obtain the target wavefront aberration output by the wavefront detection model.

[0112] In this embodiment, the first fluorescence intensity sequence can be input into the trained wavefront detection model. The wavefront detection model can then perform wavefront detection processing based on the first fluorescence intensity sequence to determine the wavefront phase difference corresponding to the first fluorescence intensity sequence. The wavefront phase difference corresponding to the first fluorescence intensity sequence is then used as the target wavefront aberration corresponding to the region of interest in the first fluorescence image.

[0113] The training methods for wavefront detection models include:

[0114] At least two training data pairs are acquired, each training data pair including wavefront aberration and a second fluorescence intensity sequence. The second fluorescence intensity sequence includes at least two second fluorescence intensities. The at least two second fluorescence intensities are obtained by applying at least two probe wavefronts to each pixel of interest in the region of interest of the second fluorescence image. The at least two second fluorescence intensities correspond one-to-one with at least two probe wavefronts. The second fluorescence intensity is related to the fluorescence intensity of each pixel of interest when the same probe wavefront is applied to the region of interest of the second fluorescence image. The second fluorescence image is generated based on at least two probe wavefronts.

[0115] A recurrent neural network is trained using at least two training data pairs to obtain a wavefront detection model.

[0116] In this embodiment, optionally, a probe wavefront Wp1 can first be applied to each pixel of interest within the region of interest (ROI) of the second fluorescence image; then, the fluorescence intensity of each pixel of interest is recorded when the probe wavefront Wp1 is applied to the ROI of the second fluorescence image; and finally, the second fluorescence intensity corresponding to the probe wavefront Wp1 is determined based on the fluorescence intensity of each pixel of interest. Then, a probe wavefront Wp2 can be applied to each pixel of interest within the ROI of the second fluorescence image; then, the fluorescence intensity of each pixel of interest is recorded when the probe wavefront Wp2 is applied to the ROI of the second fluorescence image; and finally, the second fluorescence intensity corresponding to the probe wavefront Wp2 is determined based on the fluorescence intensity of each pixel of interest. This process is repeated, with each probe wavefront having a corresponding second fluorescence intensity. Thus, at least two corresponding second fluorescence intensities can be obtained based on at least two probe wavefronts, resulting in a second fluorescence intensity sequence.

[0117] In this embodiment, a recurrent neural network is trained using at least two training datasets. After the recurrent neural network model is trained, it can be used as the wavefront detection model in this embodiment.

[0118] The technical solution of this embodiment improves the accuracy of wavefront detection by combining recurrent neural networks with data pairs because recurrent neural networks can extract temporal and semantic information from data pairs.

[0119] Optionally, the first fluorescence intensity is linearly positively correlated with the sum of the fluorescence intensities of the pixels of interest in the region of interest of the first fluorescence image.

[0120] Optionally, the second fluorescence intensity is linearly positively correlated with the sum of the fluorescence intensities of the pixels of interest in the region of interest of the second fluorescence image.

[0121] It should be noted that the criteria for determining the region of interest in the first fluorescence image can be the same as those for determining the region of interest in the second fluorescence image, and the shape and size of the region of interest in the first fluorescence image and the region of interest in the second fluorescence image are the same.

[0122] In one possible implementation, the first and second fluorescence images are generated based on a pre-defined number of terms of at least two probe wavefronts and Zelnik polynomials.

[0123] Optionally, the preset number is less than or equal to the total number of terms in the Zelnik polynomial.

[0124] In this embodiment, the first and second fluorescence images are generated by at least two probe wavefronts and a pre-preset number of terms of the Zernike polynomial, which can balance image generation quality and efficiency.

[0125] Please refer to Figure 5, which is a schematic diagram illustrating the working principle of wavefront prediction based on a recurrent neural network according to an embodiment of this application. As shown in Figure 5, the wavefront aberration in the training data pair can be used as the real target, and the second fluorescence intensity sequence can be used as the training data. The training data pair is then encoded. The encoded data is modulated by a light modulator, and then the modulated data is data-grabbing and input into the recurrent neural network model for training. After the recurrent neural network model makes predictions based on the input data, it obtains the predicted label. By comparing the real predicted label with the predicted label, the model loss of the recurrent neural network model can be obtained. Based on the model loss, it can be determined whether the recurrent neural network model has been trained successfully. When training is successful, the trained recurrent neural network model is output as the wavefront detection model.

[0126] The following examples illustrate how DMD is used to randomly scan a binary hologram to obtain a fluorescence image, and how artificial intelligence is used for wavefront detection, based on any of the above examples.

[0127] S0. Using the Lee hologram generation scheme (see Figure 4), generate scanning points that cover the entire microscope field of view to obtain a fluorescence image of the entire field of view. Strong fluorescence intensity indicates a dense aggregation of fluorescent molecules, meaning the experimental object we want to observe is present. This area is designated as the region of interest (ROI), and its analysis is performed to obtain the RIO. It is worth noting that the working area only needs to be within the irregularly enclosed region of the field of view shown in Figure 2; complete coverage of the entire internal area is not required.

[0128] The Lee hologram is a binary image with alternating bright and dark stripes. The 1 / 0 values ​​represent the brightness, and its mathematical expression is as follows:

[0129]

[0130]

[0131] R(x,y)=x·sin(α)+y·cos(α)

[0132] Where R(x, y) is the term controlling the lateral scanning of the focus, and φ(x, y) is the term controlling the axial movement of the focus. p (x, y) represents the actively added item (the controlled object in this scheme). The meanings and functions of other parameters are as follows: λ represents the wavelength of the system, f represents the equivalent focal length of the control wavefront, q (0≤q≤1 / 2) is a constant that controls the fringe width, T is a constant that controls the distance of the generated hologram order, α represents the parameter that controls the hologram angle, and k represents an integer.

[0133] S1. The wavefront aberration to be detected is defined as W. t The probe wavefront sequentially applied to this wavefront is W. p .

[0134] S2. Design a probe wavefront W for a fixed-length sequence. p : Specifies the order of the Zelnik polynomials that the system needs to correct, such as the first 17 terms (using the Noll sequence); Specifies the sequence length of the probe wavefront, such as 1000 terms, i.e., W p，i ={Z1, Z2, ..., Z 17}, max(i) = 1000.

[0135] S3. Generate the probe wavefront hologram designed in S1 using the Lee holographic generation scheme and upload it to the DMD. Substitute each probe wavefront into the formula, i.e. This method allows you to add a probe wavefront at a specified spatial location. Note that this method is applicable to a single spatial location. If you need to generate the wavefront at multiple spatial locations, you can repeat this step for each location.

[0136] S4. Based on S3, scan the region of interest obtained in S0 at 22kHz, keeping the scanned region unchanged, and apply a different probe wavefront W to each point within the region of interest in each scan. p The fluorescence intensities obtained from the scanned region are summed to obtain the intensity of that scan. By changing the probe wavefront loaded in the region of interest in turn and repeating this process, the intensity sequence can be obtained.

[0137] And record the intensity sequence I at each point. pp,i max(pp) = point number max(i) = 1000.

[0138] S5, Change W as defined in S1 t But keep W p Repeat S2-S5 as before to collect training data.

[0139] Training phase (see Figure 5):

[0140] S6. Determine the data pairs for the input network. The original data pairs are in the format Data. pp ={label:W t,pp |value:I pp,1 ,I pp,2 ,…,I pp,1000}, where pp is the training sample number.

[0141] S7. Adjust I according to the actual network needs. pp The values ​​are standardized to a form acceptable to the network for training, and the parameters are adjusted to obtain the final convergent model. RNNAO Its implemented mapping is f:I pp →W t .

[0142] Optionally, the intensity sequence of length L needs to be transformed into an array of length M×N (=L), where M represents the step size of the RNN and N represents the feature size, so as to input the signal into the network for training.

[0143] Usage phase:

[0144] S8. Following the method in S0, obtain the region of interest requiring correction. Add a probe wavefront to the scanned single-point location according to method S3, and acquire intensity data at a rate of 22 kHz. In practical implementation, since the wavefront phase of light is usually a complex curved surface, optical realization can be effectively achieved by selecting an orthogonal basis to decompose it into a sum of mutually orthogonal functions. In holographic fluorescence microscopy, Zernike coefficients are a widely used set of orthogonal bases that can effectively simulate wavefront types of different orders, achieving good fitting results. Therefore, adding the equivalent Zernike coefficients to the wavefront generates a two-dimensional phase distribution function of the light wavefront surface, and using a Lee hologram to transform the phase distribution into the corresponding hologram. When the laser point with the probe wavefront is applied to the fluorescent molecule, the fluorescence intensity is correlated with the incident light intensity (the intensity and the applied wavefront have a non-linear relationship). Therefore, the fluorescence intensity is acquired as the intensity of our acquired signal.

[0145] S9. The collected single-point intensity sequences are organized into a form acceptable to the network and input into the network to obtain the target wavefront information at the point where wavefront correction is performed, i.e., W. p,test =Model RNNAO (I test ).

[0146] Please refer to Figure 6. Figure 6 is a schematic diagram of a holographic projection-based recurrent neural network-assisted wavefront sensing device provided in an embodiment of this application. As shown in Figure 6, the holographic projection-based recurrent neural network-assisted wavefront sensing device 60 may include an image acquisition module 601, a region of interest determination module 602, and a wavefront detection module 603, wherein:

[0147] Image acquisition module 601 is used to acquire a first fluorescence image;

[0148] The region of interest determination module 602 is used to determine the region of interest in the first fluorescence image;

[0149] The wavefront detection module 603 is used to perform wavefront detection processing on the region of interest of the first fluorescence image based on a recurrent neural network to obtain the target wavefront information corresponding to the region of interest of the first fluorescence image, the target wavefront information including the target wavefront aberration.

[0150] In one possible implementation, the conditions of interest include:

[0151] The proportion of target pixels within the region is greater than a preset proportion threshold; and / or,

[0152] The proportion of target pixels within a region is not less than the proportion of target pixels within any other region;

[0153] The target pixel value is used to indicate that the pixel is the foreground of the first fluorescence image.

[0154] In one possible implementation, the first fluorescence image is generated based on at least two probe wavefronts;

[0155] The wavefront detection module 603 is used to acquire a first fluorescence intensity sequence, which includes at least two first fluorescence intensities. The at least two first fluorescence intensities are obtained by applying at least two detection wavefronts to each pixel of interest in the region of interest of the first fluorescence image. The at least two first fluorescence intensities correspond one-to-one with at least two detection wavefronts. The first fluorescence intensity is related to the fluorescence intensity of each pixel of interest when the same detection wavefront is applied to the region of interest of the first fluorescence image.

[0156] The target wavefront information corresponding to the region of interest in the first fluorescence image is determined based on a recurrent neural network and the first fluorescence intensity sequence.

[0157] In one possible implementation, the wavefront detection module 603 is used to input the first fluorescence intensity sequence into the trained wavefront detection model to obtain the target wavefront aberration output by the wavefront detection model.

[0158] The training methods for wavefront detection models include:

[0159] At least two training data pairs are acquired, each training data pair including wavefront aberration and a second fluorescence intensity sequence. The second fluorescence intensity sequence includes at least two second fluorescence intensities. The at least two second fluorescence intensities are obtained by applying at least two probe wavefronts to each pixel of interest in the region of interest of the second fluorescence image. The at least two second fluorescence intensities correspond one-to-one with at least two probe wavefronts. The second fluorescence intensity is related to the fluorescence intensity of each pixel of interest when the same probe wavefront is applied to the region of interest of the second fluorescence image. The second fluorescence image is generated based on at least two probe wavefronts.

[0160] A recurrent neural network is trained using at least two training data pairs to obtain a wavefront detection model.

[0161] In one possible implementation, the neural network model includes a recurrent neural network model.

[0162] In one possible implementation, the first fluorescence intensity is linearly positively correlated with the sum of the fluorescence intensities of the pixels of interest in the region of interest of the first fluorescence image;

[0163] The second fluorescence intensity is linearly positively correlated with the sum of the fluorescence intensities of the pixels of interest in the region of interest of the second fluorescence image.

[0164] In one possible implementation, the first and second fluorescence images are generated based on a pre-defined number of terms of at least two probe wavefronts and Zelnik polynomials.

[0165] The holographic projection-based recurrent neural network-assisted wavefront sensing device of this embodiment can execute the steps of the holographic projection-based recurrent neural network-assisted wavefront sensing method shown in the foregoing embodiments of this application. The implementation principle is similar and will not be repeated here.

[0166] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the holographic projection-based recurrent neural network-assisted wavefront sensing method described above.

[0167] In one optional embodiment, an electronic device is provided, as shown in FIG7. The electronic device 7000 shown in FIG7 includes a processor 7001 and a memory 7003. The processor 7001 and the memory 7003 are connected, for example, via a bus 7002. Optionally, the electronic device 7000 may further include a transceiver 7004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 7004 is not limited to one type, and the structure of the electronic device 7000 does not constitute a limitation on the embodiments of this application.

[0168] Processor 7001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 7001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0169] Bus 7002 may include a pathway for transmitting information between the aforementioned components. Bus 7002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 7002 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 7, but this does not indicate that there is only one bus or one type of bus.

[0170] The memory 7003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0171] The memory 7003 is used to store computer programs that execute the embodiments of this application, and its execution is controlled by the processor 7001. The processor 7001 is used to execute the computer programs stored in the memory 7003 to implement the steps shown in the foregoing method embodiments.

[0172] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0173] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0174] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0175] The above are only optional implementation methods for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application, without departing from the technical concept of this application, also fall within the protection scope of the embodiments of this application.

Claims

1. A recurrent neural network-assisted wavefront sensing method based on holographic projection, characterized in that, include: Acquire the first fluorescence image; Determine the region of interest in the first fluorescence image; Based on a recurrent neural network, wavefront detection processing is performed on the region of interest of the first fluorescence image to obtain the target wavefront information corresponding to the region of interest of the first fluorescence image, wherein the target wavefront information includes target wavefront aberrations; The first fluorescence image was generated based on at least two probe wavefronts; The step of performing wavefront detection processing on the region of interest (ROI) of the first fluorescence image based on a recurrent neural network to obtain target wavefront information corresponding to the ROI of the first fluorescence image includes: acquiring a first fluorescence intensity sequence, the first fluorescence intensity sequence including at least two first fluorescence intensities, the at least two first fluorescence intensities being obtained by applying the at least two detector wavefronts to each pixel of ROI in the first fluorescence image, the at least two first fluorescence intensities corresponding one-to-one with the at least two detector wavefronts, and the first fluorescence intensity being related to the fluorescence intensity of each pixel of ROI when the same detector wavefront is applied to the ROI of the first fluorescence image; determining the target wavefront information corresponding to the ROI of the first fluorescence image based on the recurrent neural network and the first fluorescence intensity sequence; and determining the target wavefront information corresponding to the ROI of the first fluorescence image based on the recurrent neural network and the first fluorescence intensity sequence. The information includes: inputting the first fluorescence intensity sequence into a trained wavefront detection model to obtain the target wavefront aberration output by the wavefront detection model; wherein the training method of the wavefront detection model includes: acquiring at least two training data pairs, each training data pair including a wavefront aberration and a second fluorescence intensity sequence, the second fluorescence intensity sequence including at least two second fluorescence intensities, the at least two second fluorescence intensities being obtained by applying the at least two detection wavefronts to each pixel of the region of interest in the second fluorescence image, the at least two second fluorescence intensities corresponding one-to-one with the at least two detection wavefronts, the second fluorescence intensity being related to the fluorescence intensity of each pixel of interest when the same detection wavefront is applied to the region of interest in the second fluorescence image, the second fluorescence image being generated based on the at least two detection wavefronts; and training a recurrent neural network using the at least two training data pairs to obtain the wavefront detection model.

2. The method according to claim 1, characterized in that, Determining the region of interest (ROI) of the first fluorescence image includes: determining the ROI of the first fluorescence image in response to a selection operation on a display interface displaying the first fluorescence image; and / or, determining the ROI of the first fluorescence image based on regions in the first fluorescence image that satisfy preset ROI conditions.

3. The method according to claim 2, characterized in that, The interest conditions include at least one of the following: the target proportion corresponding to the region is greater than a preset proportion threshold; the target proportion corresponding to the region is not lower than the target proportion corresponding to other regions; wherein, the target proportion is the ratio of the number of connected target pixels in the region to the total number of pixels in the region, and the pixel value of the target pixel is greater than a set intensity threshold.

4. The method according to claim 1, characterized in that, The first fluorescence intensity is linearly positively correlated with the sum of the fluorescence intensities of the pixels of interest in the region of interest of the first fluorescence image; the second fluorescence intensity is linearly positively correlated with the sum of the fluorescence intensities of the pixels of interest in the region of interest of the second fluorescence image.

5. The method according to claim 1, characterized in that, The first fluorescence image and the second fluorescence image are generated based on a pre-preset number of terms of at least two probe wavefronts and Zernike polynomials.

6. A recurrent neural network-assisted wavefront sensing device based on holographic projection, characterized in that, include: The image acquisition module is used to acquire the first fluorescence image; A region of interest determination module is used to determine the region of interest in the first fluorescence image; A wavefront detection module is used to perform wavefront detection processing on the region of interest of the first fluorescence image based on a recurrent neural network to obtain target wavefront information corresponding to the region of interest of the first fluorescence image, wherein the target wavefront information includes target wavefront aberrations. The first fluorescence image was generated based on at least two probe wavefronts; The step of performing wavefront detection processing on the region of interest (ROI) of the first fluorescence image based on a recurrent neural network to obtain target wavefront information corresponding to the ROI of the first fluorescence image includes: acquiring a first fluorescence intensity sequence, the first fluorescence intensity sequence including at least two first fluorescence intensities, the at least two first fluorescence intensities being obtained by applying the at least two detector wavefronts to each pixel of ROI in the first fluorescence image, the at least two first fluorescence intensities corresponding one-to-one with the at least two detector wavefronts, and the first fluorescence intensity being related to the fluorescence intensity of each pixel of ROI when the same detector wavefront is applied to the ROI of the first fluorescence image; determining the target wavefront information corresponding to the ROI of the first fluorescence image based on the recurrent neural network and the first fluorescence intensity sequence; and determining the target wavefront information corresponding to the ROI of the first fluorescence image based on the recurrent neural network and the first fluorescence intensity sequence. The information includes: inputting the first fluorescence intensity sequence into a trained wavefront detection model to obtain the target wavefront aberration output by the wavefront detection model; wherein the training method of the wavefront detection model includes: acquiring at least two training data pairs, each training data pair including a wavefront aberration and a second fluorescence intensity sequence, the second fluorescence intensity sequence including at least two second fluorescence intensities, the at least two second fluorescence intensities being obtained by applying the at least two detection wavefronts to each pixel of the region of interest in the second fluorescence image, the at least two second fluorescence intensities corresponding one-to-one with the at least two detection wavefronts, the second fluorescence intensity being related to the fluorescence intensity of each pixel of interest when the same detection wavefront is applied to the region of interest in the second fluorescence image, the second fluorescence image being generated based on the at least two detection wavefronts; and training a recurrent neural network using the at least two training data pairs to obtain the wavefront detection model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1-5.

Images