Structured illumination obvious micro-reconstruction method based on physical model prior neural network

Abstract

The invention provides a structural illumination obvious micro-reconstruction method based on a physical model prior neural network, and the method comprises the steps: obtaining experimental data which comprises an image original input sequence; inputting experimental data into the convolutional neural network initialization model to obtain model initial output; performing parameter iterative optimization training on the convolutional neural network initialization model by taking Fourier transform and spatial similarity as constraint conditions and combining with initial output of the model to obtain a target convolutional neural network model; and outputting a model output image corresponding to the target convolutional neural network model as a target image. According to the method, on one hand, the convolutional neural network is adopted for re-parameterization, the robustness of reconstruction under the noise condition is improved, and meanwhile the consistency of a reconstruction result and an input data source is guaranteed; and on the other hand, the convergence speed and the convergence stability of the whole reconstruction process are improved by adopting a two-stage strategy of pre-training and fine tuning.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a structured illumination microscopic reconstruction method based on a physical model priori neural network. Background Art

[0002] In recent years, the field of super-resolution microscopy has received extensive attention. Because it can break through the diffraction limit and observe finer-grained objects or microscopic structures, it has important applications in many fields such as microbiology, medicine, and life sciences. Among them, structured light illumination microscopy technology has become one of the mainstream super-resolution microscopy technologies due to its advantages such as low loss of optical power and fast imaging speed.

[0003] However, the super-resolution effect of structured illumination microscopy is often affected by the reconstruction algorithm. Most of the early reconstruction algorithms of structured illumination microscopy were based on numerical calculation or iterative optimization design, which had problems such as slow convergence, sensitivity to noise, and many hyperparameters that need to be adjusted. With the rise of machine learning technology, this technology can learn specific pattern features from images and mapping rules established by physical models, which can solve many inverse process problems under pathological conditions, and therefore has received widespread attention. At present, although the use of machine learning for structured illumination microscopy reconstruction has played some advantages, the scheme of reconstructing structured illumination microscopy data based on trained single-stage neural networks still has certain problems in generalization and reliability. Summary of the invention

[0004] The purpose of the present invention is to provide a structured illumination microscopic reconstruction method based on a physical model priori neural network, so as to solve the problem of structured illumination microscopic reconstruction based on a physical model priori neural network.

[0005] In a first aspect, the present application provides a structured illumination microscopic reconstruction method based on a physical model prior neural network, the method comprising:

[0006] Acquiring experimental data, wherein the experimental data includes an original input sequence of images;

[0007] Inputting the experimental data into the convolutional neural network initialization model to obtain the model initial output;

[0008] Based on Fourier transform and spatial similarity as constraints and combined with the initial output of the model, the convolutional neural network initialization model is trained with parameter iterative optimization, wherein:

[0009] When the loss function value corresponding to the convolutional neural network reaches the target value, the iterative training is stopped to obtain the target convolutional neural network model;

[0010] The model output image corresponding to the target convolutional neural network model is output as the target image.

[0011] In a possible implementation of the present application, the method further includes generating the convolutional neural network initialization model, specifically including:

[0012] Generate paired simulated data from a natural image database;

[0013] After the paired simulation data are grouped according to the direction and phase of the illumination light field, they are input into the preset convolutional neural network model for training.

[0014] The supervised training loss function is used for training, and after the training is completed, the model weight parameters of the preset convolutional neural network model are obtained;

[0015] Construct a basic network model with the same structure as the preset convolutional neural network model, and initialize the basic network model based on the model weight parameters to obtain the convolutional neural network initialization model, wherein the structure of the basic network model is a group convolution residual neural network.

[0016] In a possible implementation of the present application, the obtaining of experimental data specifically includes:

[0017] Get input data entered by the user;

[0018] Based on the input data, data type identification is performed to obtain a frame image;

[0019] The frame images are grouped to obtain an original image input sequence, and the experimental data is obtained based on the original image input sequence.

[0020] In a possible implementation of the present application, inputting the experimental data into a convolutional neural network initialization model to obtain an initial output of the model specifically includes:

[0021] The grouped original image input sequence is input into the convolutional neural network initialization model for training, wherein:

[0022] Based on the frame images in the original input sequence of the image, the intra-group feature extraction, feature fusion and reconstruction are performed to obtain the initial output of the model.

[0023] In a possible implementation of the present application, the intra-group feature extraction processing step specifically includes:

[0024] Performing group feature extraction on the feature map, wherein the feature map includes an original input sequence of images;

[0025] performing data normalization on the feature map; and

[0026] activating the feature map to introduce nonlinearity; and

[0027] The size of the feature map is adjusted to adjust the receptive field.

[0028] In a possible implementation of the present application, the feature fusion and reconstruction processing steps specifically include:

[0029] Performing an overall feature transformation on the feature map;

[0030] performing data normalization on the feature map; and

[0031] activating the feature map to introduce nonlinearity; and

[0032] The feature map is resized to restore the feature map size.

[0033] In a possible implementation of the present application, constraints are performed based on Fourier transform and spatial similarity, specifically including:

[0034] Get the model output image

[0035] Based on the model output image, the target prediction sequence is obtained by using matrix algorithm changes The calculation formula is as follows:

[0036]

[0037] Based on the target prediction sequence and the experimental data, Fourier transform and spatial similarity constraint are performed, wherein the calculation formula is as follows:

[0038]

[0039]

[0040] Among them, x, y represent each frame of experimental data image and its corresponding predicted image, μ x represents the image mean of the experimental data image, μ y represents the mean of the predicted image, σ x represents the standard deviation of the experimental data image, σ y represents the standard deviation of the predicted image, L fourier represents Fourier transform, L SSIM represents the spatial similarity, I n represents the original input sequence of images, represents the target prediction sequence, P n represents the lighting stripes, H(x,y) represents the point spread function, and n represents the number of original input sequence values ​​of the image.

[0041] In a second aspect, the present application provides a structured illumination microscopic reconstruction system based on a physical model prior neural network, the system comprising:

[0042] An acquisition module, used for acquiring experimental data, wherein the experimental data includes an original input sequence of images;

[0043] An input module, used to input the experimental data into the convolutional neural network initialization model to obtain the initial output of the model;

[0044] A training module, used to perform parameter iterative optimization training on the initialization model of the convolutional neural network based on Fourier transform and spatial similarity as constraints combined with the initial output of the model, wherein the iterative training is stopped when the loss function value corresponding to the convolutional neural network reaches a target value to obtain a target convolutional neural network model;

[0045] An output module is used to output the model output image corresponding to the target convolutional neural network model as a target image.

[0046] In a third aspect, the present application provides a computer-readable storage medium as described above, on which a computer program is stored, and when the program is executed by a processor, the structured illumination microscopic reconstruction method based on a physical model prior neural network is implemented.

[0047] In a fourth aspect, the present application provides an electronic device as mentioned above, comprising: a processor and a memory; wherein the memory is used to store a computer program, and the processor is used to load and execute the computer program so that the electronic device executes the structured illumination microscopic reconstruction method based on a physical model prior neural network.

[0048] As described above, the structured illumination microscopic reconstruction method based on the physical model prior neural network of the present invention, on the one hand, adopts a convolutional neural network for re-parameterization, which improves the robustness of reconstruction under noisy conditions, while ensuring the consistency of the reconstruction result with the input data source; on the other hand, by adopting a two-stage strategy of pre-training plus fine-tuning, the convergence speed and convergence stability of the overall reconstruction process are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram showing the method steps of a structured illumination microscopic reconstruction method based on a physical model prior neural network according to an embodiment of the present invention;

[0050] Figure 2 It is a schematic diagram showing the method steps of a structured illumination microscopic reconstruction method based on a physical model prior neural network according to an embodiment of the present invention;

[0051] Figure 3It is a schematic diagram showing the method steps of a structured illumination microscopic reconstruction method based on a physical model prior neural network according to an embodiment of the present invention;

[0052] Figure 4 It is a schematic diagram showing the training of a preset convolutional neural network model in an embodiment of the structured illumination microscopic reconstruction method based on a physical model prior neural network of the present invention;

[0053] Figure 5 It is a schematic diagram showing the method steps of a structured illumination microscopic reconstruction method based on a physical model prior neural network according to an embodiment of the present invention;

[0054] Figure 6 It is a schematic diagram showing the method steps of a structured illumination microscopic reconstruction method based on a physical model prior neural network according to an embodiment of the present invention;

[0055] Figure 7 It is a schematic diagram showing the structure of a residual convolutional neural network of group convolution in one embodiment of the structured illumination microscopic reconstruction method based on a physical model prior neural network of the present invention;

[0056] Figure 8 It is a schematic structural diagram of a structured illumination microscopic reconstruction system based on a physical model prior neural network according to an embodiment of the present invention;

[0057] Fig. 9 Shown is a schematic structural diagram of an electronic device in an embodiment of the present invention.

[0058] Component number description

[0059] Steps S102 to S110

[0060] Steps S202 to S206

[0061] Steps S302 to S308

[0062] Steps S502 to S508

[0063] Steps S602 to S608

[0064] 80 Structured illumination microscopic reconstruction system based on physical model prior neural network

[0065] 81 Get Module

[0066] 82 Input Module

[0067] 83 Training Modules

[0068] 84 Output Module DETAILED DESCRIPTION

[0069] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0070] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0071] Traditional structured light illumination microscopic reconstruction methods are mostly based on numerical calculation or iterative optimization design. Among them, numerical calculation methods generally construct a linear equation system for solution, and iterative optimization methods establish the optimal form, make an initial guess for the high-resolution target, and then gradually mine the original data information to update the target guess. Traditional numerical calculation methods or optimization methods often have problems such as slow convergence speed, sensitivity to noise, and many hyperparameters that need to be adjusted.

[0072] As an artificial intelligence technology, machine learning aims to mine the intrinsic associations of data through learning and approximate specific physical models or functional mapping relationships. Machine learning can be roughly divided into supervised learning and unsupervised learning. Supervised learning provides data and paired real labels to the model, and the model learns features from a large number of data pairs and establishes a mapping from data to labels. In unsupervised learning, there are generally no real data labels. The model analyzes the intrinsic structure of the data through potential prior constraints (such as physical models, closed loops, etc.) to complete specific tasks.

[0073] The application of machine learning technology to computational imaging problems is generally carried out in a data-driven manner using convolutional neural networks as carriers. The input of the convolutional neural network is experimental data, such as image data or one-dimensional signals obtained by the detector, and the output is generally an imaging target (such as super-resolution images in structured illumination microscopy). The convolutional neural network input and output often constitute an inverse process mapping on a physical model. In recent years, there have been schemes that apply machine learning technology to the process of structured illumination microscopy reconstruction using trained single-stage networks, but there are deficiencies in generalization and reliability.

[0074] In view of the problems existing in the prior art, the present application provides a structured illumination microscopic reconstruction method based on a physical model prior neural network. On the one hand, the present application adopts a convolutional neural network for heavy parameterization, which improves the robustness of reconstruction in noisy scenes while ensuring the reliability of the reconstruction results; on the other hand, a trained convolutional neural network with the same structure is used for weight initialization, which improves the stability and convergence speed of the iteration and reconstruction process. The image will be clearer, more resistant to noise, and have a better signal-to-noise ratio, which has important application significance in biomedicine, life sciences and other fields.

[0075] See also Figure 1 In one embodiment of the invention, the structured illumination microscopic reconstruction method based on physical model prior neural network of the present invention comprises the following steps:

[0076] Step S102, acquiring experimental data, wherein the experimental data includes an original input sequence of images;

[0077] Step S104, inputting the experimental data into a convolutional neural network initialization model to obtain an initial output of the model;

[0078] Step S106, performing iterative parameter optimization training on the convolutional neural network initialization model based on Fourier transform and spatial domain similarity as constraint conditions combined with the initial output of the model;

[0079] Step S108, when the loss function value corresponding to the convolutional neural network reaches the target value, the iterative training is stopped to obtain the target convolutional neural network model;

[0080] Step S110, outputting the model output image corresponding to the target convolutional neural network model as the target image.

[0081] It should be noted that obtaining experimental data, such as Figure 2 As shown, the specific steps include:

[0082] Step S202, obtaining input data input by the user terminal;

[0083] Step S204, identifying the data type based on the input data to obtain a frame image;

[0084] Step S206: grouping the frame images to obtain an original input sequence of images, and obtaining the experimental data based on the original input sequence of images.

[0085] Specifically, in this embodiment, for the data sequence collected by the user experiment (this embodiment takes the low-resolution images including "9" images and each group of "3" images as an example) I n, the user needs to input it into the neural network for processing. Therefore, the input data input by the user end must be obtained first, and then the frame image is obtained by type recognition based on the input data. The frame image is grouped to obtain the original input sequence of the image (that is, it corresponds to three groups of sequences, each group of sequences includes three low-resolution images), and then the experimental data is input into the test, where each group of images will obtain a set of high-resolution image predictions. The high-resolution image prediction is obtained by the forward physical model of structured light illumination to obtain a low-resolution prediction sequence, which corresponds to the target prediction sequence. Among them, the forward physical model is a known quantity, and then the parameters of the convolutional neural network initialization model are adjusted based on the Fourier transform and spatial domain similarity as constraints combined with the target prediction sequence. Specifically, the Fourier domain consistency and spatial domain similarity between the low-resolution prediction sequence and the collected real data sequence are calculated: Where x, y represent each frame of experimental data image and its corresponding predicted image, μ x represents the image mean of the experimental data image, μ y represents the mean of the predicted image, σ x represents the standard deviation of the experimental data image, σ y represents the standard deviation of the predicted image, L fourier represents Fourier transform, L SSIM represents the spatial similarity, I n represents the original input sequence of images, represents the target prediction sequence, P n represents the lighting stripe, H(x,y) represents the point spread function, and n represents the number of original input sequences of the image to minimize L fourier +L SSIM The neural network parameters are fine-tuned for the objective function. Specifically, the deep learning framework Pytorch and the optimizer Adam are used to calculate the corresponding gradients of the parameters in the neural network under the optimization objective in each round, and the learning rate "0.0001" is used to iteratively fine-tune the parameters. The parameter update amount in each iteration is the product of the learning rate and the gradient. When the loss function value corresponding to the convolutional neural network reaches the target value, the iterative training is stopped to obtain the target convolutional neural network model, or when the number of iterations reaches the target number of iterations, the iterative training is stopped, for example, when the number of iterations reaches "500" times or the objective function does not decrease for "20" consecutive rounds, the iteration is stopped to obtain the network parameters of the final target convolutional neural network model, and the model output image corresponding to the target convolutional neural network model is output as the target image to obtain the corresponding high-resolution image reconstruction result.

[0086] Furthermore, in one embodiment of the invention, the method further comprises generating the convolutional neural network initialization model, such as Figure 3 As shown, the specific steps include:

[0087] Step S302, generating paired simulation data from a natural image database;

[0088] Step S304, after the paired simulation data are grouped according to the direction and phase of the illumination light field, the data are input into a preset convolutional neural network model for training;

[0089] Step S306, using a supervised training loss function to perform training, and after the training is completed, obtaining the model weight parameters of the preset convolutional neural network model;

[0090] Step S308, constructing a basic network model with the same structure as the preset convolutional neural network model, and initializing the basic network model based on the model weight parameters to obtain the convolutional neural network initialization model;

[0091] It should be noted that in this embodiment, a large number of structured illumination microscopic paired data under different experimental settings (such as illumination light fields with different sinusoidal orientations and different sinusoidal phases, different levels of Gaussian noise, etc.) are generated from a natural image database by simulation, wherein each data pair includes "1" high-resolution image and "9" low-resolution image sequences of "3" images each. Specifically, Figure 4 As shown, it shows the preset convolutional neural network model training process, in which the high-resolution image I obj Forward physical model with structured light illumination Get low resolution sequence I n (n=1,…,9), where P n ,H(x,y),N(x,y) are the illumination stripes, point spread function and Gaussian noise term respectively. The specific grouping can be based on the illumination stripes direction and phase n Group them and get every "3" pieces as a group Among them, the structure of the basic network model f θ (·) Specifically, it is a group convolutional residual neural network. Since the Gaussian noise term cannot be obtained when using experimental data for training, when using experimental data for processing, the target prediction sequence The processing of Gaussian noise terms is not included.

[0092] Furthermore, in one embodiment of the invention, the experimental data is input into a convolutional neural network initialization model to obtain an initial output of the model, specifically comprising the following steps: inputting the grouped original input sequence of images into the convolutional neural network initialization model for training; and performing intra-group feature extraction based on frame images in the original input sequence of images, as well as feature fusion and reconstruction to obtain an initial output of the model.

[0093] It should be noted that when the experimental data is input into the convolutional neural network initialization model for iterative training, there will be a model output for each training, which will be used as the initial output of the model, wherein the grouped original input sequence of the image is input into the convolutional neural network initialization model for training, so as to perform intra-group feature extraction based on the frame images in the original input sequence of the image, and obtain the initial output of the model by feature fusion and reconstruction, and then each iterative training can be performed, and the parameters of the neural network model can be adjusted to obtain the final target convolutional neural network model and target image.

[0094] Specifically, in one embodiment of the invention, Figure 5 As shown, the intra-group feature extraction processing step specifically includes the following steps:

[0095] Step S502, performing group feature extraction on the feature map, wherein the feature map includes an original input sequence of images;

[0096] Step S504, performing data normalization on the feature map;

[0097] Step S506, activating the feature map to introduce nonlinearity;

[0098] Step S508: adjusting the size of the feature map to adjust the receptive field.

[0099] And, if Figure 6 As shown, the feature fusion and reconstruction processing steps specifically include the following steps:

[0100] Step S602, performing overall feature transformation on the feature map;

[0101] Step S604, performing data normalization on the feature map;

[0102] Step S606, activating the feature map to introduce nonlinearity;

[0103] Step S608, adjusting the size of the feature map to restore the size of the feature map.

[0104] It should be noted that, in this embodiment, Figure 7As shown, it is a schematic diagram of the residual convolutional neural network structure of group convolution, wherein, when performing intra-group feature extraction, the corresponding module of the network structure specifically includes "7" group convolution blocks with the number of groups being "3" and the total number of channels being "9, 18, 18, 36, 36, 72, 144" respectively. Each group convolution block includes a group convolution layer, a batch normalization layer and an activation function layer, and a maximum pooling layer with a step size of "2" is added at the end of every two group convolution blocks. Specifically, the group convolution block performs learnable feature extraction and feature transformation on the input image or feature map, batch normalization normalizes the data distribution of the feature map to between 0 and 1, the activation function layer activates the feature map to introduce nonlinearity, and the maximum pooling layer reduces the spatial size of the feature map and increases the receptive field.

[0105] Furthermore, when performing feature fusion and reconstruction, the corresponding module of the network structure specifically includes "7" convolution blocks with channel numbers of "144, 72, 72, 72, 54, 54, 36", each convolution block contains a convolution layer, a batch normalization layer and an activation function layer, and an upsampling layer with a step size of "2" is added at the end of every two convolution blocks. The convolution block performs an overall feature transformation on the feature map, batch normalization normalizes the data distribution of the feature map to between 0 and 1, the activation function layer activates the feature map to introduce nonlinearity, and the upsampling layer restores the spatial size of the feature map. In addition, the intermediate layer features obtained by the corresponding module of feature extraction within the group are sequentially spliced ​​into the feature map fusion module through jump connections.

[0106] An embodiment of the present application also provides a structured illumination microscopic reconstruction system based on a physical model prior neural network. The structured illumination microscopic reconstruction system based on a physical model prior neural network can implement the structured illumination microscopic reconstruction method based on a physical model prior neural network described in the present application. However, the implementation device of the structured illumination microscopic reconstruction method based on a physical model prior neural network described in the present application includes but is not limited to the structure of the structured illumination microscopic reconstruction system based on a physical model prior neural network listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0107] See also Figure 8 In one embodiment, this embodiment provides a structured illumination microscopic reconstruction system 80 based on a physical model prior neural network, the system comprising:

[0108] An acquisition module 81 is used to acquire experimental data, wherein the experimental data includes an original input sequence of images;

[0109] An input module 82, used to input the experimental data into the convolutional neural network initialization model to obtain the model initial output;

[0110] A training module 83 is used to perform parameter iterative optimization training on the initialization model of the convolutional neural network based on Fourier transform and spatial structural similarity as constraints combined with the initial output of the model, wherein the iterative training is stopped when the loss function value corresponding to the convolutional neural network reaches a target value to obtain a target convolutional neural network model;

[0111] The output module 84 is used to output the model output image corresponding to the target convolutional neural network model as the target image.

[0112] Since the specific implementation of this embodiment corresponds to the aforementioned method embodiment, the same details will not be repeated here, and those skilled in the art should also understand that Figure 8 The division of the modules in the embodiment is merely a division of logical functions, and in actual implementation, all or part of them can be integrated into one or more physical entities, and these modules can be implemented entirely in the form of software called by processing elements, or entirely in the form of hardware, or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware.

[0113] See also Fig. 9 This embodiment provides an electronic device. Specifically, the electronic device includes at least a memory and a processor connected via a bus, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to execute all or part of the steps in the aforementioned method embodiment.

[0114] In summary, on the one hand, the present invention adopts convolutional neural network for re-parameterization, which improves the robustness of reconstruction under noisy conditions, while ensuring the consistency of the reconstruction result with the input data source; on the other hand, by adopting a two-stage strategy of pre-training plus fine-tuning, the convergence speed and convergence stability of the overall reconstruction process are improved.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0116] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0117] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] The present application embodiment also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by a program to instruct the processor, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state hard disk (SSD)), etc.

[0119] The present application embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer or data center.

[0120] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.

[0121] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0122] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A structured illumination microscopic reconstruction method based on a physical model prior neural network, characterized in that: The method comprises: Acquiring experimental data, wherein the experimental data includes an original input sequence of images; Inputting the experimental data into the convolutional neural network initialization model to obtain the model initial output; Based on Fourier transform and spatial similarity as constraints and combined with the initial output of the model, the convolutional neural network initialization model is trained with parameter iterative optimization, wherein: When the loss function value corresponding to the convolutional neural network reaches the target value, the iterative training is stopped to obtain the target convolutional neural network model; The model output image corresponding to the target convolutional neural network model is output as the target image.

2. The structured light illumination microscopic reconstruction method based on physical model prior neural network according to claim 1, characterized in that: The method further includes generating the convolutional neural network initialization model, specifically including: Generate paired simulated data from a natural image database; After the paired simulation data are grouped according to the direction and phase of the illumination light field, they are input into the preset convolutional neural network model for training. The supervised training loss function is used for training, and after the training is completed, the model weight parameters of the preset convolutional neural network model are obtained; Construct a basic network model with the same structure as the preset convolutional neural network model, and initialize the basic network model based on the model weight parameters to obtain the convolutional neural network initialization model, wherein the structure of the basic network model is a group convolution residual neural network.

3. The structured light illumination microscopic reconstruction method based on physical model prior neural network according to claim 1, characterized in that: The obtaining of experimental data specifically includes: Get input data entered by the user; Based on the input data, data type identification is performed to obtain a frame image; The frame images are grouped to obtain an original image input sequence, and the experimental data is obtained based on the original image input sequence.

4. The structured light illumination microscopic reconstruction method based on physical model prior neural network according to claim 3 is characterized in that: The step of inputting the experimental data into the convolutional neural network initialization model to obtain the initial output of the model specifically includes: The grouped original input sequence of images is input into the convolutional neural network initialization model for training, wherein, based on the frame images in the original input sequence of images, intra-group feature extraction is performed, and feature fusion and reconstruction are performed to obtain the initial output of the model.

5. The structured light illumination microscopic reconstruction method based on physical model prior neural network according to claim 4 is characterized in that: The intra-group feature extraction processing steps specifically include: Performing group feature extraction on the feature map, wherein the feature map includes an original input sequence of images; performing data normalization on the feature map; and activating the feature map to introduce nonlinearity; and The size of the feature map is adjusted to adjust the receptive field.

6. The structured light illumination microscopic reconstruction method based on physical model prior neural network according to claim 5, characterized in that: The feature fusion and reconstruction processing steps specifically include: Performing an overall feature transformation on the feature map; performing data normalization on the feature map; and activating the feature map to introduce nonlinearity; and The feature map is resized to restore the feature map size.

7. The structured light illumination microscopic reconstruction method based on physical model prior neural network according to claim 6, characterized in that: Constraints are imposed based on Fourier transform and spatial similarity, including: Get the model output image Based on the model output image, the target prediction sequence is obtained by using matrix algorithm changes The calculation formula is as follows: Based on the target prediction sequence and the experimental data, Fourier transform and spatial similarity constraint are performed, wherein the calculation formula is as follows: Among them, x, y represent each frame of experimental data image and its corresponding predicted image, μ x represents the image mean of the experimental data image, μ y represents the mean of the predicted image, σ x Represents the standard deviation of the experimental data image, σ y represents the standard deviation of the predicted image, L fourier represents Fourier transform, L SSIM represents the spatial similarity, I n represents the original input sequence of images, represents the target prediction sequence, P n represents the lighting stripes, H(x,y) represents the point spread function, and n represents the number of original input sequence values ​​of the image.

8. A structured illumination microscopic reconstruction system based on a physical model prior neural network, characterized in that: The system comprises: An acquisition module, used for acquiring experimental data, wherein the experimental data includes an original input sequence of images; An input module, used to input the experimental data into the convolutional neural network initialization model to obtain the initial output of the model; A training module, used to perform parameter iterative optimization training on the initialization model of the convolutional neural network based on Fourier transform and spatial similarity as constraints combined with the initial output of the model, wherein the iterative training is stopped when the loss function value corresponding to the convolutional neural network reaches a target value to obtain a target convolutional neural network model; An output module is used to output the model output image corresponding to the target convolutional neural network model as a target image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the structured illumination microscopic reconstruction method based on a physical model prior neural network as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the structured illumination microscopic reconstruction method based on a physical model prior neural network as described in any one of claims 1 to 7.

Images