Feature domain full-view-field reconstruction system containing adjustable suppression factors

Abstract

The invention provides a feature domain full-view-field reconstruction system containing an adjustable suppression factor, and the system comprises the steps: firstly introducing an adjustable suppression factor into the loss of a feature domain, and constructing a gradient modulation item dominated by the adjustable suppression factor; secondly, providing a suggested adjustment range and a suggested adjustment strategy of an inhibition factor; finally, the effect of the inhibition factor in feature utilization and gradient updating is elaborated. Therefore, the utilization degree of the sample image features is flexibly adjusted by introducing the adjustable suppression factor, the problem that details of a reconstruction result are fuzzy or distorted due to the fact that the image features cannot be fully utilized in an existing full-view-field reconstruction technology is solved, and meanwhile the full-view-field reconstruction quality reaches the level of a traditional method.

Description

Technical Field

[0001] This invention belongs to the field of microscopic imaging technology, and particularly relates to a feature domain full-field reconstruction system containing an adjustable suppression factor. Background Technology

[0002] Fourier transform imaging (FTIM) is a quantitative phase imaging technique that can simultaneously achieve large field of view and high-resolution imaging, overcoming the limitations of traditional optical microscopes. The basic process of FTIM involves illuminating the sample with an LED array at different angles, capturing a set of low-resolution images, then using Fast Fourier Transform (FFT) to transform them to the corresponding region in the Fourier domain, and finally reconstructing the high-resolution complex amplitude of the sample through inverse Fourier transform.

[0003] In digital pathology, full-field reconstruction is a crucial requirement, especially high-quality full-field reconstruction, which is of great significance to the application and development of Fourier transform microscopes. Due to limitations in imaging models, a common approach to achieving full-field reconstruction is to use block processing and stitching strategies, but this suffers from computational redundancy and digital stitching artifacts. Another feasible approach is a full-field reconstruction method that reconstructs the entire field of view in a single operation.

[0004] A feature-domain-based full-field-of-view reconstruction method utilizes image features for phase retrieval, bypassing vignetting artifacts to achieve full-field-of-view reconstruction in a single operation. Specifically, this method extracts image features from low-resolution images acquired by a microscope, calculates an error function in the feature domain, and then uses an optimizer to update the gradient of the error function, updating the entire field of view in one operation, thereby achieving reliable full-field-of-view reconstruction.

[0005] In other words, while existing feature domain-based full-field-of-view reconstruction methods can utilize image feature information, they have limitations in utilizing different image features, leading to either excessive or insufficient detail in the reconstruction results. Therefore, the reconstruction quality achieved by these methods lags behind traditional block processing algorithms. Reconstruction quality is a crucial criterion for evaluating the performance of reconstruction algorithms. Although vignetting artifacts are bypassed, the limitations in reconstruction quality have prevented the widespread practical application of existing feature domain-based full-field-of-view reconstruction methods. Furthermore, the lack of flexible feature utilization capabilities when dealing with different samples to be reconstructed limits the methods' cross-domain adaptability. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a feature domain full-field-of-view reconstruction system containing an adjustable suppression factor. An adjustable suppression factor is introduced into the feature domain loss, constructing a gradient modulation term dominated by the adjustable suppression factor, which enables high-quality full-field-of-view reconstruction.

[0007] A feature domain full-field reconstruction system with an adjustable inhibition factor includes a microscope, an imaging model, a feature extraction module, a gradient acquisition and modulation module, a gradient update module, a reconstruction module, and a judgment module.

[0008] The feature extraction module is used to extract features from the low-resolution sample observation image sequence O acquired by the microscope and the sample prediction image sequence P calculated by the imaging model, respectively, to obtain the observation feature image sequence ▽O and the prediction feature image sequence ▽P.

[0009] The gradient acquisition and modulation module is used to acquire the spatial gradient of the loss function constructed based on the absolute error between ▽O and ▽P, and to multiply it with the spatial gradient according to the set modulation term to obtain the modulated spatial gradient.

[0010] The gradient update module is used to perform gradient inversion operation on the sample observation image sequence O according to the modulated spatial gradient to obtain the Fourier spectrum sequence corresponding to the sample observation image sequence O.

[0011] The reconstruction module is used to perform inverse Fourier transform on the Fourier spectrum sequence to obtain a high-resolution image sequence of the reconstructed sample.

[0012] The judgment module is used to determine whether the sample reconstructed high-resolution image sequence meets the set requirements. If yes, the reconstruction is completed; if no, the modulation term is updated, and the sample reconstructed high-resolution image sequence is used as the sample prediction image sequence P for the next cycle to re-acquire the sample reconstructed high-resolution image sequence for the next cycle.

[0013] Furthermore, the method for calculating the modulation term is as follows:

[0014]

[0015] Among them, P n W is the nth predicted image in the predicted image sequence P, where n∈{1,2,3,...,N}, N is the total number of LEDs in the LED array used by the microscope, and W n Predict image P for the nth sample. n The corresponding modulation term, This is an adaptive multiplication operator, where ρ is the inhibition factor.

[0016] Furthermore, the method for updating the modulation term is as follows:

[0017] The modulation terms are updated based on the detail clarity, contrast level, and dark field of the reconstructed image sequence. Specifically, the suppression factor is reduced when the detail clarity is lower than the set requirement, the suppression factor is also reduced when the contrast is lower than the set requirement, the suppression factor is increased when the dark field image is lower than the set requirement, and the suppression factor is increased when the number of image details is lower than the set requirement.

[0018] Furthermore, the suppression factor ρ ranges from -0.1 to 0.8.

[0019] Furthermore, according to and The loss function L(P') constructed from the absolute error between them is as follows:

[0020]

[0021] Where P' is the theoretical sample reconstructed image sequence, n∈{1,2,3,...,N}, and N is the total number of LEDs in the LED array used by the microscope.

[0022] Furthermore, the method for calculating the modulated spatial gradient is as follows:

[0023]

[0024] in, Predict image P for the nth sample. n The corresponding spatial gradient, denoted as the modulated spatial gradient, and · represents matched multiplication.

[0025] Furthermore, the feature extraction module uses a first-order gradient operator, Sobel operator, Prewitt operator, Canny operator, or a pre-trained deep learning network to extract features from the sample observation image sequence O and the sample prediction image sequence P. The extracted features are edge features, key point features, gradient features (HOG), or deep learning features.

[0026] Furthermore, the gradient update module is implemented using the AdaBelifDelta adaptive optimizer, the Adam optimizer, or the RMSprop optimizer.

[0027] Beneficial effects:

[0028] This invention provides a feature domain full-field reconstruction system with an adjustable suppression factor. First, an adjustable suppression factor is introduced into the feature domain loss, constructing a gradient modulation term dominated by the adjustable suppression factor. Second, suggested adjustment range and strategies for the suppression factor are given. Finally, the role of the suppression factor in feature utilization and gradient update is explained. Thus, this invention solves the problem of blurred or distorted details in existing full-field reconstruction techniques due to the inability to fully utilize image features by flexibly adjusting the utilization degree of sample image features through the introduction of an adjustable suppression factor, while achieving the level of traditional methods in full-field reconstruction quality. Attached Figure Description

[0029] Figure 1 A schematic diagram of the feature domain full-field reconstruction system containing an adjustable inhibition factor provided by the present invention.

[0030] Figure 2 The feature extraction module structure diagram provided by this invention;

[0031] Figure 3 The gradient acquisition and modulation module provided by this invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0033] like Figure 1 As shown, a feature domain full-field reconstruction system with an adjustable inhibition factor includes a microscope, an imaging model, a feature extraction module, a gradient acquisition and modulation module, a gradient update module, a reconstruction module, and a judgment module.

[0034] The feature extraction module is used to extract features from the low-resolution sample observation image sequence O acquired by the microscope and the sample prediction image sequence P calculated by the imaging model, respectively, to obtain the observation feature image sequence ▽O and the prediction feature image sequence ▽P.

[0035] Specifically, such as Figure 2 As shown, a set of low-resolution observation images O of the sample acquired by the microscope and a set of predicted images P corresponding to O calculated by the Fourier layered microscopy imaging model are processed by the feature extraction module to obtain the corresponding feature images ▽O and ▽P. The number of O, P, ▽O, and ▽P is N, and the value of N is the same as the number of LEDs used in the LED array of the microscope system. The feature extraction module is as follows: Figure 2 As shown. The feature extraction module includes a first-order gradient operator, and the extracted features are the edge features of the observed image and the predicted image, respectively;

[0036] It should be noted that, apart from the edge features used in this invention, which can be used throughout the entire reconstruction process, more advanced image features, such as keypoint features, gradient features (HOG), and deep learning features, are feasible for calculating feature loss and participating in reconstruction. On the other hand, regarding the selection of feature extractors, this invention uses a first-order gradient operator to extract image edge features, but other gradient operators, such as the Sobel operator, Prewitt operator, and Canny operator, can also be used to extract edge features. At the same time, pre-trained deep learning networks can also be used to extract more refined edge features.

[0037] like Figure 3As shown, the gradient acquisition and modulation module is used to acquire the spatial gradient of the loss function constructed based on the absolute error between ▽O and ▽P, and multiplies it with the spatial gradient according to a set modulation term to obtain the modulated spatial gradient; wherein, according to and The loss function L(P') constructed from the absolute error between them is as follows:

[0038]

[0039] Where P' is the theoretical sample reconstructed image sequence, n∈{1,2,3,...,N}, and N is the total number of LEDs in the LED array used by the microscope.

[0040] The modulated spatial gradient is calculated as follows:

[0041]

[0042] in, Predict image P for the nth sample. n The corresponding spatial gradient, The modulated spatial gradient is represented by ·, which indicates matched multiplication; that is, the modulation term W is multiplied by ·. n and spatial gradient By performing matched multiplication, we can obtain the modulated spatial gradient that can be used for subsequent optimizer updates;

[0043] Thus, this invention achieves the transformation of the loss function from the image domain to the feature domain, and calculates the spatial gradient of the loss function with respect to P'.

[0044] The gradient update module is used to perform gradient inversion operation on the sample observation image sequence O according to the modulated spatial gradient to obtain the Fourier spectrum sequence corresponding to the sample observation image sequence O. It should be noted that the gradient update module is implemented using the AdaBelifDelta adaptive optimizer, the Adam optimizer, or the RMSprop optimizer. The specific process is to update the Fourier spectrum of the low-resolution image through the optimization algorithm of the optimizer, thereby realizing an update of the entire field of view. The AdaBelifDelta optimizer has two learning rate parameters r1 and r2 and an adaptive step size δ. The learning rate in the optimizer is preset, and the adaptive step size is automatically updated during the operation of the optimizer.

[0045] The reconstruction module is used to perform inverse Fourier transform on the Fourier spectrum sequence to obtain a high-resolution image sequence of the reconstructed sample.

[0046] The judgment module is used to determine whether the sample reconstructed high-resolution image sequence meets the set requirements. If yes, the reconstruction is completed; if no, the modulation term is updated, and the sample reconstructed high-resolution image sequence is used as the sample prediction image sequence P for the next cycle to re-acquire the sample reconstructed high-resolution image sequence for the next cycle.

[0047] It should be noted that the method for calculating the modulation term is as follows:

[0048]

[0049] Among them, P n W is the nth predicted image in the predicted image sequence P, where n∈{1,2,3,...,N}, N is the total number of LEDs in the LED array used by the microscope, and W n Predict image P for the nth sample. n The corresponding modulation term, This is an adaptive multiplication operator, where ρ is a suppression factor, typically ranging from -0.1 to 0.8.

[0050] The method for updating the modulation term is as follows:

[0051] The modulation terms are updated based on the detail clarity, contrast level, and dark field of the reconstructed image sequence. Specifically, the suppression factor ρ is reduced when the detail clarity is lower than the set requirement, the suppression factor ρ is also reduced when the contrast is lower than the set requirement, the suppression factor ρ is increased when the dark field image is lower than the set requirement, and the suppression factor ρ is increased when the number of image details is lower than the set requirement.

[0052] Therefore, this invention introduces a suppression factor ρ, and uses the sample predicted image P and the suppression factor to construct the gradient modulation term W. n Modulation term W n This is achieved through an adaptive multiplication operation between the two terms, which is used for subsequent modulation of the loss function gradient. The specific modulation process involves matching and multiplying the modulation term and the spatial gradient of the loss function, essentially adjusting the predicted image features |P| based on the pixel distribution. n | ρ Weights are assigned to the spatial gradient so that it tends to advance in a direction that facilitates the recovery of image details during subsequent optimization, thereby improving the overall reconstruction quality of the algorithm. The selection of the suppression factor is empirical, usually with an initial value set in advance. Furthermore, the suppression term is constructed based on the amplitude and unit phase of the predicted image. The suppression factor can adjust the pixel distribution of the image, which can be considered as adjusting the degree of utilization of the features of the bright and dark fields of the image.

[0053] In other words, this invention obtains a reconstructed high-resolution image by performing an inverse Fourier transform on the updated Fourier spectrum, and then fine-tunes the suppression factor based on the reconstruction results. This adjustment is typically done manually, specifically through empirical fine-tuning based on factors such as the clarity of detail and contrast in the reconstructed image. Generally, the suppression factor is adjusted according to the principles of decreasing the suppression factor when the detail clarity is low, decreasing the suppression factor when the contrast is low, increasing the suppression factor when the dark field image is insufficient, and increasing the suppression factor when the image detail is insufficient. It is important to note that the increments in the suppression factor should not be too large, as drastic changes can lead to fluctuations in algorithm performance. The reconstructed image is then used as the initial image for the next iteration, repeating the above steps. Since the fine-tuning of the suppression factor is usually done manually, a suggested range of -0.1 to 0.8 has been provided through extensive simulations and testing. A dynamic automatic adjustment strategy can also be implemented, but testing has shown that dynamically changing the suppression factor does not affect the highest reconstruction performance in practical use, but it does affect the best achievable result. Adjustments can be made after multiple iterations.

[0054] Repeat the above process until the entire reconstruction algorithm converges iteratively or the reconstruction result satisfies the needs of human visual observation, at which point the iteration can be terminated.

[0055] In summary, by introducing an adjustable inhibition factor, this invention can achieve high-quality full-field reconstruction in different environments (such as different systems and different samples), and has a wider range of application scenarios than existing technologies.

[0056] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A feature domain full-field reconstruction system containing an adjustable suppression factor, characterized in that, It includes a microscope, an imaging model, a feature extraction module, a gradient acquisition and modulation module, a gradient update module, a reconstruction module, and a judgment module; The feature extraction module is used to extract features from the low-resolution sample observation image sequence O acquired by the microscope and the sample prediction image sequence P calculated by the imaging model, respectively, to obtain the corresponding observation feature image sequence. and predict feature image sequences The gradient acquisition and modulation module is used to acquire based on... and The spatial gradient of the loss function is constructed from the absolute error between the two values, and the modulated spatial gradient is obtained by matching and multiplying the spatial gradient with the set modulation term. The gradient update module is used to perform gradient inversion operation on the sample observation image sequence O according to the modulated spatial gradient to obtain the Fourier spectrum sequence corresponding to the sample observation image sequence O. The reconstruction module is used to perform inverse Fourier transform on the Fourier spectrum sequence to obtain a high-resolution image sequence of the reconstructed sample. The judgment module is used to determine whether the sample reconstructed high-resolution image sequence meets the set requirements. If yes, the reconstruction is completed; if no, the modulation term is updated, and the sample reconstructed high-resolution image sequence is used as the sample prediction image sequence P for the next cycle to re-acquire the sample reconstructed high-resolution image sequence for the next cycle.

2. The feature domain full-field reconstruction system containing an adjustable suppression factor as described in claim 1, characterized in that, The method for calculating the modulation term is as follows: Among them, P n W is the nth predicted image in the predicted image sequence P, where n∈{1,2,3,...,N}, N is the total number of LEDs in the LED array used by the microscope, and W n Predict image P for the nth sample. n The corresponding modulation term, This is an adaptive multiplication operator, where ρ is the inhibition factor.

3. The feature domain full-field reconstruction system containing an adjustable suppression factor as described in claim 2, characterized in that, The method for updating the modulation term is as follows: The modulation terms are updated based on the detail clarity, contrast level, and dark field of the reconstructed image sequence. Specifically, the suppression factor is reduced when the detail clarity is lower than the set requirement, the suppression factor is also reduced when the contrast is lower than the set requirement, the suppression factor is increased when the dark field image is lower than the set requirement, and the suppression factor is increased when the number of image details is lower than the set requirement.

4. The feature domain full-field reconstruction system containing an adjustable suppression factor as described in claim 2, characterized in that, The inhibition factor ρ ranges from -0.1 to 0.

8.

5. A feature domain full-field reconstruction system containing an adjustable suppression factor as described in claim 2, characterized in that, The loss function L(P') constructed based on the absolute error between ▽O and ▽P is as follows: Where P' is the theoretical sample reconstructed image sequence, n∈{1,2,3,...,N}, and N is the total number of LEDs in the LED array used by the microscope.

6. A feature domain full-field reconstruction system containing an adjustable suppression factor as described in claim 2, characterized in that, The modulated spatial gradient is calculated as follows: in, Predict image P for the nth sample. n The corresponding spatial gradient, denoted as the modulated spatial gradient, and · represents matched multiplication.

7. The feature domain full-field reconstruction system containing an adjustable suppression factor as described in claim 1, characterized in that, The feature extraction module uses a first-order gradient operator, Sobel operator, Prewitt operator, Canny operator, or a pre-trained deep learning network to extract features from the sample observation image sequence O and the sample prediction image sequence P. The extracted features are edge features, key point features, gradient features (HOG), or deep learning features.

8. The feature domain full-field reconstruction system containing an adjustable suppression factor as described in claim 1, characterized in that, The gradient update module is implemented using the AdaBelifDelta adaptive optimizer, the Adam optimizer, or the RMSprop optimizer.

Images