Real-time imaging method and system based on speckle correlation and transfer learning

Abstract

The application discloses a kind of based on speckle correlation and real-time imaging method and system of transfer learning, real-time imaging method includes: constructing source domain dataset is used for pre-training;Target domain dataset is constructed for fine-tuning;Respectively train pre-training network and fine-tuning network;Complete speckle reconstruction model is constructed;Test stage, real-time reconstruction, the present application is aimed at the deficiency of existing deep learning imaging method, by combining data model and physical model, the generalization ability of model is improved, transfer learning strategy is used to give full play to the data advantage of pre-training stage, also by decoupling pre-training and fine-tuning into independent module, the flexibility and expansibility of system are enhanced, the neural network based on Transformer architecture can capture and analyze complex features, realize high-fidelity reconstruction, and in test stage, the average reconstruction time of each image is less than 100 milliseconds, meet the real-time imaging needs of video stream, applicable to complex underwater imaging and turbid water monitoring and other scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical imaging and image processing, in particular to a real-time imaging method and system based on speckle correlation and transfer learning in turbid water scene. BACKGROUND

[0002] Optical imaging is one of the key ways for humans to obtain information from the outside world. In order to realize imaging through scattering medium, in the past ten years, researchers have proposed optical gating, optical coherence tomography, wavefront modulation, optical phase conjugation, and transmission matrix measurement methods. However, due to the complexity of the imaging system and the harsh conditions for application, these methods have gradually faded out of sight. At present, the mainstream imaging method based on single-frame speckle is the imaging technology based on speckle correlation. The amplitude of the target image can be extracted using single-frame speckle, and then the reconstruction of the occluded target is realized through phase recovery algorithm. However, this method still has limitations such as poor reconstruction quality, slow reconstruction speed, and poor iteration stability.

[0003] In recent years, the rapid development of deep learning technology has brought new breakthroughs to the field of imaging. As a new method for solving inverse problems, it has been widely applied. By constructing a one-to-one mapping data set between speckle images and original target images, the hidden scattering characteristics between data can be simulated by using the information mining and nonlinear fitting capabilities of neural networks, without the need for complex and high-precision imaging equipment to achieve good imaging results. However, existing deep learning methods usually require large-scale training data and have limited generalization ability, making it difficult to cope with unknown imaging environments. In addition, in actual imaging environments, the medium that hinders optical imaging often exists in a turbid and dynamic form, such as imaging through turbid water. Unlike static solid media such as frosted glass, the scattering characteristics of dynamic media are constantly changing. Traditional end-to-end deep learning methods are difficult to apply in such complex and dynamic scattering scenarios. SUMMARY

[0004] The purpose of the present application is to provide a real-time imaging method and system based on speckle correlation and transfer learning. To solve the problems raised in the background art, especially for the corresponding problems in the technical field of complex underwater imaging and turbid water monitoring, the present application proposes targeted improvements to the defects of existing deep learning imaging methods. The present application provides the following technical solutions: A real-time imaging method based on speckle correlation and transfer learning, comprising the following steps:

[0005] S1: Constructing a source domain data set for pre-training, i.e. performing autocorrelation operation on the original target in the original target data set, without the need for optical devices to calculate and generate each target autocorrelation. Make a data pair of target autocorrelation and target, and construct a source domain data set for pre-training;

[0006] Preferably, the original target dataset should contain as many types of original targets as possible to enrich the diversity of pre-training data, and the target autocorrelation acquisition process is completely performed by program calculation without the need for any actual optical imaging device, so the size of this source domain dataset has no upper limit and can be chosen according to the training cost of the network model and the target categories of the application field to ensure that the data advantage of the neural network and the effect of the pre-training task are fully played.

[0007] S2: Constructing a target domain dataset for fine-tuning, that is, in a laboratory environment, a dynamic scattering field is built, a speckle image is collected by using a reflective imaging system, autocorrelation is calculated, a data pair of speckle autocorrelation and target autocorrelation is made, and a target domain dataset is constructed for fine-tuning;

[0008] Preferably, the reflective imaging system comprises an incoherent LED light source, a lens, a diaphragm, a digital micromirror device, an original target loaded thereon, a turbid water body, a band-pass filter, a CMOS camera, and a speckle reconstruction model for imaging through the turbid water body. The target domain dataset making process needs to be actually collected by an experimental device. In order to reduce the complexity of collection, the size of the target domain dataset is one-tenth of the size of the source domain dataset in step S1.

[0009] S3: Training the pre-training network and the fine-tuning network respectively, that is, training the pre-training network and the fine-tuning network model based on the source domain dataset and the target domain dataset respectively, and the neural network model is a degenerate perception Transformer neural network;

[0010] Preferably, the neural network adopts a neural network based on a Transformer architecture to obtain a high-quality reconstruction result. The network can contain an attention mechanism, layer normalization, residual connection and a feedforward module. The pre-training network and the fine-tuning network can use the same network model or can be customized and replaced according to actual application requirements. The trained pre-training network has the ability to reconstruct the original target from the autocorrelation image, and the trained fine-tuning network has the ability of feature adaptation, which can align the autocorrelation of the actually collected speckle image with the ideal target autocorrelation.

[0011] S4: Constructing a complete speckle reconstruction model, that is, connecting the trained fine-tuning network and the pre-training network in a cascaded form to construct a complete speckle reconstruction model;

[0012] Preferably, the cascade mode is to take the output result of the fine-tuning network as the input data of the pre-training network, and for the input speckle image, after the autocorrelation operation, the speckle image is first processed by the fine-tuning network to adapt the feature, so that the feature distribution of the refined autocorrelation image is closer to the requirement of the pre-training model, and the pre-training model can more effectively process and adapt to the characteristics of the target domain after receiving the refined autocorrelation, thereby significantly improving the overall performance of the complete reconstruction model in the target domain, especially in the case that the data distribution of the source domain and the target domain is greatly different.

[0013] S5: test phase, real-time reconstruction, that is, the collected speckle image is real-time returned to the reconstruction model, and the collected result is continuously reconstructed by the reconstruction model to obtain real-time imaging results through turbid water;

[0014] As preferred, the test phase is to realize real-time reconstruction, that is, in an unknown turbid water test scene, a reflection imaging system is built, a non-coherent LED device is used for illumination, a digital micromirror device is controlled to load an unknown test original target, a light beam carrying target information transmits through turbid water, passes through an aperture and a filter, and is finally captured by a CMOS camera and real-time returned to the reconstruction model, and the camera collected result is continuously reconstructed by the reconstruction model to obtain real-time imaging results through turbid water.

[0015] According to the above technical scheme, the present application provides the following technical scheme: a real-time imaging system based on speckle correlation and transfer learning, comprising: a pre-training preparation module, a fine-tuning preparation module, a neural network model training module, a speckle reconstruction model construction module, and a model test module.

[0016] Preferably, the pre-training preparation module is used for autocorrelation operation on the original target in the original target data set, making a data pair of target autocorrelation and target, and constructing a source domain data set for pre-training.

[0017] The fine-tuning preparation module is used for combining speckle correlation imaging and transfer learning strategy, wherein a small amount of collected speckle autocorrelation is used for fine-tuning to realize the use of less speckle training data to obtain results.

[0018] The neural network model training module is used for training the pre-training network and the fine-tuning network model on the source domain data set and the target domain data set, respectively.

[0019] The speckle reconstruction model construction module is used for connecting the trained fine-tuning network and the pre-training network in a cascade form to construct a complete speckle reconstruction model.

[0020] The model testing module is used for a two-stage speckle reconstruction process, in the first stage, the test speckle is refined first by a fine-tuning network to obtain an autocorrelation, and in the second stage, the refined autocorrelation is further processed by a second network, i.e., a pre-training network, to obtain a reconstruction result.

[0021] Compared with the prior art, the present application has the following beneficial effects:

[0022] 1. The present application successfully realizes real-time imaging of hidden targets in turbid water bodies, and can efficiently and high-quality reconstruct the obscured original target from the degraded speckle image captured in a complex dynamic imaging environment, without the need for high-precision imaging equipment to obtain high-quality imaging results.

[0023] 2. The method of the present application addresses the defects of end-to-end deep learning imaging methods, combines neural network data models with physical models, uses the link of speckle correlation imaging, and replaces the speckle-target pair in most scattering imaging methods with a more robust and robust autocorrelation-target data pair and a speckle autocorrelation-target autocorrelation data pair, ensuring the generalization performance of the model.

[0024] 3. The method of the present application is based on a transfer learning strategy, the source domain dataset is composed of target autocorrelation and target, and can be easily obtained without any optical equipment, which can fully play the characteristics of the pre-training stage and the data advantages of the network, ensuring the reconstruction performance of the method.

[0025] 4. The method of the present application separates the pre-training and fine-tuning, and deploys them on two models, instead of operating on the same model twice, decouples the traditional pre-training-fine-tuning framework into two independent steps, explores the synergy between the two steps, and allows each model to be optimized independently without interference, making the system more flexible in dealing with different tasks and datasets, for example, new target domain data can directly adjust the fine-tuning network without retraining the entire system. This modular design not only improves the adaptability of the system, but also facilitates future expansion and upgrading.

[0026] 5. The method of the present application designs a neural network based on the Transformer architecture in the pre-training and fine-tuning tasks, which can effectively capture and analyze complex features to obtain high-fidelity reconstruction results by using unified context refinement attention mechanism, layer normalization, residual connection and feedforward module.

[0027] 6. The present application proposes targeted improvements to the defects of existing deep learning imaging methods, significantly improves the generalization ability of the network model while ensuring the high fidelity of the reconstruction result, and the reconstruction model has good calculation speed in the test stage, with an average reconstruction time of less than 100 milliseconds per test image, which can meet the real-time imaging requirements of more than 10 frames of video stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0029] Figure 1 is a flow chart of a real-time imaging method based on speckle correlation and transfer learning provided by an embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of a reflective imaging system device provided by an embodiment of the present invention;

[0031] Figure 3 This is a statistical characteristic analysis diagram of speckle and autocorrelation images provided by an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of two training data sets obtained after executing steps S1 and S2 according to an embodiment of the present invention;

[0033] Figure 5 This is a block diagram of a degradation-aware Transformer network provided by an embodiment of the present invention;

[0034] Figure 6 is a speckle reconstruction flow chart of the testing phase provided by an embodiment of the present invention;

[0035] Figure 7 Graphs showing the results of a reconstruction experiment on some unknown test objects in the Mnist and Fashion-mnist datasets, provided by an embodiment of the present invention;

[0036] Figure 8 3 is a schematic diagram of the composition structure of a real-time imaging system based on speckle correlation and transfer learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] The present invention provides a technical solution: a real-time imaging method based on speckle correlation and transfer learning, specifically, Figure 1 As shown, the embodiment of the present invention includes the following steps:

[0039] S1: constructing a source domain dataset for pre-training, i.e. performing autocorrelation operation on original targets in the original target dataset, i.e. making data pairs of target autocorrelation and targets, and constructing a source domain dataset for pre-training;

[0040] As preferred, in the present embodiment, each target autocorrelation is calculated without optical device, data pairs of target autocorrelation and targets are made, and the source domain dataset is constructed for pre-training, wherein the original target dataset can be selected from Mnist and Fashion-mnist datasets, in order to fully exert the characteristics of the pre-training stage and the data advantages of the network model, the entire Mnist and Fashion-mnist datasets are selected for autocorrelation operation, i.e. the scale of the source domain dataset is 120000 pairs of training numbers.

[0041] S2: constructing a target domain dataset for fine-tuning, i.e. in a laboratory environment, a dynamic scattering scene is built, a speckle image is collected by using a reflection imaging system, autocorrelation operation is performed to obtain each speckle autocorrelation, data pairs of speckle autocorrelation and target autocorrelation are made, and a target domain dataset is constructed for fine-tuning;

[0042] As preferred, the reflection imaging system, as shown in Figure 2 , includes an incoherent LED light source, a lens, a diaphragm, a digital micromirror device, original targets loaded thereon, turbid water, a band-pass filter, a CMOS camera, and a speckle reconstruction model for imaging through turbid water.

[0043] In the present embodiment, the incoherent light beam generated by the incoherent LED light source passes through the lens and the diaphragm 1 to irradiate on the digital micromirror device, illuminates the original targets loaded thereon, and then the light beam carrying the target information passes through the turbid water and the diaphragm 2, is captured by the industrial CMOS camera after passing through a band-pass filter, and then the collection of a speckle image is completed;

[0044] More specifically, the incoherent LED light source can be selected as a narrow-band incoherent light source with a wavelength of 625 nanometers, such as Thorlabs M625L4 light source, the diameters of the diaphragm 1 and the diaphragm 2 are 5 millimeters, the turbid water can be selected from polystyrene microsphere suspension, calcium carbonate solution, fat emulsion, sea salt solution or mixed solution of multiple media, the center wavelength of the band-pass filter is 632 nanometers, and the resolution of the CMOS camera can be selected as 1920*1200 pixels with a pixel size of 5.86 micrometers / pixel;

[0045] Further, according to the principle of speckle correlation imaging, within the range of optical memory effect, the point spread function PSF of the imaging system satisfies the linear shift-invariant characteristic, and the speckle image collected on the imaging surface can be expressed as the convolution of the target and the point spread function as shown below:

[0046] I(x) = O(x) * S(x) (1)

[0047] where "*" denotes the convolution operator, O(x) and S(x) denote the object and the point spread function respectively, using the collected speckle, using the convolution theorem, the speckle autocorrelation is equal to the convolution of the object autocorrelation and the autocorrelation of the PSF of the imaging system, and the imaging model is shown as formula (2):

[0048] I(x) * I(x) = [O(x) * S(x)] * [O(x) * S(x)] = [O(x) * O(x)] *

[0049] [S(x) * S(x)] (2)

[0050] where "*" denotes the autocorrelation operator, and the autocorrelation of the point spread function can be further expressed as:

[0051]

[0052] where J1 denotes the first kind of Bessel function, ∝ means proportional, d, λ and v respectively denote the aperture diameter, wavelength and imaging distance, based on formula (3), the autocorrelation S(x) * S(x) of the point spread function of the imaging system is a peak function with a certain width, which affects the resolution of the autocorrelation, and the width is related to the size of the speckle particles, within the optical memory range, the point spread function shows high similarity, so that the autocorrelation of two approximately identical point spread functions can be regarded as a δ function which has a negative impact on the reconstruction result, and can be approximated as a background noise constant C.

[0053] Therefore, the speckle correlation imaging model can be further expressed as:

[0054] I(x) * I(x) = O(x) * O(x) + C (4)

[0055] where the background noise constant C is the difference between the object autocorrelation and the speckle autocorrelation, and the difference will be affected by the scattering medium transformation and the experimental environment.

[0056] In this embodiment, the application scenario of the present application is a dynamic scattering scene of turbid water body, and the typical feature is that the scattering characteristics are constantly changing. In order to further analyze and verify the influence of the scattering medium transformation and the experimental environment on the speckle reconstruction, the statistical characteristic analysis results of the speckle and the autocorrelation image collected by the same original object through different scattering media are shown as Figure 3 Figure 3 (a) shows the original object and the object autocorrelation as the true value under the darkroom environment, and the speckle and the autocorrelation image collected after the medium transformation, by drawing the statistical characteristics along the dashed line in the image, Figure 3 ​(c) and (d) respectively show the statistical characteristics of speckle images and speckle autocorrelation images. It can be observed that even for the same original target, the statistical characteristics of different speckles collected after the medium is transformed have great differences. It is this difference that limits the generalization ability of the end-to-end imaging method based on deep learning. After autocorrelation operation, the speckle images with different distributions have relatively uniform structural characteristics, such as Figure 3 (d) shows. Because of the structural characteristics of the target autocorrelation, the imaging method based on speckle correlation proposed in the present application also has good generalization performance. At the same time, Figure 3 (d) also presents individual differences between the characteristic curves. As shown in the speckle correlation imaging model, these differences are the embodiment of the background noise constant, representing the change of the medium scattering characteristics and the influencing factors such as the imaging environment, which affect the final imaging effect. And with the increase of the intensity of external interference, the difference between the speckle autocorrelation and the target autocorrelation will also increase. For example, the low signal-to-noise ratio speckle and its autocorrelation image collected under external interference light are shown in Figure 3 (b), which is significantly different from the true value of the autocorrelation image.

[0057] Further, after performing steps S1 and S2 of the embodiment of the present application, two training data sets can be obtained as shown in Figure 4 The larger-scale source domain data set generated directly through program calculation and the smaller-scale target domain data set made by experimental collection.

[0058] S3: Train the pre-training network and the fine-tuning network respectively, that is, train the pre-training network and the fine-tuning network model on the source domain data set and the target domain data set respectively.

[0059] As a preferred, the neural network can adopt a neural network based on the Transformer architecture to obtain high-quality reconstruction results. The network can include attention mechanisms, layer normalization, residual connections, and feedforward modules. In the embodiment of the present application, a neural network named degradation-aware Transformer designed by the inventors is used, and its network block diagram is shown in Figure 5 .

[0060] In this embodiment, a unified context refinement attention module, layer normalization, residual connection, and feedforward module are included in each block of the network.

[0061] Further, the unified context refinement attention mechanism performs high-order feature correlation regulation at the channel and spatial levels to realize the collaborative regulation of channel features and spatial structures, which comprehensively considers the importance weight of each feature channel and the internal correlation and importance between different spatial positions in the image. Therefore, the unified context refinement attention calculates the global importance score of each channel and adjusts the feature representation of each channel according to the score, and also calculates the similarity or correlation between pixel positions to guide the model to focus on specific regions and details in the image. This integrated attention mechanism helps the network effectively capture the relationship between different channels and spatial positions in the image, and improves the discriminability and generalization ability of feature representation; and in order to further improve the efficiency and retain important feature information, a depth separable convolution is introduced to decompose the standard convolution into two smaller convolution operations, which significantly reduces the computational amount.

[0062] For example, by using layer normalization in the network, it helps to normalize the input of each module, thereby accelerating the training speed and improving the model performance. The use of residual connection can effectively transmit and learn the information between the input and the output, ensuring the stability and efficiency of the training process. The use of feedforward module can perform nonlinear transformation and extraction on the features, improving the understanding ability and generalization ability of the network for image structure.

[0063] Preferably, the pre-training network and the fine-tuning network can use the same network model, or can be customized and replaced according to actual application requirements. In the embodiment of the present application, the pre-training network and the fine-tuning network both use the degenerative perception Transformer network designed by us.

[0064] Specifically, the pre-training network and the fine-tuning network respectively reach a convergence state on the source domain data set D pre and the target domain data set D fine , that is, the convergence states of the pre-training network and the fine-tuning network are respectively represented as:

[0065]

[0066] Where arg min represents the minimum value of the objective function, Θ represents the model parameter set, represents the solution expectation, represents the network loss function, f pre and f fine represent the pre-training network and the fine-tuning network respectively, X represents the input data, and Y represents the label data. Therefore, in the training process of the pre-training and fine-tuning network, the parameter set Θ of the neural network is continuously optimized to minimize the loss function between the predicted output of the network on the training data set and the label data Y, that is, to reach a convergence state.

[0067] In the embodiment of the present application, the loss function of the network adopts the combination of Charbonnier loss and edge loss, which is expressed in the following form:

[0068] L = L char + λL edge (7)

[0069]

[0070] wherein the parameter λ determines the weight of the two losses and is selected as 0.05, X GT and respectively represent the generated image and the true value, and the constant ε is set as 1e-3;

[0071] As preferred, in the embodiment of the present application, the trained pre-training network has the ability to reconstruct the original target from the autocorrelation image, and the trained fine-tuning network has the ability of feature adaptation, which can align the autocorrelation of the actually collected speckle image with the ideal target autocorrelation.

[0072] S4: constructing a complete speckle reconstruction model, that is, connecting the trained fine-tuning network and the pre-training network in a cascaded form to construct a complete speckle reconstruction model;

[0073] As preferred, in the embodiment of the present application, the output result of the fine-tuning network is taken as the input data of the pre-training network to construct a complete speckle reconstruction model in a cascaded form. After the construction is completed, as shown in Figure 6 for the input speckle image, after autocorrelation operation, the fine-tuning network is first used for feature adaptation, so that the feature distribution of the refined autocorrelation image is closer to the requirement of the pre-training model. The pre-training model can more effectively process and adapt to the characteristics of the target domain after receiving the refined autocorrelation, thereby significantly improving the overall performance of the model in the target domain, especially in the case of large difference between the data distribution of the source domain and the target domain.

[0074] As shown in the speckle correlation imaging model of formula (4) and Figure 3 (d), there is a data distribution difference between the speckle autocorrelation and the ideal target autocorrelation. Therefore, the fine-tuning network effectively plays the role of a feature adapter, which first pre-processes the target domain data to reduce the data heterogeneity between the speckle autocorrelation and the target autocorrelation, unifies the features of different speckle autocorrelations, aligns their feature representations with the target autocorrelation, and makes the target domain features in the refined autocorrelation image more effectively migrate to the pre-training model, as shown in Figure 3As shown in (a), (b) and (e), the fine-tuning network output has a more unified structure characteristic, and has a higher fitting degree between the refined autocorrelation and the target autocorrelation, so that the pre-training model can better adapt to the requirements of the target domain when processing these data, thereby significantly improving the performance of the overall model in the target domain.

[0075] S5: test phase, real-time reconstruction, that is, in an unknown turbid water test scene, a reflection imaging system is built, a non-coherent LED device is used for illumination, a digital micromirror device is controlled to load unknown test original targets, a light beam carrying target information is transmitted through the turbid water body, passes through an aperture and a filter, and is finally captured by a CMOS camera and the collected speckle images are real-time returned to the reconstruction model, the reconstruction model is used to continuously cycle the camera acquisition results, and real-time imaging results through the turbid water body are obtained;

[0076] As preferred, the reflection imaging system built by the embodiment of the application is as shown in Figure 2 The reconstruction experimental results of some unknown test targets in the Mnist and Fashion-mnist data sets are as shown in Figure 7 It can be observed that whether the handwritten digit target or the Fashion target, the reconstruction results of the application are excellent in details and overall structural aspects, and have a relatively high similarity degree with the original target, and high-fidelity imaging results through the turbid water body are achieved.

[0077] The embodiment of the application also provides a real-time imaging system based on speckle correlation and transfer learning, as shown in Figure 8 The embodiment includes a pre-training preparation module, a fine-tuning preparation module, a neural network model training module, a speckle reconstruction model construction module and a model test module.

[0078] For example, the pre-training preparation module is used for autocorrelation operation on original targets in an original target data set, making a data pair of target autocorrelation and target, and constructing a source domain data set for pre-training.

[0079] The fine-tuning preparation module is used for combining speckle correlation imaging and transfer learning strategies, wherein a small amount of collected speckle autocorrelation is used for fine-tuning, so as to realize the use of less speckle training data to obtain results.

[0080] The neural network model training module is used for training pre-training network and fine-tuning network models on the source domain data set and the target domain data set, respectively.

[0081] The speckle reconstruction model construction module is used for connecting the trained fine-tuning network and pre-training network in a cascade form to construct a complete speckle reconstruction model.

[0082] The model test module is used for a two-stage reconstruction process, in the first stage, speckle is tested to obtain refined autocorrelation through fine-tuning network, and in the second stage, refined autocorrelation is further obtained through a second network, i.e., a pre-training network, to obtain a reconstruction result.

[0083] Further, the pre-training preparation module further comprises an original data set module and an autocorrelation operation module.

[0084] Specifically, the original data set module is used for selecting original target data sets of Mnist and Fashion-mnist data sets; and the autocorrelation operation module is used for performing autocorrelation operation based on the entire Mnist and

[0085] Fashion-mnist data sets.

[0086] The fine-tuning preparation module further comprises a speckle autocorrelation module and a data pair module.

[0087] The speckle autocorrelation module is used for performing autocorrelation operation based on the collected speckle images to obtain each speckle autocorrelation, and the data pair module is used for making data pairs of speckle autocorrelation and target autocorrelation.

[0088] The neural network model training module further comprises a degradation perception Transformer neural network module, which is used for obtaining a high-quality reconstruction result by using a degradation perception Transformer neural network based on a Transformer architecture.

[0089] The speckle reconstruction model construction module further comprises a cascade connection interaction module, which is used for connecting and interacting the trained fine-tuning network and the pre-training network in a cascade form.

[0090] The model test module further comprises an imaging result construction module, which is used for continuously recycling camera acquisition results by using the reconstruction model to obtain real-time imaging results through turbid water.

[0091] In summary, the embodiment of the present application successfully realizes real-time imaging of the occluded target in a complex dynamic scattering scene, and can efficiently and high-quality reconstruct the original target occluded in a complex dynamic imaging environment through the degraded speckle images captured in the complex dynamic imaging environment, without the need of high-precision imaging equipment to obtain high-quality imaging results.

[0092] The present application combines a neural network data model and a speckle correlation physical model, uses robust autocorrelation data pairs to improve the generalization performance of the model, and based on the transfer learning strategy, the source domain data set can be easily obtained without optical equipment, fully plays the characteristics of the pre-training stage and the data advantages of the network, ensures the reconstruction performance of the method, and by decoupling the pre-training and fine-tuning into independent steps and deploying them in different models, not only enhances the system flexibility and modular design, but also facilitates expansion and upgrading, and the neural network based on the Transformer architecture can effectively capture and analyze complex features to obtain high-fidelity reconstruction results.

[0093] Therefore, the present application improves the defects of the existing deep learning imaging method, significantly improves the generalization ability of the network model, and can ensure the high fidelity of the reconstruction result, the reconstruction model has good calculation speed in the test stage, the average reconstruction time of each test image is within 100 milliseconds, and can face the real-time imaging demand of more than 10 frames of video stream, that is, the embodiment of the present application specifically realizes a real-time imaging based on speckle correlation and transfer learning through turbid water, which can be applied to complex underwater imaging, turbid water monitoring and other technical fields.

[0094] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0095] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and does not limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A real-time imaging method based on speckle correlation and transfer learning, characterized by: The method comprises the following steps: S1: Construct a source domain dataset for pre-training. The source domain dataset contains the original target and the target autocorrelation. The pre-training is one of the key steps of the transfer learning strategy, that is, training the pre-trained network based on the source domain dataset. S2: Constructing a target domain dataset for fine-tuning. The target domain dataset contains speckle autocorrelation and target autocorrelation. Fine-tuning is another key step in the transfer learning strategy, that is, training and fine-tuning the network based on the target domain dataset. S3: training the pre-trained network and the fine-tuned network separately, that is, training the pre-trained network and the fine-tuned network model based on the source domain dataset and the target domain dataset respectively, wherein the network model is a degradation-aware Transformer neural network; S4: Build a complete speckle reconstruction model, that is, connect the trained fine-tuning network and the pre-trained network in a cascade form to build a complete speckle reconstruction model; S5: Testing phase, real-time reconstruction, that is, testing is performed based on the speckle reconstruction model, the collected speckle image is transmitted back to the reconstruction model in real time, and the collection result is continuously reconstructed through the reconstruction model in a cycle to obtain a real-time imaging result through turbid water.

2. The real-time imaging method based on speckle correlation and transfer learning according to claim 1, characterized in that: In steps S1 and S2, pre-training and fine-tuning further include combining speckle correlation imaging with a transfer learning strategy. During the fine-tuning process, the transfer learning strategy involves less training data, that is, the large-scale source domain dataset obtained through simulation in pre-training is used to reduce the actually collected speckle data.

3. The real-time imaging method based on speckle correlation and transfer learning according to claim 2, characterized in that: After executing steps S1 and S2, two training datasets are obtained, including a larger-scale source domain dataset directly generated by program calculation and a smaller-scale target domain dataset produced by experimental collection.

4. The real-time imaging method based on speckle correlation and transfer learning according to claim 1, characterized in that: The pre-trained network and the fine-tuned network use the same network model, which is a degradation-aware Transformer network. Each block of the degradation-aware Transformer network includes a unified context-refined attention module, layer normalization, residual connection, and feedforward module. The unified contextual refinement attention module regulates high-order feature associations at the channel and spatial levels to achieve coordinated regulation of channel features and spatial structures. It calculates and scores the importance weight of each feature channel and the intrinsic correlation and importance between different spatial positions in the image, adjusts the feature representation of each channel based on the score, and simultaneously calculates the similarity or correlation between pixel positions, guiding the model to focus on specific areas and details in the image. It further decomposes the standard convolution into two smaller convolution operations based on depthwise separable convolution to reduce the amount of computation. The layer normalization is used to normalize the input of each module, speeding up training and improving model performance. The residual connection is used to transmit and learn information between input and output. The feedforward module is used to perform nonlinear transformation and extraction on features.

5. The real-time imaging method based on speckle correlation and transfer learning according to claim 4, characterized in that: The training of the pre-trained network and the fine-tuned network is as follows: pre and the target domain dataset D fine The convergence state is reached, that is, the convergence state reached by the pre-trained network and the fine-tuned network is expressed as: Among them, arg min represents the minimum value of the objective function, Θ represents the model parameter set, represents the expectation of solution, Represents the network loss function, f pre With f fine Represent the pre-trained network and the fine-tuned network respectively, X represents the input data, and Y represents the label data. Therefore, during the training process of the pre-trained and fine-tuned networks, the parameter set Θ of the neural network will be continuously optimized to make the loss function between the predicted output of the network on the training data set and the label data Y Reaching minimization, that is, reaching the convergence state; The loss function of the network adopts the combination of Charbonnier loss and edge loss, which is expressed as follows: L=L char +λL edge Among them, the parameter λ determines the weight of the two losses and is selected as 0.

05. GT and represent the generated image and the true value respectively, and the ε constant is set to 1e-3.

6. The real-time imaging method based on speckle correlation and transfer learning according to claim 5, characterized in that: The training of the pre-trained network and the fine-tuning network also includes: the trained pre-trained network has the ability to reconstruct the original target from the autocorrelation image, and the trained fine-tuning network has the ability of feature adaptation, which can align the autocorrelation of the actually collected speckle image with the ideal target autocorrelation.

7. The real-time imaging method based on speckle correlation and transfer learning according to claim 6, characterized in that: In constructing the complete speckle reconstruction model, the output result of the fine-tuning network is used as the input data of the pre-training network, and the complete speckle reconstruction model is constructed in a cascade form. The target domain data is preprocessed based on the trained fine-tuning network, the features of different speckle autocorrelations are unified, and their feature representations are aligned with the target autocorrelation, so that the target domain features in the refined autocorrelation image are effectively transferred to the pre-training model.

8. The real-time imaging method based on speckle correlation and transfer learning according to claim 1, characterized in that: The test process includes: building a reflective imaging system in an unknown turbid water test scene, using an incoherent LED device for illumination, controlling a digital micromirror device to load an unknown test original target, and allowing a light beam carrying target information to pass through the turbid water, pass through an aperture and a filter, and finally be captured by a CMOS camera. The collected speckle image is transmitted back to a reconstruction model in real time, and the reconstruction model is used to continuously reconstruct the camera acquisition results in a loop to obtain real-time imaging results through the turbid water.

9. A real-time imaging system based on speckle correlation and transfer learning, characterized by: The system comprises: The pre-training preparation module is used to perform autocorrelation operations on the original targets in the original target dataset, generate data pairs of target autocorrelation and target, and construct the source domain dataset for pre-training; A fine-tuning preparation module is used to combine speckle correlation imaging with a transfer learning strategy, in which a small amount of acquired speckle autocorrelations are used for fine-tuning, thereby achieving results with less speckle training data; Neural network model training module, used to train pre-trained networks and fine-tune network models on source domain datasets and target domain datasets respectively; The neural network model training module also includes a degradation-aware Transformer neural network module for obtaining high-quality reconstruction results using a degradation-aware Transformer neural network based on a Transformer architecture; The speckle reconstruction model construction module is used to connect the trained fine-tuning network and the pre-trained network in a cascade form to build a complete speckle reconstruction model; The model testing module is used for a two-stage speckle reconstruction process. In the first stage, the test speckle is first refined by the fine-tuning network to obtain the autocorrelation. In the second stage, the refined autocorrelation is then passed through a second network - the pre-trained network to obtain the reconstruction result.

Images