Real-time imaging method and system based on speckle correlation and transfer learning

Abstract

The invention discloses a real-time imaging method and system based on speckle correlation and transfer learning. The real-time imaging method comprises the following steps: constructing a source domain data set for pre-training; constructing a target domain data set for fine tuning; respectively training the pre-training network and the fine tuning network; constructing a complete speckle reconstruction model; aiming at the defects of an existing deep learning imaging method, the generalization ability of the model is improved by combining a data model and a physical model, the data advantages of the pre-training stage are fully played by adopting a transfer learning strategy, and pre-training and micro-demodulation are coupled into independent modules, so that the real-time reconstruction is realized. Compared with the prior art, the system has the advantages that the flexibility and expansibility of the system are enhanced, the neural network based on the Transform architecture can capture and analyze complex features and realize high-fidelity reconstruction, the average reconstruction time of each image is less than 100 milliseconds in the test stage, the real-time imaging requirement of video streaming is met, and the system is suitable for scenes such as complex underwater imaging and turbid water body monitoring.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging and image processing, and in particular to a real-time imaging method and system based on speckle correlation and transfer learning in a turbid water scene. Background Art

[0002] Optical imaging is one of the key ways for humans to obtain external information. In order to achieve imaging through scattering media, researchers have proposed methods such as optical gating, optical coherence tomography, wavefront modulation, optical phase conjugation, and transmission matrix measurement in the past decade. However, due to the complexity of the imaging system and the harsh application conditions, these methods have gradually faded out of sight. The current mainstream imaging method based on single-frame speckle is the speckle-related imaging technology. The amplitude of the target image can be extracted using a single-frame speckle, and then the occluded target can be reconstructed through a phase recovery algorithm. However, this method still has limitations such as poor reconstruction quality, slow reconstruction speed, and poor iterative stability.

[0003] In recent years, the rapid development of deep learning technology has brought new breakthroughs in the field of imaging. As an emerging method for solving inverse problems, it has been widely used. By constructing a one-to-one mapping data set between speckle images and original target images, the information mining and nonlinear fitting capabilities of neural networks are used to simulate the hidden scattering characteristics between data, and better imaging effects can be achieved without complex and high-precision imaging equipment. However, existing deep learning methods usually require the collection of large-scale training data and have limited generalization capabilities, making it difficult to cope with unknown imaging environments. In addition, in actual imaging environments, media that hinder optical imaging often exist in a turbid dynamic form. For example, imaging through turbid water is different from static solid static media such as frosted glass. The scattering characteristics of dynamic media are constantly changing. Traditional end-to-end deep learning methods are difficult to apply in such complex dynamic scattering scenarios. Summary of the invention

[0004] The object of the present invention is to provide a real-time imaging method and system based on speckle correlation and transfer learning. In order to solve the problems raised in the above background technology, especially the corresponding problems in the technical field of complex underwater imaging and turbid water monitoring, the present invention proposes targeted improvements to the defects of the existing deep learning imaging method. The present invention provides the following technical solutions: A real-time imaging method based on speckle correlation and transfer learning, comprising the following steps:

[0005] S1: constructing a source domain data set for pre-training, that is, performing autocorrelation operations on the original targets in the original target data set, calculating and generating the autocorrelation of each target without an optical device, making data pairs of target autocorrelation and target, and constructing a source domain data set for pre-training;

[0006] Preferably, the original target data set should contain as many types of original targets as possible to enrich the diversity of pre-training data, and the process of obtaining target autocorrelation is completely executed by program calculation without going through any actual optical imaging device. Therefore, there is no upper limit to the size of this source domain data set, and it can be selected based on the training cost of the network model and the target category of the application field, ensuring that the data advantages of the neural network and the effect of the pre-training task are fully utilized.

[0007] S2: Construct a target domain dataset for fine-tuning, that is, in a laboratory environment, build a dynamic scattering scene, use a reflective imaging system to collect speckle images, perform autocorrelation operations to obtain the autocorrelations of each speckle, make data pairs of speckle autocorrelations and target autocorrelations, and construct a target domain dataset for fine-tuning;

[0008] Preferably, the reflective imaging system comprises an incoherent LED light source, a lens, an aperture, a digital micromirror device, an original target loaded thereon, a turbid water body, a bandpass filter, a CMOS camera, and a speckle reconstruction model for performing imaging through turbid water bodies. The target domain data set production process requires actual collection through an experimental device. In order to reduce the complexity of collection, the scale of this target domain data set is one tenth of the scale of the source domain data set described in step S1.

[0009] S3: training the pre-trained network and the fine-tuned network respectively, that is, training the pre-trained network and the fine-tuned network model based on the source domain dataset and the target domain dataset respectively, wherein the neural network model is a degradation-aware Transformer neural network;

[0010] Preferably, the neural network adopts a neural network based on a Transformer architecture to obtain high-quality reconstruction results. The network may include an attention mechanism, layer normalization, residual connection and a feedforward module. The pre-trained network and the fine-tuning network may use the same network model, or may be customized and replaced according to actual application requirements. The trained pre-trained network has the ability to reconstruct the original target from the autocorrelation image, and the trained fine-tuning network has the ability of feature adaptation, and may align the autocorrelation of the actually collected speckle image with the ideal target autocorrelation.

[0011] S4: construct a complete speckle reconstruction model, that is, connect the trained fine-tuning network and the pre-trained network in a cascade form to construct a complete speckle reconstruction model;

[0012] Preferably, the cascade method is to use the output result of the fine-tuning network as the input data of the pre-training network. For the input speckle image, after the autocorrelation operation, the feature adaptation is first performed through the fine-tuning network, so that the feature distribution of the refined autocorrelation image is closer to the requirements of the pre-training model. After receiving the refined autocorrelation, the pre-training model can more effectively process and adapt to the characteristics of the target domain, thereby significantly improving the overall performance of the complete reconstruction model in the target domain, especially when the data distribution difference between the source domain and the target domain is large.

[0013] S5: Testing phase, real-time reconstruction, that is, the collected speckle image is transmitted back to the reconstruction model in real time, and the collection results are continuously reconstructed through the reconstruction model to obtain real-time imaging results through turbid water bodies;

[0014] Preferably, the test phase is to achieve real-time reconstruction, that is, in an unknown turbid water test scene, a reflective imaging system is built, an incoherent LED device is used for lighting, and the digital micromirror device is controlled to load an unknown test original target. The light beam carrying the target information passes through the turbid water, passes through the aperture and the filter, and is finally captured by the CMOS camera. The collected speckle image is transmitted back to the reconstruction model in real time, and the reconstruction model is used to continuously reconstruct the camera acquisition results in a loop to obtain real-time imaging results through the turbid water.

[0015] According to the above technical solution, the present invention provides the following technical solution: a real-time imaging system based on speckle correlation and transfer learning, comprising: a pre-training preparation module, a fine-tuning preparation module, a neural network model training module, a speckle reconstruction model building module, and a model testing module;

[0016] Preferably, the pre-training preparation module is used to perform autocorrelation operations on original targets in the original target data set, produce data pairs of target autocorrelation and target, and construct a source domain data set for pre-training;

[0017] The fine-tuning preparation module is used to achieve the result with less speckle training data by combining speckle correlation imaging with transfer learning strategy, in which a small amount of speckle autocorrelation collected is used for fine-tuning;

[0018] The neural network model training module is used to train the pre-trained network and fine-tune the network model on the source domain dataset and the target domain dataset respectively;

[0019] The speckle reconstruction model building module is used to connect the trained fine-tuning network and the pre-trained network in a cascade form to build a complete speckle reconstruction model;

[0020] The model testing module is used for a two-stage speckle reconstruction process. In the first stage, the test speckle is first refined by a fine-tuning network to obtain a refined autocorrelation. In the second stage, the refined autocorrelation is then passed through a second network - a pre-trained network to obtain the reconstruction result.

[0021] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0022] 1. The present invention successfully realizes real-time imaging of hidden targets in turbid water bodies, and can efficiently and high-quality reconstruct the occluded original targets through degraded speckle images captured in complex dynamic imaging environments, and can obtain high-quality imaging results without the need for high-precision imaging equipment.

[0023] 2. The method of the present invention addresses the defects of the end-to-end deep learning imaging method, combines the neural network data model with the physical model, utilizes the link of speckle correlation imaging, and replaces the speckle-target pairs in most scattering imaging methods with more robust autocorrelation-target data pairs and speckle autocorrelation-target autocorrelation data pairs, thereby ensuring the generalization performance of the model.

[0024] 3. The method of the present invention is based on a transfer learning strategy. The source domain data set consists of target autocorrelation and target, and can be easily acquired without any optical equipment. It can give full play to the characteristics of the pre-training stage and the data advantages of the network, ensuring the reconstruction performance of the method.

[0025] 4. The method of the present invention processes pre-training and fine-tuning separately and deploys them on two models respectively, instead of performing two operations on the same model. The traditional pre-training-fine-tuning framework is decoupled into two independent steps, and the synergy between the two steps is explored. In addition, the design in the method of the present invention allows each model to be optimized independently without interfering with each other, making the system more flexible in dealing with different tasks and data sets. For example, new target domain data can directly adjust the fine-tuning network without retraining the entire system. This modular design not only improves the adaptability of the system, but also facilitates future expansion and upgrading.

[0026] 5. In the pre-training and fine-tuning tasks, the method of the present invention designs a neural network based on the Transformer architecture. By utilizing the unified context refinement attention mechanism, layer normalization, residual connection and feedforward module, the network can effectively capture and parse complex features to obtain high-fidelity reconstruction results.

[0027] 6. The present invention proposes targeted improvements to the defects of existing deep learning imaging methods, significantly improving the generalization ability of the network model while ensuring the high fidelity of the reconstruction results. The reconstruction model has good computing speed in the testing phase. The average reconstruction time for each test image is within 100 milliseconds, which can meet the real-time imaging needs of video streams with more than 10 frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0029] In the attached picture:

[0030] Figure 1 is a flow chart of a real-time imaging method based on speckle correlation and transfer learning provided by an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of a reflective imaging system device provided by an embodiment of the present invention;

[0032] Figure 3 is a statistical characteristic analysis diagram of speckle and autocorrelation image provided by an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of two training data sets obtained after executing steps S1 and S2 provided in an embodiment of the present invention;

[0034] Figure 5 is a block diagram of a degradation-aware Transformer network provided by an embodiment of the present invention;

[0035] Figure 6 is a speckle reconstruction flow chart of the testing phase provided by an embodiment of the present invention;

[0036] Figure 7 It is a result diagram of a reconstruction experiment on some unknown test targets in the Mnist and Fashion-mnist data sets provided by an embodiment of the present invention;

[0037] Figure 8 It is a schematic diagram of the composition structure of a real-time imaging system based on speckle correlation and transfer learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The present invention provides a technical solution: a real-time imaging method based on speckle correlation and transfer learning, specifically, as Figure 1 As shown, the embodiment of the present invention includes the following steps:

[0040] S1: Construct a source domain dataset for pre-training, that is, perform autocorrelation operations on the original targets in the original target dataset, that is, make data pairs of target autocorrelation and target, and construct a source domain dataset for pre-training;

[0041] Preferably, in this embodiment, the autocorrelation of each target can be calculated and generated without an optical device, and a data pair of the target autocorrelation and the target is produced. A source domain data set is constructed for pre-training, wherein the original target data set can use the Mnist and Fashion-mnist data sets. In order to give full play to the characteristics of the pre-training stage and the data advantages of the network model, the entire Mnist and Fashion-mnist data sets are selected for autocorrelation operations, that is, the scale of the source domain data set is 120,000 training pairs.

[0042] S2: Construct a target domain dataset for fine-tuning, that is, in a laboratory environment, build a dynamic scattering scene, use a reflective imaging system to collect speckle images, perform autocorrelation operations to obtain the autocorrelations of each speckle, make data pairs of speckle autocorrelations and target autocorrelations, and construct a target domain dataset for fine-tuning;

[0043] Preferably, the reflective imaging system is Figure 2 As shown, it includes: an incoherent LED light source, a lens, an aperture, a digital micromirror device and the original target loaded thereon, turbid water, a bandpass filter, a CMOS camera, and a speckle reconstruction model for performing imaging through turbid water.

[0044] In this embodiment, the incoherent light beam generated by the incoherent LED light source passes through the lens and aperture 1 to illuminate the digital micromirror device, illuminating the original target loaded thereon, and then the light beam carrying the target information passes through the turbid water and aperture 2, and is captured by the industrial CMOS camera after passing through a bandpass filter, thus completing the acquisition of a speckle image;

[0045] More specifically, the incoherent LED light source can be a narrow-band incoherent light source with a wavelength of 625 nanometers, such as Thorlabs' M625L4 light source, the diameters of aperture 1 and aperture 2 are 5 mm, the turbid water body can be made of a polystyrene microsphere suspension, a calcium carbonate solution, a fat emulsion, a sea salt solution or a mixed solution of multiple media, the central wavelength of the bandpass filter is 632 nanometers, and the resolution of the CMOS camera can be 1920*1200 pixels, and the pixel size is 5.86 microns / pixel;

[0046] Furthermore, according to the principle of speckle correlation imaging, within the range of the optical memory effect, the point spread function PSF of the imaging system satisfies the linear translation invariance characteristic, and the speckle image collected on the imaging surface can be expressed as the convolution of the target and the point spread function as shown below:

[0047] Ι(x)=O(x)*S(x) (1)

[0048] Wherein, “*” represents the convolution operator, O(x) and S(x) represent the target and point spread function respectively. Using the collected speckle and applying the convolution theorem, the speckle autocorrelation is equal to the convolution of the target autocorrelation and the autocorrelation of the PSF of the imaging system. The imaging model is shown in formula (2):

[0049] Ι(x)★Ι(x)=[O(x)*S(x)]★[O(x)*S(x)]=[O(x)★O(x)]*[S(x)★S(x)] (2)

[0050] Where ★ represents the autocorrelation operator, and the autocorrelation of the point spread function can be further expressed as:

[0051]

[0052] Where J1 represents the first kind of Bessel function, ∝ means proportional, d, λ and v represent the aperture diameter, wavelength and imaging distance respectively. Based on formula (3), it can be shown that the autocorrelation S(x)★S(x) of the point spread function of the imaging system is a peak function with a certain width that affects the autocorrelation resolution. Its width is related to the size of the speckle particles. Within the optical memory range, the point spread function shows a high degree of similarity, so that the autocorrelation of two approximately identical point spread functions can be regarded as a delta function that has a negative impact on the reconstruction result, which can be approximated as the background noise constant C;

[0053] Therefore, the speckle correlation imaging model can be further expressed as:

[0054] Ι(x)★Ι(x)=O(x)★O(x)+C (4)

[0055] The background noise constant C is the difference between the target autocorrelation and the speckle autocorrelation, and this difference will be affected by the change of the scattering medium and the experimental environment.

[0056] In this embodiment, the application scenario of the present invention is a dynamic scattering scenario of turbid water, whose typical feature is that the scattering characteristics are constantly changing. In order to further analyze and verify the influence of scattering medium transformation and experimental environment on speckle reconstruction, the speckles and autocorrelation images collected from the same original target through different scattering media are statistically analyzed. The results are as follows: Figure 3 As shown, Figure 3 (a) shows the original target and target autocorrelation as the true value in a darkroom environment, as well as the speckle and their autocorrelation images collected after the medium transformation. By plotting their statistical characteristics along the dotted lines in the image, Figure 3(c) and (d) show the statistical characteristics of the speckle image and the speckle autocorrelation image, respectively. It can be observed that even for the same original target, the statistical characteristics of different speckles collected after the medium transformation are very different. It is this difference that limits the generalization ability of the end-to-end imaging method based on deep learning. After the autocorrelation operation, the speckle images with different distributions have relatively unified structural characteristics, such as Figure 3 As shown in (d), due to the structural characteristics of the target autocorrelation, the imaging method based on speckle correlation proposed in the present invention also has good generalization performance. Figure 3 (d) also shows the individual differences between the characteristic curves, as shown in the speckle correlation imaging model. These differences are the embodiment of the background noise constant, representing the influencing factors such as the change in the medium scattering characteristics and the imaging environment, which affect the final imaging effect. As the intensity of external interference increases, the difference between the speckle autocorrelation and the target autocorrelation will also increase. For example, the low signal-to-noise ratio speckle and its autocorrelation image collected under external interference light is shown in Figure 3 As shown in (b), there is a significant difference between the true value of the autocorrelation image and the true value of the autocorrelation image.

[0057] Further, after executing steps S1 and S2 of the embodiment of the present invention, the following can be obtained: Figure 4 The two training datasets shown are: a larger source domain dataset generated directly through program calculation and a smaller target domain dataset made through experimental collection.

[0058] S3: Train the pre-trained network and the fine-tuned network separately, that is, train the pre-trained network and the fine-tuned network models on the source domain dataset and the target domain dataset respectively;

[0059] Preferably, the neural network can adopt a neural network based on the Transformer architecture to obtain high-quality reconstruction results. The network can include an attention mechanism, layer normalization, residual connection and a feedforward module. In the embodiment of the present invention, a neural network named degradation-aware Transformer designed by the inventor is used, and its network block diagram is as follows: Figure 5 shown.

[0060] In this embodiment, each block of the network includes a unified context refinement attention module, layer normalization, residual connection, and feedforward module.

[0061] Furthermore, the unified contextual refinement attention mechanism regulates high-order feature associations at the channel and spatial levels to achieve coordinated regulation of channel features and spatial structures. It comprehensively considers the importance weight of each feature channel and the intrinsic correlation and importance between different spatial positions in the image. Therefore, the unified contextual refinement attention calculates the global importance score of each channel and adjusts the feature representation of each channel based on these scores. It also calculates the similarity or correlation between pixel positions to guide the model to focus on specific areas and details in the image. This integrated attention mechanism helps the network effectively capture the relationship between different channels and spatial positions in the image, and improves the discriminative and generalization capabilities of feature representation. In order to further improve efficiency and retain important feature information, the standard convolution is decomposed into two smaller convolution operations by introducing deep separable convolution, which significantly reduces the amount of calculation.

[0062] For example, by using layer normalization in the network, it helps to normalize the input of each module, thereby speeding up training and improving model performance. The use of residual connections can effectively transmit and learn information between input and output, ensuring the stability and efficiency of the training process. The use of feedforward modules can perform nonlinear transformation and extraction on features, thereby improving the network's understanding and generalization capabilities of image structures.

[0063] Preferably, the pre-trained network and the fine-tuned network can use the same network model, or can be customized and replaced according to actual application requirements. In the embodiment of the present invention, the pre-trained network and the fine-tuned network both use the degradation-aware Transformer network designed by ourselves;

[0064] Specifically, the pre-trained network and the fine-tuned network will be respectively pre and the target domain dataset D fine The convergence state is reached, that is, the convergence state reached by the pre-trained network and the fine-tuned network is expressed as:

[0065]

[0066] Among them, arg min represents the minimum value of the objective function, Θ represents the model parameter set, represents the expectation of solution, Represents the network loss function, f pre With f fine Represent the pre-trained network and the fine-tuned network, X represents the input data, and Y represents the label data. Therefore, during the training process of the pre-trained and fine-tuned networks, the parameter set Θ of the neural network will be continuously optimized so that the loss function between the predicted output of the network on the training data set and the label data Y Reaching the minimum means reaching the convergence state.

[0067] In the embodiment of the present invention, the loss function of the network adopts a combination of Charbonnier loss and edge loss, which is expressed as follows:

[0068]

[0069] The parameter λ determines the weight of the two losses and is selected as 0.05. GT and denote the generated image and the true value respectively, and the ε constant is set to 1e-3;

[0070] Preferably, in the embodiment of the present invention, the trained pre-trained network has the ability to reconstruct the original target from the autocorrelation image, and the trained fine-tuning network has the ability of feature adaptation, which can align the autocorrelation of the actually collected speckle image with the ideal target autocorrelation.

[0071] S4: construct a complete speckle reconstruction model, that is, connect the trained fine-tuning network and the pre-trained network in a cascade form to construct a complete speckle reconstruction model;

[0072] Preferably, in the embodiment of the present invention, the output result of the fine-tuning network is used as the input data of the pre-training network to construct a complete speckle reconstruction model in a cascade form. After the construction is completed, in the testing phase, Figure 6 As shown in the figure, for the input speckle image, after the autocorrelation operation, the feature adaptation is first performed through the fine-tuning network, so that the feature distribution of the refined autocorrelation image is closer to the requirements of the pre-trained model. After receiving the refined autocorrelation, the pre-trained model can more effectively process and adapt to the characteristics of the target domain, thereby significantly improving the overall performance of the model in the target domain, especially when the data distribution difference between the source domain and the target domain is large.

[0073] For example, the speckle correlation imaging model of formula (4) is Figure 3 As shown in (d), there are data distribution differences between speckle autocorrelations and ideal target autocorrelations. Therefore, the fine-tuning network effectively plays the role of a feature adapter. First, the target domain data is preprocessed to reduce the data heterogeneity between speckle autocorrelations and target autocorrelations, unify the features of different speckle autocorrelations, and align their feature representations with the target autocorrelations, so that the target domain features in the refined autocorrelation image can be more effectively migrated to the pre-trained model, as shown in Figure 3 As shown in (a), (b) and (e), the refined autocorrelation of the fine-tuned network output has more unified structural characteristics and a higher degree of fit with the target autocorrelation, ensuring that the pre-trained model can better adapt to the needs of the target domain when processing these data, thereby significantly improving the performance of the overall model in the target domain.

[0074] S5: Testing phase, real-time reconstruction, that is, in an unknown turbid water test scene, a reflective imaging system is built, an incoherent LED device is used for illumination, and the digital micromirror device is controlled to load an unknown test original target. The light beam carrying the target information passes through the turbid water, passes through the aperture and filter, and is finally captured by the CMOS camera. The collected speckle image is transmitted back to the reconstruction model in real time. The reconstruction model is used to continuously reconstruct the camera acquisition results in a loop to obtain real-time imaging results through the turbid water.

[0075] Preferably, the reflective imaging system constructed in the embodiment of the present invention is as follows Figure 2 As shown in Figure 2, the reconstruction results of some unknown test targets in the Mnist and Fashion-mnist datasets are shown in Figure 2. Figure 7 As shown, it can be observed that no matter it is a handwritten digital target or a Fashion target, the reconstruction results of the present invention perform excellently in terms of details and overall structure, maintain a fairly high degree of similarity with the original target, and achieve high-fidelity imaging results through turbid water bodies.

[0076] The embodiment of the present invention also provides a real-time imaging system based on speckle correlation and transfer learning, such as Figure 8 As shown, this embodiment includes: a pre-training preparation module, a fine-tuning preparation module, a neural network model training module, a speckle reconstruction model building module, and a model testing module;

[0077] Exemplarily, the pre-training preparation module is used to perform autocorrelation operations on original targets in the original target data set, produce data pairs of target autocorrelation and target, and construct a source domain data set for pre-training;

[0078] The fine-tuning preparation module is used to achieve the result with less speckle training data by combining speckle correlation imaging with transfer learning strategy, in which a small amount of speckle autocorrelation collected is used for fine-tuning;

[0079] The neural network model training module is used to train the pre-trained network and fine-tune the network model on the source domain dataset and the target domain dataset respectively;

[0080] The speckle reconstruction model building module is used to connect the trained fine-tuning network and the pre-trained network in a cascade form to build a complete speckle reconstruction model;

[0081] The model testing module is used for a two-stage reconstruction process. In the first stage, the test speckle is first refined by a fine-tuning network to obtain a refined autocorrelation. In the second stage, the refined autocorrelation is then passed through a second network - a pre-trained network to obtain the reconstruction result.

[0082] Furthermore, the pre-training preparation module also includes: an original data set module and an autocorrelation operation module;

[0083] Specifically, the original data set module is used to select the original target data set of the Mnist and Fashion-mnist data sets; the autocorrelation operation module is used to perform autocorrelation operations based on the entire Mnist and Fashion-mnist data sets;

[0084] The fine-tuning preparation module also includes: a speckle autocorrelation module and a data pair module;

[0085] The speckle autocorrelation module is used to perform autocorrelation calculation based on the acquired speckle image to obtain the autocorrelation of each speckle, and the data pair module is used to produce data pairs of speckle autocorrelation and target autocorrelation;

[0086] The neural network model training module also includes a degradation-aware Transformer neural network module, which is used to obtain high-quality reconstruction results using a degradation-aware Transformer neural network based on the Transformer architecture;

[0087] The speckle reconstruction model building module also includes a cascade connection interaction module, which is used to connect and interact the trained fine-tuning network and the pre-trained network in a cascade form;

[0088] The model testing module also includes an imaging result building module, which is used to use the reconstruction model to continuously and cyclically reconstruct the camera acquisition results to obtain real-time imaging results through turbid water bodies.

[0089] In summary, the real-time imaging method and system based on speckle correlation and transfer learning in the embodiments of the present invention successfully realize the real-time imaging of occluded targets in complex dynamic scattering scenarios, and can efficiently and high-quality reconstruct the occluded original target through the degraded speckle image captured in the complex dynamic imaging environment, and can obtain high-quality imaging results without the need for high-precision imaging equipment.

[0090] The present invention combines the neural network data model with the speckle-related physical model and utilizes robust autocorrelation data pairs to improve the generalization performance of the model. Based on the transfer learning strategy, the source domain data set can be easily acquired without optical equipment, giving full play to the characteristics of the pre-training stage and the data advantages of the network to ensure the reconstruction performance of the method. By coupling pre-training and fine-tuning into independent steps and deploying each in different models, not only the system flexibility and modular design are enhanced, but also expansion and upgrading are facilitated. The proposed neural network based on the Transformer architecture can effectively capture and parse complex features to obtain high-fidelity reconstruction results.

[0091] Therefore, the present invention proposes targeted improvements to the defects of existing deep learning imaging methods, significantly improving the generalization ability of the network model while ensuring the high fidelity of the reconstruction results. The reconstruction model has good computing speed in the test phase, and the average reconstruction time of each test image is within 100 milliseconds, which can meet the real-time imaging requirements of video streams of more than 10 frames. That is, the embodiment of the present invention specifically realizes a real-time imaging based on speckle correlation and transfer learning through turbid water, which can be applied to technical fields such as complex underwater imaging and turbid water monitoring.

[0092] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0093] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time imaging method based on speckle correlation and transfer learning, characterized in that: The method comprises the following steps: S1: construct a source domain dataset for pre-training, wherein the source domain dataset contains the original target and the target autocorrelation. The pre-training is one of the key steps of the transfer learning strategy, that is, training the pre-trained network based on the source domain dataset; S2: constructing a target domain dataset for fine-tuning, wherein the target domain dataset includes speckle autocorrelation and target autocorrelation. The fine-tuning is another key step of the transfer learning strategy, i.e., training a fine-tuning network based on the target domain dataset; S3: training the pre-trained network and the fine-tuned network respectively, that is, training the pre-trained network and the fine-tuned network model based on the source domain dataset and the target domain dataset respectively, wherein the network model is a degradation-aware Transformer neural network; S4: construct a complete speckle reconstruction model, that is, connect the trained fine-tuning network and the pre-trained network in a cascade form to construct a complete speckle reconstruction model; S5: Testing phase, real-time reconstruction, that is, testing is performed based on the speckle reconstruction model, the collected speckle image is transmitted back to the reconstruction model in real time, and the collection result is continuously reconstructed through the reconstruction model in a cycle to obtain real-time imaging results through turbid water.

2. The real-time imaging method based on speckle correlation and transfer learning according to claim 1, characterized in that: In the steps S1 and S2, the pre-training and fine-tuning further include combining the speckle correlation imaging with the transfer learning strategy. In the fine-tuning process, the transfer learning strategy involves less training data, that is, the large-scale source domain data set obtained by simulation in the pre-training reduces the actually collected speckle data.

3. The real-time imaging method based on speckle correlation and transfer learning according to claim 2, characterized in that: After executing steps S1 and S2, two training data sets are obtained, including a larger-scale source domain data set generated directly by program calculation and a smaller-scale target domain data set produced by experimental collection.

4. The real-time imaging method based on speckle correlation and transfer learning according to claim 1, characterized in that: The pre-trained network and the fine-tuned network use the same network model, which is a degradation-aware Transformer network. Each block of the degradation-aware Transformer network includes a unified context-refined attention module, layer normalization, residual connection, and feedforward module. The unified contextual refinement attention module performs high-order feature correlation regulation at the channel and spatial levels to achieve coordinated regulation of channel features and spatial structures. By calculating and scoring the importance weight of each feature channel and the intrinsic correlation and importance between different spatial positions in the image, the feature representation of each channel is adjusted according to the score, and the similarity or correlation between pixel positions is calculated at the same time, guiding the model to focus on specific areas and details in the image, and further decomposing the standard convolution into two smaller convolution operations based on deep separable convolution to reduce the amount of calculation; the layer normalization is used to normalize the input of each module, speed up the training speed and improve the model performance; the residual connection is used to transmit and learn the information between input and output; The feedforward module is used to perform nonlinear transformation and extraction on features.

5. The real-time imaging method based on speckle correlation and transfer learning according to claim 4, characterized in that: The training of the pre-trained network and the fine-tuned network is as follows: pre and the target domain dataset D fine The convergence state is reached, that is, the convergence state reached by the pre-trained network and the fine-tuned network is expressed as: Among them, arg min represents the minimum value of the objective function, Θ represents the model parameter set, represents the expectation of solution, Represents the network loss function, f pre With f fine Represent the pre-trained network and the fine-tuned network, X represents the input data, and Y represents the label data. Therefore, during the training process of the pre-trained and fine-tuned networks, the parameter set Θ of the neural network will be continuously optimized to make the loss function between the predicted output of the network on the training data set and the label data Y Reaching the minimum, that is, reaching the convergence state; The network loss function is a combination of Charbonnier loss and edge loss, expressed as follows: L=L char +λL edge The parameter λ determines the weight of the two losses and is selected as 0.

05. GT and represent the generated image and the true value respectively, and the ε constant is set to 1e-3.

6. The real-time imaging method based on speckle correlation and transfer learning according to claim 5, characterized in that: The training of the pre-trained network and the fine-tuning network also includes: the trained pre-trained network has the ability to reconstruct the original target from the autocorrelation image, and the trained fine-tuning network has the ability of feature adaptation, and can align the autocorrelation of the actually collected speckle image with the ideal target autocorrelation.

7. The real-time imaging method based on speckle correlation and transfer learning according to claim 6, characterized in that: In the construction of the complete speckle reconstruction model: the output result of the fine-tuning network is used as the input data of the pre-training network, and the complete speckle reconstruction model is constructed in a cascade form. The target domain data is pre-processed based on the trained fine-tuning network, the features of different speckle autocorrelations are unified, and their feature representations are aligned with the target autocorrelations, so that the target domain features in the refined autocorrelation image are effectively migrated to the pre-training model.

8. The real-time imaging method based on speckle correlation and transfer learning according to claim 1, characterized in that: The test process includes: in an unknown turbid water test scene, building a reflective imaging system, using an incoherent LED device for lighting, controlling the digital micromirror device to load an unknown test original target, and allowing a light beam carrying target information to pass through the turbid water, through an aperture and a filter, and finally be captured by a CMOS camera and the collected speckle image is transmitted back to a reconstruction model in real time, and the reconstruction model is used to continuously and cyclically reconstruct the camera acquisition results to obtain real-time imaging results through the turbid water.

9. A real-time imaging system based on speckle correlation and transfer learning, characterized in that: The system comprises: A pre-training preparation module is used to perform autocorrelation operations on the original targets in the original target data set, generate data pairs of target autocorrelation and target, and construct a source domain data set for pre-training; A fine-tuning preparation module for obtaining results with less speckle training data by combining speckle correlation imaging with a transfer learning strategy, wherein a small amount of acquired speckle autocorrelations are used for fine-tuning; A neural network model training module is used to train the pre-trained network and fine-tune the network model on the source domain dataset and the target domain dataset respectively; The neural network model training module also includes a degradation-aware Transformer neural network module, which is used to obtain high-quality reconstruction results using a degradation-aware Transformer neural network based on a Transformer architecture; A speckle reconstruction model building module is used to connect the trained fine-tuning network and the pre-trained network in a cascade form to build a complete speckle reconstruction model; The model testing module is used for the two-stage speckle reconstruction process. In the first stage, the test speckle is first refined by the fine-tuning network to obtain the refined autocorrelation. In the second stage, the refined autocorrelation is then passed through the second network - the pre-trained network to obtain the reconstruction result.

Images