A dynamic scattering imaging method based on a lightweight network and space-time prior

Abstract

The application discloses a dynamic scattering imaging method based on a light-weight network and space-time prior, relates to the field of computational imaging and deep learning, and the technical scheme of the application comprises the following steps: S1, constructing a physical model for imaging through a dynamic scattering medium; S2, deep learning processing based on a light-weight network and space-time prior; and S3, outputting a reconstructed image of a target behind the dynamic scattering medium. The application can complete network training only by using a small amount of data, reduces model parameter quantity and computation quantity by about 90%, significantly improves processing speed, and effectively improves the imaging precision and generalization ability under a dynamic scattering condition.

Description

Technical Field

[0001] This invention relates to the fields of computational imaging and deep learning, and in particular to a dynamic scattering imaging method based on lightweight networks and spatiotemporal priors. Background Technology

[0002] Imaging through scattering media is an important research direction in computational imaging, with wide applications in polarization dehazing, biological tissue imaging, underwater imaging, and dynamic environmental monitoring. When light passes through scattering media such as fog, turbid water, and biological tissue, it interacts strongly with the medium and is randomly scattered. The spatial information carried by the target is severely disturbed during propagation, ultimately forming a messy speckle image on the detection surface, making it difficult to obtain a clear target image directly. Therefore, how to overcome light scattering and achieve high-quality imaging through scattering media has always been a pressing problem to be solved in this field.

[0003] Existing scattering imaging techniques can be broadly classified into two categories.

[0004] The first category is classical scattering imaging methods, including wavefront shaping, transfer matrix reconstruction, single-pixel imaging, and speckle autocorrelation based on optical memory effects. These methods can achieve good reconstruction results when the target is known a priori and the scattering conditions are relatively stable, but they usually require complex iterative calculations and have poor transferability to different scattering media. Among them, methods based on memory effects are also strictly limited by the range of memory effects, making them difficult to apply to dynamic, unknown, or complex scattering environments.

[0005] The second category is deep learning-based scattering imaging methods. These methods leverage the powerful feature extraction and mapping capabilities of neural networks to quickly and accurately reconstruct targets after model training, overcoming some of the limitations of classical methods. However, in dynamic scattering imaging tasks, existing deep learning solutions generally suffer from the "data dependency-computational cost-dynamic real-time performance" trilemma, meaning they struggle to simultaneously meet the requirements of low data dependency, low computational cost, and dynamic real-time processing.

[0006] Regarding data dependency, purely data-driven methods require a large number of paired speckle images and target images as training samples. These training samples need to cover different scattering media, scattering intensities, target shapes, and motion states. When the scattering medium is disturbed, the target's motion state changes, or system parameters such as imaging distance and wavelength deviate from the training distribution, the network is prone to domain shift, leading to decreased reconstruction accuracy. For real-world scenarios such as biological tissues, underwater turbid media, and hazy environments, it is difficult to obtain large-scale, high-quality paired samples, limiting the transferability and generalization ability of such methods in real-world dynamic scenarios.

[0007] In terms of computational cost, while physics-driven or physics-embedded deep learning methods can reduce their dependence on training data by utilizing scattering models, they often require repeatedly embedding diffraction propagation, transfer matrix solving, phase retrieval, or iterative optimization processes within the network. These methods typically involve a large number of parameters, complex forward-backward propagation links, and multiple iterative calculations per frame, leading to a significant increase in memory usage, floating-point operations, and single-frame processing time, making them difficult to deploy in imaging devices on mobile, embedded, or edge devices.

[0008] In terms of dynamic real-time performance, dynamic scattering media and moving targets cause speckle patterns to continuously change over time. If optimization is performed from a random initialization or a fixed initial state for each frame, the computation time will be insufficient to meet the requirements of real-time imaging. If a static model is directly reused without updating the network parameters, errors will accumulate due to scattering state drift and target motion, leading to flickering, loss of detail, or structural distortion in the video sequence reconstruction. Existing methods often focus on single-frame spatial priors and do not fully utilize the temporal continuity and redundant information between adjacent frames, thus making it difficult to balance reconstruction quality and frame rate under dynamic conditions.

[0009] Therefore, existing technologies typically require a trade-off among these three factors: increasing training data reduces the difficulty of single-frame inference, but data acquisition costs are high and generalization is insufficient; strengthening the physical model reduces data dependence, but significantly increases computational complexity; pursuing real-time performance usually requires simplifying the model or reducing the number of iterations, thereby sacrificing reconstruction quality. How to ensure reconstruction reliability using physical constraints under limited data conditions, while simultaneously reducing computational overhead through lightweight networks and utilizing the spatiotemporal prior information of video to achieve dynamic real-time imaging, is a pressing technical challenge in this field. Summary of the Invention

[0010] This invention provides a dynamic scattering imaging method based on lightweight networks and spatiotemporal priors. It aims to resolve the technical contradiction in existing solutions where data dependence, computational cost, and dynamic real-time performance are difficult to balance simultaneously.

[0011] The technical solution adopted by this invention to solve its technical problem is: a dynamic scattering imaging method based on lightweight networks and spatiotemporal priors, comprising the following steps: S1, Construct a physical model for imaging through a dynamic scattering medium; S2, based on lightweight networks and spatiotemporal priors for deep learning processing; S3 outputs a reconstructed image of the target behind a dynamically scattering medium.

[0012] A three-layer progressive technical architecture is proposed: a physical modeling layer (S1), a deep learning processing layer (S2), and an output layer (S3). Its core principle lies in the deep integration of physical prior constraints and data-driven learning—instead of simply relying on neural networks to learn mapping relationships from data, it embeds the physical laws of the scattering process (angular spectrum diffraction theory) into the optimization framework, so that network training is constrained by physical consistency.

[0013] It breaks away from the dependence of traditional pure data-driven methods on large-scale paired training data, ensures that the reconstruction results conform to the laws of optical scattering, improves interpretability, and balances reconstruction accuracy and processing speed under dynamic scattering conditions.

[0014] Preferably, step S1 includes: S1.1 decomposes the light scattering process into three independent stages: the incident light carrying target information arrives at the dynamic scattering medium plane, the light passes through the dynamic scattering medium with random phase and amplitude modulation, and the modulated scattered light field propagates to the detection surface in free space. S1.2, the propagation of the light field in free space follows the angular spectrum diffraction theory, and the speckle image I is obtained by the detector on the detection surface; S1.3, the relationship between the speckle image and the target image is regarded as the process of forward encoding the target information into the speckle intensity map.

[0015] The complex light scattering process is modularly decomposed into three independent stages: target light field propagation → random modulation of the scattering medium → free propagation of the scattered light field. This decomposition is based on angular spectrum diffraction theory, discretizing the continuous physical process into computable and modelable mathematical relationships. Among them, S1.3 reformulates the imaging inverse problem as a "forward encoding" problem, laying the foundation for subsequent physics-driven optimization.

[0016] The inverse scattering problem, which is difficult to solve directly, is transformed into a well-state problem that can be optimized through forward model iteration. This provides clear physical constraints and loss calculation basis for subsequent network training, and allows different scattering conditions (medium type, distance, wavelength) to be flexibly adapted by adjusting model parameters.

[0017] Preferably, step S1 establishes a forward scattering relationship between the scattered light field and the target light field that satisfies angular spectral diffraction, the expression of which is: in, and These represent the light field information before and after scattering, respectively. , Let represent the angular spectral transfer functions of the distance from the target to the dynamic scattering medium and the distance from the dynamic scattering medium to the detector, respectively. This represents the convolution operation. is the amplitude transmittance function, used to describe the amplitude and phase modulation of incident light by the scattering medium.

[0018] This formula is a specific application of angular spectrum diffraction theory in scattering imaging. The two convolutions in the formula correspond to the free-space propagation from the target to the scattering medium and from the scattering medium to the detector, respectively (determined by the angular spectrum transfer function). (Description), the point-by-point multiplication in the middle corresponds to the amplitude transmittance modulation of the scattering medium. This model rigorously describes the scattering process as a cascade structure of linear propagation + nonlinear modulation + linear propagation. This shifts the network optimization objective from "matching manual annotations" to "satisfying physical consistency," significantly reducing the reliance on labeled data.

[0019] Preferably, step S2 includes: S2.1 Construct a lightweight network ST-StarNet. The network adopts an encoder-decoder structure and includes a Stem module, several Stage modules, a StarBlock module, an upsampling module, and skip connections in sequence. Its input layer receives the speckle image I collected by the detector, and its output layer outputs the predicted target reconstruction image. S2.2, The acquired speckle image I is input into the lightweight network for forward propagation to obtain the predicted target reconstruction image; S2.3, using a known scattering model, perform forward calculations on the predicted target reconstruction image to obtain the corresponding calculated speckle intensity distribution; S2.4, calculate the error between the input speckle image I and the calculated speckle intensity distribution, and use it as the loss function of the network; S2.5, The weights of the network are updated according to the loss function through the backpropagation algorithm to reduce the error; S2.6 Repeat steps S2.2 to S2.5 until the loss function reaches the preset threshold or the number of iterations, thus completing the network optimization.

[0020] A physically embedded self-supervised training paradigm is adopted: the network predicts and reconstructs the image → the forward scattering model calculates the simulated speckle pattern → the error is compared with the real speckle pattern → backpropagation updates the weights. This is an "analysis-synthesis" closed-loop optimization strategy, where the network can complete self-supervised training using only the speckle image itself and a known physical model, without any target image annotations. ST-StarNet's encoder-decoder structure is specifically designed for image reconstruction tasks, unlike classification networks.

[0021] Training on zero-labeled data is achieved, requiring only speckle images for network optimization. The encoder-decoder structure, combined with skip connections, effectively recovers spatial details lost during scattering. An iterative optimization mechanism ensures the network output progressively approximates the physically accurate solution.

[0022] Preferably, in step S2, the error between the input speckle image I and the speckle intensity distribution calculated by the scattering model is expressed as: in, Defined as the L2 norm between the measured value and the scattering model prediction. For scattering models, Represents the pre-trained decoder. For the first The decoder has been optimized and updated. For the first Frame-by-frame true speckle image.

[0023] The L2 norm is used to measure physical consistency, which is the difference between the actual speckle and the speckle predicted by the model. Represents the current decoder state. This is a fixed forward scattering model. The essence of this loss function is to search for the optimal parameters in the decoder parameter space that minimize the forward reprojection error, which belongs to the deep learning implementation of model-driven inverse problem solving.

[0024] L2 norm calculation is simple and efficient, facilitating gradient backpropagation. It directly measures physical consistency rather than visual similarity, avoiding pixel-level annotation.

[0025] Preferably, the StarBlock module differs from the four-stage downsampling star module used for image classification in the general StarNet. It is embedded in the multi-scale feature extraction path and decoding refinement path of the encoder-decoder reconstruction network, and does not have a global pooling layer or a classification head. For input features The StarBlock includes a gating mechanism and depthwise separable convolution. First, a large-kernel depthwise convolution expands the receptive field, then a multi-branch star-shaped operation performs high-dimensional nonlinear feature mapping. The star-shaped operation is an element-wise multiplication operation. Finally, residual connections and depthwise separable convolution refine the path and optimize the feature representation, satisfying the following: in express Large kernel depthwise convolution, and These represent channel expansion and channel compression projection, respectively. This represents a nonlinear activation or gate function. This indicates element-wise multiplication. This represents depthwise separable convolution. This indicates the output features.

[0026] Task-oriented structural modifications to the general-purpose StarNet: 1. Large kernel depthwise convolution ( ): Expanding the receptive field to capture long-range spatial correlations of speckle patterns.

[0027] 2. Star operation (element-by-element multiplication) ): Introducing second-order polynomial feature interactions without increasing the number of explicit channels to form a high-dimensional implicit nonlinear expression.

[0028] 3. Depthwise Separable Convolution (DSConv): Decomposes standard convolution into depthwise convolution + pointwise convolution, reducing the number of parameters and computational cost.

[0029] 4. Residual connections: alleviate gradient vanishing and promote deep network training.

[0030] From a mathematical perspective, star operations Expandable to include Second-order polynomials, when stacked in multiple layers, can form higher-order implicit feature spaces. This allows for the expression of complex nonlinear mappings in scattering imaging with low computational cost. Compared to UNet, ST-StarNet reduces both parameter and computational cost by approximately 90%, facilitating embedded deployment. It maintains high reconstruction fidelity (SSIM of 0.90 and PSNR of 26.74 dB). Simultaneously integrated into the encoding, decoding, and skip paths, it balances multi-scale feature extraction and detail restoration.

[0031] Preferably, step S2 further introduces a spatiotemporal prior training strategy, including: S2.7, use the spatial prior information learned from the previous frame of the video sequence as the prior for the reconstruction of the current frame to establish the spatiotemporal relationship between adjacent frames; S2.8, Set threshold Calculate the metric value of the current frame. ,when Greater than Update decoder weights in real time, when Not greater than To maintain the decoder weights unchanged and allow subsequent frames to be reconstructed using the current decoder, the expression is: in, For the first The second-optimized decoder This is a preset threshold.

[0032] The temporal redundancy between video frames is used as a regularization prior. In dynamic videos, adjacent frames are highly correlated, and the changes in the scattering medium are gradual. Therefore, the current frame does not need to be reconstructed from scratch; instead, incremental fine-tuning is performed based on the decoder weights that have converged in the previous frame. A threshold is set... Adaptive determination of whether weights need to be updated: when physical consistency error Use it directly when Optimization is triggered only when the time is right. This significantly shortens iteration time and improves frame processing speed (approximately 60% increase in FPS) – avoiding computational redundancy caused by independent frame-by-frame optimization. It ensures the temporal continuity of the video sequence, reducing inter-frame flicker and abrupt changes – enabling high frame rate dynamic reconstruction even when scattering states change slowly.

[0033] Preferably, step S3 includes: S3.1, using the optimized lightweight network, the speckle video sequence acquired behind the dynamic scattering medium is reconstructed frame by frame to obtain the target reconstruction image of each frame; S3.2, based on the spatiotemporal prior training strategy, adaptively reuses or updates the decoder weights for adjacent frames to achieve dynamic reconstruction at a high frame rate; S3.3 outputs the reconstructed image of the target behind the dynamic scattering medium, and evaluates and verifies the performance of the reconstruction results.

[0034] The trained lightweight network is combined with spatiotemporal prior policies to form an online inference process: frame-by-frame input speckle → adaptive reuse / update of the decoder → output of reconstructed image. This process transfers the physical consistency and spatiotemporal continuity constraints obtained from offline training to online dynamic inference.

[0035] A dynamic scattering imaging device based on lightweight networks and spatiotemporal priors includes: The physical model building module is used to establish the forward scattering relationship of incident light carrying target information, which is randomly modulated by a dynamic scattering medium and propagates to the detector surface in free space, thereby generating a speckle intensity distribution on the detector surface. A data acquisition module is used to connect to the detection surface of the physical model construction module to acquire the speckle intensity distribution; A lightweight network processing module is used to receive the speckle intensity distribution output by the data acquisition module as input, and output the target reconstructed image through a lightweight network; the lightweight network processing module is configured with a backpropagation mechanism, which adjusts the network weights to reduce the error between the input speckle image I and the speckle intensity distribution calculated by the scattering model; A scattering model calculation module is used to connect to the lightweight network processing module and calculate the corresponding speckle intensity distribution from the target reconstruction image output by the lightweight network processing module through a scattering model. The error measurement and feedback module is used to measure the error between the speckle image I acquired by the data acquisition module and the speckle intensity distribution calculated by the scattering model calculation module, and feeds the error signal back to the lightweight network processing module for weight update until the error meets the preset threshold, thereby completing the network optimization.

[0036] The physical model building module provides interpretable forward constraints, the lightweight network processing module enables efficient inverse problem solving, the error feedback module forms a self-supervised optimization closed loop, and the whole device can be deployed on resource-constrained platforms (mobile devices, embedded devices).

[0037] Preferably, the lightweight network processing module employs an encoder-decoder network to extract features from the speckle image I and output the target reconstructed image; the encoder-decoder network includes depthwise separable convolution, gating mechanism, residual connection and skip connection; The error measurement and feedback module uses mean squared error loss, structural similarity loss, or other loss functions suitable for measuring the differences in speckle intensity distribution to calculate the error; It also includes a spatiotemporal prior processing module, which is connected to the lightweight network processing module. This module is used to use the temporal redundancy information between adjacent frames in the video sequence as a priori to guide the reconstruction of the current frame, and to determine whether to update the decoder weights based on a preset threshold, so as to simultaneously satisfy physical constraints and temporal consistency constraints.

[0038] Depthwise separable convolutions further reduce computational overhead. SSIM loss improves structural similarity, while MSE loss guarantees pixel-level accuracy. The spatiotemporal prior module enables the system to dynamically adapt, achieving a balance between physical constraints and temporal consistency.

[0039] Advantages of this invention: This approach uses a scattering model as a forward constraint, guiding the network to optimize under physical consistency, thus enabling training without relying on a large amount of paired training data. This overcomes the shortcomings of traditional data-driven methods, such as over-reliance on data and limited generalization ability, while ensuring that the reconstruction results conform to the laws of scattering optics. This allows the trained network to maintain strong adaptability when facing unknown or changing scattering environments.

[0040] This invention employs the lightweight network ST-StarNet, modifying the structure of the general StarNet to transform it from a hierarchical discriminative backbone oriented towards classification into an encoder-decoder reconstruction network oriented towards the inverse scattering problem. The StarBlock within the network constructs high-dimensional implicit nonlinear features through element-wise multiplication, captures long-range speckle correlations through large-kernel depthwise convolutions, and reduces computational cost while preserving spatial details through depthwise separable convolutions, residual connections, and skip connections. While maintaining high reconstruction fidelity, it reduces the number of model parameters and computational cost by approximately 90%, thus overcoming the high computational cost problem of existing physical embedded methods and facilitating deployment on resource-constrained platforms such as mobile or embedded systems.

[0041] This invention introduces a spatiotemporal prior training strategy to fully exploit the temporal redundancy information between adjacent frames in a video sequence. The network does not need to reconstruct each frame from scratch; instead, it learns only the subtle changes between frames using the prior information from previous frames, thus significantly shortening the iteration time and improving frame processing speed. Even in low signal-to-noise ratio environments, the network prediction can simultaneously satisfy both physical and temporal consistency constraints, achieving high-precision dynamic imaging under dynamic scattering conditions.

[0042] The core innovation of this invention lies in its effective combination of the advantages of spatiotemporal priors and lightweight networks. It utilizes both the prior knowledge and interpretability of physical models and the efficient feature extraction and generalization capabilities of lightweight networks. In dynamic and complex scattering scenarios such as polarization defogging, underwater turbidity imaging, and endoscopic biological tissue imaging, it can achieve dynamic imaging that balances high fidelity and high frame rate, providing a new approach for the application of physics-driven deep learning technology in the field of optical imaging.

[0043] By employing a triple synergy of physical model constraints, lightweight networks, and spatiotemporal prior strategies, this triangular dilemma—the impossible triangle of "data dependence—computational cost—dynamic real-time performance"—is broken. Using a scattering physics model as a forward constraint, the optimization objective is transformed from "matching manually labeled data" to "the physical consistency error between real and computed speckle patterns," thus achieving self-supervised training. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only 6 of the drawings in the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the physical model of imaging through a scattering medium according to the present invention; Figure 2This is a schematic diagram of the processing part of the dynamic scattering imaging method of the present invention, wherein (a) represents a schematic diagram of a model-driven network based on spatial priors, and (b) represents a schematic diagram of a model-driven network based on spatiotemporal priors. Figure 3 This is a schematic diagram of the lightweight network ST-StarNet of the present invention, where (a) represents the overall network architecture; (b) represents the spatiotemporal prior learning strategy; (c) represents the StarBlock module structure; (d) represents the depthwise separable convolutional structure; and (e) represents the composition of each module of the network. Figure 4 This is a schematic diagram of the reconstruction effect of speckle image under dynamic scattering conditions according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the reconstruction effect of a dynamic video sequence according to an embodiment of the present invention; Figure 6 This is a schematic diagram of an ablation experiment according to an embodiment of the present invention. Detailed Implementation

[0046] To enhance understanding of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. These embodiments are only used to explain the invention and do not limit the scope of protection of the invention.

[0047] Example 1: A Dynamic Scattering Imaging Method Based on Lightweight Networks and Spatiotemporal Priors I. System Setup and Experimental Environment Configuration This embodiment uses a He-Ne laser source with a wavelength of 632.8 nm as the illumination source to build an experimental system for imaging through a dynamic scattering medium. The experimental system includes: a laser source, a beam expander and collimator system, a target object (resolution target or actual scene), a dynamic scattering medium simulation device (such as a rotating frosted glass or turbulence simulation cell), a CCD / CMOS detector (camera), and a data processing computer.

[0048] The system optical path parameters are set as follows: distance between the target plane and the scattering medium plane. The distance between the scattering medium plane and the detector plane The camera's pixel resolution is... The pixel size is matched with the optical magnification to ensure complete acquisition of the resulting speckle image.

[0049] II. Step S1: Constructing a physical model for imaging through a dynamic scattering medium Stage decomposition of the S1.1 scattering process The complete propagation process of light through a dynamic scattering medium can be broken down into three independent and continuous physical stages: Phase 1: Incident light field carrying target spatial information It propagates in free space and reaches the plane of the dynamic scattering medium. During this stage, the propagation of the light field obeys the theory of free-space angular spectrum diffraction.

[0050] The second stage: The incident light field passes through a dynamic scattering medium with random phase and amplitude modulation. The scattering medium produces complex random modulations on the light field, including amplitude absorption and phase perturbation.

[0051] The third stage: The scattered light field, modulated by the scattering medium, continues to propagate in free space to the detection surface, where the detector acquires the speckle intensity image. .

[0052] S1.2 Angular Diffraction Theory and Speckle Image Acquisition The propagation of a light field in free space strictly obeys the theory of angular spectrum diffraction. The light field travels from a plane... propagation to the plane The angular spectral transfer function is expressed in the frequency domain as: in, The distance light travels in free space. The wavelength of light These are the spatial frequency coordinates in the Fourier domain. The angular spectral transfer function in the spatial domain. Through the It is obtained by performing a two-dimensional inverse Fourier transform.

[0053] Speckle images are acquired by a camera on the detection surface. In this embodiment, by adjusting the light source power and camera exposure time, the speckle image is ensured to have a sufficient signal-to-noise ratio while avoiding detector saturation.

[0054] S1.3 Establishment of Forward Encoding Relationship The relationship between the speckle image and the target image is considered as a process of forward encoding target information into the speckle intensity map. A forward scattering relationship satisfying angular spectral diffraction is established between the scattered light field and the target light field, and its mathematical expression is: in: and These represent the light field information before and after scattering (target light field and observed speckle). , These represent the distance from the target to the dynamic scattering medium (range). ) and dynamic scattering medium to detector (distance) ) angular spectrum transfer function; This represents the convolution operation; is the amplitude transmittance function of the scattering medium, used to describe the amplitude modulation and phase modulation of the incident light by the scattering medium.

[0055] The amplitude transmittance function of the scattering medium is specifically expressed as: in: This is the amplitude modulation function, used to simulate the absorption of the light field amplitude by the scattering medium; This is a phase modulation function used to simulate the perturbation of the optical field phase by the scattering medium. By adjusting... and By determining the random distribution range (such as uniform distribution, Gaussian distribution) and statistical parameters (mean, variance), dynamic scattering media of different complexities can be simulated.

[0056] In this embodiment, by rotating the frosted glass or controlling the turbulence intensity, It changes slowly over time, creating dynamic scattering conditions.

[0057] III. Step S2: Deep learning processing based on lightweight networks and spatiotemporal priors Construction of the S2.1 Lightweight Network ST-StarNet A lightweight network, ST-StarNet (Spatial-TemporalStarNet), is constructed. This network adopts a highly symmetric encoder-decoder structure and is specifically designed for the inverse problem of scattering imaging.

[0058] The overall network architecture is as follows (see Figure 3 (a) Input layer: Receive resolution is Single-channel speckle intensity image

[0059] Stem module: First, the Stem module performs initial feature extraction and channel adjustment, mapping the input speckle image to a high-dimensional feature space; Encoder stage: This is followed by an encoder consisting of five consecutive Stage layers, progressively reducing the feature map resolution from... downsampling to In each stage, multiple StarBlocks are stacked for deep feature extraction, and the number of channels increases accordingly. Decoder stage: Image resolution is restored step by step through symmetrical upsampling modules (transposed convolution or interpolation + convolution); Skip connections: Establish cross-scale skip connections between the encoder and decoder to fuse multi-scale features of the corresponding scale in the encoding stage with the decoding layer in order to compensate for the loss of detail caused by the scattering process; Output layer: Outputs the reconstructed target image corresponding to the spatial dimensions of the input speckle image.

[0060] Key differences from the general-purpose StarNet: The general-purpose StarNet is typically used as the backbone network for image classification, employing a structure of Stem layers, multi-stage hierarchical downsampling, repeated StarBlocks, global pooling, and a classification head. This embodiment of ST-StarNet incorporates the following targeted modifications: 1. Remove global pooling and classification head: This transforms the network from a classification-discrimination structure to a pixel-by-pixel image reconstruction structure; 2. Add a symmetric decoder: restore spatial resolution through an upsampling module; 3. Add cross-scale skip connections: fuse shallow texture details and deep semantic features; 4. Network relocation: StarBlock is simultaneously deployed in the encoding path, decoding path, and skip fusion path.

[0061] Detailed structure of the StarBlock module (see...) Figure 3 (c) The StarBlock module mainly consists of large-kernel depthwise convolutions, star operations, gating mechanisms, residual connections, and depthwise separable convolutional refinement paths. For input features... The processing flow is as follows: 1. Large kernel depthwise convolution expands the receptive field: in express Depth-wise convolution with large kernels is used to expand the receptive field and capture the long-range spatial correlation of speckle texture.

[0062] 2. Channel expansion and branching: in This represents point-wise convolution, which expands the number of channels and divides the data into two branches based on the number of channels. and .

[0063] 3. Star operation (non-linear eigenmap): in This represents the ReLU6 nonlinear activation function (gating function). This represents element-wise multiplication (Hadamard product), also known as star operation.

[0064] The mathematical principle of the star schema operation: This operation is not a simple feature concatenation or addition. For a feature vector at a single location, if the two branches are... , Then the star output is For the first The output channels can be expanded as follows: Therefore, element-wise multiplication introduces a second-order characteristic term without significantly increasing the explicit channel width. Stacking multiple StarBlocks can further form higher-order implicit interactions, enabling the network to express complex nonlinear mappings in dynamic scattering imaging at a lower computational cost.

[0065] 4. Channel compression and depthwise separable convolution thinning: in This indicates channel compression projection. This represents depthwise separable convolution (composed of depthwise convolution and pointwise convolution). Output features. Residual connections. It helps alleviate gradient vanishing and promotes deep network training.

[0066] Depthwise separable convolutional structures (see...) Figure 3 (d) Depthwise separable convolution decomposes standard convolution into two steps: 1) Depth-wise Convolution: Spatial convolution is performed on each input channel separately, without mixing channel information; 2) Pointwise Convolution: Using Convolutional mixing channel information.

[0067] This decomposition significantly reduces the number of parameters and computational cost. For an input channel number of... The number of output channels is The kernel size is Situation: Standard convolution parameter count:

[0068] Number of parameters in depthwise separable convolution:

[0069] when and When the value is large, the parameter reduction ratio is close to .

[0070] S2.2 Forward propagation of speckle image The acquired speckle image Input randomly initialized decoder After one forward propagation, a preliminary target reconstruction image estimate is obtained: Forward calculation of the S2.3 scattering model Using the established physical scattering model (i.e., the forward scattering relationship in step S1), to reconstruct the image of the predicted target. Perform forward calculations to obtain the corresponding calculated speckle intensity distribution: S2.4 Error Calculation and Loss Function Calculate the input speckle image The calculated speckle intensity distribution The error between the two values ​​is used as the network's loss function. This embodiment uses mean squared error (MSE) as the loss function: in: It is defined as the L2 norm between the measured value and the scattering model prediction; Forward scattering model; Represents the pre-trained decoder (or the current decoder state); For the first The decoder has been optimized and updated. For the first Frame-by-frame true speckle image; The physical meaning of this loss function is to search for the optimal parameters in the decoder parameter space that minimize the forward reprojection error. The optimization objective is not to match the manually labeled target image, but to make the result of the network output after forward propagation through the physical model consistent with the actual observed speckle pattern, that is, to satisfy the physical consistency constraint.

[0071] S2.5 Backpropagation and Weight Update Through the backpropagation algorithm, based on the loss function Network weights Calculate the gradient and update the network weights to reduce the error: in The learning rate is used. This embodiment employs the AdamW optimizer, with an initial learning rate of... The weight decay coefficient is .

[0072] S2.6 Iterative optimization until convergence Repeat steps S2.2 to S2.5, iteratively performing forward propagation, physical model calculation, error assessment, and backpropagation update until the loss function is updated. The preset threshold or the maximum number of iterations has been reached.

[0073] In this embodiment, a preset threshold is used. The maximum number of iterations per frame is 300, and the batch size is 1.

[0074] S2.7-S2.8 Spatiotemporal Prior Training Strategy Furthermore, a spatiotemporal prior training strategy is introduced, utilizing the high temporal correlation between adjacent frames in the video sequence (see...). Figure 3 (b) and Figure 2 (b) S2.7 Spatial Prior Transfer: Spatial prior information learned from the previous frame of the video sequence is used as the prior for reconstruction of the current frame, establishing spatiotemporal relationships between adjacent frames. This allows the network to focus on learning subtle changes between frames without having to reconstruct the entire imaging process from scratch.

[0075] S2.8 Adaptive Weight Update: Setting a Threshold Calculate the metric value of the current frame. .when If this happens, it indicates that the current decoder weights do not match the scattering state of the current frame, and the decoder weights need to be updated. ;when At this time, the decoder weights remain unchanged, allowing subsequent frames to be reconstructed using the current decoder. Its mathematical expression is: in For the first The second-optimized decoder This is a preset threshold.

[0076] The technical principle of the spatiotemporal prior strategy is as follows: Since the first frame of a video sequence lacks reference information from previous frames, its iterative optimization usually requires many steps to converge. However, subsequent frames, being highly correlated with the previous frame (the changes in the target and scattering medium are usually gradual), only require a few iterations to converge. By adaptively using or updating the decoder weights, the processing time is significantly reduced while ensuring reconstruction accuracy, achieving high frame rate dynamic reconstruction.

[0077] IV. Step S3: Output the reconstructed image of the target behind the dynamic scattering medium. S3.1 Frame-by-Frame Reconstruction Using the optimized lightweight network ST-StarNet, the speckle video sequence acquired behind the dynamic scattering medium is reconstructed frame by frame to obtain the target reconstruction image of each frame.

[0078] S3.2 Adaptive Weight Update Based on the spatiotemporal prior training strategy, the decoder weights are adaptively reused or updated for adjacent frames. The specific process is as follows: 1. For the speckle image of the current frame, first use the current decoder to perform forward inference to obtain a preliminary reconstruction; 2. The reconstructed results are used to perform forward calculations using a scattering model to obtain simulated speckle patterns; 3. Calculate the error between the actual speckle pattern and the actual speckle pattern. ; 4. If The current reconstruction result is output directly, and the decoder weights remain unchanged; 5. If This triggers iterative optimization, updates the decoder weights, and then outputs the reconstruction result.

[0079] S3.3 Performance Evaluation and Application Verification Output the reconstructed image of the target after dynamic scattering medium, and evaluate the performance of the reconstruction results. Evaluation metrics include: Structural Similarity (SSIM): Measures the structural similarity between a reconstructed image and a reference image; Peak signal-to-noise ratio (PSNR): measures the pixel-level fidelity of a reconstructed image; Root mean square error (RMSE): measures the magnitude of reconstruction error.

[0080] V. Experimental Results and Comparative Verification UNet STeP 28.95M 209.17G 28.92 16.24s 0.85±0.10 23.09±5.94dB 0.09±0.06 EfficientNet STeP 4.03M 3.17G 51.56 11.80s 0.53±0.15 15.71±3.35dB 0.18±0.07 MobileNetV4 STeP 0.88M 1.52G 104.66 3.22s 0.79±0.10 21.44±5.14dB 0.10±0.06 P-StarNet Physics-Driven 2.06M 13.15G 16.98 18.54s 0.51±0.20 18.45±4.58dB 0.14±0.08 ST-StarNet (ours) STeP 2.06M 13.15G 82.05 5.94s 0.90±0.08 26.74±6.23dB 0.06±0.05 Results analysis: 1. Compared with UNet: ST-StarNet maintains higher reconstruction quality (SSIM 0.90 vs 0.85, PSNR 26.74dB vs 23.09dB) while reducing the number of parameters from 28.95M to 2.06M (a reduction of about 93%), FLOPs from 209.17G to 13.15G (a reduction of about 94%), and single frame processing time from 16.24s to 5.94s (an improvement of about 63%).

[0081] 2. Compared with MobileNetV4: Although MobileNetV4 has fewer parameters and fewer FLOPs, its reconstruction quality is significantly worse than ST-StarNet (SSIM 0.79 vs 0.90), indicating that simply pursuing lightweight design while ignoring task characteristics leads to performance degradation. ST-StarNet achieves the optimal accuracy-efficiency tradeoff with a moderate number of parameters through star-shaped operations and physical embedding.

[0082] 3. Compared with P-StarNet: The purely physics-driven P-StarNet requires no training data, but its reconstruction quality is poor (SSIM only 0.51) and its processing speed is slow (18.54s / frame). ST-StarNet, by introducing a spatiotemporal prior training strategy, significantly improves both reconstruction accuracy and frame processing speed while maintaining the same network size.

[0083] 3. Overall advantages: ST-StarNet outperforms all comparison methods in the three image quality metrics of SSIM, PSNR, and RMSE, while maintaining a high processing speed (82.05 FPS), verifying the effectiveness of this embodiment in balancing high fidelity and high frame rate in dynamic scattering imaging tasks.

[0084] VI. Ablation Experiment Verification To further verify the contribution of each technical component, ablation experiments were conducted (see...). Figure 6 ): 1. Removing physical forward constraints (purely data-driven): requires a large amount of paired training data, and the generalization ability is significantly reduced; 2. Removing star operations (replacing them with simple addition): Reduced non-linear expressive power, loss of reconstruction details, and a decrease in SSIM of approximately 0.08; 3. Remove large kernel depthwise convolutions (reuse standard kernels) Convolution: Insufficient capture of long-range speckle correlation, resulting in a PSNR decrease of approximately 2 dB; 4. Removal of spatiotemporal prior strategy (frame-by-frame independent optimization): Processing time increased by approximately 4 times, and inter-frame flickering appeared in the video sequence; 5. Remove depthwise separable convolutions (replace with standard convolutions): The number of parameters increases by about 8 times, and FLOPs increase by about 6 times.

[0085] The ablation experiment results show that each technical component in this embodiment (physical forward constraint, star operation, large kernel depthwise convolution, spatiotemporal prior strategy, and depthwise separable convolution) makes a significant contribution to the final performance and none of them can be omitted.

[0086] VII. Application Scenarios The method in this embodiment can be applied to the following scenarios: Polarization defogging: Imaging through atmospheric scattering media such as fog and smoke; Underwater turbidity imaging: Target detection and identification through turbid water; Endoscopic biological tissue imaging: non-invasive or minimally invasive internal imaging through biological tissues; Dynamic environmental monitoring: Real-time monitoring of target changes through dynamic scattering media.

[0087] Example 2: Dynamic Scattering Imaging Device Based on Lightweight Networks and Spatiotemporal Priors Based on the method of Embodiment 1, this embodiment provides a corresponding imaging device, including the following functional modules: 1. Physical Model Building Module This module is used to establish the forward scattering relationship of incident light carrying target information, which is randomly modulated by a dynamic scattering medium and propagates in free space to the detector surface. It internally stores the angular spectral transfer function. , and the transmissivity model of scattering media A speckle intensity distribution corresponding to the target is generated on the detection surface.

[0088] 2. Data Acquisition Module The data interface is connected to the detection surface (camera) corresponding to the physical model building module to collect the speckle intensity distribution on the detection surface. The collected data is then transmitted to the subsequent processing module in digital image format.

[0089] 3. Lightweight network processing module The module receives the speckle intensity distribution from the data acquisition module as input and outputs a reconstructed target image through the ST-StarNet lightweight network. This module is internally configured with a backpropagation mechanism, which calculates the target image based on the input speckle image. The network weights are adjusted to reduce the error between the discrepancy and the speckle intensity distribution calculated by the scattering model.

[0090] The network architecture employs an encoder-decoder network that incorporates depthwise separable convolutions, gating mechanisms, residual connections, and skip connections to extract speckle images. The multi-scale features in the image are used to output the target reconstructed image.

[0091] 4. Scattering Model Calculation Module Connected to the lightweight network processing module, the target reconstruction image output by the network is processed through a scattering model. Perform forward calculations to obtain the corresponding calculated speckle intensity distribution. .

[0092] 5. Error Measurement and Feedback Module Used to measure speckle images acquired by the data acquisition module The speckle intensity distribution calculated by the scattering model calculation module Error between This module uses mean squared error (MSE), structural similarity loss (SSIM), or other loss functions suitable for measuring differences in speckle intensity distribution to calculate the error.

[0093] The error signal is fed back to the lightweight network processing module for weight updates until the error meets the preset threshold. This allows for network optimization.

[0094] 6. Spatiotemporal Prior Processing Module Connected to a lightweight network processing module, this module uses temporal redundancy information between adjacent frames in a video sequence as a priori information to guide the reconstruction of the current frame. This module is based on a preset threshold. Determine whether to update the decoder weights: Calculate the current frame metric. : like Keep the decoder weights unchanged, and use the current decoder in subsequent frames; like This triggers a decoder weight update.

[0095] This satisfies both physical and temporal consistency constraints, enabling high frame rate dynamic reconstruction.

[0096] Device working process: 1. The physical model building module generates a forward scattering model based on system parameters; 2. The data acquisition module acquires dynamic speckle video sequences in real time; 3. The first frame speckle input is processed by a lightweight network module, which is iteratively optimized until convergence. 4. Subsequent frames are adaptively reconstructed using the spatiotemporal prior processing module; 5. The scattering model calculation module and the error measurement feedback module form a closed-loop optimization; 6. The final output is a high-fidelity, time-continuous dynamic reconstructed video.

[0097] The entire device requires only a small amount of data to complete network training under the constraints of the physical model, without relying on a large amount of paired data. While significantly reducing the number of parameters and computation (by about 90%), it effectively improves the accuracy and generalization ability of imaging under dynamic scattering environments.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A dynamic scattering imaging method based on lightweight networks and spatiotemporal priors, characterized in that, Includes the following steps: S1, Construct a physical model for imaging through a dynamic scattering medium; S2, based on lightweight networks and spatiotemporal priors for deep learning processing; S3 outputs a reconstructed image of the target behind a dynamically scattering medium.

2. The dynamic scattering imaging method based on lightweight networks and spatiotemporal priors according to claim 1, characterized in that, Step S1 includes: S1.1 decomposes the light scattering process into three independent stages: the incident light carrying target information arrives at the dynamic scattering medium plane, the light passes through the dynamic scattering medium with random phase and amplitude modulation, and the modulated scattered light field propagates to the detection surface in free space. S1.2, the propagation of the light field in free space follows the angular spectrum diffraction theory, and the speckle image I is obtained by the detector on the detection surface; S1.3, the relationship between the speckle image and the target image is regarded as the process of forward encoding the target information into the speckle intensity map.

3. A dynamic scattering imaging method based on lightweight networks and spatiotemporal priors according to claim 1 or 2, characterized in that, Step S1 establishes a forward scattering relationship between the scattered light field and the target light field that satisfies angular spectrum diffraction, and its expression is: in, and These represent the light field information before and after scattering, respectively. , Let represent the angular spectral transfer functions of the distance from the target to the dynamic scattering medium and the distance from the dynamic scattering medium to the detector, respectively. This represents the convolution operation. is the amplitude transmittance function, used to describe the amplitude and phase modulation of incident light by the scattering medium.

4. The dynamic scattering imaging method based on lightweight networks and spatiotemporal priors according to claim 1, characterized in that, Step S2 includes: S2.1 Construct a lightweight network ST-StarNet. The network adopts an encoder-decoder structure and includes a Stem module, several Stage modules, a StarBlock module, an upsampling module, and skip connections in sequence. Its input layer receives the speckle image I collected by the detector, and its output layer outputs the predicted target reconstruction image. S2.2, The acquired speckle image I is input into the lightweight network for forward propagation to obtain the predicted target reconstruction image; S2.3, using a known scattering model, perform forward calculations on the predicted target reconstruction image to obtain the corresponding calculated speckle intensity distribution; S2.4, calculate the error between the input speckle image I and the calculated speckle intensity distribution, and use it as the loss function of the network; S2.5, The weights of the network are updated according to the loss function through the backpropagation algorithm to reduce the error; S2.6 Repeat steps S2.2 to S2.5 until the loss function reaches the preset threshold or the number of iterations, thus completing the network optimization.

5. The dynamic scattering imaging method based on lightweight networks and spatiotemporal priors according to claim 4, characterized in that, In step S2, the error between the input speckle image I and the speckle intensity distribution calculated by the scattering model is measured, and its expression is: in, Defined as the L2 norm between the measured value and the scattering model prediction. For scattering models, Represents the pre-trained decoder. For the first The decoder has been optimized and updated. For the first Frame-by-frame true speckle image.

6. A dynamic scattering imaging method based on lightweight networks and spatiotemporal priors according to claim 4 or 5, characterized in that, The StarBlock module differs from the four-stage downsampling star module used for image classification in the general StarNet. It is embedded in the multi-scale feature extraction path and decoding refinement path of the encoder-decoder reconstruction network, and does not have a global pooling layer or a classification head. For input features The StarBlock includes a gating mechanism and depthwise separable convolution. First, a large-kernel depthwise convolution expands the receptive field, then a multi-branch star-shaped operation performs high-dimensional nonlinear feature mapping. The star-shaped operation is an element-wise multiplication operation. Finally, residual connections and depthwise separable convolution refine the path and optimize the feature representation, satisfying the following: in express Large kernel depthwise convolution, and These represent channel expansion and channel compression projection, respectively. This represents a nonlinear activation or gate function. This indicates element-wise multiplication. This represents depthwise separable convolution. This indicates the output features.

7. The dynamic scattering imaging method based on lightweight networks and spatiotemporal priors according to claim 1, characterized in that, Step S2 further introduces a spatiotemporal prior training strategy, including: S2.7, use the spatial prior information learned from the previous frame of the video sequence as the prior for the reconstruction of the current frame to establish the spatiotemporal relationship between adjacent frames; S2.8, Set threshold Calculate the metric value of the current frame. ,when Greater than Update decoder weights in real time, when Not greater than To maintain the decoder weights unchanged and allow subsequent frames to be reconstructed using the current decoder, the expression is: in, For the first The optimized decoder This is a preset threshold.

8. The dynamic scattering imaging method based on lightweight networks and spatiotemporal priors according to claim 1, characterized in that, Step S3 includes: S3.1, using the optimized lightweight network, the speckle video sequence acquired behind the dynamic scattering medium is reconstructed frame by frame to obtain the target reconstruction image of each frame; S3.2, based on the spatiotemporal prior training strategy, adaptively reuses or updates the decoder weights for adjacent frames to achieve dynamic reconstruction at a high frame rate; S3.3 outputs the reconstructed image of the target behind the dynamic scattering medium, and evaluates and verifies the performance of the reconstruction results.

9. A dynamic scattering imaging device based on lightweight networks and spatiotemporal priors, characterized in that, include: The physical model building module is used to establish the forward scattering relationship of incident light carrying target information, which is randomly modulated by a dynamic scattering medium and propagates to the detector surface in free space, thereby generating a speckle intensity distribution on the detector surface. A data acquisition module is used to connect to the detection surface of the physical model construction module to acquire the speckle intensity distribution; A lightweight network processing module is used to receive the speckle intensity distribution output by the data acquisition module as input, and output the target reconstructed image through a lightweight network; the lightweight network processing module is configured with a backpropagation mechanism, which adjusts the network weights to reduce the error between the input speckle image I and the speckle intensity distribution calculated by the scattering model; A scattering model calculation module is used to connect to the lightweight network processing module and calculate the corresponding speckle intensity distribution from the target reconstruction image output by the lightweight network processing module through a scattering model. The error measurement and feedback module is used to measure the error between the speckle image I acquired by the data acquisition module and the speckle intensity distribution calculated by the scattering model calculation module, and feeds the error signal back to the lightweight network processing module for weight update until the error meets the preset threshold, thereby completing the network optimization.

10. A dynamic scattering imaging device based on lightweight networks and spatiotemporal priors according to claim 9, characterized in that, The lightweight network processing module employs an encoder-decoder network to extract features from the speckle image I and output the target reconstructed image; the encoder-decoder network includes depthwise separable convolution, gating mechanism, residual connection and skip connection; The error measurement and feedback module uses mean squared error loss, structural similarity loss, or other loss functions suitable for measuring the differences in speckle intensity distribution to calculate the error; It also includes a spatiotemporal prior processing module, which is connected to the lightweight network processing module. This module is used to use the temporal redundancy information between adjacent frames in the video sequence as a priori to guide the reconstruction of the current frame, and to determine whether to update the decoder weights based on a preset threshold, so as to simultaneously satisfy physical constraints and temporal consistency constraints.

Images