A method and device for multi-mode fiber speckle image reconstruction based on polarization token attention

Abstract

The application discloses a kind of based on polarization token attention multi-mode optical fiber speckle image reconstruction method and device;The method comprises: multi-polarization speckle acquisition, shared encoder feature extraction, polarization token construction, cross-polarization token attention fusion, spatial pixel-by-pixel attention fusion, decoding reconstruction.The method compresses the feature of multi-polarization speckle into polarization token sequence and superimposes learnable polarization identity code, generates channel-level fusion weight by modeling the global complementary relationship between each polarization channel through multi-head self-attention, and further generates spatial pixel-by-pixel fusion weight in the decoder layer, realizing two-level adaptive fusion from global channel to local space;Multi-polarization speckle shares the same set of encoder weights, and each polarization channel is independently processed for the first layer convolution to suppress the uniform degradation of attention weight.The reconstruction performance of the present application is still better than the multi-network independent weighting scheme with several times the parameter quantity of the present application under the condition of significantly reducing the parameter quantity.

Description

Technical Field

[0001] This invention relates to the fields of fiber optic imaging and computational imaging technology, and more specifically, to a method and apparatus for multimode fiber speckle image reconstruction based on polarization token attention. Background Technology

[0002] Multimode fiber, capable of supporting hundreds to thousands of propagation modes simultaneously within a single fiber core, holds significant promise for applications in ultra-fine endoscopic imaging, neuroscience detection, and fiber optic sensing. When an optical field couples into a multimode fiber, the propagation modes coherently superimpose at the output due to differences in propagation constants, degrading the original image into a visually unrelated random speckle image. Recovering the original image from the speckle image is essentially a highly ill-posed inverse problem; in recent years, end-to-end speckle reconstruction methods based on deep neural networks have become the mainstream technological approach.

[0003] When a light field propagates in a multimode fiber, the speckle patterns formed at the output end due to the birefringence effect and the difference in mode coupling paths corresponding to different polarization states exhibit a complementary statistical distribution. The speckles corresponding to different polarization analysis directions carry complementary projection information of the original image under different polarization substrates. Therefore, theoretically, jointly utilizing multi-path polarization speckles can overcome the information bottleneck of single-path speckle observation and improve reconstruction fidelity. However, the effective fusion of multi-path polarization speckles faces the following technical challenges.

[0004] First, the contributions of each polarization channel to the reconstruction quality are not equal, and each channel contains both complementary and redundant information. Existing techniques that directly superimpose multiple polarization speckle paths at the input before feeding them into a single network result in irreversible feature mixing at the network's first layer. The network lacks a mechanism to distinguish the differences in contributions from each channel, and these differential statistical characteristics actually act as interference, reducing reconstruction performance.

[0005] Secondly, another approach in the existing technology is to use independent complete reconstruction networks for each polarization speckle path, and then weight and combine the reconstruction results of multiple paths. In this approach, each channel is always isolated from each other during the feature extraction stage, and the deep complementary information across polarization channels cannot be mined in the feature space. Moreover, regardless of whether fixed weights or scalar weights output by gating networks are used, the fusion occurs at the output end, which is essentially still a linear weighted combination of the output results, and fails to model the complementary relationship between channels in the feature space. At the same time, multiple independent networks lead to a several-fold increase in the number of parameters, resulting in large computational and storage overhead.

[0006] Third, the effective information contribution of multi-polarized speckle to different spatial regions of the same input image varies with spatial location. For example, the dominant polarization components suitable for high-frequency edge regions and uniform low-frequency regions of an image may not be the same. The existing output scalar weighting method applies the same weight to the entire feature map, which cannot achieve spatial local adaptive fusion.

[0007] In summary, there is currently a lack of a multimode fiber multipolar speckle fusion reconstruction method that can adaptively sense the differences in contributions of each polarization channel in the feature space, deeply mine cross-polarization complementary information, and take into account parameter efficiency. Summary of the Invention

[0008] The purpose of this invention is to provide a method and apparatus for multimode fiber speckle image reconstruction based on polarization token attention, so as to solve the technical problems in the prior art where the contribution of each channel cannot be adaptively sensed during multi-path polarization speckle fusion, the cross-polarization complementary information cannot be deeply mined in the feature space, and the number of parameters in multi-network schemes is too large.

[0009] To achieve the above objectives, the present invention adopts the following technical solution.

[0010] In a first aspect, the present invention provides a method for reconstructing multimode fiber speckle images based on polarization token attention, comprising the following steps:

[0011] Step S1: Multi-polarization optical signal acquisition. A laser outputs monochromatic coherent light, which is incident on a spatial light modulator. After the spatial light modulator modulates the original input image, the modulated light field is coupled into a multimode fiber. At the output end of the multimode fiber, N polarization components are obtained through a multi-polarization beam splitter. N polarized speckle images are acquired by N image sensors respectively.

[0012] Step S2: Polarization-Specific First-Layer Convolution and Shared Encoding Backbone Feature Extraction. The N polarization speckle images are input into the same encoder with shared weights. Each polarization speckle image is independently forward-propagated to obtain multi-level feature maps and bottleneck layer features corresponding to each polarization speckle image. The first-layer convolution of the encoder uses independent convolution weights for each polarization channel of the N polarization components. Subsequent layers after the first convolution share convolution weights among the N polarization components. This shared weight design places the features of different polarization components in the same representation space, providing a semantic alignment basis for subsequent attention comparisons between cross-polarization tokens, while significantly reducing the number of parameters. The independent design of each polarization channel in the first-layer convolution allows for the extraction of polarization-specific local texture features for each polarization component at the first layer, amplifying the initial discriminative power of different polarization components at the feature level.

[0013] Step S3: Polarization Token Construction. Global average pooling is performed on the bottleneck layer features corresponding to each polarization speckle image to obtain the global description vector of each polarization component. The global description vectors of each polarization component are mapped to the attention computation space via a learnable linear projection, and a learnable polarization identity encoding vector corresponding to each polarization component is superimposed to obtain a polarization token sequence of length N. The global average pooling compresses the spatial distribution of each polarization speckle into a compact representation of its global statistical characteristics; the linear projection amplifies the discriminative power of different polarization components in the attention computation space; and the learnable polarization identity encoding vector injects a unique identifier into each polarization component, preventing self-attention from degenerating into uniform weighting due to excessively high token similarity.

[0014] Step S4: Cross-polarization token attention fusion. The polarization token sequence is input into the multi-head self-attention module. The polarization token sequence is linearly projected to obtain a query vector, a key vector, and a value vector. The scaled dot product of the query vector and the key vector is calculated using a normalized exponential function to obtain an N×N attention weight matrix. The multi-head self-attention module performs the above calculation in parallel in multiple independent feature subspaces. The attention weight matrices of each feature subspace are concatenated and linearly projected to obtain the final attention weight matrix. This invention does not use the attention weight matrix for attention value transformation. Instead, it takes the mean along one dimension and normalizes it using a normalized exponential function to obtain the channel-level fusion weights corresponding to each of the N polarization components. The bottleneck layer features of the N polarization components (not the value tensor after the value vector transformation) are weighted and summed using the channel-level fusion weights to obtain the fused bottleneck layer features.

[0015] Step S5: Spatial pixel-wise attention fusion. In the middle layer of the decoder, the feature maps of the N polarization components in the middle layer of the encoder are transmitted via the shared coding backbone through skip connections; the N feature maps are processed by convolution and normalized exponential function along the polarization channel dimension to obtain spatial pixel-wise fusion weight maps of the N polarization components; the spatial pixel-wise fusion weight maps of each path are multiplied element-wise with the middle layer feature maps of the corresponding polarization components, and the N results are summed to obtain the fused middle layer feature map.

[0016] Step S6: Decoding and Reconstruction. The fused bottleneck layer features and the fused middle layer feature map are input into the decoder, and the reconstructed image is output after processing by the decoder.

[0017] In this invention, in step S1, the multi-path polarization beam splitter is used to generate polarization components in four linear polarization directions: 0°, 45°, 90°, and 135°, or to generate polarization components in the above four linear polarization directions and a total intensity channel without polarization analysis; the multi-path polarization beam splitter includes a combination of a non-polarizing beam splitter prism, a polarizing beam splitter prism, and a polarizer.

[0018] In this invention, in step S1, the spatial light modulator is a digital micromirror device (DMD), a reflective liquid crystal on silicon spatial light modulator (LCoS-SLM), a transmissive liquid crystal spatial light modulator (LC-SLM), a deformable mirror array, or other two-dimensional optical devices capable of spatially distributing amplitude or phase modulation of the incident light field; the multimode fiber can be a step-index multimode fiber or a graded-index multimode fiber, or can be replaced with a few-mode fiber or a multi-core fiber.

[0019] In this invention, in step S2, the backbone of the encoder with shared weights is the encoder part of a deep network architecture such as U-Net Convolutional Neural Network, standard U-Net network, ResNet (Residual Neural Network), or DenseNet (Densely Connected Network); the decoder and the encoder form an encoder-decoder structure, and skip connections are set between corresponding layers of the encoder and the decoder.

[0020] In this invention, in step S4, the channel-level fusion weight is further used in step S6 to perform cross-polarization weighted fusion of the high-level features transmitted by the skip connection, so that the complementary information of each polarization component can be utilized at multiple levels of the decoding path.

[0021] Secondly, the present invention provides a multimode fiber speckle image reconstruction apparatus based on polarization token attention, which is used to implement the above-mentioned multimode fiber speckle image reconstruction method, including:

[0022] Multipolar optical transmission and acquisition module: including laser, spatial light modulator, multimode fiber, multi-path polarization beam splitter and N image sensors, used to spatially modulate the original input image and inject it into the multimode fiber, and acquire N polarization speckle images at the output end of the multimode fiber;

[0023] Reconstruction network system: used to carry the deep neural network to perform end-to-end reconstruction of the N-channel polarization speckle images, the reconstruction network system includes the following sub-modules:

[0024] The shared weight encoder submodule is used to independently propagate the N polarization speckle images forward and extract the multi-level feature maps and bottleneck layer features corresponding to each polarization component. The first layer of the encoder uses independent convolution weights for each polarization channel for the N polarization components, and subsequent layers share convolution weights among the N polarization components.

[0025] Cross-polarization token attention submodule: Deployed in the bottleneck layer of the encoder, it is used to perform global average pooling, linear projection and superposition of learnable polarization identity codes on the bottleneck layer features of each polarization component to obtain a polarization token sequence, generate channel-level fusion weights through multi-head self-attention calculation, and weighted fuse the N bottleneck layer features to obtain the fused bottleneck layer features.

[0026] Spatial attention fusion submodule: Deployed in the middle layer of the decoder (feature layer with the same resolution as the original input image), it is used to process the middle layer feature maps corresponding to each polarization component by convolution and normalized exponential function along the polarization channel dimension to obtain the spatial pixel-wise fusion weight map of N polarization components, and then sum them by element-wise multiplication with the corresponding middle layer feature map to obtain the fused middle layer feature map.

[0027] Shared weight decoder submodule: contains skip connections, used to upsample the fused bottleneck layer features and the fused middle layer feature maps step by step to output the reconstructed image.

[0028] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the processor executes a computer program stored in the memory to implement the above-described multimode fiber speckle image reconstruction method.

[0029] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described multimode fiber speckle image reconstruction method.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) Adaptive sensing of cross-polarization channel contribution in feature space. This invention serializes multi-path polarization speckle features into polarization tokens. Through multi-head self-attention, the relationship between each polarization token is dynamically calculated by the dot product of the query vector and the key vector. Thus, the weight of the polarization component with richer effective information is adaptively increased according to the statistical characteristics of the current speckle. This overcomes the limitations of direct superposition at the input end, which cannot distinguish channel contribution, and fixed or scalar weighting at the output end, which cannot adaptively sense the contribution.

[0032] (2) Deep mining of cross-polarization complementary information in the feature space. The fusion of the present invention occurs in the deep feature space of the encoder bottleneck layer and the middle layer of the decoder, rather than the weighted result at the output end, so that the potential deep complementary features of each polarization channel can be mined within the network.

[0033] (3) Achieve two-level adaptive fusion from global channels to local space. This invention applies channel-level scalar fusion weights to the bottleneck layer and further applies spatial pixel-by-pixel fusion weights to the middle layer of the decoder, enabling different spatial locations to adaptively select the dominant polarization component, thus achieving spatial local adaptive fusion that cannot be achieved by the scalar weighting method at the output end.

[0034] (4) Suppressing uniform degradation of attention weights. This invention uses independent convolution weights for each polarization channel in the first layer convolution of the encoder, and introduces linear projection and learnable polarization identity encoding in the construction of polarization tokens, which amplifies the initial discriminative power of each polarization token and effectively suppresses the phenomenon of self-attention degrading to uniform weighting under multi-branch input, thereby activating the dynamic fusion capability of each polarization channel.

[0035] (5) Significantly improves parameter efficiency. The N polarization components of the present invention share the same set of encoder weights. Compared with the scheme of configuring independent and complete networks for each polarization component, it significantly reduces the number of parameters and computational overhead, while ensuring that the features of different polarization components are in the same representation space, providing a semantically consistent basis for the comparison of cross-polarization attention.

[0036] (6) Experimental verification. In the preferred embodiment, the method of the present invention was used in the face image reconstruction task of 100 m step-index multimode fiber. Compared with the comparison fusion methods such as single-path polarization reconstruction and input-end superposition and output-end weighting, the Pearson correlation coefficient (PCC) and structural similarity (SSIM) index of the test set were steadily improved, and the number of parameters was significantly reduced compared with the independent network scheme. Attached Figure Description

[0037] Figure 1: Flowchart of the method according to an embodiment of the present invention.

[0038] Figure 2: Schematic diagram of the polarization token attention fusion network structure of the present invention. Detailed Implementation

[0039] The embodiments of the present invention will be further described below with reference to the accompanying drawings. These embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0040] To facilitate understanding of the following descriptions, the main symbols used in this specification and their meanings are summarized in Table 1:

[0041] Table 1 Summary of Symbols and Their Meanings

[0042]

[0043] As shown in Figures 1 and 2, this invention provides a method for reconstructing multimode fiber speckle images based on polarization token attention, comprising the following steps:

[0044] Step S1: Multi-polarization optical signal acquisition. A laser outputs monochromatic coherent light, which, after beam expansion and collimation, is incident on a spatial light modulator. The spatial light modulator is loaded with a modulation pattern corresponding to the original input image. After spatial modulation of the monochromatic coherent light, the modulated light field is coupled into a multimode fiber via an objective lens. At the output end of the multimode fiber, the output light field, after collimation by the objective lens, is split into N paths by a multi-polarization beam splitter. These paths pass through linear polarizers at 0°, 45°, 90°, and 135°, as well as a total intensity channel without polarization analysis. These N paths are then simultaneously acquired by N image sensors to obtain N polarization speckle images. Each image sensor is triggered by hardware to achieve frame-level synchronous acquisition, ensuring that the input image corresponds to each polarization component completely identically.

[0045] Step S2: Polarization-specific first-layer convolution and shared-encoding backbone feature extraction. A deep neural network containing an encoder and a decoder is constructed, where the encoder shares the same set of convolution weights for N polarization speckle images. The N polarization speckle images are independently input into the encoder with shared weights to obtain multi-order feature maps and bottleneck layer features corresponding to each polarization component. In this encoder, the first layer convolution employs independent convolution weights for each of the N polarization channels, enabling the extraction of polarization-specific local texture features for each polarization component at the first layer. Subsequent layers after the first convolution share convolution weights among the N polarization components, placing the features of different polarization components in the same representation space.

[0046] Step S3: Polarization token construction. Bottleneck layer features for each polarization component. Global average pooling is performed to obtain a global description vector, which is then mapped to the attention computation space via a learnable linear projection, and finally superimposed with a learnable polarization identity encoding vector corresponding one-to-one with each polarization component. Obtain the p-th polarization token. , This constitutes a polarization token sequence of length N. The construction process of the polarization tokens is as follows: global average pooling, linear projection, and superimposed polarization identity encoding. Global average pooling compresses the spatial distribution of each polarization speckle into a compact representation of its global statistical characteristics. Linear projection amplifies the distinguishability of different polarization components at the feature level. Polarization identity encoding injects the prior identity information of the polarization components into each polarization token to prevent self-attention from degenerating into uniform weighting due to the high similarity of each token.

[0047] Step S4: Cross-polarization token attention fusion. The polarization token sequence T is normalized and then input into the multi-head self-attention module. T is linearly projected into the query vector Q, key vector K, and value vector V. The attention is calculated using the following formula:

[0048]

[0049] Where d is the dimension of the key vector. The scaling factor is used. The multi-head self-attention module performs the above calculations in parallel in h independent feature subspaces. The outputs of each subspace are concatenated and linearly projected to obtain the attention weight matrix A. The mean of the attention weight matrix A along its last dimension is taken, and then normalized by a normalized exponential function to obtain the channel-level fusion weights corresponding to each of the N polarization components. ,satisfy The bottleneck layer features are weighted and summed using channel-level fusion weights to obtain the fused bottleneck layer features:

[0050]

[0051] Step S5: Spatial pixel-wise attention fusion. In the middle layer of the decoder, the middle layer feature maps corresponding to each polarization component are processed by 1×1 convolution, and a normalized exponential function is applied along the polarization channel dimension to obtain the spatial pixel-wise fusion weight map of N polarization components. p=1,2,…,N; The corresponding mid-layer feature map is multiplied element-wise and then summed to obtain the fused mid-layer feature map. The spatial pixel-by-pixel fusion weight map gives each of the N polarization components an independent fusion weight at different spatial locations, thereby selecting the appropriate dominant polarization component in the high-frequency edge region and the uniform low-frequency region of the image, respectively.

[0052] Step S6: Decoding and Reconstruction. The fused bottleneck layer features are input into the decoder. The channel-level fusion weight α is further used to perform cross-polarization weighted fusion of the high-level features transmitted by the skip connections. The fused mid-layer feature map merges with the decoding path in the middle layer of the decoder. After being upsampled step by step by the decoder, the reconstructed image Y is output.

[0053] Furthermore, the present invention provides an apparatus for implementing the above-described method, comprising: a multi-polarization optical transmission and acquisition module, a shared encoding module, a polarization token construction module, a cross-polarization token attention module, a spatial attention fusion module, and a decoding and reconstruction module; the function of each module corresponds one-to-one with the steps of the above-described method. The present invention also provides an electronic device, comprising a memory and a processor, wherein the processor executes a program stored in the memory to implement the above-described method; and a computer-readable storage medium storing a computer program, which, when executed, implements the steps of the above-described method.

[0054] I. Experimental System and Training Configuration

[0055] The experimental setup of this invention mainly includes: a laser source (OBIS 561 nm); a spatial light modulator (DMD, pulse width modulation grayscale modulation); a step-index multimode fiber (length 100 m, core diameter 200 μm, numerical aperture NA≈0.22); five polarization-splitting optical paths (0°, 45°, 90°, 135° linear polarization channels and one unanalyzed total intensity channel); and five complementary metal-oxide-semiconductor (CMOS) image sensors (resolution 256×256, 20 frames / second), which are hardware-triggered to achieve frame-level synchronous acquisition. After collimation by a lens group, the laser image is modulated into the light field by the DMD, coupled through the objective lens into the multimode fiber, and the output end is simultaneously acquired by the five CMOS sensors via the five polarization-splitting optical paths. The acquired image resolution is 256×256, 8-bit grayscale, and the original input image is a 64×64, 8-bit grayscale face image.

[0056] The reconstruction network used in this invention is a U-Net based on an attention mechanism, with a 5-stage encoder and 48 basic channels. Each polarization component shares encoder weights, while the polarization channels in the first convolutional layer are independent. Training employs the AdamW (Adam with Decoupled Weight Decay) optimizer with an initial learning rate of 1×10⁻⁻. 4The batch size is 4, the training hardware is a single NVIDIA RTX 4090 Graphics Processing Unit (GPU), and the training run is limited to a maximum of 60 epochs. An early stopping strategy based on the Pearson correlation coefficient of the validation set is employed. The training set consists of 2000 samples, the validation set 200 samples, and the test set 200 samples. The above experimental configuration is only for illustrating the implementation effect of this invention and does not constitute a limitation on parameters such as network structure, optimizer type, learning rate, number of training epochs, batch size, hardware platform, dataset size, and number of polarization component paths N.

[0057] II. Reconstructing the Overall Network Structure

[0058] The reconstruction network described in this invention adopts an encoder-decoder architecture, including: (i) a multi-polarization shared weight encoder, used to extract multi-level feature maps and bottleneck layer features from N polarization speckle paths respectively; (ii) a cross-polarization token attention module, deployed in the bottleneck layer of the encoder, to perform channel-level adaptive fusion of N bottleneck features; (iii) a spatial pixel-by-pixel attention fusion module, deployed in the middle layer of the decoder, to achieve pixel-level adaptive fusion; and (iv) a shared weight decoder, containing skip connections, to upsample the fused features step by step and output the reconstructed image.

[0059] The encoder consists of 5 downsampling blocks, with the number of feature map channels in each stage being {48, 96, 192, 384, 768}, and the base number of channels being 48. Each stage contains two 3×3 convolutional blocks, each with batch normalization and ReLU activation. The stages are downsampled by a factor of 2 using 2×2 max pooling.

[0060] The encoder's first-layer convolution uses independent convolution weights for the N polarization components, meaning each polarization component is equipped with a separate 3×3 convolution kernel; subsequent coding layers share the same set of convolution weights across the N polarization components.

[0061] The encoder has a bottleneck layer at its bottom with a spatial resolution of 8×8 and 1536 channels. Following the bottleneck layer is the cross-polarization token attention module, the specific structure and calculation process of which are shown in Figure 2 and the following section "III. Specific Implementation of Polarization Token Attention Fusion".

[0062] The decoder also consists of 5 stages. It performs upsampling by 2x through 2×2 transposed convolution and passes the features of the corresponding stage of the encoder to the decoder through skip connections. The middle layer of the decoder (i.e., the 64×64 resolution stage) is followed by the spatial pixel-by-pixel attention fusion module, which performs pixel-level fusion of the middle layer features of N polarization components.

[0063] The decoder outputs a 256×256 reconstructed feature map with the same acquisition resolution, which is then downsampled to 64×64 using adaptive average pooling as the final reconstructed image output.

[0064] III. Specific Implementation of Polarized Token Attention Fusion

[0065] 1) Construction of polarization tokens. Features of the bottleneck layer for each polarization component. Global average pooling is performed to obtain a global description vector, which is then mapped to the attention computation space via a fully connected linear projection layer, and a learnable polarization identity code is superimposed on it. Obtain the polarization token In this embodiment, the number of polarization component paths N is 4 (corresponding to four line polarization components of 0°, 45°, 90°, and 135°), and the length of the polarization token sequence is 4.

[0066] 2) Cross-polarization token attention. The polarization token sequence is fed into the multi-head self-attention module and feedforward network after layer normalization. The number of multi-heads is 4, resulting in the attention weight matrix A. The mean along the last dimension is taken and normalized by the normalized exponential function to obtain the channel-level fusion weight α. The features of the 4 bottleneck layers are weighted and summed.

[0067] 3) Spatial pixel-wise attention. In the middle layer of the decoder (feature layer with the same resolution as the original input image), the four middle layer feature maps are processed by 1×1 convolution and normalized exponential function along the polarization channel dimension to obtain four spatial pixel-wise fused weight maps, which are then summed after element-wise multiplication with the corresponding middle layer feature maps.

[0068] 4) Independent first-layer convolution. The encoder uses independent convolution weights for the four polarization components in the first layer convolution, and the subsequent coding layers share the convolution weights among the four polarization components.

[0069] Example 1: Performance Comparison of Different Fusion Methods

[0070] This embodiment represents a typical implementation configuration: N=4, the encoder shares weights and each polarization channel of the first convolutional layer is independent, the bottleneck layer applies cross-polarization token attention, and the middle layer of the decoder applies spatial pixel-wise attention. The control group maintains the same network structure and training hyperparameters, only changing the multi-polarization fusion method: Control A uses single-path polarization reconstruction; Control B uses the input of four polarization speckle paths directly superimposed into a single network; Control C uses four polarization components reconstructed separately using independent complete networks and then weighted and combined with fixed weights; Control D uses four polarization components reconstructed separately using independent complete networks and then weighted and combined with scalar weights output from a gated network.

[0071] Table 2 Performance comparison of Example 1 and the control group test set

[0072]

[0073] Note: The values ​​in the table above are the mean and standard deviation of repeated experiments.

[0074] The experimental results are shown in Table 2. Control B, due to irreversible feature mixing occurring at the first layer of the network, has a lower PCC on the test set than single-path polarization control A. Controls C and D achieve some improvement through output-side weighting, but the improvement is limited because the fusion occurs at the output and is scalar-weighted. This embodiment uses polarization token attention fusion, and the PCC and SSIM on the test set are optimal among all control configurations. Compared to single-path control A and the input-side superposition and output-side weighting control methods, it achieves a stable improvement, verifying the advantage of modeling the cross-channel relationship of polarization token sequences in the feature space over the input-side superposition and output-side weighting methods.

[0075] Example 2: The impact of first-layer convolutional independence on attention weights

[0076] This embodiment examines the impact of whether the first-layer convolution of the encoder is made independent for each polarization channel on the cross-polarization token attention weights and reconstruction performance. Other configurations are the same as in Embodiment 1.

[0077] Experiments show that when the encoder's first-layer convolution shares weights among the N polarization components, the feature representations of each polarization speckle after processing by the same first-layer convolution are highly aligned at the first layer. This leads to the subsequent multi-head self-attention failing to distinguish the semantic differences between polarization tokens. The channel-level fusion weights degenerate into an approximately uniform distribution (each channel is close to 1 / N) and remain essentially unchanged across different input samples, resulting in a loss of the adaptive capability of multi-polarization fusion. When the first-layer convolution uses independent convolution weights for the N polarization components, each polarization component extracts polarization-specific local texture features at the first layer. The initial discriminative power of each polarization token is amplified, and the channel-level fusion weights exhibit a differentiated distribution that dynamically changes with the input samples. The adaptive capability of multi-polarization fusion is effectively activated, and the reconstruction performance is correspondingly improved. This embodiment verifies the effect of making each polarization channel independent in the first-layer convolution on suppressing the uniform degradation of attention weights and activating the cross-polarization dynamic fusion capability.

[0078] Example 3: Ablation of Two-Level Attention Fusion

[0079] This embodiment examines the contributions of bottleneck layer channel-level attention and decoder layer spatial pixel-by-pixel attention to reconstruction performance. Other configurations are the same as in Embodiment 1.

[0080] Experiments show that applying channel-level attention with cross-polarization tokens only at the bottleneck layer, or applying spatial pixel-by-pixel attention only at the middle layer of the decoder, can achieve improvements compared to input-side stacking and output-side weighting. Furthermore, when both channel-level attention at the bottleneck layer and spatial pixel-by-pixel attention at the middle layer of the decoder are applied simultaneously, the reconstruction performance surpasses either configuration alone. This indicates that the global channel-level fusion at the bottleneck layer primarily models the overall complementary relationship between polarization components at the most abstract feature level, while the spatial pixel-by-pixel fusion at the middle layer of the decoder models the differentiated dependence of different spatial locations on each polarization component at the local spatial level. These two approaches complement each other, forming a two-level adaptive fusion from global channels to local space. Their synergistic effect is key to achieving optimal reconstruction performance in this invention.

[0081] Example 4: Parametric Efficiency of Shared Encoders

[0082] This embodiment examines the parameter efficiency of a shared encoder design relative to a multi-network independent scheme. Other configurations are the same as in Embodiment 1. The parameters are values ​​determined by the network structure and can be directly calculated from the network structure, independent of training randomness.

[0083] Table 3 Comparison of Parameter Quantities for Different Schemes

[0084]

[0085] As shown in Table 3, this invention employs a design where N polarization components share the same set of encoder weights, resulting in approximately one-quarter of the parameter count of a four-network independent scheme. The parameter differences are determined by the network structure and can be directly calculated from the network structure, independent of the training process. The shared encoder significantly reduces the number of parameters and computational overhead while placing features of different polarization components in the same representation space, providing a semantically consistent basis for attention comparisons between cross-polarization tokens. Combined with Example 1, it can be seen that this invention, while significantly reducing the number of parameters, still achieves better reconstruction performance than the multi-network independent weighted scheme with several times the number of parameters.

[0086] The above embodiments provide a typical implementation configuration of the present invention (N=4, shared encoder with independent polarization channels in the first layer convolution, and two-level fusion of bottleneck layer channel-level attention and mid-layer spatial pixel-by-pixel attention in the decoder). Embodiment 1 demonstrates that the present invention achieves a stable performance improvement in face image reconstruction tasks using 100 m step-index multimode fiber compared to methods such as single-path polarization and input-end superposition and output-end weighting. Embodiments 2, 3, and 4 systematically verify the effectiveness, generalizability, and parameter efficiency advantages of the present invention from three dimensions: independent first-layer convolution, two-level attention fusion, and shared encoder parameter efficiency. All performance values ​​listed in this specification are based on actual experimental records.

[0087] The above description is merely an exemplary embodiment of the present invention and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made in accordance with the present invention are still within the scope of the present invention. Other embodiments of the invention will readily conceive of those skilled in the art upon consideration of the specification and practice of the disclosure herein. These variations follow the general principles of the invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. A method for reconstructing speckle images in multimode fiber based on polarization token attention, characterized in that, Includes the following steps: Step S1: Acquisition of multi-polarization optical signals A laser outputs monochromatic coherent light, which is incident on a spatial light modulator. After the spatial light modulator modulates the original input image, the modulated light field is coupled into a multimode fiber. At the output end of the multimode fiber, N polarization components are obtained through a multi-path polarization splitter. N polarization speckle images are acquired by N image sensors respectively; Step S2: Polarization-specific first-layer convolution and shared coding backbone feature extraction The N polarization speckle images are respectively input into an encoding network consisting of polarization-specific first-layer convolutions and a shared encoding backbone. The polarization-specific first-layer convolutions apply independent convolution weights to each of the N polarization components, and the shared encoding backbone shares convolution weights among the N polarization components. The encoding network independently propagates forward through each polarization speckle image to obtain multi-level feature maps and bottleneck layer features corresponding to each polarization speckle image. Step S3: Polarization Token Construction Global average pooling is performed on the bottleneck layer features corresponding to each polarization speckle image to obtain the global description vector of each polarization component. The global description vector of each polarization component is linearly projected onto the attention computation space and superimposed with the learnable polarization identity encoding vector corresponding to each polarization component to obtain a polarization token sequence of length N. Step S4: Cross-polarization token attention fusion The polarization token sequence is input into the multi-head self-attention module, and the polarization token sequence is linearly projected to obtain the query vector, key vector and value vector respectively. An attention weight matrix of shape N×N is obtained by scaling the dot product of the query vector and the key vector and then normalizing it using a normalized exponential function. Instead of using the attention weight matrix for attention value transformation, this invention takes the mean along one dimension of the attention weight matrix and then normalizes it using a normalized exponential function to obtain the channel-level fusion weights corresponding to each of the N polarization components. The bottleneck layer features of the N polarization components (not the value tensor after the value vector transformation) are weighted and summed using the channel-level fusion weights to obtain the fused bottleneck layer features. Step S5: Spatial pixel-wise attention fusion In the middle layer of the decoder, the N polarization components are transmitted via skip connections to the feature maps of the encoder layer. The N feature maps are processed by convolution and normalized exponential function along the polarization channel dimension to obtain spatial pixel-wise fusion weight maps of the N polarization components; the spatial pixel-wise fusion weight maps of each path are multiplied element-wise with the corresponding intermediate feature map of the polarization component, and the N results are summed to obtain the fused intermediate feature map. Step S6: Decoding and Reconstruction The fused bottleneck layer features and the fused middle layer feature map are input into the decoder, and the reconstructed image is output after processing by the decoder.

2. The multimode fiber speckle image reconstruction method according to claim 1, characterized in that, In step S1, the multi-path polarization beam splitter is used to generate polarization components in four linear polarization directions: 0°, 45°, 90°, and 135°; or it is used to generate polarization components in the four linear polarization directions and a total intensity channel that has not undergone polarization analysis.

3. The multimode fiber speckle image reconstruction method according to claim 1, characterized in that, In step S1, the spatial light modulator is selected from any one of digital micromirror device (DMD), reflective liquid crystal silicon-based spatial light modulator (LCoS-SLM), transmissive liquid crystal spatial light modulator (LC-SLM), or deformable mirror array; the multimode fiber is a step-index multimode fiber or a graded-index multimode fiber, or can be replaced with a few-mode fiber or a multi-core fiber.

4. The multimode fiber speckle image reconstruction method according to claim 1, characterized in that, In step S2, the shared encoding backbone is selected from the encoder portion of any one of Attention U-Net, Standard U-Net, Residual Neural Network ResNet, and Dense Network; the decoder and the encoder with shared weights form an encoder-decoder structure, and skip connections are set between corresponding layers of the encoder and the decoder.

5. The multimode fiber speckle image reconstruction method according to claim 1, characterized in that, In step S3, the linear projection is a learnable fully connected mapping layer; the learnable polarization identity encoding vector is a unique identity tensor for each polarization component that is updated during training, and is added element-wise to the global description vector in the polarization token construction step.

6. The multimode fiber speckle image reconstruction method according to claim 1, characterized in that, In step S4, the multi-head self-attention module calculates the attention weights of the polarization token sequence in parallel in multiple independent feature subspaces, and splices the outputs of each feature subspace and outputs them through linear projection; the channel-level fusion weights obtained in step S4 are further used in step S6 to perform cross-polarization weighted fusion of the high-level features transmitted by the skip connection.

7. The multimode fiber speckle image reconstruction method according to claim 1, characterized in that, In step S5, the spatial pixel-wise fusion weight map is applied to the feature layer in the decoder that has the same resolution as or an integer multiple of the original input image; the spatial pixel-wise fusion weight map gives the N polarization components independent fusion weights at different spatial locations.

8. A multimode fiber speckle image reconstruction device based on polarization token attention, characterized in that, The apparatus is used to implement the multimode fiber speckle image reconstruction method according to any one of claims 1-7, the apparatus comprising: Multipolar optical transmission and acquisition module: used to spatially modulate the original input image and inject it into a multimode fiber, and acquire N polarization speckle images at the output end of the multimode fiber through a multi-path polarization beam splitter. Reconstruction network system: used for end-to-end reconstruction of the N-channel polarization speckle images, the reconstruction network system includes the following sub-modules: Shared weight encoder submodule: used to extract multi-level feature maps and bottleneck layer features from the N-channel polarization speckle images respectively, and its first-layer convolution uses independent convolution weights for each polarization channel; Cross-polarization token attention submodule: Deployed in the bottleneck layer of the encoder, it is used to perform global average pooling, linear projection and superimpose learnable polarization identity codes on the features of each bottleneck layer, generate channel-level fusion weights through multi-head self-attention calculation, and perform weighted fusion of the N bottleneck layer features; Spatial attention fusion submodule: Deployed in the middle layer of the decoder, used to generate a spatial pixel-by-pixel fusion weight map and sum the middle layer feature maps of each polarization channel after element-wise multiplication; Shared weight decoder submodule: contains skip connections, used to upsample the fused bottleneck layer features and the fused middle layer feature maps step by step to output the reconstructed image.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the processor executes a computer program stored in the memory to implement the multimode fiber speckle image reconstruction method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the multimode fiber speckle image reconstruction method according to any one of claims 1-7.

Images