Multi-mode fiber speckle physical state sensing and image reconstruction integrated method and system

Abstract

The application discloses a multimode fiber speckle physical state sensing and image reconstruction integrated method and system, relates to the technical field of optical speckle imaging and machine learning, constructs a multi-state speckle dataset, and selects a pilot codebook based on state separability evaluation; in the reasoning stage, the corresponding speckle sequence is obtained by loading the pilot image, the state representation vector is obtained through the shared encoder and the convergence operator, and the curvature and stress category are output; further, the error between the pilot true value and the current pilot reconstruction output is used to construct an online calibration signal, only a few step updates are performed on the FiLM parameter generation unit, adaptive compensation of the continuously drifting unknown physical state is realized; at the same time, the state representation vector is used to perform conditional modulation on the reconstruction decoder, and state-driven image reconstruction is realized; finally, the integration of physical state sensing and image reconstruction is realized. The application can effectively improve the adaptability of the system to the continuously drifting state, and improve the reconstruction quality and stability under the multi-state condition.

Description

Technical Field

[0001] This invention relates to the fields of optical speckle imaging and machine learning technology, and more specifically to an integrated method and system for sensing the physical state of multimode fiber speckle and reconstructing images. Background Technology

[0002] Multimode fiber, due to its small diameter, high flexibility, and rich mode capacity, has significant application value in optical imaging, neural interfaces, and industrial endoscopy. The near-field speckle intensity image at the output end face of a multimode fiber contains a comprehensive encoding of the input optical field information and the fiber transmission characteristics, and can be reconstructed and restored using machine learning methods.

[0003] However, in practical deployments, optical fibers are subject to disturbances such as curvature changes, mechanical stress, attitude shifts, and temperature drift, leading to changes in their transmission characteristics and shifts in the speckle distribution corresponding to the same input image. This causes reconstruction models trained based on fixed states to experience problems such as reduced reconstruction quality and inconsistent outputs in cross-state scenarios. At the same time, the physical state of the optical fiber itself is also an important sensing object, but existing reconstruction methods typically lack the ability to explicitly sense and output its properties.

[0004] Existing technologies mainly address the above challenges through the following three types of solutions:

[0005] (1) Offline recalibration scheme: Data is reacquired and the model is trained after the fiber state changes. This scheme has poor real-time performance, high calibration cost, and cannot adapt to continuous drift scenarios.

[0006] (2) Retrieval-based calibration scheme: Construct a multi-state speckle database and retrieve neighboring states during inference, but the storage overhead is large, the generalization ability to unknown states is limited, and state mismatch is still easy to occur.

[0007] (3) Multi-state hybrid training scheme: All state data are mixed to train a single model, but the mapping of different state domains is prone to mutual interference, the reconstruction quality and cross-state stability decrease, and it is difficult to explicitly output the current physical state.

[0008] The common limitations of the above schemes are: firstly, they lack explicit sensing output of the physical state of the optical fiber, making it impossible to achieve the integration of sensing and reconstruction; secondly, the reconstruction network lacks effective adaptive modulation and online compensation capabilities for state changes. Especially in real deployment, the actual transmission state of multimode optical fiber is often difficult to strictly correspond to a certain discrete joint physical state sampled during the training phase. Therefore, even if an approximate estimate of the current state is obtained through multi-state training, state mismatch may still exist, leading to poor cross-state reconstruction consistency and a decrease in reconstruction quality.

[0009] Therefore, there is an urgent need for an integrated method that can simultaneously complete physical state perception and image reconstruction within a unified framework, and can perform online compensation for continuously drifting unknown states. Summary of the Invention

[0010] In view of the above problems, the present invention is proposed to provide an integrated method and system for multimode fiber speckle physical state sensing and image reconstruction that overcomes or at least partially solves the above problems.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] In a first aspect, embodiments of the present invention provide an integrated method for sensing the physical state of multimode fiber speckle and reconstructing an image, comprising:

[0013] S1: Construct a multimode fiber speckle imaging system, acquire the near-field speckle intensity image of the output end face after the input image is transmitted through the multimode fiber speckle imaging system, and record the corresponding curvature state and stress state to construct a multi-state speckle dataset; use the multi-state speckle dataset to perform offline training on the shared encoder, classification head and reconstruction decoder to obtain a pre-trained model.

[0014] S2: Select Top-K images from the candidate input image set as pilot codebooks based on state separability evaluation;

[0015] S3: During the inference phase, the pilot images in the pilot codebook are loaded sequentially and the corresponding pilot speckle sequence is acquired. Features are extracted by the shared encoder in the pre-trained model and the state representation vector is obtained through the convergence operator. The corresponding curvature category and stress category are output by the classification head in the pre-trained model.

[0016] S4: Based on the error between the pilot image and the current output of the system, an online calibration loss is constructed. While keeping the main parameters of the shared encoder and reconstruction decoder of the pre-trained model unchanged, only a few steps of gradient update are performed on the FiLM parameter generation unit to perform online self-calibration compensation for the unknown physical state of the continuous drift of the multimode fiber.

[0017] S5: Construct a conditional image reconstruction module, map the state representation vector to modulation parameters, perform FiLM conditional modulation on the intermediate features of the reconstruction decoder, and make the reconstruction process adaptively adjust according to the current physical state;

[0018] S6: After completing online self-calibration compensation, load speckle is acquired and input into the conditional image reconstruction module, and the reconstructed image is output. At the same time, combined with the curvature category and stress category output by S3, the integrated output of physical state perception and image reconstruction is realized.

[0019] Preferably, the specific implementation process of S2 is as follows:

[0020] For the candidate input images in the candidate input image set, their speckle sets are acquired under different joint physical states, and feature vectors are extracted using a feature extractor to obtain the feature set;

[0021] Calculate the feature center distance and intra-class dispersion of the feature set;

[0022] The state separability of the candidate input images is scored based on the feature center distance and intra-class dispersion, and the top K images with the highest scores are selected as the pilot codebook based on the scoring results.

[0023] Preferably, the formula for calculating the feature center distance is:

[0024]

[0025] The formula for calculating the intra-class dispersion is:

[0026]

[0027] The formula for calculating the state separability score is as follows:

[0028]

[0029] in, Represents the feature set, and These represent the feature center distances of the corresponding feature sets of the candidate input image under joint physical states u and v, respectively. Indicates the within-class dispersion. To prevent the division by zero small constant, Score the separability of states. This represents averaging the multiple feature samples corresponding to index t. This indicates averaging across all joint physical states. This represents the average of (u,v) for all different joint physical state pairs that satisfy u≠v.

[0030] Preferably, S5 specifically includes:

[0031] The state representation vector is passed through two independent fully connected layers to calculate the channel scaling parameter and the channel bias parameter, respectively:

[0032]

[0033]

[0034] in, and They represent the first l Layer channel scaling parameter generation matrix and channel offset parameter generation matrix; and Representing the l Layer scaling parameters generate bias and bias parameters generate bias. Indicates the first l Layer channel scaling parameters, Indicates the first l Layer channel offset parameters;

[0035] In the reconstructed decoder l The layer performs an affine transformation on the normalized intermediate features using channel scaling and channel bias parameters generated from the state representation vector:

[0036]

[0037] in, l c represents the reconstructed decoder index; 'c' represents the channel index. h , w Indicates spatial index; Indicates the decoder's first l The layer is in channel c, spatial location ( h , w Intermediate features at ) These are the features after adaptive modulation; Norm(·) is the normalization operator; and These are the first two terms generated by the state representation vector z. l Scaling parameters and channel offset parameters for the c-th channel of layer.

[0038] Preferably, S4 includes:

[0039] Construct an online calibration loss based on the error between the pilot image and the current system output:

[0040]

[0041] in, To calibrate the balance coefficient of the structure term in the loss, K is the number of pilot images. This is a pilot image. For system output;

[0042] Perform minimal gradient updates only on the FiLM parameter generation unit:

[0043]

[0044] in, This indicates that all FiLM parameters are used to generate element parameters. For online calibration of the learning rate, T is the number of online update steps. This represents the parameter set of the FiLM parameter generation unit during the t-th online calibration iteration. This represents the set of parameters for the FiLM parameter generation unit after the current gradient update.

[0045] Preferably, S4 further includes:

[0046] The total calibration loss is constructed using the parameter offset regularization term:

[0047]

[0048]

[0049] in, , , , This represents the initial parameters saved after offline training is completed. For regularization weights, and They represent the first l Layer channel scaling parameter generation matrix and channel offset parameter generation matrix; and Representing the l Layer scaling parameters generate bias and bias parameters generate bias. Indicates the total calibration loss. This indicates the parameter offset regularization term.

[0050] Preferably, the total loss function of the model is:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] in, The actual input image, To reconstruct the image, yes Norm, For balance coefficient, and represents the predicted probability output for the curvature category and the stress category, respectively, where c and s represent the corresponding ground truth labels. For the total loss function, To reconstruct the loss, For state classification loss, For cross-state consistency loss, The loss is the feature alignment loss, where z is the state representation vector. , , , This represents the weighting coefficient corresponding to the loss.

[0057] Secondly, embodiments of the present invention provide an integrated system for multimode fiber speckle physical state perception and image reconstruction, comprising:

[0058] System construction and multi-state dataset acquisition module: used to construct a multimode fiber speckle imaging system, acquire the near-field speckle intensity image of the output end face after the input image is transmitted through the multimode fiber speckle imaging system, and record the corresponding curvature state and stress state to construct a multi-state speckle dataset; use the multi-state speckle dataset to perform offline training on the shared encoder, classification head and reconstruction decoder to obtain a pre-trained model;

[0059] Pilot codebook construction module: Selects Top-K images from the candidate input image set as pilot codebooks based on state separability evaluation;

[0060] Physical state perception module: During the inference phase, the pilot images in the pilot codebook are loaded sequentially and the corresponding pilot speckle sequence is acquired. Features are extracted by the shared encoder in the pre-trained model and the state representation vector is obtained by the convergence operator. The corresponding curvature category and stress category are output by the classification head in the pre-trained model.

[0061] Online self-calibration module: Based on the error between the pilot image and the current output of the system, an online calibration loss is constructed. While keeping the main parameters of the shared encoder and reconstruction decoder of the pre-trained model unchanged, only a few steps of gradient update are performed on the FiLM parameter generation unit to perform online self-calibration compensation for the unknown physical state of the continuous drift of the multimode fiber.

[0062] Conditional image reconstruction module: used to map the state representation vector to modulation parameters, perform FiLM conditional modulation on the intermediate features of the reconstruction decoder, so that the reconstruction process is adaptively adjusted according to the current physical state;

[0063] Conditional image output module: After completing online self-calibration compensation, it acquires load speckle and inputs it into the conditional image reconstruction module to output the reconstructed image. At the same time, it combines the curvature category and stress category output by the physical state perception module to realize the integrated output of physical state perception and image reconstruction.

[0064] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an integrated method for sensing the physical state of multimode fiber speckle and reconstructing an image.

[0065] Fourthly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements an integrated method for multimode fiber speckle physical state perception and image reconstruction.

[0066] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an integrated method and system for multimode fiber speckle physical state perception and image reconstruction, which has the following advantages:

[0067] 1) By using pilot state sensing and online self-calibration mechanisms, the offset of the real state relative to the training discrete state set is compensated, thereby improving the system's adaptability to continuously drifting unknown states.

[0068] 2) Improve the quality and stability of image reconstruction under multi-state conditions through state-driven conditional modulation;

[0069] 3) The integrated output of state recognition and image reconstruction is completed simultaneously in the same inference process, and only a small number of parameters need to be updated online, resulting in low computational overhead. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0071] Figure 1 This is a flowchart of an integrated method for sensing the physical state of multimode fiber speckle and reconstructing an image, provided in an embodiment of the present invention.

[0072] Figure 2 This is a schematic diagram of the structure of the multimode fiber speckle imaging system provided in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram of the physical state sensing module provided in an embodiment of the present invention;

[0074] Figure 4 This is a schematic diagram of the conditional image reconstruction module provided in an embodiment of the present invention;

[0075] Figure 5This is a schematic diagram of an integrated system for sensing the physical state of multimode fiber speckle and reconstructing images, provided in an embodiment of the present invention.

[0076] Figure 6 This is a schematic diagram of the conditional reconstruction decoder and FiLM modulation insertion position in an embodiment of the present invention.

[0077] The components include: 1. Light source module; 2. Lens; 3. Image loading module; 4. Coupling module; 5. Multimode fiber; 6. Speckle acquisition module; 7. Bending control component; and 8. Host computer processing module. Detailed Implementation

[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] This invention discloses an integrated method for multimode fiber speckle physical state perception and image reconstruction, such as... Figure 1 As shown, it includes:

[0080] S1: Construct a multimode fiber speckle imaging system, acquire near-field speckle intensity images of the output end face after the input image is transmitted through the multimode fiber speckle imaging system, and record the corresponding curvature state and stress state to construct a multi-state speckle dataset; use the multi-state speckle dataset to train the shared encoder, classification head and reconstruction decoder offline to obtain a pre-trained model.

[0081] S2: Select Top-K images from the candidate input image set as pilot codebooks based on state separability evaluation; where the candidate input image set is a set constructed based on the input images in S1;

[0082] S3: During the inference phase, the pilot images in the pilot codebook are loaded sequentially and the corresponding pilot speckle sequence is acquired. The features are extracted by the shared encoder in the pre-trained model and the state representation vector is obtained through the convergence operator. The corresponding curvature category and stress category are output by the classification head in the pre-trained model.

[0083] S4: Based on the error between the pilot image and the current output of the system, an online calibration loss is constructed. While keeping the main parameters of the shared encoder and reconstruction decoder of the pre-trained model unchanged, only a few steps of gradient update are performed on the FiLM parameter generation unit to perform online self-calibration compensation for the unknown physical state of the continuous drift of the multimode fiber.

[0084] S5: Construct a conditional image reconstruction module, map the state representation vector to modulation parameters, perform FiLM conditional modulation on the intermediate features of the reconstruction decoder, and make the reconstruction process adaptively adjust according to the current physical state;

[0085] S6: After completing online self-calibration compensation, load speckle is acquired and input into the conditional image reconstruction module, and the reconstructed image is output. At the same time, combined with the curvature category and stress category output by S3, the integrated output of physical state perception and image reconstruction is realized.

[0086] Furthermore, such as Figure 2 As shown, the multimode fiber speckle sensing integrated imaging system constructed by the present invention includes a light source module 1, a lens 2, an image loading module 3, a coupling module 4, a multimode fiber 5, a speckle acquisition module 6, and a host computer processing module 8. The light source module 1 provides a coherent illumination beam; the lens 2 is used to collimate and / or focus the coherent illumination beam emitted by the light source module 1, so that the illumination beam illuminates the image loading module 3 with a predetermined spot size and incident state, thereby improving the image loading effect; the image loading module 3 (preferably a spatial light modulator, SLM) is used to load the image to be transmitted; the coupling module 4 couples the beam carrying image information into the input end of the multimode fiber, wherein the coupling module 4 includes a lens, a beam splitter, and an objective lens, wherein the lens is used to converge and shape the beam, the beam splitter is used to adjust the propagation direction of the optical path, and the objective lens is set in front of the incident end of the multimode fiber 5 to focus and couple the beam to the incident end face of the multimode fiber 5; the multimode fiber 5 interferes in multiple propagation modes, forming a speckle pattern on the output end face; the speckle acquisition module 6 (CCD / CMOS camera) records the near-field speckle intensity image of the output end face; the host computer processing module 8 completes data storage, state perception, and reconstruction reasoning.

[0087] It also includes a bending control component 7, which is located outside the multimode fiber 5, and is used to adjust the bending radius, bending position and spatial attitude of the fiber to simulate various physical states in actual use.

[0088] Joint physical state settings: Select the approximately straight state of the optical fiber, 4 curvature levels (bending diameters of 1.2cm, 2.4cm, 3.0cm, and 5.0cm respectively) and 4 stress levels (applied stresses of 0.05N, 0.01N, 0.2N, and 0.5N) to form 9 joint physical states, with state index u∈{1,…,9}.

[0089] Dataset composition: Input images can be derived from the MNIST handwritten digit dataset, the Fashion-MNIST clothing dataset, and a subset of natural scene images. Multiple sets of speckle-image paired samples and corresponding curvature and stress labels are collected for each joint state for model training and testing.

[0090] Speckle preprocessing includes dark field subtraction, region of interest alignment, intensity normalization, and logarithmic transformation. This reduces dynamic range differences and improves training stability.

[0091] In this embodiment, the present invention automatically selects K images with optimal state separability from the candidate input image set X as pilot codebooks. The specific steps are as follows:

[0092] (1) Feature extraction. For candidate input images... For each joint physical state u∈{1,…,M}, its speckle set is collected, and feature vectors are extracted using a feature extractor F (which can be principal component analysis, random projection, or the global pooling output of a lightweight convolutional network). .

[0093] (2) Feature center and intra-class divergence calculation. For candidate input images The feature set extracted under joint state u Calculate its feature center With intra-class dispersion :

[0094]

[0095]

[0096] (3) Definition of state separability score. Based on the feature center distance and intra-class dispersion between different joint states, the candidate input image is defined. State separability score:

[0097]

[0098] in To prevent the division by zero small constant, Score the separability of states. This represents averaging the multiple feature samples corresponding to index t. This indicates averaging across all joint physical states. This represents the average of (u,v) for all different joint physical state pairs that satisfy u≠v. In the formula, the numerator represents the candidate input image. The ratio represents the average distance between feature centers under different joint physical states, with the denominator representing the average intra-class dispersion under each joint state. A larger ratio indicates that the candidate image is more effective in distinguishing different joint states.

[0099] (4) Codebook selection. [The codebook is then...] The images are sorted from highest to lowest quality, and the top K images are selected to form the pilot codebook P. Optionally, the joint state recognition accuracy is not lower than a preset threshold. Under the constraint, the minimum K value is searched to reduce the proportion of pilot frames and improve the real-time performance of the system.

[0100] In this embodiment, as Figure 3 As shown, the state-aware module is used to extract the physical state representation vector z from the pilot speckle sequence and output the curvature / stress category, rather than to recover the target image content.

[0101] (1) Module structure

[0102] The state-aware module includes a shared encoder E(·) and a convergence operator Agg(·). The shared encoder E(·) consists of four convolutional layers, each with a kernel size of 3×3, and output channels of 16, 32, 64, and 128 respectively. Each convolutional layer is followed by batch normalization and ReLU activation function. The final output is a 128-dimensional feature vector after global average pooling.

[0103] Let the pilot codebook be During the inference phase, K pilot patterns from the pilot codebook are loaded sequentially, and corresponding pilot speckle patterns are acquired. For the pilot speckle of the k-th frame The pilot speckle output features of the k-th frame are extracted by the shared encoder. , where k takes the values ​​1, 2, ..., K.

[0104] (2) Construction of state representation vector

[0105] The convergence operator Agg(·) performs average convergence on the pilot speckle features of K frames to obtain the state representation vector z of the current joint physical state:

[0106]

[0107] The dimension of the state representation vector z can be set to 16 to 64, preferably 32; it simultaneously encodes curvature level, stress level and its continuous offset information.

[0108] (3) Classification head design

[0109] After obtaining the state representation vector z, the curvature classification head is set respectively. With stress classification head The current joint physical state is determined. This includes the curvature classification head. With stress classification head Each layer consists of two fully connected layers with a ReLU activation function in between, and the output layer is connected to a Softmax function to output the curvature category probabilities. With stress category probability .

[0110] (4) Reference state setting

[0111] Optionally, the mean of the state characterization vector of the multimode fiber in an approximately straight state is denoted as the reference feature prototype. ;The z-symmetric vector in the current state and The Euclidean distance between them can be used as a quantitative indicator of the degree of bending offset, and can be used to calculate the subsequent feature alignment loss.

[0112] In this embodiment, as Figure 4 As shown, the conditional image reconstruction module uses the state representation vector z to perform state adaptive modulation on the intermediate features of the reconstruction decoder, so that the reconstruction mapping under different physical states can be automatically switched, reducing the reconstruction degradation caused by multi-state mixing.

[0113] (1) Shared encoder multiplexing

[0114] Loaded speckle The load characteristics are obtained by sharing the same encoder E(·) with the pilot speckle. The shared encoder is reused in the state-aware branch and the conditional reconstruction branch, thereby reducing the number of model parameters.

[0115] (2) Generation of modulation parameters

[0116] Channel scaling parameters With bias parameters It is generated by mapping from the state representation vector z by two independent fully connected layers:

[0117]

[0118]

[0119] in, and They represent the first Layer scaling parameter generation matrix and bias parameter generation matrix; and Representing the Layer scaling parameters generate bias and bias parameters generate bias.

[0120] (3) FiLM conditional modulation

[0121] In the reconstruction decoder D(·) Layers generate channel-wise scaling parameters using the state representation vector z. With bias parameters For the normalized intermediate features Perform an affine transformation:

[0122]

[0123] in, c represents the decoder index; h and w represent the channel index; and h and w represent the spatial index. Indicates the decoder's first The intermediate features of the layer at spatial location (h, w) in channel c; These are features after adaptive modulation; Norm(·) is a normalization operator (which can be batch normalization or instance normalization); and These are the first two terms generated by the state representation vector z. Scaling parameters and channel offset parameters for the c-th channel of layer.

[0124] (4) FiLM modulation insertion position

[0125] FiLM modulation is inserted at the intermediate features of multiple layers in the decoder to balance global structural recovery and local detail reconstruction. For example... Figure 6 As shown, FiLM modulation is inserted at the intermediate feature positions of multiple decoding layers in the conditional reconstruction decoder. The state representation vector z output by the state-aware module is input to the FiLM parameter generation unit, which generates corresponding channel scaling parameters for different decoding layers of the conditional reconstruction decoder. and channel offset parameters In the actual execution of the load phase, the load speckle, after being extracted by the shared encoder, is input into the conditional reconstruction decoder. In each decoding layer, the input features of the current layer are preferably first processed by convolution and / or upsampling, then normalized, and finally processed using the corresponding channel scaling parameters. and channel offset parameters FiLM affine modulation is performed on the normalized intermediate features to obtain modulated features adapted to the current physical state of the multimode fiber, and then passed to the subsequent decoding layer.

[0126] Furthermore, different decoding layers receive different sets of modulation parameters to adaptively adjust the reconstructed features at different scales. Among them, deeper features focus more on overall structure restoration, while shallower features focus more on local detail reconstruction. Therefore, by inserting FiLM modulation into multiple decoding layers, both global structure restoration and local detail reconstruction can be taken into account, thereby improving the stability and accuracy of image reconstruction in multimode fiber under different physical states.

[0127] (5) Reconstruct the output

[0128] After the load feature u is extracted by the shared encoder, it is input into the conditional reconstruction decoder D( The reconstruction is completed under the condition of the state representation vector z, and the final output is the reconstructed image. The decoder uses the U-Net decoding architecture.

[0129] In this embodiment, the state-aware module and the conditional reconstruction module are jointly trained end-to-end, and the total loss function is:

[0130]

[0131] The loss terms are defined as follows:

[0132] (1) Reconstruction loss

[0133] Reconstruction loss is defined as:

[0134]

[0135] in, The actual input image, To reconstruct the output image, yes Norm, This is the balance coefficient; the example value is 0.5. The structural similarity index measures the degree of similarity between the reconstructed image and the original input image in terms of brightness, contrast, and structure. The reconstruction loss considers pixel-level similarity as well. Accuracy and structural similarity.

[0136] (2) State classification loss

[0137] The state classification loss is defined as:

[0138]

[0139] The state classification loss calculates the cross-entropy for both the curvature and stress categories, which drives the discriminative learning of the classification head. and represents the predicted probability output for the curvature category and the stress category, respectively, where c and s represent the corresponding ground truth labels. The cross-entropy loss is used to measure the difference between the curvature category prediction output and the corresponding true curvature label, and the stress category prediction output and the corresponding true stress label, in order to supervise the state-aware branch learning to discriminate the state representation of the joint physical state.

[0140] (3) Cross-state consistency loss

[0141] Cross-state consistency loss is defined as:

[0142]

[0143] This represents the reconstructed image corresponding to the same input under the joint physical state u. This represents the reconstructed image corresponding to the same input in another joint physical state v. The cross-state consistency loss utilizes the data structure of repeatedly acquiring the same input image in different joint physical states to constrain the reconstruction results in different states to tend to be consistent, thereby improving cross-state stability.

[0144] (4) Feature alignment loss (optional)

[0145] Feature alignment loss is defined as:

[0146]

[0147] The feature loss is used to constrain the current state representation vector toward the reference state feature prototype. Approaching to assist the convergence of the state-aware module; in scenarios without a reference state, it can be set... , This is the state representation vector under the preset reference physical state.

[0148] The weights of each loss term and training parameters can be set and adjusted based on the validation set results. The model training can be performed end-to-end using the Adam optimizer.

[0149] In this embodiment, the complete inference phase is executed in the following sequence:

[0150] (1) Pilot stage: K patterns in pilot codebook P are loaded sequentially. The speckle acquisition module records the corresponding pilot speckle. State perception module calculation The classification head outputs the curvature category probability respectively. With stress category probability .

[0151] (2) Online self-calibration stage: After obtaining the current state representation vector z, the final reconstruction of the load speckle is not performed immediately. Instead, the pilot pattern input is known as a priori condition. The difference between the output obtained by the system reconstruction of the current pilot pattern and the pilot true value is constructed as an online self-supervised calibration signal. Only a few steps of gradient update are performed on the FiLM parameter generation unit in S5 to correct the mapping relationship between the current state representation vector z and the decoder modulation parameters.

[0152] (3) Loading stage: After completing online self-calibration, the image loading module loads the load image x and acquires the corresponding load speckle. The conditional reconstruction module utilizes the calibrated parameter generation unit to perform conditional reconstruction of the load speckle based on the current state representation vector z, and outputs the reconstructed image. .

[0153] (4) Synesthesia integrated output: Simultaneously output the reconstructed image in the same inference process. With physical state category ( , This enables the integrated output of physical perception and image reconstruction.

[0154] Each inference window may include several pilot frames and several payload frames to balance state awareness accuracy and system real-time performance. Online self-calibration, as a necessary step in the inference process of this invention, is performed after obtaining the current state representation vector in the pilot stage. Furthermore, the trigger threshold and update frequency of online self-calibration can be set according to the pilot reconstruction error, state classification confidence, or preset inference window interval to balance state adaptability and real-time requirements.

[0155] In this embodiment, when the multimode fiber slowly drifts due to environmental changes, although the state characterization vector z obtained by S3 can provide an initial estimate of the current physical state, the decoder modulation parameters it drives may still mismatch with the actual transmission characteristics. Therefore, for the pilot pattern... The corresponding output is obtained after the current system is reconstructed. Based on the known input from the pilot signals Compared with the current reconstruction output The difference between them is used to construct the calibration loss:

[0156]

[0157] in, To calibrate the balance coefficient of the structural term in the loss, the example value is 0.5.

[0158] During the online self-calibration phase, gradient updates are performed only on the FiLM parameter generation unit, and the updated parameters include... , , , Based on calibration loss Perform gradient updates in 1 to 3 steps on the parameter generation unit mentioned above:

[0159]

[0160] in, This indicates that all FiLM parameters are used to generate unit parameters. For online calibration of the learning rate, it is exemplarily advisable to T represents the number of online update steps, preferably 1≤T≤3.

[0161] To suppress over-updates with a small number of pilot samples, a parameter offset regularization term can be added:

[0162]

[0163] Therefore, the total calibration loss is written as:

[0164]

[0165] in, , , , This represents the initial parameters saved after offline training is completed. The values ​​are regularization weights, with examples ranging from 0.01 to 0.1. During online self-calibration, the main parameters of the shared encoder and reconstructed decoder are not updated; only the FiLM parameter generation unit is updatable. This allows for rapid adaptation to new states while effectively avoiding catastrophic forgetting.

[0166] Optionally, for highly complex inputs such as natural images, m known masks can be superimposed without changing the payload content. m-frame speckle is generated and stacked into a multi-channel input to be fed into a shared encoder. At the same time, the state representation vector z is used to perform FiLM conditional modulation, thereby improving the detail recovery capability while keeping the synesthetic integration framework unchanged.

[0167] Based on the same inventive concept, embodiments of the present invention also provide an integrated system for multimode fiber speckle physical state perception and image reconstruction, such as... Figure 5 As shown, it includes:

[0168] System construction and multi-state dataset acquisition module: used to build a multimode fiber speckle imaging system, acquire the near-field speckle intensity image of the output end face after the input image is transmitted through the multimode fiber speckle imaging system, and record the corresponding curvature state and stress state to build a multi-state speckle dataset; use the multi-state speckle dataset to train the shared encoder, classification head and reconstruction decoder offline to obtain a pre-trained model;

[0169] Pilot codebook construction module: Selects Top-K images from the candidate input image set as pilot codebooks based on state separability evaluation;

[0170] Physical state perception module: During the inference phase, the pilot images in the pilot codebook are loaded sequentially and the corresponding pilot speckle sequence is acquired. Features are extracted by the shared encoder in the pre-trained model and the state representation vector is obtained through the convergence operator. The corresponding curvature category and stress category are output by the classification head in the pre-trained model.

[0171] Online self-calibration module: Based on the error between the pilot image and the current output of the system, an online calibration loss is constructed. While keeping the main parameters of the shared encoder and reconstruction decoder of the pre-trained model unchanged, only a few steps of gradient update are performed on the FiLM parameter generation unit to perform online self-calibration compensation for the unknown physical state of the continuous drift of the multimode fiber.

[0172] Conditional image reconstruction module: used to map the state representation vector to modulation parameters, perform FiLM conditional modulation on the intermediate features of the reconstruction decoder, so that the reconstruction process can be adaptively adjusted according to the current physical state;

[0173] Conditional image output module: After completing online self-calibration compensation, it acquires load speckle and inputs it into the conditional image reconstruction module to output the reconstructed image. At the same time, it combines the curvature category and stress category output by the physical state perception module to realize the integrated output of physical state perception and image reconstruction.

[0174] Since the principle behind the problem solved by the system is similar to that of the aforementioned methods, the specific implementation can be found in the implementation of the aforementioned methods, and the repeated parts will not be repeated.

[0175] This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements an integrated method for multimode fiber speckle physical state perception and image reconstruction.

[0176] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements an integrated method for multimode fiber speckle physical state perception and image reconstruction.

[0177] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0179] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for integrating multimode fiber speckle physical state perception and image reconstruction, characterized in that, include: S1: Construct a multimode fiber speckle imaging system, acquire the near-field speckle intensity image of the output end face after the input image is transmitted through the multimode fiber speckle imaging system, and record the corresponding curvature state and stress state to construct a multi-state speckle dataset; use the multi-state speckle dataset to perform offline training on the shared encoder, classification head and reconstruction decoder to obtain a pre-trained model. S2: Select Top-K images from the candidate input image set as pilot codebooks based on state separability evaluation; S3: During the inference phase, the pilot images in the pilot codebook are loaded sequentially and the corresponding pilot speckle sequence is acquired. Features are extracted by the shared encoder in the pre-trained model and the state representation vector is obtained through the convergence operator. The corresponding curvature category and stress category are output by the classification head in the pre-trained model. S4: Based on the error between the pilot image and the current output of the system, an online calibration loss is constructed. While keeping the main parameters of the shared encoder and reconstruction decoder of the pre-trained model unchanged, only a few steps of gradient update are performed on the FiLM parameter generation unit to perform online self-calibration compensation for the unknown physical state of the continuous drift of the multimode fiber. S5: Construct a conditional image reconstruction module, map the state representation vector to modulation parameters, perform FiLM conditional modulation on the intermediate features of the reconstruction decoder, and make the reconstruction process adaptively adjust according to the current physical state; S6: After completing online self-calibration compensation, load speckle is acquired and input into the conditional image reconstruction module, and the reconstructed image is output. At the same time, combined with the curvature category and stress category output by S3, the integrated output of physical state perception and image reconstruction is realized.

2. The method as described in claim 1, characterized in that, The specific implementation process of S2 is as follows: For the candidate input images in the candidate input image set, their speckle sets are acquired under different joint physical states, and feature vectors are extracted using a feature extractor to obtain the feature set; Calculate the feature center distance and intra-class dispersion of the feature set; The state separability of the candidate input images is scored based on the feature center distance and intra-class dispersion, and the top K images with the highest scores are selected as the pilot codebook based on the scoring results.

3. The method as described in claim 2, characterized in that, The formula for calculating the feature center distance is: The formula for calculating the intra-class dispersion is: The formula for calculating the state separability score is as follows: in, Represents the feature set, and These represent the feature center distances of the corresponding feature sets of the candidate input image under joint physical states u and v, respectively. Indicates the within-class dispersion. To prevent the division by zero small constant, Score the separability of states. This represents averaging the multiple feature samples corresponding to index t. This indicates averaging across all joint physical states. This represents the average of (u,v) for all different joint physical state pairs that satisfy u≠v.

4. The method as described in claim 1, characterized in that, S5 specifically includes: The state representation vector is passed through two independent fully connected layers to calculate the channel scaling parameter and the channel bias parameter, respectively: in, and They represent the first l Layer channel scaling parameter generation matrix and channel offset parameter generation matrix; and Representing the l Layer scaling parameters generate bias and bias parameters generate bias. Indicates the first l Layer channel scaling parameters, Indicates the first l Layer channel offset parameters; In the reconstructed decoder l The layer performs an affine transformation on the normalized intermediate features using channel scaling and channel bias parameters generated from the state representation vector: in, l c represents the reconstructed decoder index; 'c' represents the channel index. h , w Indicates spatial index; Indicates the decoder's first l The layer is in channel c, spatial location ( h , w Intermediate features at ) These are the features after adaptive modulation; Norm(·) is the normalization operator; and These are the first two terms generated by the state representation vector z. l Scaling parameters and channel offset parameters for the c-th channel of layer.

5. The method as described in claim 1, characterized in that, S4 includes: Construct an online calibration loss based on the error between the pilot image and the current system output: in, To calibrate the balance coefficient of the structure term in the loss, K is the number of pilot images. This is a pilot image. For system output; Perform minimal gradient updates only on the FiLM parameter generation unit: in, This indicates that all FiLM parameters are used to generate unit parameters. For online calibration of the learning rate, T is the number of online update steps. This represents the parameter set of the FiLM parameter generation unit during the q-th online calibration iteration. This represents the set of parameters for the FiLM parameter generation unit after the current gradient update.

6. The method as described in claim 5, characterized in that, S4 further includes: The total calibration loss is constructed using the parameter offset regularization term: in, , , , This represents the initial parameters saved after offline training is completed. For regularization weights, and They represent the first l Layer channel scaling parameter generation matrix and channel offset parameter generation matrix; and Representing the l Layer scaling parameters generate bias and bias parameters generate bias. Indicates the total calibration loss. This indicates the parameter offset regularization term.

7. The method as described in claim 1, characterized in that, The model's total loss function is: in, The actual input image, To reconstruct the image, yes Norm, For balance coefficient, and represents the predicted probability output for the curvature category and the stress category, respectively, where c and s represent the corresponding ground truth labels. For the total loss function, To reconstruct the loss, For state classification loss, For cross-state consistency loss, The loss is the feature alignment loss, where z is the state representation vector. , , , This represents the weighting coefficient corresponding to the loss. This represents the reconstructed image corresponding to the same input under the joint physical state u. This represents the reconstructed image corresponding to the same input under another joint physical state v. This is the state representation vector under a predefined reference physical state. Represents cross-entropy loss, This represents the structural similarity index.

8. A multimode fiber speckle physical state sensing and image reconstruction integrated system, characterized in that, include: System construction and multi-state dataset acquisition module: used to construct a multimode fiber speckle imaging system, acquire the near-field speckle intensity image of the output end face after the input image is transmitted through the multimode fiber speckle imaging system, and record the corresponding curvature state and stress state to construct a multi-state speckle dataset; use the multi-state speckle dataset to perform offline training on the shared encoder, classification head and reconstruction decoder to obtain a pre-trained model; Pilot codebook construction module: Selects Top-K images as pilot codebooks from the candidate input image set based on state separability evaluation; Physical state perception module: During the inference phase, the pilot images in the pilot codebook are loaded sequentially and the corresponding pilot speckle sequence is acquired. Features are extracted by the shared encoder in the pre-trained model and the state representation vector is obtained by the convergence operator. The corresponding curvature category and stress category are output by the classification head in the pre-trained model. Online self-calibration module: Based on the error between the pilot image and the current output of the system, an online calibration loss is constructed. While keeping the main parameters of the shared encoder and reconstruction decoder of the pre-trained model unchanged, only a few steps of gradient update are performed on the FiLM parameter generation unit to perform online self-calibration compensation for the unknown physical state of the continuous drift of the multimode fiber. Conditional image reconstruction module: used to map the state representation vector to modulation parameters, perform FiLM conditional modulation on the intermediate features of the reconstruction decoder, so that the reconstruction process is adaptively adjusted according to the current physical state; Conditional image output module: After completing online self-calibration compensation, it acquires load speckle and inputs it into the conditional image reconstruction module to output a reconstructed image. At the same time, it combines the curvature category and stress category output by the physical state perception module to realize the integrated output of physical state perception and image reconstruction.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements an integrated method for multimode fiber speckle physical state perception and image reconstruction as described in any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements an integrated method for multimode fiber speckle physical state perception and image reconstruction as described in any one of claims 1 to 7.

Images