Dual-analyte concentration deep learning calculation method and system based on heterogeneous sensor data fusion

Abstract

The embodiment of the invention provides a dual-analyte concentration deep learning calculation method and system based on heterogeneous sensor data fusion, and relates to the technical field of analyte concentration detection. The method comprises the following steps: acquiring an image sequence acquired by an optical sensor and a time sequence signal acquired by an electrochemical sensor, and preprocessing the image sequence and the time sequence signal; extracting space-time correlation characteristics of the image sequence and dynamic time sequence characteristics of the time sequence signal; performing cross-modal feature fusion on the space-time correlation features and the dynamic time sequence features to obtain cross-modal fusion features; after the cross-modal fusion features are input into a shared encoder for encoding, concentration predicted values and uncertainty parameters of the two to-be-analyzed objects are output; performing Bayesian uncertainty quantization through Monte Carlo sampling, and outputting a concentration mean value and a confidence interval; and constructing a multi-objective function, and realizing end-to-end training optimization through the multi-objective function. The key technical problems of heterogeneous sensor data fusion, analyte cross interference and the like can be effectively solved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of analyte concentration detection, and in particular to a deep learning calculation method and system for dual analyte concentrations based on heterogeneous sensor data fusion. Background Art

[0002] Current analyte concentration detection mainly relies on a single sensor (such as electrochemical, optical sensors) or simply connects multiple sensors in series. Common methods include:

[0003] 1. Single sensor independent calculation: Use a dedicated sensor for each analyte and output the concentration value separately;

[0004] 2. Traditional fusion methods: weighted averaging or linear regression modeling of multi-sensor data;

[0005] 3. Single-task deep learning: Use CNN / RNN to process single sensor data to predict a single analyte.

[0006] The above method has the following disadvantages:

[0007] 1. Cross-interference problem: A single sensor cannot distinguish the cross-response signals of two analytes, resulting in deviation in concentration calculation;

[0008] 2. Data heterogeneity remains unresolved: Heterogeneous sensor data, such as optical and electrochemical sensors, vary widely in temporal and spatial resolution, dimensionality, and noise patterns, making it difficult for traditional fusion methods to effectively integrate them.

[0009] 3. Low computational efficiency: Dual analytes require independent modeling, ignoring correlations and increasing computational redundancy;

[0010] 4. Poor dynamic adaptability: Traditional models are not robust enough to sensor drift and environmental noise. Summary of the Invention

[0011] The deep learning calculation method for dual analyte concentration based on heterogeneous sensor data fusion provided in the present disclosure can effectively solve key technical problems such as heterogeneous sensor data fusion and analyte cross-interference.

[0012] According to a first aspect of an embodiment of the present disclosure, a deep learning calculation method for dual analyte concentrations based on heterogeneous sensor data fusion is provided, the method comprising:

[0013] Acquire an image sequence acquired by an optical sensor and a time sequence signal acquired by an electrochemical sensor, and preprocess the image sequence and the time sequence signal;

[0014] Extracting spatiotemporal correlation features of the image sequence and dynamic temporal features of the temporal signal;

[0015] Performing cross-modal feature fusion on the spatiotemporal correlation feature and the dynamic time series feature to obtain a cross-modal fusion feature;

[0016] After the cross-modal fusion features are input into a shared encoder for encoding, the concentration prediction values ​​and uncertainty parameters of the two analytes are output;

[0017] Bayesian uncertainty quantification is performed through Monte Carlo sampling, and the concentration mean and confidence interval are output;

[0018] A multi-objective function is constructed, and end-to-end training optimization is achieved through the multi-objective function.

[0019] In one embodiment, the preprocessing of the image sequence and the timing signal includes:

[0020] Downsampling the image sequence to match the 3D-CNN input specification; performing wavelet denoising on the time series signal to eliminate original data noise;

[0021] The down-sampled image sequence is subjected to pixel normalization processing to eliminate reflectivity deviation caused by ambient lighting differences; the time series signal after wavelet denoising processing is subjected to z-score normalization processing to ensure that the multimodal data are in the same dimensional range.

[0022] In one embodiment, extracting the spatiotemporal correlation features of the image sequence and the dynamic temporal features of the temporal signal includes:

[0023] Using 3D-CNN architecture processing the image sequence, wherein the three-dimensional convolution kernel simultaneously captures the spatial texture change and the temporal diffusion process, models the convection-diffusion effect of the analyte in the microfluidic chip, and obtains the spatiotemporal correlation characteristics of the image sequence;

[0024] Using two-way The dynamic time series features are processed, and the forward and reverse dynamics of the redox reaction are learned using a forgetting gating mechanism to obtain the dynamic time series features of the time series signal.

[0025] In one embodiment, performing cross-modal feature fusion on the spatiotemporal correlation feature and the dynamic time series feature to obtain a cross-modal fusion feature includes:

[0026] Unifying the time scales of the spatiotemporal correlation features and the dynamic temporal series features through a dimension alignment operation;

[0027] The attention mechanism is used to calculate the interaction weights of the spatiotemporal correlation features and the dynamic time series features to achieve weighted fusion of the spatiotemporal correlation features and the dynamic time series features.

[0028] In one embodiment, unifying the time scales of the spatiotemporal correlation features and the dynamic temporal series features through a dimensional alignment operation includes:

[0029] The dynamic timing characteristics Extended along the time dimension to ,in, is the optical characteristic time scale;

[0030] The association strength between the optical feature point i and the electrochemical time step j is quantified based on a key-value attention mechanism; the key-value attention mechanism is:

[0031] ;

[0032] in, is the feature vector of the i-th spatial-temporal position of the optical feature (3D-CNN output); is the characteristic vector of the jth time step after the electrochemical characteristics are aligned and expanded, and The dimensions are the same; is the dimension of the attention mechanism’s key; and Both are electrochemical features, but they are generated by independent linear projection layers; is the attention weight, normalized by Softmax We get , which represents the degree of dependence of position i on time step j;

[0033] To align the features, it weights all electrochemical Value vectors for each position i .

[0034] In one embodiment, after inputting the cross-modal fusion feature into a shared encoder for encoding, outputting the concentration prediction values ​​and uncertainty parameters of the two analytes includes:

[0035] Construct a two-branch decoder, one branch decoder is used to output the predicted value of the first analyte concentration and uncertainty , another branch decoder is used to output the second analyte concentration prediction value and uncertainty ;

[0036] Common features are adopted via a shared base layer Extract reaction kinetics commonalities;

[0037] The dual-branch decoder uses ;

[0038] in, , corresponding to different analytes; is the true value of the regression concentration; The logarithmic standard deviation is fitted by the Softplus function.

[0039] In one embodiment, the Bayesian uncertainty quantification is performed by Monte Carlo sampling, and the output concentration mean and confidence interval include:

[0040] Enable Monte Carlo Dropout for 100 random samplings and calculate the mean of the concentration predictions. and standard deviation , the inference phase performs Monte Carlo sampling:

[0041] ;

[0042] Generate a 95% confidence interval:

[0043] .

[0044] In one embodiment, constructing a multi-objective function includes:

[0045] The multi-objective function is constructed by the first formula; wherein the first formula is:

[0046] ;

[0047] in, To minimize the concentration prediction error, To force the model uncertainty to match the empirical error, .

[0048] According to a second aspect of an embodiment of the present disclosure, a dual-analyte concentration deep learning calculation system based on heterogeneous sensor data fusion is provided, the system comprising: an acquisition module, an extraction module, a fusion module, an encoding module, a sampling module and an optimization module; wherein,

[0049] The acquisition module is used to acquire the image sequence acquired by the optical sensor and the time sequence signal acquired by the electrochemical sensor, and pre-process the image sequence and the time sequence signal;

[0050] The extraction module is used to extract the spatiotemporal correlation features of the image sequence and the dynamic time series features of the time series signal;

[0051] The fusion module is used to perform cross-modal feature fusion on the spatiotemporal correlation feature and the dynamic time series feature to obtain a cross-modal fusion feature;

[0052] The encoding module is used to input the cross-modal fusion features into a shared encoder for encoding, and then output the concentration prediction values ​​and uncertainty parameters of the two analytes;

[0053] The sampling module is used to perform Bayesian uncertainty quantification through Monte Carlo sampling and output the concentration mean and confidence interval;

[0054] The optimization module is used to construct a multi-objective function and implement end-to-end training optimization through the multi-objective function.

[0055] The present invention provides a deep learning calculation method for dual-analyte concentration based on heterogeneous sensor data fusion. The method first receives image sequences acquired by an optical sensor and time series signals acquired by an electrochemical sensor, and preprocesses them through normalization and wavelet denoising. Heterogeneous features are then extracted using a 3D convolutional network and a bidirectional LSTM, respectively. A cross-modal attention mechanism is then innovatively used to align and fusion feature spaces. The fused features are then compactly represented by a shared encoder and then input into a dual-task decoder to synchronously output the predicted concentration values ​​and uncertainty parameters of the two analytes. Bayesian uncertainty quantification is finally performed through Monte Carlo Dropout sampling, and the mean concentration value and 95% confidence interval are output. The entire model is then end-to-end trained and optimized using a multi-objective loss function, effectively solving key technical problems such as heterogeneous sensor data fusion and analyte cross-interference.

[0056] In addition, the deep learning calculation method for dual-analyte concentration based on heterogeneous sensor data fusion provided by the present disclosure has the following advantages: 1. It improves the calculation accuracy of dual-analyte concentration and can reduce the average error by ≥32%; 2. It solves the problems of dimensional mismatch and noise pattern differences in heterogeneous sensor data; 3. A single model synchronously outputs dual-analyte results, reducing computing resource consumption by 50%; 4. It enhances clinical scenario reliability through uncertainty quantification. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of a deep learning calculation method for dual analyte concentrations based on heterogeneous sensor data fusion provided for the implementation of the present disclosure.

[0058] Figure 2 Architectural diagram of a dual-analyte concentration deep learning computing system based on heterogeneous sensor data fusion provided for the implementation of the present disclosure. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of systems consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0060] Figure 1This is a flow chart of a deep learning calculation method for dual analyte concentration based on heterogeneous sensor data fusion provided by an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0061] Step 101: Acquire an image sequence acquired by an optical sensor and a time sequence signal acquired by an electrochemical sensor, and preprocess the image sequence and time sequence signal;

[0062] In one embodiment, the preprocessing of the image sequence and the timing signal includes:

[0063] Downsampling the image sequence to match the 3D-CNN input specification; performing wavelet denoising on the time series signal to eliminate original data noise;

[0064] The down-sampled image sequence is subjected to pixel normalization processing to eliminate reflectivity deviation caused by ambient lighting differences; the time series signal after wavelet denoising processing is subjected to z-score normalization processing to ensure that the multimodal data are in the same dimensional range.

[0065] In this step, the optical sensor image sequence and electrochemical sensor time series signals are input. Noise is removed from the raw data through spatial downsampling (optical data) and wavelet denoising (electrochemical data). Pixel normalization and z-score standardization are also performed to bring the multimodal data into the same dimensional range. This step provides a unified benchmark for subsequent feature extraction and addresses input bias caused by inherent sensor differences.

[0066] For optical sensor image sequences , spatially downsampled to 224×224 resolution to match the 3D-CNN input specification (VGG architecture baseline), followed by dynamic range normalization , eliminating the reflectivity deviation caused by ambient light differences. Electrochemical signal Symlets-4 wavelet basis is used for 3-layer decomposition and reconstruction to filter out high-frequency noise. The theoretical basis is that wavelet threshold denoising improves the signal-to-noise ratio by 18dB in the 0.1-100Hz frequency band, and then Z-score normalization is used. Unified dimensions. This process converges the multimodal data distribution to the intervals [0, 1] and [-1, 1], reducing gradient instability during subsequent network training.

[0067] Step 102: extracting the spatiotemporal correlation features of the image sequence and the dynamic time series features of the time series signal;

[0068] In one embodiment, extracting the spatiotemporal correlation features of the image sequence and the dynamic temporal features of the temporal signal includes:

[0069] Using 3D-CNN architecture processing the image sequence, wherein the three-dimensional convolution kernel simultaneously captures the spatial texture change and the temporal diffusion process, models the convection-diffusion effect of the analyte in the microfluidic chip, and obtains the spatiotemporal correlation characteristics of the image sequence;

[0070] Using two-way The dynamic time series features are processed, and the forward and reverse dynamics of the redox reaction are learned using a forgetting gating mechanism to obtain the dynamic time series features of the time series signal.

[0071] In this step, specialized networks adapted to physical properties are used to process multimodal data separately: a 3D convolutional network extracts the spatiotemporal correlations of optical data, while a bidirectional LSTM captures the dynamic temporal dependencies of electrochemical signals. These separate feature extraction pathways fully preserve the spatial distribution characteristics of optical imaging and the continuous variations of electrochemical reactions, laying the foundation for cross-modal fusion.

[0072] Specifically, the optical data adopts the 3D-CNN architecture , whose 3D convolution kernel (5×3×3) simultaneously captures the spatial texture changes and the temporal diffusion process (output ), modeling the convection-diffusion effect of analytes in microfluidic chips. The electrochemical signal is bidirectionally , using the forget gating mechanism to learn the forward and reverse dynamics of redox reactions (output 128-dimensional vector).

[0073] Step 103: performing cross-modal feature fusion on the spatiotemporal correlation feature and the dynamic time series feature to obtain a cross-modal fusion feature;

[0074] In one embodiment, performing cross-modal feature fusion on the spatiotemporal correlation feature and the dynamic time series feature to obtain a cross-modal fusion feature includes:

[0075] Unifying the time scales of the spatiotemporal correlation features and the dynamic temporal series features through a dimension alignment operation;

[0076] In one embodiment, unifying the time scales of the spatiotemporal correlation features and the dynamic temporal series features through a dimensional alignment operation includes:

[0077] The dynamic timing characteristics Extended along the time dimension to ,in, is the optical characteristic time scale;

[0078] The association strength between the optical feature point i and the electrochemical time step j is quantified based on a key-value attention mechanism; the key-value attention mechanism is:

[0079] ;

[0080] in, is the feature vector of the i-th spatial-temporal position of the optical feature (3D-CNN output); is the characteristic vector of the jth time step after the electrochemical characteristics are aligned and expanded, and The dimensions are the same; is the dimension of the attention mechanism’s key; and Both are electrochemical features, but they are generated by independent linear projection layers; is the attention weight, normalized by Softmax We get , which represents the degree of dependence of position i on time step j;

[0081] To align the features, it weights all electrochemical Value vectors for each position i .

[0082] The attention mechanism is used to calculate the interaction weights of the spatiotemporal correlation features and the dynamic time series features to achieve weighted fusion of the spatiotemporal correlation features and the dynamic time series features.

[0083] In this step, dimensional alignment is used to unify the feature timescales, and an attention mechanism is used to calculate the interaction weights between optical and electrochemical features, achieving adaptive feature weighted fusion. This step innovatively addresses the spatial-temporal mismatch in heterogeneous data, highlighting key information about the dual-analyte response.

[0084] Specifically, the electrochemical characteristics are first Extended along the time dimension to , solve the frame rate mismatch problem (T' is the optical feature time scale). Based on the key-value attention mechanism:

[0085] ;

[0086] in, From optical feature spatial position encoding, is the electrochemical feature projection. Weight Quantify the correlation strength between the reflectivity change per unit area of ​​optical imaging and the current response (experimentally measured r = 0.88). Final stitching features Covering 192 channels of spatiotemporal information.

[0087] is the feature vector of the i-th spatial-temporal position of the optical feature (3D-CNN output). Its physical meaning is: it represents the visual features of the optical sensor at a specific time frame and spatial position (such as pixel block), such as the local intensity change of the color reaction.

[0088] is the characteristic vector of the jth time step of the electrochemical feature (after alignment and expansion), ,in, is the dimension of the attention mechanism key. and The dimensions are the same. It represents the signal characteristics of the electrochemical sensor at time step j, such as the current transient response or impedance change.

[0089] and are all electrochemically characterized, but generated by separate linear projection layers, (Value dimension Can be equal to or different from ), which carries the deep information of electrochemical signals and is used for the actual content vector of weighted fusion.

[0090] is the scaling factor, i.e. the dimension size of the Key vector and the Query vector. , to prevent the dot product value from being too large due to high dimension, thereby stabilizing the gradient of Softmax output.

[0091] is the attention weight, which is normalized by Softmax It is obtained, which represents the degree of dependence of position i on time step j.

[0092] Nature: Satisfaction ,and , quantifies the strength of the association between optical feature point i and electrochemical time step j. For example: high → The color change at optical position i is strongly correlated with the electrochemical response j (such as the peak current of the enzyme reaction), → There is no significant coupling between the two.

[0093] To align the features, for each position i, the weighted comprehensive electrochemical Value vector , which is the same as the input Query feature (number of spatial locations × value dimension ), represents the cross-modal contextual information injected into the optical feature. For example, the optical feature of spatial position (i) is fused with the electrochemical correction signal associated with it.

[0094] Step 104: Input the cross-modal fusion features into a shared encoder for encoding, and output the concentration prediction values ​​and uncertainty parameters of the two analytes;

[0095] In one embodiment, after inputting the cross-modal fusion feature into a shared encoder for encoding, outputting the concentration prediction values ​​and uncertainty parameters of the two analytes includes:

[0096] Construct a two-branch decoder, one branch decoder is used to output the predicted value of the first analyte concentration and uncertainty , another branch decoder is used to output the second analyte concentration prediction value and uncertainty ;

[0097] Common features are adopted via a shared base layer Extract reaction kinetics commonalities;

[0098] The dual-branch decoder uses: ;

[0099] in, , corresponding to different analytes; is the true value of the regression concentration; The logarithmic standard deviation is fitted by the Softplus function.

[0100] In this step, the fused high-dimensional features are flattened, and a fully connected layer is used to extract high-level cross-modal abstract features. Layer normalization is then performed to stabilize the data distribution. This generates a low-dimensional, compact feature vector (dmodel = 256), eliminating redundant information while preserving the correlation between the two analyte concentrations and providing a shared knowledge representation for collaborative decoding.

[0101] Specifically, the fusion features are first flattened into (1568=28×28×2), input fully connected layer:

[0102] , this compression ratio (1568→256) forces the network to learn the latent variables of the dual analyte concentration coupling, followed by layer normalization: , in order to eliminate the distribution shift between samples.

[0103] Based on the shared underlying features, a two-branch decoder is constructed: branch 1 outputs the predicted value of analyte 1 concentration and uncertainty , branch 2 synchronously outputs analyte 2 and Parameter sharing reduces computational redundancy (50% more efficient than dual-independent models), while independent weight matrices are used to ensure analyte specificity.

[0104] Common features first pass through the shared base layer Extract common characteristics of reaction dynamics. The dual-branch decoder uses: ;

[0105] in, For different analytes, is the true value of the regression concentration, By fitting the logarithmic standard deviation through the Softplus function (guaranteed ).

[0106] Step 105: Perform Bayesian uncertainty quantification through Monte Carlo sampling and output the concentration mean and confidence interval;

[0107] In one embodiment, the Bayesian uncertainty quantification is performed by Monte Carlo sampling, and the output concentration mean and confidence interval include:

[0108] Enable Monte Carlo Dropout for 100 random samplings and calculate the mean of the concentration predictions. and standard deviation , the inference phase performs Monte Carlo sampling:

[0109] ;

[0110] Generate a 95% confidence interval:

[0111] .

[0112] In this step, Monte Carlo Dropout is enabled to perform 100 random samplings and calculate the mean of the concentration prediction. and standard deviation , the inference phase performs Monte Carlo sampling:

[0113] , generating a 95% confidence interval ,This Bayesian inference mechanism quantifies the model’s prediction reliability and provides a ,credibility assessment basis for clinical decision making.

[0114] Step 106: construct a multi-objective function, and implement end-to-end training optimization through the multi-objective function.

[0115] In one embodiment, constructing a multi-objective function includes:

[0116] The multi-objective function is constructed by the first formula; wherein the first formula is:

[0117] ;

[0118] in, To minimize the concentration prediction error, To force the model uncertainty to match the empirical error, .

[0119] In this step, a multi-objective loss function is constructed:

[0120] ;

[0121] in, To minimize the concentration prediction error, To force the model uncertainty to match the empirical error, , concentration error term Ensure prediction accuracy, uncertainty calibration term To improve the reliability of confidence interval, the regularization term ( ) Prevent overfitting, and back propagation realizes the coordinated update of parameters in the whole process.

[0122] The present invention provides a deep learning calculation method for dual-analyte concentration based on heterogeneous sensor data fusion. The method first receives image sequences acquired by an optical sensor and time series signals acquired by an electrochemical sensor, and preprocesses them through normalization and wavelet denoising. Heterogeneous features are then extracted using a 3D convolutional network and a bidirectional LSTM, respectively. A cross-modal attention mechanism is then innovatively used to align and fusion feature spaces. The fused features are then compactly represented by a shared encoder and then input into a dual-task decoder to synchronously output the predicted concentration values ​​and uncertainty parameters of the two analytes. Bayesian uncertainty quantification is finally performed through Monte Carlo Dropout sampling, and the mean concentration value and 95% confidence interval are output. The entire model is then end-to-end trained and optimized using a multi-objective loss function, effectively solving key technical problems such as heterogeneous sensor data fusion and analyte cross-interference.

[0123] In addition, the deep learning calculation method for dual-analyte concentration based on heterogeneous sensor data fusion provided by the present disclosure has the following advantages: 1. It improves the accuracy of dual-analyte concentration calculation and can reduce the average error by ≥32%; 2. It solves the problems of dimensional mismatch and noise pattern differences in heterogeneous sensor data; 3. A single model synchronously outputs dual-analyte results, reducing computing resource consumption by 50%; 4. It enhances clinical scenario reliability through uncertainty quantification.

[0124] Figure 2 This is an architecture diagram of a dual analyte concentration deep learning computing system based on heterogeneous sensor data fusion provided by an embodiment of the present disclosure. Figure 2As shown, the system includes: an acquisition module 201, an extraction module 202, a fusion module 203, an encoding module 204, a sampling module 205 and an optimization module 206; wherein the acquisition module 201 is used to acquire an image sequence acquired by an optical sensor and a time series signal acquired by an electrochemical sensor, and preprocess the image sequence and time series signal; the extraction module 202 is used to extract the spatiotemporal correlation features of the image sequence and the dynamic time series features of the time series signal; the fusion module 203 is used to perform cross-modal feature fusion on the spatiotemporal correlation features and the dynamic time series features to obtain cross-modal fusion features; the encoding module 204 is used to input the cross-modal fusion features into a shared encoder for encoding, and output the concentration prediction values ​​and uncertainty parameters of the two analytes; the sampling module 205 is used to perform Bayesian uncertainty quantification through Monte Carlo sampling, and output the concentration mean and confidence interval; the optimization module 206 is used to construct a multi-objective function and realize end-to-end training optimization through the multi-objective function.

[0125] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be pre-stored in a computer-readable storage medium. When executed, the program performs the steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0126] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0127] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A deep learning calculation method for dual analyte concentration based on heterogeneous sensor data fusion, characterized in that: The method comprises: Acquire an image sequence acquired by an optical sensor and a time sequence signal acquired by an electrochemical sensor, and preprocess the image sequence and the time sequence signal; Extracting spatiotemporal correlation features of the image sequence and dynamic temporal features of the temporal signal; Performing cross-modal feature fusion on the spatiotemporal correlation feature and the dynamic time series feature to obtain a cross-modal fusion feature; After the cross-modal fusion features are input into a shared encoder for encoding, the concentration prediction values ​​and uncertainty parameters of the two analytes are output; Bayesian uncertainty quantification is performed through Monte Carlo sampling, and the concentration mean and confidence interval are output; A multi-objective function is constructed, and end-to-end training optimization is achieved through the multi-objective function.

2. The method according to claim 1, characterized in that The preprocessing of the image sequence and the time sequence signal comprises: Downsampling the image sequence to match the 3D-CNN input specification; performing wavelet denoising on the time series signal to eliminate original data noise; The down-sampled image sequence is subjected to pixel normalization processing to eliminate reflectivity deviation caused by ambient lighting differences; the time series signal after wavelet denoising processing is subjected to z-score normalization processing to ensure that the multimodal data are in the same dimensional range.

3. The method according to claim 1, characterized in that The step of extracting the spatiotemporal correlation features of the image sequence and the dynamic time series features of the time series signal comprises: Using 3D-CNN architecture processing the image sequence, wherein the three-dimensional convolution kernel simultaneously captures the spatial texture change and the temporal diffusion process, models the convection-diffusion effect of the analyte in the microfluidic chip, and obtains the spatiotemporal correlation characteristics of the image sequence; Using two-way The dynamic time series features are processed, and the forward and reverse dynamics of the redox reaction are learned using a forgetting gating mechanism to obtain the dynamic time series features of the time series signal.

4. The method according to claim 1, wherein The cross-modal feature fusion of the spatiotemporal correlation feature and the dynamic time series feature to obtain the cross-modal fusion feature includes: Unifying the time scales of the spatiotemporal correlation features and the dynamic temporal series features through a dimension alignment operation; The attention mechanism is used to calculate the interaction weights of the spatiotemporal correlation features and the dynamic time series features to achieve weighted fusion of the spatiotemporal correlation features and the dynamic time series features.

5. The method according to claim 4, characterized in that The unification of the spatiotemporal correlation feature and the dynamic temporal feature time scale by the dimension alignment operation includes: The dynamic timing characteristics Extended along the time dimension to ,in, is the optical characteristic time scale; The association strength between the optical feature point i and the electrochemical time step j is quantified based on a key-value attention mechanism; the key-value attention mechanism is: ； in, is the feature vector of the i-th spatial-temporal position of the optical feature (3D-CNN output); is the characteristic vector of the jth time step after the electrochemical characteristics are aligned and expanded, and The dimensions are the same; is the dimension of the attention mechanism’s key; and Both are electrochemical features, but they are generated by independent linear projection layers; is the attention weight, normalized by Softmax We get , which represents the degree of dependence of position i on time step j; To align the features, it weights all electrochemical Value vectors for each position i .

6. The method according to claim 1, wherein After inputting the cross-modal fusion features into a shared encoder for encoding, outputting the concentration prediction values ​​and uncertainty parameters of the two analytes includes: Construct a two-branch decoder, one branch decoder is used to output the predicted value of the first analyte concentration and uncertainty , another branch decoder is used to output the second analyte concentration prediction value and uncertainty ; Common features are adopted via a shared base layer Extract reaction kinetics commonalities; The dual-branch decoder uses ; in, , corresponding to different analytes; is the true value of the regression concentration; The logarithmic standard deviation is fitted by the Softplus function.

7. The method according to claim 1, characterized in that The Bayesian uncertainty quantification is performed by Monte Carlo sampling, and the output concentration mean and confidence interval include: Enable Monte Carlo Dropout for 100 random samplings and calculate the mean of the concentration predictions. and standard deviation , the inference phase performs Monte Carlo sampling: ； Generate a 95% confidence interval: 。 8. The method according to claim 1, characterized in that The constructing of the multi-objective function comprises: The multi-objective function is constructed by the first formula; wherein the first formula is: ； in, To minimize the concentration prediction error, To force the model uncertainty to match the empirical error, .

9. A deep learning calculation system for dual analyte concentration based on heterogeneous sensor data fusion, characterized in that: The system includes: an acquisition module, an extraction module, a fusion module, an encoding module, a sampling module and an optimization module; wherein, The acquisition module is used to acquire the image sequence acquired by the optical sensor and the time sequence signal acquired by the electrochemical sensor, and pre-process the image sequence and the time sequence signal; The extraction module is used to extract the spatiotemporal correlation features of the image sequence and the dynamic time series features of the time series signal; The fusion module is used to perform cross-modal feature fusion on the spatiotemporal correlation feature and the dynamic time series feature to obtain a cross-modal fusion feature; The encoding module is used to input the cross-modal fusion features into a shared encoder for encoding, and then output the concentration prediction values ​​and uncertainty parameters of the two analytes; The sampling module is used to perform Bayesian uncertainty quantification through Monte Carlo sampling and output the concentration mean and confidence interval; The optimization module is used to construct a multi-objective function and implement end-to-end training optimization through the multi-objective function.

Images