An energy storage multi-modal data fusion method, system, device, medium and product

Abstract

This invention discloses a method, system, device, medium, and product for multimodal data fusion in energy storage, relating to the field of energy storage system data processing technology. It involves extracting time-series features from acquired multimodal data of an energy storage system, outputting feature vectors corresponding to each mode; calculating statistical quality indices based on the original sampled values ​​of each mode; inputting the statistical quality indices and feature vectors corresponding to each mode into a quality assessment network to generate quality scores for each mode; using the quality scores as bias terms for soft attention operations to obtain fusion weights, and then weighted and aggregated to generate fusion feature vectors; inputting the fusion feature vectors into a downstream task network, using the joint weighted value of task loss and entropy regularization loss as the total loss adjustment parameter, and using it for the next feature extraction. This invention adaptively allocates mode weights based on the true quality of the data, and corrects weight biases through end-to-end joint constraints of task loss and entropy regularization, thereby improving the quality of multimodal data fusion in energy storage.

Description

Technical Field

[0001] This invention relates to the field of energy storage system data processing technology, and in particular to a method, system, device, medium and product for multimodal data fusion in energy storage. Background Technology

[0002] The safe operation and condition assessment of energy storage power stations heavily rely on the comprehensive analysis of multi-modal sensing data, such as voltage, current, temperature, and state of charge. How to effectively integrate these multi-source heterogeneous data to support applications such as condition estimation and fault diagnosis is a core technical challenge in the field of intelligent operation and maintenance of energy storage. Existing multi-modal data fusion methods mainly include early fusion, late fusion, fixed-weight fusion, and fusion methods based on traditional attention mechanisms. Early fusion directly concatenates the original multi-modal data at the input layer and extracts features through a unified neural network, but it ignores the differences in data quality among different modalities and is easily contaminated by low-quality modalities. Late fusion extracts features from each modality separately and then performs simple averaging or voting at the decision layer. While this can isolate inter-modal interference, it is difficult to fully capture cross-modal correlation information. Fixed-weight fusion assigns preset fixed weights to each modality, which cannot adapt to dynamic changes in data quality. Although traditional attention mechanisms can automatically calculate modal weights, they do not consider data quality factors and may assign excessive weights to low-quality data, thus exacerbating fusion bias.

[0003] The aforementioned methods exhibit increasingly prominent inherent shortcomings when dealing with the complex and ever-changing operating environments of energy storage power stations. When modal data is missing, noisy, or experiences abrupt changes due to sensor malfunctions, packet loss, or strong electromagnetic interference, existing fusion methods lack effective weighting or elimination mechanisms, allowing low-quality data to directly contaminate the fusion features. Furthermore, different modal data vary in sensor accuracy and acquisition stability, and these differences dynamically change with the operating environment. However, existing methods either preset fixed fusion weights or calculate weights solely based on feature content, lacking the ability to dynamically perceive the quality of the modal data itself.

[0004] More critically, existing methods often employ a phased training strategy, training the feature extraction network first and then training the fusion module separately. This results in a lack of coordination between the quality assessment criteria and the downstream task objectives. Even when some methods adopt an end-to-end architecture, they often lack effective constraint mechanisms for the fusion weights (such as entropy regularization), preventing the model from accurately correcting the quality assessment network's misjudgments of low-quality data based on the task loss. This lack of a coordinated optimization mechanism between quality assessment and downstream task objectives directly leads to the inability to dynamically adjust quality perception based on task feedback, and a continuous deviation between the fusion weights and the data's true reliability, severely limiting the reliability of the fusion results. Summary of the Invention

[0005] This invention provides a method, system, device, medium, and product for multimodal data fusion in energy storage. It solves the technical problem that existing multimodal data fusion methods lack an end-to-end collaborative optimization mechanism between quality assessment standards and downstream task objectives, resulting in the inability to dynamically correct quality perception capabilities based on task feedback, continuous deviation between fusion weights and data authenticity and credibility, and consequently, insufficient reliability of fusion results.

[0006] The first aspect of this invention provides a method for multimodal data fusion in energy storage, comprising: Multiple modal data of the energy storage system are acquired, and time-series features are extracted from each modal data to output the feature vector corresponding to each mode. Based on the original sampled values ​​corresponding to each modal data, calculate the statistical quality index corresponding to each modality; The statistical quality index and feature vector corresponding to each mode are respectively input into the quality assessment network to perform mode quality assessment and generate the quality score corresponding to each mode; The quality scores corresponding to each modality are converted into quality bias terms, and soft attention operations are performed on the feature vectors corresponding to each modality to obtain the fusion weights corresponding to each modality. Based on the fusion weights corresponding to each modality, all the feature vectors are weighted and aggregated to generate a fused feature vector; The fused feature vector is input into the downstream task network, the task loss value output by the downstream task network is calculated, and the joint weighted value of the task loss value and the entropy regularization loss value of the fused weight is used as the total loss value. The parameters involved in the temporal feature extraction, modality quality assessment, soft attention operation, and downstream task network are adjusted based on the total loss value, and the adjusted parameters are used for the next execution of the temporal feature extraction.

[0007] Optionally, the step of extracting temporal features from each of the modal data and outputting the feature vector corresponding to each modality includes: The sampling frequency of each modal data is judged for consistency, and the modal data with inconsistent sampling frequencies are time-aligned to obtain the target modal data; Each target modality data is convolutionally mapped and nonlinearly transformed using a one-dimensional convolutional neural network corresponding to each target modality data, thereby generating a first temporal feature map corresponding to each modality. Each of the first temporal feature maps is downsampled to generate a second temporal feature map corresponding to each modality; Global downsampling is performed on each of the second time-series feature maps to generate global feature vectors corresponding to each mode; Each global feature vector is input into a fully connected layer for dimensionality transformation, and the feature vector corresponding to each modality is output.

[0008] Optionally, the step of calculating the statistical quality index corresponding to each modality based on the original sampled values ​​corresponding to each modality data includes: Calculate the ratio between the signal variance and the noise variance of each modal data, and use it as the signal-to-noise ratio for each modality. The ratio of missing sampling points to the total number of sampling points for each modality is used as the data integrity rate for each modality. The original sampled values ​​of each modality data are divided into sliding windows, the standard deviation of the original sampled values ​​within each sliding window is calculated, and the variance stability of each modality is determined based on the variance of each standard deviation. Obtain the current measured value of each modality data, and determine the physical consistency of each modality based on the deviation between the current measured value and the expected value of the physical model, combined with the reasonable value range corresponding to the modality to which the current measured value belongs.

[0009] Optionally, the step of inputting the statistical quality index and feature vector corresponding to each mode into the quality assessment network for mode quality assessment and generating a quality score corresponding to each mode includes: The statistical quality indices corresponding to each modality are combined into a statistical quality vector corresponding to each modality; The statistical quality vector corresponding to each mode is concatenated with the corresponding feature vector to generate the concatenated vector corresponding to each mode. The concatenated vectors corresponding to each modality are input into a multilayer perceptron for feature mapping to generate intermediate evaluation values ​​corresponding to each modality. An S-shaped function is used to map the intermediate evaluation values ​​corresponding to each mode to a preset interval to obtain the quality score corresponding to each mode.

[0010] Optionally, the step of converting the quality scores corresponding to each modality into quality bias terms, performing soft attention operations on the feature vectors corresponding to each modality, and obtaining the fusion weights corresponding to each modality includes: A global concatenation vector is constructed by concatenating the feature vectors corresponding to all the aforementioned modes. A linear mapping is performed on the global concatenation vector to generate a global query vector; The feature vectors corresponding to each mode are simultaneously used as the key vectors and value vectors corresponding to each mode; Stack the key vectors corresponding to all the modes to generate a key matrix, and stack the value vectors corresponding to all the modes to generate a value matrix; Add a preset value to the mass fraction corresponding to each mode and take the logarithm to generate the mass bias term corresponding to each mode; The product of the global query vector and the key matrix is ​​calculated, scaled, and added to the quality bias term corresponding to each modality to obtain the attention score corresponding to each modality. The attention scores corresponding to each modality are normalized to obtain the fusion weights corresponding to each modality.

[0011] Optionally, the step of inputting the fused feature vector into the downstream task network, calculating the task loss value output by the downstream task network, and using the joint weighted value of the task loss value and the entropy regularization loss value of the fused weights as the total loss value includes: Calculate the difference between the output data of the downstream task network and the corresponding task value to generate the task loss value; Calculate the negative of the entropy of the fusion weights corresponding to each mode to generate the entropy regularization loss value; The total loss value is generated by weighted summation of the task loss value and the entropy regularization loss value.

[0012] A second aspect of the present invention provides an energy storage multimodal data fusion system, comprising: The feature extraction module is used to acquire multiple modal data of the energy storage system, extract time-series features from each modal data, and output the feature vector corresponding to each mode. The statistical quality calculation module is used to calculate the statistical quality index corresponding to each modality based on the original sampled values ​​corresponding to each modality data. The quality assessment module is used to input the statistical quality indicators and feature vectors corresponding to each mode into the quality assessment network to perform mode quality assessment and generate a quality score for each mode. The attention calculation module is used to convert the quality scores corresponding to each of the modes into quality bias terms, perform soft attention operations on the feature vectors corresponding to each of the modes, and obtain the fusion weights corresponding to each of the modes. The fusion module is used to perform weighted aggregation on all the feature vectors according to the fusion weights corresponding to each modality to generate a fused feature vector; The loss calculation module is used to input the fused feature vector into the downstream task network, calculate the task loss value output by the downstream task network, and use the joint weighted value of the task loss value and the entropy regularization loss value of the fused weights as the total loss value. The parameter update module is used to adjust the parameters involved in performing the temporal feature extraction, the modality quality assessment, the soft attention operation, and the downstream task network based on the total loss value. The adjusted parameters are used for the next execution of the temporal feature extraction.

[0013] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the energy storage multimodal data fusion method described above.

[0014] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the energy storage multimodal data fusion method as described above.

[0015] The fifth aspect of the present invention provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer performs the energy storage multimodal data fusion method as described above.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: This invention achieves dynamic optimization of fusion weights and adaptive suppression of anomalous interference through a quality-aware driven soft attention mechanism and an end-to-end joint training strategy. First, temporal features are extracted from each modality's data, and statistical quality indicators such as signal-to-noise ratio, completeness, variance stability, and physical consistency are calculated based on the original sampled values ​​to quantitatively assess the reliability of each modality's data. Then, the statistical quality indicators and feature vectors are input into the quality assessment network to generate a quality score for each modality, which directly reflects the reliability of each modality's data at the current moment. Based on this, the method introduces the quality score as a bias term into the soft attention calculation, automatically assigning lower fusion weights to low-quality modalities with low quality scores and higher fusion weights to high-quality modalities, thus achieving dynamic adaptive adjustment of fusion weights according to the quality of the modality's data. When a modality is affected by sensor failure, packet loss, or strong electromagnetic interference, resulting in missing data, noise, or anomalous changes, its statistical quality indicators deteriorate and its corresponding quality score decreases. The fusion weight of this modality is automatically reduced, suppressing the contamination of fusion features by low-quality or anomalous data and avoiding the shortcomings of traditional fusion methods that are easily interfered with by inferior data. Simultaneously, this invention constructs a unified end-to-end computation graph integrating temporal feature extraction, modality quality assessment, soft attention weight calculation, and the downstream task network. The joint weighted value of the downstream task loss and the fusion weight entropy regularization loss is used as the total loss to complete global parameter iterative optimization. On one hand, the task loss gradient can be backpropagated to the quality assessment network and temporal feature extraction module, achieving coordinated optimization of quality assessment standards and downstream energy storage operation and maintenance task objectives, solving the problems of module disconnection and uncorrectable weight bias caused by traditional phased training. On the other hand, the entropy regularization loss constrains the distribution of fusion weights, preventing the model from over-relying on a single modality.

[0017] Through the aforementioned quality-aware-driven soft attention mechanism, the fusion weights no longer rely on manual presets or fixed rules, but are automatically adjusted according to the real-time changes in the data quality of each modality. Low-quality modalities are dynamically downweighted, while high-quality modalities are fused first, effectively solving the problem of the difficulty in dynamically optimizing fusion weights. Simultaneously, end-to-end collaborative training endows the quality assessment network with task-oriented perception capabilities, reducing the deviation between fusion weights and the true reliability of the data. This improves the robustness and reliability of multimodal fusion results in complex, variable, and abnormally induced energy storage operation scenarios. Finally, this invention outputs a highly reliable quality-aware fusion feature vector, providing reliable support for downstream tasks and improving the technical shortcomings of existing multimodal fusion methods, such as insufficient reliability and poor adaptability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the steps of a multimodal data fusion method for energy storage provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a multimodal data fusion method for energy storage provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the quality-perceived soft attention mechanism provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the 1D-CNN feature extractor network structure provided in an embodiment of the present invention; Figure 5 This is a schematic diagram comparing the performance of the multimodal fusion method provided in the embodiments of the present invention; Figure 6 This is a schematic diagram illustrating the robustness comparison under data anomaly scenarios provided in this embodiment of the invention. Figure 7 This is a schematic diagram of modal weight distribution under different operating conditions provided in the embodiments of the present invention; Figure 8 A structural block diagram of an energy storage multimodal data fusion system provided in an embodiment of the present invention; Figure 9 This is a structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0020] This invention provides a method, system, device, medium, and product for multimodal data fusion in energy storage, which addresses the technical problems in existing multimodal data fusion methods, such as the difficulty in dynamically adapting fusion weights to data quality and the lack of collaborative optimization mechanisms among feature extraction, quality assessment, and fusion decision modules. These problems result in model parameters being unable to adaptively match complex and ever-changing operating environments, leading to insufficient reliability of fusion results and low accuracy of state assessment.

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see Figure 1 , Figure 1 A flowchart illustrating the steps of a multimodal data fusion method for energy storage provided in an embodiment of the present invention.

[0023] This invention provides a method for multimodal data fusion in energy storage, comprising: Step 101: Obtain multiple modal data of the energy storage system, extract time-series features from each modal data, and output the feature vector corresponding to each mode.

[0024] It should be noted that energy storage system operation monitoring involves data in multiple modes, including voltage, current, temperature, and state of charge. Multimodal data refers to the collection of multi-source physical quantity data obtained from various sensors deployed at different monitoring nodes during the operation of an energy storage power station, specifically including voltage... Current ,temperature and historical state of charge Data with different modalities. Temporal feature extraction refers to using independent one-dimensional convolutional neural networks to extract features from the temporal sequences of data from each modality, converting the original signal into a fixed-dimensional feature vector representation. The feature vector is a low-dimensional dense vector representation obtained after mapping each modality's data through the feature extractor, used for subsequent quality assessment and attention calculation.

[0025] Figure 2 The overall system architecture of the present invention is shown. The system adopts a process architecture of multimodal input, feature extraction, quality assessment, attention weighting, and fusion output to downstream tasks. The feature extraction layer is composed of one-dimensional convolutional neural networks that are independent of each modality. Figure 4The document specifically demonstrates the network structure of a feature extractor that is independent for each modality. A single-modal feature extractor is constructed for single-modal input time-series signals, employing a three-layer one-dimensional convolutional neural network (1D-CNN) structure. The model input is single-modal time-series data. Specifically, the first layer is a Conv1D-32-k3 layer with ReLU activation function and a max pooling layer MaxPool(2) with a stride of 2, and the output size is compressed to... The second layer is a Conv1D-64-k3 layer with ReLU activation and a max-pooling layer MaxPool(2) with a stride of 2, further compressing the output size to... The third layer uses Conv1D-64-k3 with ReLU activation, without any further pooling operations. After extracting temporal local features from the three convolutional layers, a global average pooling layer (GAP) is applied to compress the length dimension into a 64-dimensional one-dimensional feature vector. This feature is then mapped through a fully connected FC-64 layer, ultimately outputting a single-modal feature vector with a fixed dimension of 64. .

[0026] In this embodiment of the invention, the specific parameter configuration of the feature extractor is as follows: the input temporal length is T. The first convolutional layer uses 128 convolutional kernels, and after convolution and pooling, the output dimension is T / 2×32, with 128 parameters; the second convolutional layer uses 64 convolutional kernels, and the output dimension is T / 4×64, with 6208 parameters; the third convolutional layer uses 64 convolutional kernels, and the output dimension is T / 4×64, with 12352 parameters; the fully connected layer maps the features to 64 dimensions, with an output dimension of 64 and 4160 parameters; the total number of parameters for the entire feature extractor is 22848.

[0027] Further, step 101 includes the following sub-steps: S11. Perform consistency judgment on the sampling frequency of each modal data, and perform time-series alignment on the modal data with inconsistent sampling frequencies to obtain the target modal data.

[0028] It should be noted that sampling frequency refers to the number of data points collected per second by each modal sensor. The sampling frequency may differ between different modalities (e.g., voltage and current typically use high-frequency sampling at the millisecond level, while temperature and historical SOC typically use low-frequency sampling at the second level). Consistency judgment refers to checking whether the sampling frequencies of the data from each modality are the same. Timing alignment refers to processing modal data with inconsistent sampling frequencies through interpolation or downsampling to ensure that the length of the input sequences for each modality is consistent.

[0029] In this embodiment of the invention, the original sampling frequency of each modality data is first obtained, and it is determined whether the sampling frequencies of each modality are consistent. If they are inconsistent, the modality data with the higher sampling frequency is downsampled, and the modality data with the lower sampling frequency is interpolated to make the sequence length of all modalities the same, thus obtaining the target modality data. Through temporal alignment, the length difference of each modality data in the time dimension is eliminated, providing a uniform scale input for the subsequent feature extraction of the one-dimensional convolutional neural network.

[0030] S12. Using a one-dimensional convolutional neural network corresponding to each target modality data, perform convolutional mapping and nonlinear transformation on each target modality data to generate the first temporal feature map corresponding to each modality.

[0031] Each modality corresponds to an independent one-dimensional convolutional neural network; that is, voltage, current, temperature, and historical SOC each have independent network parameters that are not shared. Convolutional mapping refers to the operation of extracting local features from the input sequence by sliding the convolutional kernel along the temporal dimension. Nonlinear transformation refers to applying a nonlinear mapping to the convolutional output through an activation function to enhance the network's expressive power.

[0032] In this embodiment of the invention, the data of each target modality are input into its corresponding one-dimensional convolutional neural network. For example... Figure 4 In the network structure shown, each modality feature extractor adopts the same convolution-activation-pooling stacked architecture. It extracts the local temporal features of each modality by sliding the convolution kernel in the temporal dimension, and enhances the feature expression through a nonlinear activation function to generate the first temporal feature map corresponding to each modality.

[0033] S13. Downsample each of the first time-series feature maps to generate the second time-series feature maps corresponding to each mode.

[0034] It should be noted that downsampling refers to reducing the temporal resolution of the feature map through pooling operations (such as max pooling), thereby reducing the amount of data and computational complexity while preserving the main features.

[0035] In this embodiment of the invention, the first temporal feature map is input into the pooling layer for downsampling. For example... Figure 4 As shown, by sliding the pooling window along the temporal dimension, the maximum value within the window is selected as the output of that window, thus achieving feature dimensionality reduction. After downsampling, a second temporal feature map corresponding to each modality is generated, whose temporal length is further reduced compared to the first temporal feature map.

[0036] S14. Perform global downsampling on each of the second time-series feature maps to generate global feature vectors corresponding to each mode.

[0037] It should be noted that global downsampling refers to compressing the entire temporal feature map into a single vector along the time dimension using global average pooling. The global feature vector is a fixed-length vector obtained after compressing the entire temporal dimension, which captures the overall feature information of the modality data across all time steps.

[0038] In this embodiment of the invention, the second temporal feature map is input into a global average pooling layer. For example... Figure 4 As shown, the global average pooling layer averages the values ​​of each feature channel across all time steps, generating a global feature vector for each modality. The dimension of the global feature vector is the same as the number of feature channels, and it does not depend on the length of the input sequence, thus realizing the transformation from variable-length time series to fixed-length feature vectors.

[0039] S15. Input each global feature vector into the fully connected layer for dimensionality transformation, and output the feature vector corresponding to each modality.

[0040] It should be noted that a fully connected layer is a linear transformation layer that establishes a full connection between each element of the input vector and each element of the output vector. Dimensional transformation refers to transforming the global feature vector from its original dimension to a predetermined target dimension through the linear mapping of the fully connected layer.

[0041] In this embodiment of the invention, the global feature vectors of each modality are respectively input into the corresponding fully connected layers. For example... Figure 4 As shown, the fully connected layer maps the global feature vector to a preset target dimension through a learnable weight matrix. Each modality outputs a feature vector of a fixed dimension, which is 64. This can be expressed by the formula: ; in, For modality m The extracted feature vector; For modality m The corresponding one-dimensional convolutional neural network feature extractor; The input timing data for mode m; m For modal indexes, the value set is: , representing voltage, current, temperature, and historical SOC, respectively. The specific feature vectors of each mode output are: ; in, These are the eigenvectors corresponding to the voltage modes; These are the eigenvectors corresponding to the current modes; These are the eigenvectors corresponding to the temperature modes; These are the feature vectors corresponding to historical SOC modes; This indicates that each eigenvector is a 64-dimensional real vector.

[0042] Step 102: Calculate the statistical quality index corresponding to each modality based on the original sampled values ​​corresponding to each modality.

[0043] It should be noted that raw sampled values ​​refer to the unprocessed raw data directly collected by each modal sensor. Statistical quality indicators are quantitative values ​​calculated based on the raw sampled values ​​to evaluate the quality of each modal data, specifically including four dimensions: signal-to-noise ratio, data integrity, variance stability, and physical consistency. These indicators comprehensively reflect the reliability of each modal data from four aspects: signal strength, data integrity, temporal stationarity, and physical plausibility.

[0044] like Figure 3 In the dual-path quality assessment network structure shown in (a), the statistical quality index calculation branch starts from the original sampled values ​​and generates four quality indices after statistical calculation. This branch runs in parallel with the feature extraction path and together constitutes the dual-path input of the quality assessment network.

[0045] Further, step 102 includes the following sub-steps: S21. Calculate the ratio between the signal variance and the noise variance of each modality data, and use it as the signal-to-noise ratio for each modality.

[0046] Signal variance refers to the variance of the effective signal components in the original sampled values, reflecting the signal strength. Noise variance refers to the variance of the noise components in the original sampled values, reflecting the degree of interference to the signal. Signal-to-noise ratio (SNR) is the ratio of signal variance to noise variance, used to measure the strength of the signal relative to noise; a higher SNR indicates better signal quality.

[0047] In this embodiment of the invention, the signal variance and noise variance are calculated for the original sampled values ​​of each modality data, and the signal-to-noise ratio (SNR) is obtained by dividing the two. The formula for calculating the SNR is: SNR is the signal-to-noise ratio, and its unit is decibel (dB). The variance of the signal; This represents the noise variance.

[0048] S22. Calculate the ratio of missing sampling points in each modality to the total number of sampling points, and use it as the data integrity rate for each modality.

[0049] Missing sampling points refer to the number of data points that could not be successfully collected during the data acquisition process due to sensor failure, packet loss, or communication interruption. Total sampling points refer to the total number of data points that should theoretically have been collected.

[0050] In this embodiment of the invention, missing data is detected in the original sampled values ​​of each modality data. The number of missing sampled points in each modality data and the total number of sampled points are counted, and the proportion of missing sampled points to the total number of sampled points is calculated. The formula for calculating the data integrity rate is as follows: ,in For data integrity rate; This represents the number of missing sampling points. This represents the total number of sampling points.

[0051] S23. Divide the original sampled values ​​of each modality into sliding windows, calculate the standard deviation of the original sampled values ​​within each sliding window, and determine the variance stability of each modality based on the variance of each standard deviation.

[0052] Sliding window partitioning refers to gradually sliding a fixed window across time series data to divide the entire time series into multiple overlapping or non-overlapping window intervals. Standard deviation is a measure of the dispersion of the original sampled values ​​within each window relative to the window mean. Variance stability refers to the variance of the standard deviations of each window, used to measure the smoothness of the data's temporal fluctuations; the smaller the variance stability, the more stable the data.

[0053] In this embodiment of the invention, the original sampled values ​​of each modality data are divided into sliding windows according to a preset window length. The standard deviation of the sampled values ​​within each window is calculated, and then the variance of the standard deviations of all windows is calculated. The formula for calculating the variance stability is as follows: ,in, For variance stability; This represents the variance of the standard deviation within each sliding window.

[0054] S24. Obtain the current measured value of each modality data, and determine the physical consistency of each modality based on the deviation between the current measured value and the expected value of the physical model, combined with the reasonable value range corresponding to the modality to which the current measured value belongs.

[0055] The current measured value refers to the actual measured value of the modal data at the current moment. The expected value of the physical model refers to the theoretical expected value of the modal data under a given operating condition, calculated based on a physical model or empirical rules. The reasonable value range refers to the normal numerical range that the modal data may physically occur within (e.g., voltage range). Temperature range is Physical consistency refers to the degree to which current measured values ​​deviate from the expected values ​​of the physical model, and is used to measure the physical reasonableness of the data.

[0056] In this embodiment of the invention, the measured value at the current moment of the original sampled value of each modality data, along with the corresponding expected value of the physical model and the reasonable value range of that modality, are obtained. The deviation between the current measured value and the expected value of the physical model is calculated, and this deviation is normalized by dividing it by the reasonable value range of that modality. The formula for calculating physical consistency is: ,in, For physical consistency; This is the current measured value; This represents the expected value of the physical model. This represents the reasonable range of values ​​for this modal data.

[0057] Step 103: Input the statistical quality index and feature vector corresponding to each modality into the quality assessment network to perform modality quality assessment and generate the quality score corresponding to each modality.

[0058] The statistical quality indicators are the four statistical quality indicators calculated in step 102. The feature vectors are the 64-dimensional feature vectors corresponding to each modality output in step 101. The quality assessment network is a neural network model used to fuse the statistical quality indicators and feature vectors and map them to a quality score; its structure is a multilayer perceptron. The quality score is a numerical value ranging from [0, 1], representing the reliability of the data in each modality; the higher the score, the more reliable the data in that modality.

[0059] like Figure 1 In the overall system architecture shown, this step corresponds to the quality assessment module. For example... Figure 3 As shown in (a), the quality assessment network is configured with two parallel input branches, one of which inputs the raw data of a single mode. A four-dimensional statistical index vector was obtained through statistical index extraction. The modal features are input to the single-modal feature extractor. The two output features are concatenated and fed into a multilayer perceptron (MLP), then activated by a sigmoid function. The mapping yields the mass fraction corresponding to this mode. The mass fractions of all modes are combined into a mass vector. H It is then transported to the soft attention computing module via the quality transfer link.

[0060] Furthermore, step 103 includes the following sub-steps: S31. Combine the statistical quality indices corresponding to each modality into a statistical quality vector corresponding to each modality.

[0061] The statistical quality vector is a vector formed by concatenating four statistical quality indicators—signal-to-noise ratio, data integrity, variance stability, and physical consistency—in a fixed order.

[0062] In this embodiment of the invention, the four statistical quality indicators (signal-to-noise ratio, data integrity, variance stability, and physical consistency) calculated in S21 to S24 are combined sequentially to form the statistical quality vector corresponding to each modality: ,in For modality The corresponding statistical quality vector; For modality Signal-to-noise ratio; For modality Data integrity rate; For modality The variance stability; For modality Physical consistency; This indicates that the vector is a 4-dimensional real vector.

[0063] S32. Concatenate the statistical quality vector corresponding to each mode with the corresponding feature vector to generate the concatenated vector corresponding to each mode.

[0064] In this embodiment of the invention, the modal statistical quality vectors generated in S31 are concatenated with the corresponding modal feature vectors generated in step 101 to generate concatenated vectors corresponding to each modality, denoted as follows: ,in For modality Statistical quality vector; For modality eigenvectors.

[0065] S33. Input the concatenated vectors corresponding to each modality into a multilayer perceptron for feature mapping to generate intermediate evaluation values ​​corresponding to each modality.

[0066] A multilayer perceptron (MLP) is a feedforward neural network architecture that consists of an input layer, one or more hidden layers, and an output layer. It achieves a non-linear transformation of features through a non-linear activation function. Intermediate evaluation values ​​refer to the feature representations before the output layer of the MLP; they are the raw evaluation scores without the activation function mapping.

[0067] In this embodiment of the invention, the concatenated vector generated in S32 is input into a multilayer perceptron for feature mapping. For example... Figure 3 As shown in (a), the multilayer perceptron structure is 68→32→16→1, which gradually reduces the input dimension from 68 dimensions to 1 dimension. Through multilayer nonlinear transformation, the deep correlation between statistical indicators and feature vectors is extracted, and intermediate evaluation values ​​corresponding to each modality are generated.

[0068] S34. Use an S-shaped function to map the intermediate evaluation values ​​corresponding to each mode to a preset interval to obtain the quality score corresponding to each mode.

[0069] The sigmoid function is a non-linear activation function that maps any real number to the interval (0, 1), and its output monotonically increases as the input value increases. The preset interval is [0, 1], which is the range of values ​​for the quality score.

[0070] In this embodiment of the invention, the intermediate evaluation values ​​generated by S33 are input into a sigmoid function for nonlinear mapping, compressing the evaluation values ​​to the [0, 1] interval, and outputting the quality scores corresponding to each mode. The calculation method is as follows: ,in For modality The mass fraction; The function is a sigmoid function; MLP stands for Multilayer Perceptron. For modality The concatenated vectors. The sigmoid function ensures... A quality score closer to 1 indicates that the modality data is more reliable, while a quality score closer to 0 indicates that the modality data is less reliable.

[0071] It is worth mentioning that the quality score is divided into different levels according to its value range. The data quality meaning and processing strategy corresponding to each level are as follows: When the quality score is in the range of 0.8 to 1.0, it indicates that the data quality of the modality is high and it is given normal weight during fusion; when the quality score is in the range of 0.5 to 0.8, it indicates that the data quality of the modality is medium and it is moderately reduced during fusion; when the quality score is in the range of 0.2 to 0.5, it indicates that the data quality of the modality is low and it is significantly reduced during fusion; when the quality score is in the range of 0 to 0.2, it indicates that the data quality of the modality is extremely low and its weight is kept close to a silent state during fusion.

[0072] Step 104: Convert the quality scores corresponding to each modality into quality bias terms, perform soft attention operations on the feature vectors corresponding to each modality, and obtain the fusion weights corresponding to each modality.

[0073] The quality score is the quality score (range [0, 1]) corresponding to each modality generated in step 103. The bias term refers to the additional offset added during attention calculation to adjust the attention score of each modality. Soft attention operation is a differentiable attention mechanism that generates a normalized attention weight distribution by calculating the similarity between the query vector and the key vector and introducing a quality bias term. The fusion weight refers to the proportion of each modality feature in the final fusion, with the sum of all weights equal to 1.

[0074] like Figure 2 In the overall system architecture shown, this step corresponds to the attention calculation module. For example... Figure 3 As shown in (b), this module receives multimodal features. mass vector passed to the left H First, an attention query Q, key K, and value V are constructed based on multimodal features, and simultaneously, an attention vector is applied. Complete the quality modulation; then calculate the scaled dot product attention base score. The base score is added to the quality modulation term and then fed into Softmax normalization to output the soft attention weights corresponding to each modality. The overall calculation expression is: .

[0075] Furthermore, step 104 includes the following sub-steps: S41. Concatenate the feature vectors corresponding to all modes to construct a global concatenation vector.

[0076] A global concatenation vector is a long vector obtained by concatenating the feature vectors of all modalities in sequence, and is used to generate a global query vector.

[0077] In this embodiment of the invention, all modal feature vectors generated in step 101 are concatenated sequentially to construct a global concatenated vector: ,in This is a globally concatenated vector; These are the feature vectors corresponding to voltage, current, temperature, and historical SOC modes, respectively. This indicates that the vector is a 256-dimensional real vector (each of the four modes is 64-dimensional, and the concatenation results in a 256-dimensional vector).

[0078] S42. Perform a linear mapping on the global concatenation vector to generate a global query vector.

[0079] The global query vector is a query vector generated from the global concatenation vector through a linear mapping, and is used to calculate the similarity with the key vectors of each modality in the attention mechanism.

[0080] In this embodiment of the invention, the global concatenation vector constructed in S41 is... With learnable query projection matrix Multiply to generate a global query vector: ,in This is the global query vector; The projection matrix is ​​a learnable parameter matrix. This is a globally concatenated vector; In this embodiment, the feature dimension of the query vector is... .

[0081] S43. Use the feature vectors corresponding to each mode as both the key vector and the value vector for each mode.

[0082] The key vector is a vector representation used to calculate attention similarity with the query vector. The value vector is a vector representation used for weighted summation and aggregation.

[0083] In this embodiment of the invention, the feature vector of each modality simultaneously serves as both a key vector and a value vector, that is... and ,in For modality The key vector; For modality The value vector, For modality The eigenvectors. For example... Figure 3As shown in (b), to reduce the number of parameters, the additional linear projection layer is omitted, and the modal features are directly used as key vectors and value vectors. ,in The feature dimension of the value vector.

[0084] S44. Stack the key vectors corresponding to all modes to generate a key matrix, and stack the value vectors corresponding to all modes to generate a value matrix.

[0085] The key matrix is ​​a matrix formed by stacking all modal key vectors. The value matrix is ​​a matrix formed by stacking all modal value vectors.

[0086] In this embodiment of the invention, the key vectors of each mode are stacked sequentially to generate a key matrix. The value vectors of each mode are stacked in the same order to generate a value matrix. : ; ; in, K The key matrix; V It is a value matrix; This is a stacking operation, which stacks multiple vectors row-wise into a matrix; These are the feature vectors for each mode.

[0087] S45. Add the preset value to the mass fraction corresponding to each mode and take the logarithm to generate the mass bias term corresponding to each mode.

[0088] The default value is a very small positive number (typically taking the value of...). This is used to prevent numerical overflow during logarithmic operations when the quality score is close to zero. The quality bias term refers to the value obtained by adding a preset value to the quality score and then taking the logarithm, which is injected as a bias term for attention calculation.

[0089] In this embodiment of the invention, the modal mass fractions generated in step 103 are added to a preset value ε, and the natural logarithm is taken to generate the mass bias term corresponding to each mode: ,in, The mass fraction of the modality; Numerical stability constant (typical value) This is used to prevent numerical overflow during logarithmic operations when the mass fraction is close to zero; It is the natural logarithm function. For example... Figure 3 As shown in (b), the low-quality mode corresponds to The value is small (tending towards negative infinity), which suppresses its attention score.

[0090] S46. Calculate the product of the global query vector and the key matrix, scale it, and add it to the quality bias term corresponding to each modality to obtain the attention score corresponding to each modality.

[0091] Attention score is a numerical value that measures the degree of matching between each modality feature and the global query vector, and is calculated by scaling dot product attention.

[0092] In this embodiment of the invention, the product of the global query vector Q and the transpose of the key matrix K is first calculated. This yields the raw attention scores for each modality. Then, these scores are divided by the scaling factor. Scale the result to prevent the dot product from becoming too large and causing gradient vanishing. Finally, compare the scaled result with the mass bias term generated by S45. Add element by element. The calculation method is as follows: ; in, Attention score; Q This is the global query vector; This is the transpose of the key matrix; The feature dimension of the key vector; This is the scaling factor; This is the mass fraction vector for all modes. ; is the numerical stability constant.

[0093] S47. Normalize the attention scores corresponding to each modality to obtain the fusion weights corresponding to each modality.

[0094] Normalization refers to using the Softmax function to map attention scores to a probability distribution. Fusion weights are the weight values ​​corresponding to each modality after normalization; the sum of the fusion weights for all modalities is 1.

[0095] In this embodiment of the invention, the attention scores of each modality obtained in S46 are input into the Softmax function for normalization, and the scores are converted into a non-negative probability distribution that sums to 1, thus obtaining the fusion weights of each modality: ; in, To fuse the weight vectors, ; It is a normalized exponential function; This is the transpose of the key matrix; The feature dimension of the key vector; This is the scaling factor; This is the mass fraction vector for all modes. ; Let be the numerical stability constant. The fusion weights satisfy . That is, the sum of all weights is 1.

[0096] Step 105: Weight all feature vectors according to the fusion weights corresponding to each modality to generate a fused feature vector.

[0097] The fusion weights are the attention weights corresponding to each modality generated in step 104. The feature vectors are the 64-dimensional feature vectors corresponding to each modality generated in step 101. Weighted aggregation refers to multiplying the feature vectors of each modality by their corresponding fusion weights and then summing them. The fusion feature vector is the unified feature representation obtained after weighted aggregation, which integrates information from each modality.

[0098] In embodiments of the present invention, such as Figure 2 As shown, this step corresponds to the fusion layer in the overall system architecture diagram. The fusion weights of each modality generated in step 104 are multiplied by the corresponding modal feature vectors generated in step 101. Then, the weighted results of all modalities are summed to generate the fusion feature vector: ; in, To fuse feature vectors; For modality The fusion weight; For modality eigenvectors; Traversal This fused feature vector integrates information from all modalities, and the higher the quality of the modality, the greater its weight in the fused feature.

[0099] It is worth mentioning that, in addition to the basic weighted summation fusion, the present invention also provides the following optional enhanced fusion methods.

[0100] In one alternative embodiment, a gating mechanism is introduced to adjust the fused features dimension by dimension. Specifically, the feature vectors corresponding to all modalities are concatenated into a global concatenation vector. ,Will The input gate layer is mapped to the [0, 1] interval by a sigmoid function to generate the gate vector. : ,in, This is the gate layer weight matrix. For gating layer bias. It is an S-shaped function. Then the gating vector is... Multiply the result of the weighted sum element by element to generate the fused feature vector: ,in, This involves element-wise multiplication. By adjusting the gating vector dimension by dimension, the model can assign different levels of attention to different feature dimensions, enhancing the flexibility of the fusion process.

[0101] In another alternative embodiment, residual connections are introduced, and the average value of each modality feature is superimposed on the weighted summation result to preserve the basic information of each modality: ,in, This is the residual intensity coefficient, typically ranging from [0.1, 0.3], used to balance the contributions of attention-weighted fusion and mean residuals. It can be set as a fixed hyperparameter or a learnable parameter; To obtain the mean eigenvector, we take the element-wise average of the eigenvectors of each modality.

[0102] It should be noted that gated fusion and residual connection are optional enhancement schemes, not steps necessary to achieve the present invention. In the basic embodiment, quality-aware multimodal data fusion can be achieved by using weighted summation fusion; in application scenarios that require further improvement in fusion flexibility, one or more of the above enhancement schemes can be selected and used in combination according to actual needs, depending on the trade-off between fusion accuracy and computational cost in the actual application scenario.

[0103] Step 106: Input the fused feature vector into the downstream task network, calculate the task loss value output by the downstream task network, and use the joint weighted value of the task loss value and the entropy regularization loss value of the fused weights as the total loss value.

[0104] The downstream task network is a neural network used to perform specific application tasks. Depending on the application scenario, it can be a SOC estimation network or a fault detection network, etc. The task loss value is the loss calculated based on the difference between the output of the downstream task network and the true task value. The entropy regularization loss value is the negative of the entropy calculated based on the fusion weights of each modality, used to prevent the fusion weights from becoming overly concentrated on a single modality. The total loss value is the joint weighted value of the task loss value and the entropy regularization loss value, used to guide the end-to-end optimization of the entire fusion network.

[0105] like Figure 2 As shown, this step corresponds to the task layer in the overall system architecture diagram.

[0106] Furthermore, step 106 includes the following sub-steps: S61. Calculate the difference between the output data of the downstream task network and the corresponding task value to generate the task loss value.

[0107] Output data refers to the prediction results output after forward propagation of the downstream task network. Task value refers to the true label (ground value) corresponding to the prediction result. Task loss value is a quantitative value that measures the degree of difference between the prediction result and the ground value.

[0108] In this embodiment of the invention, the fused feature vector generated in step 105 is input into the downstream task network for forward computation to obtain the network's output prediction value. An appropriate loss function is selected based on the downstream task type: for a SOC estimation task, the Mean Squared Error (MSE) loss function is used. ,in, This represents the task loss value. Mean square error function The SOC estimate predicted by the model; This represents the true SOC value. For fault detection tasks, the cross-entropy loss function is used: ,in, The cross-entropy loss function; The class distribution predicted by the model; This provides the truth labels for the fault categories. The task loss value is generated by calculating the difference between the network output and the task truth.

[0109] S62. Calculate the negative of the entropy of the fusion weights corresponding to each modality, and generate the entropy regularization loss value.

[0110] Entropy is a metric in information theory that measures the uniformity of a probability distribution. The more uniform the distribution, the higher the entropy value; the more concentrated the distribution, the lower the entropy value. The entropy regularization loss is the negative of the entropy of the fused weight distribution, used to encourage the weight distribution to tend towards uniformity.

[0111] In this embodiment of the invention, based on the modal fusion weights generated in step 104 Calculate the logarithm of each weight, sum them up, and then take the negative of the sum to generate the entropy regularization loss value. ,in This is the entropy regularization loss value; For modality The fusion weight; It is the natural logarithm function; This is the numerical stability constant (typically taken as follows). This prevents logarithmic overflow. When the weights are evenly distributed, Taking the maximum value encourages the model to utilize complementary information from multiple modalities; when weights are concentrated on a single modality, Take the smaller value.

[0112] S63. The task loss value and the entropy regularization loss value are weighted and summed to generate the total loss value.

[0113] The total loss value is a weighted sum of the task loss value and the entropy regularization loss value, used for end-to-end optimization of the entire fusion network.

[0114] In this embodiment of the invention, the task loss value generated in S61 and the entropy regularization loss value generated in S62 are weighted and summed to generate the total loss value: ,in, This represents the total loss value. This represents the task loss value. This is the entropy regularization loss value; This is the entropy regularization coefficient, typically 0.1; the negative sign is used to maximize entropy (i.e., to encourage uniform weight distribution).

[0115] In an optional embodiment that includes quality consistency loss, the total loss function is: ,in This is the quality consistency loss coefficient, typically taken as 0.05; This results in a loss of quality consistency. pass The calculation yielded, where This is the binary cross-entropy (BCE). Modal output for quality assessment network The mass fraction; For modality The loss is used to guide the quality score output by the quality assessment network to be consistent with the manually labeled data quality tags.

[0116] Step 107: Adjust the parameters involved in the execution of temporal feature extraction, modality quality assessment, soft attention operation and downstream task network based on the total loss value. The adjusted parameters are used for the next execution of temporal feature extraction.

[0117] The total loss value is the total loss value generated in step 106. Parameters refer to the parameters of the one-dimensional convolutional neural network performing temporal feature extraction, the parameters of the multilayer perceptron in the quality assessment network, the query projection matrix parameters in the soft attention operation, and the parameters of the downstream task network. End-to-end optimization refers to the joint training of all modules in the entire fusion process as a whole, updating the parameters of all modules simultaneously through a single backpropagation to achieve global optimum.

[0118] like Figure 2In the overall system architecture shown, this step corresponds to the backpropagation path in the diagram. The total loss value is propagated back through the gradient layer by layer to the feature extraction layer, quality assessment module, attention calculation module, and downstream task network via the backpropagation algorithm, achieving synchronous updates of all trainable parameters. The adjusted parameters are used for the next iteration, and this process is repeated until the total loss value converges. The entire fusion process is differentiable, supports joint training with downstream tasks, and achieves global optimization. Because the quality assessment network participates in end-to-end training, the quality score no longer relies solely on purely physical indicators such as signal-to-noise ratio, but can be dynamically adjusted based on whether it is useful for the final task, achieving task-aware quality assessment.

[0119] In this embodiment of the invention, the total loss value generated in step 106 is used as the basis. The gradients of the parameters of each module are calculated using the backpropagation algorithm, and all trainable parameters of the entire fusion network are updated using an optimizer (such as Adam). The adjusted parameters are used for the next execution of the temporal feature extraction in step 101, and this process is repeated iteratively until the total loss value converges. In this embodiment of the invention, the typical parameter configurations of the above modules are as follows: the number of convolutional kernels of the feature extractor is set to [32, 64, 64], that is, the number of convolutional kernels in the three convolutional layers are 32, 64, and 64 respectively, and the output dimension is set to 64; the hidden dimension of the quality evaluation network is set to [32, 16], that is, the dimensions of the two hidden layers of the multilayer perceptron are 32 and 16 respectively; the key vector feature dimension (Key dimension) of the attention mechanism is set to 64; the activation function of the gating layer is the sigmoid function; and the learning rate during training is set to 1×10. - ³, Batch size is set to 64. The above parameters are typical preferred configurations, and those skilled in the art can make appropriate adjustments according to actual application scenarios and data scale.

[0120] To further illustrate the technical effects of the method of the present invention, the present invention will be verified in detail below with reference to specific application scenarios and experimental data.

[0121] Example 1: Multimodal fusion in SOC estimation task; This embodiment applies the method of the present invention to the task of estimating the state of charge (SOC) of an energy storage system, and integrates four-modal data of voltage, current, temperature and historical SOC to estimate the SOC at the current moment.

[0122] The data configuration is as follows: the input consists of time-series data for four modes: voltage, current, temperature, and historical SOC; the output is the estimated SOC value at the current moment. The training set contains 10,000 samples, and the test set contains 2,000 samples. The historical SOC can be derived from the estimate of the previous moment or a coarse estimate using the Coulomb integral method.

[0123] The model is configured as follows: the feature dimension is set to 64; the quality assessment network structure is 68-dimensional input, which is passed through two hidden layers of 32-dimensional and 16-dimensional dimensions and outputs 1-dimensional data, i.e., 68→32→16→1; the task head (downstream task network) structure is 64-dimensional input, which is passed through a 32-dimensional hidden layer and outputs 1-dimensional data, i.e., 64→32→1.

[0124] The verification results based on actual operating data of the energy storage power station are shown in Table 1.

[0125] Table 1. Comparison of SOC estimation performance of different fusion methods

[0126] As shown in Table 1, the method of this invention achieves a mean absolute error (MAE) of 1.5% in the SOC estimation task, significantly outperforming the 2.8% of early splicing fusion, the 2.4% of late average fusion, and the 2.2% of fixed-weight fusion. The method of this invention has approximately 110K parameters, including approximately 91.2K for the four-modal feature extractor (22.8K per modality), approximately 2.4K for the quality assessment network, and approximately 16.4K for the attention module (including global query parameters such as the query projection matrix). Experimental results demonstrate that the method of this invention achieves a significant improvement in accuracy with a moderate increase in parameter count, validating the effectiveness of the quality-aware fusion mechanism. The above comparison results can be found in [reference needed]. Figure 5 .

[0127] Example 2: Robustness verification under data anomaly scenarios; This embodiment simulates a data anomaly scenario caused by temperature sensor failure to verify the robustness of the method of the present invention to abnormal data. 20% of the samples in the test set are randomly selected, and the temperature data is replaced with random noise to simulate the injection of abnormal data caused by sensor failure or strong electromagnetic interference.

[0128] Table 2 shows a comparison of the SOC estimation MAE of different fusion methods in normal and abnormal data scenarios.

[0129] Table 2. Robustness comparison of different fusion methods under data anomaly scenarios.

[0130] The relative decrease is calculated as follows: (abnormal data MAE minus normal data MAE) divided by normal data MAE multiplied by 100%.

[0131] As shown in Table 2, when temperature data anomalies exist, the MAE of early stitching and fusion increases sharply from 2.8% to 8.5%, a relative decrease of 204%; the MAE of late-stage average fusion increases from 2.4% to 5.2%, a relative decrease of 117%; while the MAE of the method of this invention only increases from 1.5% to 2.1%, a relative decrease of 40%. Experimental results show that the SOC estimation accuracy of the method of this invention remains stable in abnormal data scenarios caused by temperature sensor failure, and the performance degradation is far lower than that of other fusion methods.

[0132] When temperature data becomes abnormal, the fusion weights of the method in this invention are automatically adjusted. Under normal circumstances, the fusion weights for the four modes—voltage, current, temperature, and SOC—are 0.35, 0.30, 0.25, and 0.10, respectively. Under abnormal circumstances, the fusion weight for the temperature mode decreases from 0.25 to 0.08, while the weight for the voltage mode increases from 0.35 to 0.42, and the weight for the current mode increases from 0.30 to 0.38. These weight adjustments demonstrate that the quality assessment network successfully identified the degradation in temperature data and automatically reduced the fusion weights of abnormal modes through a soft attention mechanism, effectively suppressing the contamination of the fusion results by abnormal data. The above comparison results can be found in [reference needed]. Figure 6 .

[0133] Example 3: Fusion weight analysis under different operating conditions; This embodiment analyzes the adaptive weight allocation of each mode under different energy storage system operating conditions using the method of the present invention, and verifies the adaptive fusion capability under different operating conditions. The test conditions include five typical operating conditions: standby, constant current charging, constant voltage charging, fast discharging, and frequency modulation response.

[0134] The attention weight distribution of each mode under different operating conditions is shown in Table 3.

[0135] Table 3. Modal fusion weight distribution under different operating conditions

[0136] As shown in Table 3, the model automatically learned the differences in the importance of each mode under different operating conditions, which is consistent with the physical characteristics of the energy storage system. In standby mode, the system is in a static state with small changes in current and voltage, while temperature changes are relatively significant; therefore, the temperature mode has the highest weight (0.45) and becomes the dominant mode. In constant current charging mode, the constant current is the control characteristic of the charging process; therefore, the current mode has the highest weight (0.40). In constant voltage charging mode, voltage is the key criterion for charging termination; therefore, the voltage mode has the highest weight (0.45). In rapid discharge and frequency modulation response modes, current is a direct reflection of power response; therefore, the current mode has the highest weight (0.35 and 0.35, respectively). The above weight distribution shows that the method of this invention does not require manual setting of operating condition rules and can automatically learn and adapt to the differences in the importance of each mode data under different operating conditions, achieving operating condition adaptive multimodal fusion. The above weight distribution results can be found in [reference needed]. Figure 7 .

[0137] Based on the aforementioned experimental results, the technical mechanism and practical benefits of this invention can be further clarified. This method independently calculates a quality score for each modality through a quality assessment network. This score is generated based on the fusion mapping of four statistical indicators—signal-to-noise ratio, data integrity rate, variance stability, and physical consistency—with the feature vector. After this quality score is injected as a bias term into the soft attention calculation, the logarithmic bias term corresponding to low-quality modalities tends towards negative infinity and is automatically assigned a lower fusion weight during the normalization process. This mechanism allows the model to adjust the fusion strategy according to the real-time state of the input data, eliminating the tedious process of manually presetting fixed weights or designing rules for different operating conditions. When sensor malfunctions lead to data loss, noise injection, or abnormal mutations, the statistical quality indicators deteriorate accordingly, and the quality score decreases. The contribution of abnormal modalities in the fusion process is automatically suppressed, thereby significantly reducing the impact on the accuracy of the fused features.

[0138] At the training level, feature extraction, quality assessment, soft attention computation, and downstream task networks are interconnected to form a unified computational graph. Backpropagation uses the joint weighted value of task loss and entropy regularization loss as the total loss, and gradients are propagated back to the quality assessment network and feature extractor, promoting coordination between modules and downstream task objectives. Specifically, the entropy regularization loss uses the negative of the entropy of the fused weight distribution as a regularization term. This loss value is smaller when weights are excessively concentrated in a single modality, and larger otherwise. By introducing a negative coefficient into the total loss, the model is encouraged to maintain a moderate dispersion of weights while ensuring accuracy, thereby promoting the utilization of complementary multimodal information.

[0139] Furthermore, the attention weights output by this invention directly quantify the contribution ratio of each modality to the fusion result. By observing the weight values ​​and their changing trends, combined with the fluctuations in the quality score, maintenance personnel can quickly locate abnormal sensor channels, providing a visualized traceability basis for system maintenance. In terms of architectural design, the fusion framework decouples from the downstream task network through a fixed-dimensional fusion feature vector. Only the task network and its loss function need to be changed to adapt to various intelligent energy storage operation and maintenance scenarios such as SOC estimation, fault detection, and health assessment, demonstrating good scalability.

[0140] In terms of hardware adaptation, this invention employs a one-dimensional convolutional neural network for temporal feature extraction, combined with a multilayer perceptron to construct a quality evaluation network. Under a typical configuration, the four-modal feature extractor occupies approximately 91.2K parameters, the quality evaluation network approximately 2.4K parameters, and the attention module approximately 16.4K parameters, totaling approximately 110K parameters. Compared to the computationally expensive Transformer-type models, this architecture significantly reduces the demand for storage space and computing resources, offers faster inference speed, and can meet the deployment requirements of edge devices in energy storage systems. Please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is a structural block diagram of an energy storage multimodal data fusion system provided in an embodiment of the present invention.

[0141] This invention provides an energy storage multimodal data fusion system, comprising: The feature extraction module 801 is used to acquire multiple modal data of the energy storage system, extract time-series features from each modal data, and output the feature vector corresponding to each mode. The statistical quality calculation module 802 is used to calculate the statistical quality index corresponding to each modality based on the original sampled values ​​corresponding to each modality data. The quality assessment module 803 is used to input the statistical quality indicators and feature vectors corresponding to each mode into the quality assessment network to perform mode quality assessment and generate the quality score corresponding to each mode. The attention calculation module 804 is used to convert the quality scores corresponding to each modality into quality bias terms, perform soft attention operations on the feature vectors corresponding to each modality, and obtain the fusion weights corresponding to each modality. The fusion module 805 is used to perform weighted aggregation of all feature vectors according to the fusion weights corresponding to each modality to generate a fused feature vector. The loss calculation module 806 is used to input the fused feature vector into the downstream task network, calculate the task loss value output by the downstream task network, and use the joint weighted value of the task loss value and the entropy regularization loss value of the fused weights as the total loss value. The parameter update module 807 is used to adjust the parameters involved in the execution of temporal feature extraction, modality quality assessment, soft attention operation and downstream task network based on the total loss value. The adjusted parameters are used for the next execution of temporal feature extraction.

[0142] Since the above is a system corresponding to a multimodal data fusion method for energy storage, its implementation principle is the same as that of a multimodal data fusion method for energy storage. For the sake of convenience and brevity, those skilled in the art can clearly understand that the specific working process of the system and modules described above can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0143] Please see Figure 9 , Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention.

[0144] An electronic device according to an embodiment of the present invention includes: a memory 901 and a processor 902. The memory 901 stores a computer program. When the computer program is executed by the processor 902, the processor 902 performs the energy storage multimodal data fusion method as described in any of the above embodiments.

[0145] Memory 901 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Memory 901 has storage space 903 for program code 913 for performing any of the method steps described above. For example, storage space 903 for program code may include various program codes 913 for implementing the various steps in the methods described above. These program codes may be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When run by a computing processing device, this code causes the computing processing device to perform the various steps in the methods described above. These program codes may be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When this code is run by a computing device, it causes the computing device to perform the various steps in the energy storage multimodal data fusion method described above.

[0146] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the energy storage multimodal data fusion method as described in any of the above embodiments.

[0147] This invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer performs the energy storage multimodal data fusion method as described in any of the above embodiments.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0153] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for multimodal data fusion in energy storage, characterized in that, include: Multiple modal data of the energy storage system are acquired, and time-series features are extracted from each modal data to output the feature vector corresponding to each mode. Based on the original sampled values ​​corresponding to each modal data, calculate the statistical quality index corresponding to each modality; The statistical quality index and feature vector corresponding to each mode are respectively input into the quality assessment network to perform mode quality assessment and generate the quality score corresponding to each mode; The quality scores corresponding to each modality are converted into quality bias terms, and soft attention operations are performed on the feature vectors corresponding to each modality to obtain the fusion weights corresponding to each modality. Based on the fusion weights corresponding to each modality, all the feature vectors are weighted and aggregated to generate a fused feature vector; The fused feature vector is input into the downstream task network, the task loss value output by the downstream task network is calculated, and the joint weighted value of the task loss value and the entropy regularization loss value of the fused weight is used as the total loss value. The parameters involved in the temporal feature extraction, modality quality assessment, soft attention operation, and downstream task network are adjusted based on the total loss value, and the adjusted parameters are used for the next execution of the temporal feature extraction.

2. The energy storage multimodal data fusion method according to claim 1, characterized in that, The step of extracting temporal features from each modality data and outputting the feature vector corresponding to each modality includes: The sampling frequency of each modal data is judged for consistency, and the modal data with inconsistent sampling frequencies are time-aligned to obtain the target modal data; Each target modality data is convolutionally mapped and nonlinearly transformed using a one-dimensional convolutional neural network corresponding to each target modality data, thereby generating a first temporal feature map corresponding to each modality. Each of the first temporal feature maps is downsampled to generate a second temporal feature map corresponding to each modality; Global downsampling is performed on each of the second time-series feature maps to generate global feature vectors corresponding to each mode; Each global feature vector is input into a fully connected layer for dimensionality transformation, and the feature vector corresponding to each modality is output.

3. The energy storage multimodal data fusion method according to claim 1, characterized in that, The step of calculating the statistical quality index corresponding to each modality based on the original sampled values ​​corresponding to each modality data includes: Calculate the ratio between the signal variance and the noise variance of each modal data, and use it as the signal-to-noise ratio for each modality. The ratio of missing sampling points to the total number of sampling points for each modality is used as the data integrity rate for each modality. The original sampled values ​​of each modality data are divided into sliding windows, the standard deviation of the original sampled values ​​within each sliding window is calculated, and the variance stability of each modality is determined based on the variance of each standard deviation. Obtain the current measured value of each modality data, and determine the physical consistency of each modality based on the deviation between the current measured value and the expected value of the physical model, combined with the reasonable value range corresponding to the modality to which the current measured value belongs.

4. The energy storage multimodal data fusion method according to claim 1, characterized in that, The step of inputting the statistical quality index and feature vector corresponding to each modality into the quality assessment network for modality quality assessment and generating a quality score corresponding to each modality includes: The statistical quality indices corresponding to each modality are combined into a statistical quality vector corresponding to each modality; The statistical quality vector corresponding to each mode is concatenated with the corresponding feature vector to generate the concatenated vector corresponding to each mode. The concatenated vectors corresponding to each modality are input into a multilayer perceptron for feature mapping to generate intermediate evaluation values ​​corresponding to each modality. An S-shaped function is used to map the intermediate evaluation values ​​corresponding to each mode to a preset interval to obtain the quality score corresponding to each mode.

5. The energy storage multimodal data fusion method according to claim 1, characterized in that, The steps of converting the quality scores corresponding to each modality into quality bias terms, performing soft attention operations on the feature vectors corresponding to each modality, and obtaining the fusion weights corresponding to each modality include: A global concatenation vector is constructed by concatenating the feature vectors corresponding to all the aforementioned modes. A linear mapping is performed on the global concatenation vector to generate a global query vector; The feature vectors corresponding to each mode are simultaneously used as the key vectors and value vectors corresponding to each mode; Stack the key vectors corresponding to all the modes to generate a key matrix, and stack the value vectors corresponding to all the modes to generate a value matrix; Add a preset value to the mass fraction corresponding to each mode and take the logarithm to generate the mass bias term corresponding to each mode; The product of the global query vector and the key matrix is ​​calculated, scaled, and added to the quality bias term corresponding to each modality to obtain the attention score corresponding to each modality. The attention scores corresponding to each modality are normalized to obtain the fusion weights corresponding to each modality.

6. The energy storage multimodal data fusion method according to any one of claims 1-5, characterized in that, The step of inputting the fused feature vector into the downstream task network, calculating the task loss value output by the downstream task network, and using the joint weighted value of the task loss value and the entropy regularization loss value of the fused weights as the total loss value includes: Calculate the difference between the output data of the downstream task network and the corresponding task value to generate the task loss value; Calculate the negative of the entropy of the fusion weights corresponding to each mode to generate the entropy regularization loss value; The total loss value is generated by weighted summation of the task loss value and the entropy regularization loss value.

7. A multimodal data fusion system for energy storage, characterized in that, include: The feature extraction module is used to acquire multiple modal data of the energy storage system, extract time-series features from each modal data, and output the feature vector corresponding to each mode. The statistical quality calculation module is used to calculate the statistical quality index corresponding to each modality based on the original sampled values ​​corresponding to each modality data. The quality assessment module is used to input the statistical quality indicators and feature vectors corresponding to each mode into the quality assessment network to perform mode quality assessment and generate a quality score for each mode. The attention calculation module is used to convert the quality scores corresponding to each of the modes into quality bias terms, perform soft attention operations on the feature vectors corresponding to each of the modes, and obtain the fusion weights corresponding to each of the modes. The fusion module is used to perform weighted aggregation on all the feature vectors according to the fusion weights corresponding to each modality to generate a fused feature vector; The loss calculation module is used to input the fused feature vector into the downstream task network, calculate the task loss value output by the downstream task network, and use the joint weighted value of the task loss value and the entropy regularization loss value of the fused weights as the total loss value. The parameter update module is used to adjust the parameters involved in performing the temporal feature extraction, the modality quality assessment, the soft attention operation, and the downstream task network based on the total loss value. The adjusted parameters are used for the next execution of the temporal feature extraction.

8. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the energy storage multimodal data fusion method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the energy storage multimodal data fusion method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the energy storage multimodal data fusion method as described in any one of claims 1-6.

Images