Operation and Maintenance Method and System for Energy Storage Power Stations Based on Multimodal Information Fusion

Abstract

This invention discloses a method and system for the operation and maintenance of energy storage power stations based on multimodal information fusion. The method includes: acquiring historical operational data of the energy storage power station; performing time synchronization and spatial registration based on a unified timestamp and a defined coordinate mapping relationship to construct a training dataset; constructing an initial semantic representation model for the operation and maintenance of the energy storage power station based on convolutional layers, pooling layers, spatial attention mechanisms, GRU units, and a query-key-value framework, and training the resulting semantic representation model; and using the obtained semantic representation model for the operation and maintenance of the target energy storage power station. This invention not only achieves operation and maintenance of energy storage power stations based on multimodal information fusion, but also offers higher reliability, better accuracy, and better performance.

Description

Technical Field

[0001] This invention belongs to the field of electrical automation, specifically relating to an operation and maintenance method and system for energy storage power stations based on multimodal information fusion. Background Technology

[0002] With economic and technological development and the improvement of people's living standards, electricity has become an indispensable secondary energy source in people's production and daily life, bringing endless convenience. Therefore, ensuring a stable and reliable supply of electricity has become one of the most important tasks of the power system.

[0003] As a critical energy regulation and load balancing unit in the power system, the safety and reliability of energy storage power stations directly affect the stability of the power grid. During long-term operation, the battery clusters in energy storage power stations are affected by factors such as changes in environmental temperature and humidity, current surges, and differences in the consistency of individual cells, leading to performance degradation or safety hazards. Therefore, monitoring and visualizing the status of the battery clusters in energy storage power stations has become one of the important tasks in the operation and maintenance of power system energy storage power stations.

[0004] Existing state monitoring solutions for battery clusters in energy storage power stations typically rely on single-modal data for status assessment. This includes using time-series data such as voltage, current, and temperature for anomaly detection and lifespan prediction, or conducting visual inspections by periodically taking images of the equipment's exterior. However, these single-data-source monitoring solutions suffer from relatively poor reliability and accuracy. While some solutions overlay sensor data with image information, these lack a unified spatiotemporal reference and cross-modal semantic alignment mechanism. This makes direct comparison and fusion between different sampling frequencies and physical quantities difficult, resulting in equally unsatisfactory performance. Summary of the Invention

[0005] One of the objectives of this invention is to provide a highly reliable, accurate, and effective operation and maintenance method for energy storage power stations based on multimodal information fusion.

[0006] The second objective of this invention is to provide a system for implementing the aforementioned operation and maintenance method for energy storage power stations based on multimodal information fusion.

[0007] The energy storage power station operation and maintenance method based on multimodal information fusion provided by this invention includes the following steps:

[0008] S1. Obtain historical operating data information of the energy storage power station;

[0009] S2. Based on the unified timestamp and the set coordinate mapping relationship, perform time synchronization and spatial registration on the data information obtained in step S1 to construct a training dataset;

[0010] S3. Based on convolutional layers, pooling layers, spatial attention mechanism, GRU unit and query-key-value framework, construct an initial model of semantic representation for the operation and maintenance of energy storage power stations;

[0011] The initial model for the semantic representation of operation and maintenance of the energy storage power station includes a visual module, a temporal module, and a fusion module;

[0012] A visual module is constructed based on convolutional layers, pooling layers, and spatial attention mechanisms. The visual module is used to extract features from the acquired image information to obtain visual feature vectors.

[0013] A temporal module is constructed based on convolutional layers and GRU units; the temporal module is used to extract features from the acquired temporal data to obtain physical feature vectors;

[0014] A fusion module is constructed based on the query-key-value framework. The fusion module is used to unify the dimensions of the obtained visual feature vectors and physical feature vectors, and to fuse the visual feature vectors and physical feature vectors according to the generated query vector, key vector and value vector, so as to obtain the semantic representation of the operation and maintenance of the energy storage power station.

[0015] S4. Using the training dataset constructed in step S2, train the initial model of the operation and maintenance semantic representation of the energy storage power station constructed in step S3 to obtain the operation and maintenance semantic representation model of the energy storage power station.

[0016] S5. Using the semantic representation model of energy storage power station operation and maintenance obtained in step S4, perform operation and maintenance of the target energy storage power station.

[0017] Step S1 specifically includes the following steps:

[0018] Acquire historical operating data information of the energy storage power station; the operating data information includes voltage and current information of each energy storage battery cluster in the energy storage power station, temperature information of the energy storage power station, humidity information of the energy storage power station, and image data information of the appearance of the battery clusters.

[0019] Based on the voltage and current information of each energy storage battery cluster, the power fluctuation characteristic of each energy storage battery cluster is calculated using the following formula:

[0020] In the formula This refers to the power fluctuation characteristic of the energy storage battery cluster; This represents the total number of sampling points; The voltage value at the i-th sampling point; Let i be the current value at the i-th sampling point; This is the average of the sampled voltage values; This is the average value of the sampled current.

[0021] Step S2 specifically includes the following steps:

[0022] The data obtained in step S1 is normalized using the maximum-minimum normalization algorithm.

[0023] The normalized data is synchronized using a predefined unified timestamp. During time synchronization, the following phase compensation function is used to align data between different sampling frequencies:

[0024] In the formula The timestamp after synchronization; The original timestamp of the k-th sampling point; M is the total number of modes; Let be the measurement value of the j-th mode at time t; Optimize variables for time offset;

[0025] After normalization, a unified spatial registration is performed between the data acquisition location and the image data. During this unified spatial registration, the following coordinate projection function is used to map the data acquisition location to the image plane:

[0026] In the formula The projection result; It is a rotation matrix; The coordinates of the location where the data was acquired in three-dimensional space; The coordinates of the image plane; It is a translation vector;

[0027] The obtained data is used to construct a training dataset.

[0028] Step S3 includes the following steps:

[0029] A visual module is constructed by sequentially connecting several convolutional layers, normalization layers, activation function layers, and pooling layers, and combining them with the spatial attention mechanism set at the end.

[0030] A temporal module is constructed by sequentially connecting several convolutional layers, normalization layers, activation function layers, GRU units, and layer normalization layers;

[0031] The fusion module's processing steps include: unifying the dimensions of visual and physical feature vectors based on a fully connected layer; calibrating the feature vectors using mean and variance; calculating attention weights and fusion features using a query-key-value framework; and finally generating a semantic representation for the operation and maintenance of energy storage power stations.

[0032] The processing of the vision module specifically includes the following steps:

[0033] The image frame data information obtained in step S1 is used as the input of the vision module;

[0034] The input data is processed sequentially through four convolutional blocks; the four convolutional blocks have the same structure, each consisting of a 2D convolutional layer, a 2D batch normalization layer, a ReLU activation function layer, and a 2D max pooling layer connected in sequence.

[0035] The feature map output by the last convolutional block is processed by the spatial attention block to obtain the output of the visual module. The processing of the spatial attention block includes: performing global average pooling and global max pooling on the input feature map respectively, concatenating the results, and then processing them sequentially through a 2D convolutional layer and a Sigmoid activation function to obtain the spatial attention weight map; and performing element-wise multiplication between the spatial attention weight map and the input feature map to obtain the weighted feature map.

[0036] Finally, the weighted feature maps are flattened and mapped through a fully connected layer to obtain a set of visual feature vectors. .

[0037] The timing module's processing procedure includes the following steps:

[0038] The timing data information obtained in step S1 is used as the input of the timing module; the timing data information includes voltage information, current information, temperature information and humidity information.

[0039] The input data is processed sequentially through two causal convolutional blocks; the two causal convolutional blocks have the same structure, each including a 1d convolutional layer, a 1d batch normalization layer and an ELU activation function layer connected in sequence.

[0040] The data from the second causal convolutional block is processed sequentially through two bidirectional GRU units, and then through a layer normalization layer to obtain the physical feature vector set. The bidirectional GRU units have the same structure, with a hidden layer dimension of 256 and a dropout rate of 0.2.

[0041] The processing steps of the fusion module specifically include the following:

[0042] physical feature vector set Processed through fully connected layers to increase the dimensionality to match the visual feature vector set. The dimensions are consistent;

[0043] Based on the mean and variance, the eigenvectors are calibrated using the following formula:

[0044] In the formula The calibrated visual feature vector; This is the scaling factor for visual features; The visual feature vector before calibration; The average value of visual features across batches; For visual features, the batch variance is used. The visual features are shifted and offset; The calibrated physical feature vector; This is the scaling factor for physical characteristics; The physical feature vector before calibration; This represents the batch average of physical characteristics. For batch variance of physical characteristics; The physical characteristics are shifted and offset;

[0045] The query vector is calculated using the following formula. Key vector Sum value vector :

[0046] In the formula This is the weight matrix for the query vector; The set prior weights; For the bias term of the query vector; The weight matrix is ​​the weight matrix of the key vectors; This is the bias term of the key vector; This is the bias term of the value vector;

[0047] Calculate the scaled dot product similarity for and will After masking, the attention weights are normalized using the softmax function. ;

[0048] Preliminary fusion features were calculated. for , Value vector The j1th element in The number of visual feature vectors. The number of values ​​in the value vector V;

[0049] Preliminary fusion characteristics The energy storage power station operation and maintenance semantic representation H is generated by processing the Swish activation function and then mapping it through a fully connected layer.

[0050] The training described in step S4 specifically includes the following steps:

[0051] The initial model of the operation and maintenance semantic representation of the energy storage power station constructed in step S3 is trained using the selected loss function.

[0052] Add the following constraint to the selected loss function:

[0053] In the formula Functions for added constraint terms; This represents the total number of visual feature vectors involved in semantic alignment calculation within the current batch. This represents the total number of physical feature vectors involved in semantic alignment calculation within the current batch. Let L2 be the norm of the vector.

[0054] Step S5 specifically includes the following steps:

[0055] Using the operation and maintenance semantic representation model of the energy storage power station obtained in step S4, the operation and maintenance semantic representation of the target energy storage power station is obtained. ;

[0056] The obtained operation and maintenance semantic representation of the target energy storage power station Rendering and dynamic visualization are used to realize the operation and maintenance of the target energy storage power station.

[0057] This invention also provides a system for implementing the energy storage power station operation and maintenance method based on multimodal information fusion, comprising a data acquisition module, a data processing module, a model building module, a model training module, and an operation and maintenance module; the data acquisition module, data processing module, model building module, model training module, and operation and maintenance module are connected in series; the data acquisition module is used to acquire historical operation data information of the energy storage power station and upload the data information to the data processing module; the data processing module is used to perform time synchronization and spatial registration on the acquired data information based on a unified timestamp and a set coordinate mapping relationship to construct a training dataset, and upload the data information to the model building module; the model building module is used to construct an initial model of the operation and maintenance semantic representation of the energy storage power station based on the received data information, using convolutional layers, pooling layers, spatial attention mechanisms, GRU units, and a query-key-value framework, and upload the data information to the model training module; wherein, the initial model of the operation and maintenance semantic representation of the energy storage power station includes a vision module, a time... The system comprises a sequence module and a fusion module; a vision module is constructed based on convolutional layers, pooling layers, and spatial attention mechanisms; the vision module extracts features from the acquired image information to obtain visual feature vectors; a temporal module is constructed based on convolutional layers and GRU units; the temporal module extracts features from the acquired temporal data information to obtain physical feature vectors; a fusion module is constructed based on a query-key-value framework; the fusion module unifies the dimensions of the obtained visual and physical feature vectors, and fuses them according to the generated query vector, key vector, and value vector to obtain a semantic representation of energy storage power station operation and maintenance; a model training module trains the initial model of the semantic representation of energy storage power station operation and maintenance using the constructed training dataset based on the received data information to obtain the semantic representation model of energy storage power station operation and maintenance, and uploads the data information to the operation and maintenance module; the operation and maintenance module performs operation and maintenance of the target energy storage power station using the obtained semantic representation model of energy storage power station operation and maintenance based on the received data information.

[0058] The energy storage power station operation and maintenance method and system provided by this invention achieves the standardization, unification and integration of basic data by acquiring, synchronizing and spatially registering historical operation data of the energy storage power station; then, through the training and application of the constructed energy storage power station operation and maintenance semantic representation model, it not only realizes the operation and maintenance of energy storage power stations based on multimodal information fusion, but also achieves higher reliability, better accuracy and better results. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0060] Figure 2 This is a schematic diagram of the functional modules of the system of the present invention. Detailed Implementation

[0061] like Figure 1 The diagram shown is a flowchart of the method of the present invention: The operation and maintenance method for energy storage power stations based on multimodal information fusion disclosed in this invention includes the following steps:

[0062] S1. Obtain historical operating data of the energy storage power station; specifically including the following steps:

[0063] Acquire historical operating data information of the energy storage power station; the operating data information includes voltage and current information of each energy storage battery cluster in the energy storage power station, temperature information of the energy storage power station, humidity information of the energy storage power station, and image data information of the appearance of the battery clusters; wherein, the sampling frequency of voltage and current information should not be lower than 1kHz, and the sampling frequency of temperature and humidity information should not be lower than 1Hz.

[0064] Based on the voltage and current information of each energy storage battery cluster, the power fluctuation characteristic of each energy storage battery cluster is calculated using the following formula:

[0065] In the formula This is a power fluctuation characteristic of energy storage battery clusters, which reflects the normalized deviation of the power fluctuation of the battery cluster, and is thus used to reveal potential individual cell anomalies and overall consistency deviations. This represents the total number of sampling points; The voltage value at the i-th sampling point; Let i be the current value at the i-th sampling point; This is the average of the sampled voltage values; This is the average of the sampled current values;

[0066] S2. Based on the data information obtained in step S1, and using a unified timestamp and a defined coordinate mapping relationship, perform time synchronization and spatial registration to construct a training dataset; specifically, this includes the following steps:

[0067] The data information obtained in step S1 is normalized using the maximum-minimum normalization algorithm. Normalization aligns the data scales between different sampling frequencies and different physical quantities, providing a unified input benchmark for the subsequent cross-modal spatiotemporal alignment process.

[0068] The normalized data is synchronized using a set unified timestamp. During time synchronization, the following phase compensation function is used to align data between different sampling frequencies, achieving strict time alignment between high-frequency voltage and current characteristics and low-frequency temperature and humidity characteristics:

[0069] In the formula The timestamp after synchronization; The original timestamp of the k-th sampling point; M is the total number of modes; Let be the measurement value of the j-th mode at time t; Optimize variables for time offset;

[0070] For the normalized data, a unified spatial registration is performed between the data acquisition location and the image data. During the unified spatial registration process, the following coordinate projection function is used to realize the mapping relationship between the data acquisition location and the image plane, resulting in a multimodal aligned data stream under a unified spatiotemporal reference:

[0071] In the formula The projection result; It is a rotation matrix; The coordinates of the location where the data was acquired in three-dimensional space; The coordinates of the image plane; It is a translation vector;

[0072] The obtained data will be used to construct a training dataset;

[0073] S3. Based on convolutional layers, pooling layers, spatial attention mechanism, GRU unit and query-key-value framework, construct an initial model of semantic representation for the operation and maintenance of energy storage power stations;

[0074] The initial model for the semantic representation of operation and maintenance of the energy storage power station includes a visual module, a temporal module, and a fusion module;

[0075] A visual module is constructed based on convolutional layers, pooling layers, and spatial attention mechanisms. The visual module is used to extract features from the acquired image information to obtain visual feature vectors.

[0076] A temporal module is constructed based on convolutional layers and GRU units; the temporal module is used to extract features from the acquired temporal data to obtain physical feature vectors;

[0077] A fusion module is constructed based on the query-key-value framework. The fusion module is used to unify the dimensions of the obtained visual feature vectors and physical feature vectors, and to fuse the visual feature vectors and physical feature vectors according to the generated query vector, key vector and value vector, so as to obtain the semantic representation of the operation and maintenance of the energy storage power station.

[0078] In practice, several convolutional layers, normalization layers, activation function layers and pooling layers are sequentially connected, and combined with the spatial attention mechanism set at the end, a visual module is constructed.

[0079] A temporal module is constructed by sequentially connecting several convolutional layers, normalization layers, activation function layers, GRU units, and layer normalization layers;

[0080] The fusion module's processing steps include: unifying the dimensions of visual and physical feature vectors based on a fully connected layer; calibrating the feature vectors using mean and variance; calculating attention weights and fusion features using a query-key-value framework; and finally generating a semantic representation for the operation and maintenance of energy storage power stations.

[0081] The processing of the vision module specifically includes the following steps:

[0082] The image frame data information obtained in step S1 is used as the input of the vision module;

[0083] The input data is processed sequentially through four convolutional blocks; the four convolutional blocks have the same structure, each consisting of a 2D convolutional layer, a 2D batch normalization layer, a ReLU activation function layer, and a 2D max pooling layer connected in sequence.

[0084] The feature map output by the last convolutional block is processed by the spatial attention block to obtain the output of the visual module. The processing of the spatial attention block includes: performing global average pooling and global max pooling on the input feature map respectively, concatenating the results, and then processing them sequentially through a 2D convolutional layer and a Sigmoid activation function to obtain the spatial attention weight map; and performing element-wise multiplication between the spatial attention weight map and the input feature map to obtain the weighted feature map.

[0085] Finally, the weighted feature maps are flattened and mapped through a fully connected layer to obtain a set of visual feature vectors. ;

[0086] The timing module's processing procedure includes the following steps:

[0087] The timing data information obtained in step S1 is used as the input of the timing module; the timing data information includes voltage information, current information, temperature information and humidity information.

[0088] The input data is processed sequentially through two causal convolutional blocks; the two causal convolutional blocks have the same structure, each including a 1d convolutional layer, a 1d batch normalization layer and an ELU activation function layer connected in sequence.

[0089] The data from the second causal convolutional block is processed sequentially through two bidirectional GRU units, and then through a layer normalization layer to obtain the physical feature vector set. The bidirectional GRU units have the same structure, with a hidden layer dimension of 256 and a dropout rate of 0.2.

[0090] The processing steps of the fusion module specifically include the following:

[0091] physical feature vector set Processed through fully connected layers to increase the dimensionality to match the visual feature vector set. The dimensions are consistent;

[0092] Based on the mean and variance, the eigenvectors are calibrated using the following formula:

[0093] In the formula The calibrated visual feature vector; This is the scaling factor for visual features; The visual feature vector before calibration; The average value of visual features across batches; For visual features, the batch variance is used. The visual features are shifted and offset; The calibrated physical feature vector; This is the scaling factor for physical characteristics; The physical feature vector before calibration; This represents the batch average of physical characteristics. For batch variance of physical characteristics; The physical characteristics are shifted and offset;

[0094] The query vector is calculated using the following formula. Key vector Sum value vector :

[0095] In the formula This is the weight matrix for the query vector; The set prior weights; For the bias term of the query vector; The weight matrix is ​​the weight matrix of the key vectors; This is the bias term of the key vector; This is the bias term of the value vector;

[0096] Calculate the scaled dot product similarity for and will After masking, the attention weights are normalized using the softmax function. ;

[0097] Preliminary fusion features were calculated. for , Value vector The j1th element in The number of visual feature vectors. The number of values ​​in the value vector V;

[0098] Preliminary fusion characteristics The energy storage power station operation and maintenance semantic representation H is generated by processing the Swish activation function and then mapping it through a fully connected layer.

[0099] H can simultaneously encode device appearance abnormalities, physical parameter fluctuations, and environmental influences, achieving a comprehensive characterization of the health status of battery devices.

[0100] S4. Using the training dataset constructed in step S2, train the initial model of the operation and maintenance semantic representation of the energy storage power station constructed in step S3 to obtain the operation and maintenance semantic representation model of the energy storage power station.

[0101] During the training process, the selected loss function is used to train the initial model of the operation and maintenance semantic representation of the energy storage power station constructed in step S3.

[0102] Add the following constraint to the selected loss function:

[0103] In the formula Functions for added constraint terms; This represents the total number of visual feature vectors involved in semantic alignment calculation within the current batch. This represents the total number of physical feature vectors involved in semantic alignment calculation within the current batch. Let L2 be the norm of the vector;

[0104] This function maps image and video features and electrochemical sensing features to a unified high-dimensional semantic space by minimizing the semantic distance between features of different modalities, thereby ensuring the semantic consistency of multimodal data.

[0105] S5. Using the semantic representation model for operation and maintenance of the energy storage power station obtained in step S4, perform operation and maintenance of the target energy storage power station; specifically including the following steps:

[0106] Using the operation and maintenance semantic representation model of the energy storage power station obtained in step S4, the operation and maintenance semantic representation of the target energy storage power station is obtained. ;

[0107] The obtained operation and maintenance semantic representation of the target energy storage power station Rendering and dynamic visualization are used to realize the operation and maintenance of the target energy storage power station;

[0108] Through visualization, maintenance personnel can intuitively perceive the equipment's form, operating status, and health level on the same interface, thereby forming a data-driven intelligent maintenance closed loop and providing input basis for subsequent predictive maintenance.

[0109] The method of the present invention will be described below with reference to an embodiment:

[0110] Dataset: Three months of measured data from a 10MW energy storage power station, including: ① High-frequency voltage / current data (1kHz sampling, 50,000 time windows, including 1,000 abnormal windows: such as voltage fluctuation >5%); ② Temperature and humidity data (1Hz sampling, 259,200 time steps); ③ Equipment appearance images (15fps, 100,000 frames, including 5,000 abnormal images: such as bulging, leakage); divided into training and testing sets in a 7:3 ratio.

[0111] Experimental environment: Hardware consists of NVIDIA RTX 3090 GPU, Intel i9-12900K CPU, and 64GB of RAM; Software consists of PyTorch 1.12, OpenCV 4.5, and Matplotlib 3.6.

[0112] Comparison of options:

[0113] Option 1 (Single-mode solution): Use only high-frequency voltage / current data and employ a GRU model (2 layers, 256 hidden layer dimensions) to assess health status;

[0114] Option 2 (Simple Fusion Solution): Combine voltage / current features, temperature and humidity features, and image CNN features (extracted by ResNet50), and output the health status through a fully connected layer;

[0115] Option 3 (the option in this application): The option in this application shall be adopted.

[0116] 2. Evaluation Indicators and Experimental Results

[0117] Evaluation metrics: ① Health status assessment accuracy (three categories: normal / minor abnormality / serious abnormality); ② Abnormality detection rate (proportion of abnormal samples identified); ③ Inference latency (single sample processing time).

[0118] The experimental results are shown in Table 1:

[0119] Results analysis:

[0120] Accuracy comparison: The accuracy of the proposed solution is 15.7% higher than that of solution 1 and 8.9% higher than that of solution 2. This is because cross-modal calibration solves the feature distribution differences and dynamic attention enhances the abnormal information.

[0121] Anomaly detection rate comparison: The proposed solution is 21.6% higher than Solution 1 and 13.3% higher than Solution 2. This is due to the anomaly-sensitive weight and activation function, which makes the detection of anomalies such as voltage drop and appearance bulge more accurate.

[0122] Latency comparison: The latency of the proposed solution is 30ms lower than that of solution 2 (due to TCN replacing part of the GRU calculation, reducing redundancy), and only 40ms higher than that of solution 1, achieving a balance between accuracy and efficiency.

[0123] In summary, the proposed solution outperforms existing solutions in terms of health status assessment accuracy, anomaly detection capability, and inference efficiency, making it more suitable for remote intelligent operation and maintenance scenarios of energy storage power stations.

[0124] This invention aligns high-frequency electrochemical data, low-frequency environmental data, and equipment appearance images using a unified spatiotemporal reference. Combined with cross-modal semantic coding and attention mechanisms, it achieves precise mapping and consistent representation of multi-source information in a high-dimensional semantic space, significantly improving the accuracy of capturing battery cluster operating status and health characteristics. The method maps high-dimensional semantic information to a three-dimensional device morphology and integrates real-time operating data and health status results, enabling dynamic and visual monitoring of battery cluster status. This allows remote maintenance personnel to obtain comprehensive health assessments instantly. Through multi-dimensional information rendering and a visualization interface, maintenance personnel can intuitively view the equipment's physical form, operating parameters, and health status on a single interface, reducing the frequency of manual inspections and improving response speed to abnormal events. The unified semantic representation and spatiotemporally aligned multimodal data provide reliable input for subsequent fault prediction, remaining life assessment, and maintenance strategy optimization, enhancing the intelligent operation and maintenance capabilities of energy storage power stations. Furthermore, this method is compatible with different types of sensors and acquisition devices, enabling flexible expansion and iterative upgrades of multimodal data to meet the monitoring and management needs of energy storage power stations of different scales.

[0125] like Figure 2The diagram shows the functional modules of the system of this invention: The system disclosed in this invention, which implements the energy storage power station operation and maintenance method based on multimodal information fusion, includes a data acquisition module, a data processing module, a model building module, a model training module, and an operation and maintenance module; these modules are connected in series. The data acquisition module acquires historical operating data information of the energy storage power station and uploads the data information to the data processing module. The data processing module, based on the received data information, performs time synchronization and spatial registration on the acquired data information according to a unified timestamp and a set coordinate mapping relationship to construct a training dataset, and uploads the data information to the model building module. The model building module, based on the received data information, constructs an initial model of the semantic representation of energy storage power station operation and maintenance using convolutional layers, pooling layers, spatial attention mechanisms, GRU units, and a query-key-value framework, and uploads the data information to the model training module. The initial model of the semantic representation of energy storage power station operation and maintenance... The system comprises a vision module, a temporal module, and a fusion module. The vision module is constructed based on convolutional layers, pooling layers, and spatial attention mechanisms. This module extracts features from acquired image information to obtain visual feature vectors. The temporal module is constructed based on convolutional layers and GRU units. This module extracts features from acquired temporal data to obtain physical feature vectors. The fusion module is constructed based on a query-key-value framework. This module unifies the dimensions of the obtained visual and physical feature vectors and fuses them according to the generated query vector, key vector, and value vector to obtain a semantic representation of energy storage power station operation and maintenance. The model training module trains the initial model of the semantic representation of energy storage power station operation and maintenance using the constructed training dataset, obtaining the semantic representation model. The data is then uploaded to the operation and maintenance module. The operation and maintenance module uses the obtained semantic representation model to perform operation and maintenance of the target energy storage power station based on the received data.

Claims

1. A method for the operation and maintenance of an energy storage power station based on multimodal information fusion, comprising the following steps: S1. Obtain historical operating data information of the energy storage power station; the historical operating data information includes voltage and current information of each energy storage battery cluster in the energy storage power station, temperature information of the energy storage power station, humidity information of the energy storage power station, and image data information of the appearance of the battery clusters; S2. Based on the unified timestamp and the set coordinate mapping relationship, perform time synchronization and spatial registration on the data information obtained in step S1 to construct a training dataset; S3. Based on convolutional layers, pooling layers, spatial attention mechanism, GRU unit and query-key-value framework, construct an initial model of semantic representation for the operation and maintenance of energy storage power stations; The initial model for the semantic representation of operation and maintenance of the energy storage power station includes a visual module, a temporal module, and a fusion module; A visual module is constructed based on convolutional layers, pooling layers, and spatial attention mechanisms; The vision module is used to extract features from the acquired image information to obtain visual feature vectors; Temporal modules are constructed based on convolutional layers and GRU units; The timing module is used to extract features from the acquired timing data to obtain physical feature vectors; the timing data includes voltage information, current information, temperature information, and humidity information. A fusion module is constructed based on the query-key-value framework. The fusion module is used to unify the dimensions of the obtained visual feature vectors and physical feature vectors, and to fuse the visual feature vectors and physical feature vectors according to the generated query vector, key vector and value vector, so as to obtain the semantic representation of the operation and maintenance of the energy storage power station. S4. Using the training dataset constructed in step S2, train the initial model of the operation and maintenance semantic representation of the energy storage power station constructed in step S3 to obtain the operation and maintenance semantic representation model of the energy storage power station; during training, the data label is set as the health status assessment result; the health status assessment result includes normal, slightly abnormal and severely abnormal; S5. Using the semantic representation model for operation and maintenance of the energy storage power station obtained in step S4, perform operation and maintenance of the target energy storage power station; specifically including the following steps: Using the operation and maintenance semantic representation model of the energy storage power station obtained in step S4, the operation and maintenance semantic representation of the target energy storage power station is obtained. ; The obtained operation and maintenance semantic representation of the target energy storage power station Rendering and dynamic visualization are used to realize the operation and maintenance of the target energy storage power station.

2. The operation and maintenance method for energy storage power stations based on multimodal information fusion according to claim 1, characterized in that... Step S2 specifically includes the following steps: The data obtained in step S1 is normalized using the maximum-minimum normalization algorithm. The normalized data is synchronized using a predefined unified timestamp. During time synchronization, the following phase compensation function is used to align data between different sampling frequencies: In the formula The timestamp after synchronization; The original timestamp of the k-th sampling point; M is the total number of modes; Let be the measurement value of the j-th mode at time t; Optimize variables for time offset; After normalization, a unified spatial registration is performed on the data acquisition location and the image data. During the unified spatial registration process, the following coordinate projection function is used to realize the mapping relationship between the data acquisition location and the image plane: In the formula The projection result; It is a rotation matrix; The coordinates of the location where the data was acquired in three-dimensional space; The coordinates of the image plane; It is a translation vector; The obtained data is used to construct a training dataset.

3. The operation and maintenance method for energy storage power stations based on multimodal information fusion according to claim 2, characterized in that... The processing of the vision module specifically includes the following steps: The image frame data information obtained in step S1 is used as the input of the vision module; The input data is processed sequentially through four convolutional blocks; the four convolutional blocks have the same structure, each consisting of a 2D convolutional layer, a 2D batch normalization layer, a ReLU activation function layer, and a 2D max pooling layer connected in sequence. The feature map output by the last convolutional block is processed by the spatial attention block to obtain the output of the visual module. The processing of the spatial attention block includes: performing global average pooling and global max pooling on the input feature map respectively, concatenating the results, and then processing them sequentially through a 2D convolutional layer and a Sigmoid activation function to obtain the spatial attention weight map; and performing element-wise multiplication between the spatial attention weight map and the input feature map to obtain the weighted feature map. Finally, the weighted feature maps are flattened and mapped through a fully connected layer to obtain a set of visual feature vectors. .

4. The operation and maintenance method for energy storage power stations based on multimodal information fusion according to claim 3, characterized in that... The timing module's processing procedure includes the following steps: The timing data information obtained in step S1 is used as the input of the timing module; The input data is processed sequentially through two causal convolutional blocks; the two causal convolutional blocks have the same structure, each including a 1d convolutional layer, a 1d batch normalization layer and an ELU activation function layer connected in sequence. The data from the second causal convolutional block is processed sequentially through two bidirectional GRU units, and then through a layer normalization layer to obtain the physical feature vector set. .

5. The operation and maintenance method for energy storage power stations based on multimodal information fusion according to claim 4, characterized in that... The processing steps of the fusion module specifically include the following: physical feature vector set Processed through fully connected layers to increase the dimensionality to match the visual feature vector set. The dimensions are consistent; Based on the mean and variance, the eigenvectors are calibrated using the following formula: In the formula The calibrated visual feature vector; This is the scaling factor for visual features; The visual feature vector before calibration; The average value of visual features across batches; For visual features, the batch variance is used. The visual features are shifted and offset; The calibrated physical feature vector; This is the scaling factor for physical characteristics; The physical feature vector before calibration; This represents the batch average of physical characteristics. For batch variance of physical characteristics; The physical characteristics are shifted and offset; The query vector is calculated using the following formula. Key vector Sum value vector : In the formula This is the weight matrix for the query vector; The set prior weights; For the bias term of the query vector; The weight matrix is ​​the weight matrix of the key vectors; This is the bias term of the key vector; The weight matrix is ​​the value vector; Calculate the scaled dot product similarity for and will After masking, the attention weights are normalized using the softmax function. ; Preliminary fusion features were calculated. for , Value vector The j1th element in The number of visual feature vectors. The number of values ​​in the value vector V; Preliminary fusion characteristics The energy storage power station operation and maintenance semantic representation H is generated by processing the Swish activation function and then mapping it through a fully connected layer.

6. The operation and maintenance method for energy storage power stations based on multimodal information fusion according to claim 5, characterized in that... The training described in step S4 specifically includes the following steps: The initial model of the operation and maintenance semantic representation of the energy storage power station constructed in step S3 is trained using the selected loss function. Add the following constraint to the selected loss function: In the formula This is a function for adding constraint terms; This represents the total number of visual feature vectors involved in semantic alignment calculation within the current batch. This represents the total number of physical feature vectors involved in semantic alignment calculation within the current batch. Let L2 be the norm of the vector.

7. A system for implementing the operation and maintenance method of an energy storage power station based on multimodal information fusion as described in any one of claims 1 to 6, characterized in that... It includes a data acquisition module, a data processing module, a model building module, a model training module, and an operation and maintenance module; the data acquisition module, data processing module, model building module, model training module, and operation and maintenance module are connected in series; the data acquisition module is used to acquire historical operation data information of the energy storage power station and upload the data information to the data processing module; The data processing module is used to perform time synchronization and spatial registration on the acquired data information based on the received data information and the set coordinate mapping relationship, so as to build a training dataset and upload the data information to the model building module. The model building module is used to construct an initial semantic representation model for the operation and maintenance of energy storage power stations based on the received data information, using convolutional layers, pooling layers, spatial attention mechanisms, GRU units, and a query-key-value framework, and then uploads the data information to the model training module. The initial semantic representation model for the operation and maintenance of energy storage power stations includes a vision module, a temporal module, and a fusion module. The vision module is constructed based on convolutional layers, pooling layers, and spatial attention mechanisms; it is used to extract features from the acquired image information to obtain visual feature vectors. The temporal module is constructed based on convolutional layers and GRU units; it is used to extract features from the acquired temporal data information to obtain physical feature vectors. A fusion module is constructed based on a query-key-value framework. This module unifies the dimensions of the obtained visual and physical feature vectors and fuses them according to the generated query vector, key vector, and value vector to obtain a semantic representation of energy storage power station operation and maintenance. A model training module trains the initial model of the constructed semantic representation of energy storage power station operation and maintenance using the received data and a constructed training dataset, obtaining the semantic representation model of energy storage power station operation and maintenance. The data is then uploaded to the operation and maintenance module. The operation and maintenance module uses the obtained semantic representation model of energy storage power station operation and maintenance to perform operation and maintenance of the target energy storage power station based on the received data.

Images