Energy storage operation and maintenance multi-modal dialogue system and method based on large language model

Abstract

The invention relates to the technical field of energy storage power station operation and maintenance, large language model reasoning and multi-mode man-machine interaction, in particular to an energy storage operation and maintenance multi-mode dialogue system and method based on a large language model, and the system comprises five layers of architectures: a data collection and multi-mode perception layer obtains BMS data, images and voice; a semantic fusion and dynamic Prompt layer constructs a Prompt containing a real-time working condition; an edge reasoning layer quantifies a 7B parameter LLM by using Q4 and is combined with LoRA fine tuning, and runs in a 10W Edge GPU; the security layer performs confidence evaluation, white list and intention classification filtering; the human-computer interaction layer supports local ASR and self-adaptive TTS, according to the energy storage operation and maintenance multi-mode dialogue system and method based on the large language model, the first response is smaller than or equal to 1.9 s, the Top-1 accuracy is larger than or equal to 92.4%, the voice bandwidth is smaller than or equal to 128 kb / s, and the word error rate is 1t; and off-line use can be realized, an unauthorized instruction is blocked, and the operation and maintenance efficiency of the energy storage power station is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of energy storage power station operation and maintenance, large language model reasoning and multi-modal human-computer interaction, and specifically relates to an energy storage operation and maintenance multi-modal dialogue system and method based on a large language model. BACKGROUND

[0002] In the field operation and maintenance work of an energy storage power station, an operation and maintenance engineer needs to quickly query a SOP (standard operating procedure), determine the meaning of an alarm code and obtain the corresponding fault handling steps to efficiently deal with various problems occurring in the process of device operation and to ensure the stable operation of the energy storage power station. This requirement is the core link of the daily operation and maintenance work of the energy storage power station and is directly related to the operation and maintenance efficiency and the safety of the power station operation. It is also a key scene that needs to be optimized in the current energy storage operation and maintenance field.

[0003] The traditional operation and maintenance information query and problem solving method mainly relies on paper or electronic manual retrieval and communication with technical experts through telephone to obtain guidance. This kind of method not only requires the operation and maintenance engineer to spend a lot of time searching or waiting for a reply, which is costly, and in the case of complicated manual information or inaccurate communication information transmission, understanding deviation and operation errors are likely to occur. Although the existing cloud chat robots can provide certain question and answer services, they are limited by the problem of insufficient network stability in the field of energy storage power stations and face the risk of privacy compliance in the data uploading process, making it difficult to be applied in the energy storage operation and maintenance scene. Therefore, in view of the above status, it is urgent to develop an energy storage operation and maintenance multi-modal dialogue system and method based on a large language model to overcome the deficiencies in the current actual application. SUMMARY

[0004] The purpose of the present application is to provide an energy storage operation and maintenance multi-modal dialogue system and method based on a large language model to solve the problems raised in the background.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] An energy storage operation and maintenance multi-modal dialogue system based on a large language model, comprising a data acquisition and multi-modal perception module, a semantic fusion and dynamic Prompt module, an edge reasoning module, a security and trust module and a human-computer interaction module;

[0007] The data acquisition and multi-modal perception module is used to acquire BMS data, image data and voice data of the energy storage power station;

[0008] The semantic fusion and dynamic Prompt module is used to associate the multi-modal data with the operation and maintenance knowledge base to construct a context Prompt containing real-time working conditions;

[0009] The edge reasoning module adopts a large language model that is quantized and fine-tuned for a scene, and realizes localized reasoning on a low-power edge GPU device;

[0010] The security and trust module is used for confidence evaluation, white list filtering and intent classification of the operation and maintenance question and answer result generated by reasoning, and intercepts unauthorized or incorrect instructions.

[0011] The human-computer interaction module supports voice and text dual-mode interaction, and the voice interaction bandwidth does not exceed a preset threshold.

[0012] As a further scheme of the application, the BMS data includes battery cell voltage, battery cluster total current, system maximum temperature and alarm code.

[0013] The image data is collected by a camera deployed in the battery compartment, with a collection frame rate of 30 frames per second and an image resolution of 1280x720.

[0014] The voice data is collected by a microphone, with a sampling rate of 16 kHz and a sampling accuracy of 16 bits.

[0015] As a further scheme of the application, the semantic fusion and dynamic Prompt module further includes a vector database, and the operation and maintenance knowledge base is stored in the vector database in a vector form after being converted by a text embedding model.

[0016] The module retrieves Top-3 related knowledge documents for constructing context Prompt by calculating the similarity of the multi-modal data fusion feature vector and the knowledge base vector.

[0017] As a further scheme of the application, the large language model in the edge reasoning module is a 7B parameter model, which adopts Q4 precision quantization and LoRA fine-tuning technology.

[0018] The low-power edge GPU device has a power consumption of 10W, a memory of not less than 8GB, and supports a single round reasoning delay of the large language model of not more than 600ms.

[0019] As a further scheme of the application, the confidence evaluation of the security and trust module obtains a confidence value by calculating the average probability of all tokens in the question and answer result.

[0020] When the confidence value is greater than or equal to 0.7, the question and answer result is directly output; when 0.5 is less than or equal to the confidence value and less than 0.7, the question and answer result needs to be output after expert review; when the confidence value is less than 0.5, the output is rejected and a prompt is given.

[0021] A multi-modal dialogue method for energy storage operation and maintenance based on a large language model, comprising the following steps:

[0022] (1) Obtain BMS data, image data and voice data of the energy storage power station through a data acquisition device;

[0023] (2) Perform semantic fusion on the multi-modal data and an operation and maintenance knowledge base to construct a dynamic Prompt containing real-time working conditions;

[0024] (3) Input the dynamic Prompt into a large language model that has been quantized and fine-tuned for a scene, perform localized inference on a low-power edge GPU device, and generate operation and maintenance question and answer results;

[0025] (4) Perform confidence evaluation, white list filtering and intent classification on the operation and maintenance question and answer results to block unauthorized or incorrect instructions;

[0026] (5) Output the question and answer results filtered through the human-computer interaction device in the form of voice or text, and the voice interaction bandwidth is ≤128 kb / s.

[0027] As a further scheme of the application, in step (1), the BMS data is collected in real time through the Modbus protocol; the image data is collected at a frame rate of 30 frames / second and a resolution of 1280*720; and the voice data is sampled at a sampling rate of 16 kHz and a sampling accuracy of 16 bits.

[0028] As a further scheme of the application, in step (2), the semantic fusion includes: converting the BMS data into a sensor summary vector, converting the image data into a visual feature vector, and converting the voice data into a semantic vector.

[0029] The dynamic Prompt is constructed by merging Top-3 retrieval knowledge documents, multi-modal fusion feature vectors and system-level instructions.

[0030] As a further scheme of the application, in step (3), the large language model adopts Q4 precision quantization and LoRA fine-tuning technology, and the inference process optimizes matrix operations through a quantized GEMM algorithm.

[0031] The low-power edge GPU device has a power consumption of 10 W, and supports a single round of inference delay of not more than 600 ms.

[0032] As a further scheme of the application, in step (4), the white list filtering is realized by comparing the operation instructions in the question and answer results with the pre-defined legal operation and maintenance instructions.

[0033] The intent classification identifies whether the question and answer results involve sensitive or dangerous operations through a pre-trained text classification model, and directly blocks the output if it is identified as a dangerous operation.

[0034] Compared with the prior art, the application has the following advantages:

[0035] 1. To achieve offline inference on the edge, this application adopts a technical solution of 7B parameter large language model with Q4 precision quantization and LoRA fine-tuning, which can run stably on 10W low power edge GPU devices without relying on cloud services. This avoids the impact of network stability on operation and maintenance response, and ensures the privacy and security of operation and maintenance data. It can still provide question and answer services normally in offline state.

[0036] 2. Achieves deep fusion of multimodal operating conditions, enabling real-time injection of alarm codes, sensor summaries (such as average battery voltage and system maximum temperature), and visual features (such as battery appearance and cooling fan status) into the model context. This allows the model-generated answers to fully integrate with the current equipment operating status, ensuring the relevance and accuracy of the answers. Experimental verification shows that the Top-1 accuracy rate can reach 92.7%, significantly better than the 61.4% of traditional manual retrieval.

[0037] 3. It has high security protection capabilities. Through a dual security filtering mechanism of confidence assessment and intent classification, it performs reliability judgment and risk interception on the answers output by the large language model. When the confidence is below 0.5, it will refuse to answer and directly block the operation when it involves unauthorized or dangerous operation (such as forced disconnection of the power grid). At the same time, it combines a whitelist strategy to ensure that only legitimate operation and maintenance instructions are output, which greatly reduces the risk of erroneous operation caused by model "illusion".

[0038] 4. To meet the low-bandwidth interaction requirements, the voice interaction link adopts the Opus encoder. With the parameter configuration of 24kb / s bit rate and 20ms frame interval, the total bandwidth of voice interaction is controlled within 128kb / s, which is suitable for the possible low-bandwidth network environment at the energy storage power station site. Moreover, the speech word error rate (WER) is as low as 5.8%, ensuring the clarity and fluency of voice interaction.

[0039] 5. Achieve high real-time response: The system's end-to-end interaction latency is only about 1.9 seconds, far lower than the 12.3 seconds of traditional manual retrieval. This allows maintenance engineers to quickly obtain key information such as alarm codes and handling steps within 2 seconds, significantly shortening emergency response time and improving the operation and maintenance efficiency of energy storage power stations. According to a satisfaction survey of maintenance personnel, 85% of them believe that it can significantly improve work efficiency. Attached Figure Description

[0040] Figure 1 This is a diagram illustrating the module composition and data interaction of a multimodal dialogue system for energy storage operation and maintenance based on a large language model, as described in this embodiment of the invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0043] Please see Figure 1 The present invention provides a multimodal dialogue system and method for energy storage operation and maintenance based on a large language model, which specifically includes the following:

[0044] I. Application Environment Setup;

[0045] This embodiment is applied to a 100MW / 200MWh energy storage power station, which is equipped with 250 battery clusters, each containing 320 lithium iron phosphate cells. To achieve localized and low-power operation of the operation and maintenance assistant system, an edge computing device (NVIDIA Jetson Orin Nano) is deployed at the operation and maintenance site. This device meets the hardware requirements of 10W power consumption and 8GB memory, and can support the entire process of this system.

[0046] The system connects to the following peripherals to achieve multimodal data acquisition and interaction:

[0047] BMS monitoring interface: Real-time collection of operating condition data such as battery cell voltage, total current, system maximum temperature and alarm codes, providing core numerical basis for subsequent operating condition integration;

[0048] Camera: Deployed inside the battery compartment to collect visual information such as battery appearance (e.g., whether it is bulging or leaking), cooling fan operation status, and indicator light status;

[0049] Microphone and speaker: The microphone is used to collect voice questions from maintenance engineers, and the speaker is used to broadcast the system's voice responses, realizing dual-mode voice interaction.

[0050] II. System Overall Architecture Implementation;

[0051] This system implements a five-layer architecture to achieve end-to-end operation and maintenance Q&A, encompassing "multimodal perception, semantic fusion, edge reasoning, security filtering, and human-computer interaction." The specific implementation methods for each layer are as follows:

[0052] 1. Data acquisition and multimodal perception layer;

[0053] The core function of this layer is to acquire multi-source operation and maintenance data, providing raw input for subsequent semantic fusion. The specific implementation method is as follows:

[0054] Data is collected in real time via the Modbus protocol of the BMS module, including: the voltage V of each battery cell. i (i=1,…,N; N is the number of cells in a single battery cluster, N=320 in this embodiment), total current of the battery cluster, and highest system temperature T. max Current alarm code c;

[0055] Images of the battery compartment are captured by a camera at a frame rate of 30 frames per second, with each frame having a resolution of 1280×720 to ensure that visual features are clearly discernible.

[0056] The voice stream of the operations engineer was captured via microphone at a sampling rate of 16kHz. t The sampling accuracy is 16 bits to ensure the integrity of the voice signal.

[0057] 2. Semantic fusion and dynamic Prompt layer;

[0058] This layer is used to semantically associate multimodal data with the operation and maintenance knowledge base, constructing a context containing real-time operating conditions. The specific implementation method is as follows:

[0059] Building an operation and maintenance knowledge base: The SOP (Standard Operating Procedures), equipment maintenance manuals, alarm code handling guidelines and other documents of the energy storage power station are structured and converted into vector form through text embedding model and stored in vector database to form a centralized knowledge base mirror (the centralized knowledge base mirror is a cloud backup of the operation and maintenance knowledge base (including SOPs / manuals), and the local mirror is stored at the edge and updated regularly with the cloud).

[0060] Text Embedding Model: The text embedding model of the operation and maintenance knowledge base adopts a dual-tower Sentence-BERT structure, with a 12-layer Transformer (768 hidden dimensions) as the base. Mean pooling and L2 normalization are used to generate 768-dimensional semantic vectors. To adapt to edge devices, a 6-layer TinyBERT quantized implementation (384 dimensions) is provided as a lightweight alternative. The model uses contrastive learning for domain fine-tuning on question-document fragment pairing data. Hard-to-fail samples are drawn from adjacent steps within the same chapter and similar alarm code entries.

[0061] Multimodal data encoding: converting numerical data collected by BMS into sensor summary vectors, converting camera images into visual feature vectors, and converting user voice / text questions into semantic vectors;

[0062] Semantic retrieval: Based on the similarity between the fused feature vector and the knowledge base vector, the most relevant knowledge documents are retrieved, providing a basis for dynamic prompts.

[0063] 3. Edge reasoning layer;

[0064] This layer uses a miniaturized large language model to achieve localized reasoning, meeting low latency requirements. The specific implementation method is as follows:

[0065] Model selection and optimization: A large language model (LLM) with 7 B parameters is adopted. Q4 precision quantization (i.e., each weight parameter is stored with 4 bits) is used to reduce memory usage. At the same time, LoRA (Low-Rank Adaptation) fine-tuning technology is adopted to inject knowledge specific to the energy storage operation and maintenance scenario (such as the meaning of alarm codes and SOP procedures) to ensure that the model output fits the operation and maintenance needs.

[0066] The precision quantization method for Q4 is as follows:

[0067] Quantization type: The quantization adopts symmetric quantization, that is, the positive and negative ranges share the same scaling factor; for abnormal distributions (such as skewed weight matrices), channel-by-channel asymmetric quantization is used as a fallback.

[0068] Quantization parameter determination method: For each layer's weight matrix W, first calculate the maximum absolute value max(|W|) of that layer, and determine the scaling factor s = max(|W|) / 7 (4 bits correspond to [-8,7]). The zero point z is set to 0 (for symmetric cases). If the distribution offset is significant, it is set according to round (-min(W) / s). Activation quantization adopts channel-by-channel dynamic calibration. During runtime, min / max is calculated within a sliding window to reduce oversaturation.

[0069] Inference optimization: Matrix multiplication is accelerated during the inference phase using the Quantized GEMM algorithm (QGEMM) while maintaining latency below 600ms.

[0070] LoRA fine-tuning techniques are as follows:

[0071] 1. Rank (r):

[0072] Set the rank r to 8 or 16, and insert a low-rank adaptation layer on the Query and Value projection matrices of the attention layer.

[0073] 2. Learning rate and training rounds:

[0074] The learning rate is set to 1×10 -4 The AdamW optimizer was used with a weight decay factor of 0.01. The training epochs were set to 35 epochs, and training was stopped early based on the convergence of the validation set.

[0075] 3. Dataset source and scale:

[0076] Sources: Internal SOP process documents, alarm code handling guidelines, equipment maintenance manuals, and Q&A pairs annotated by experts.

[0077] Scale: Approximately 5.1 million training samples, of which 70% are used for training, 15% for validation, and 15% for testing.

[0078] Preprocessing method: The document is segmented into fragments (300-500 tokens) and a "question-fragment" pairing is constructed; for alarm code data, a triplet of "code value-corresponding meaning-SOP step" is added to convert it into question-answering training samples; all text is uniformly encoded as UTF-8 and meaningless symbols and duplicate entries are removed.

[0079] In the patent implementation, Q4 precision quantization uses a symmetric quantization method, which determines the scaling factor by statistically analyzing the maximum absolute value of the weights channel by channel and then performs 4-bit mapping. For weights with skewed distributions, asymmetric quantization is used to reduce errors. Quantization matrix multiplication (QGEMM) is used during inference to achieve low-latency computation. LoRA fine-tuning employs a low-rank adaptation layer with rank r=8, inserted into the Query and Value projection matrices of the attention module, with a fine-tuning learning rate of 1×10⁻⁶. -4 The training consisted of 35 epochs, and the training data came from the SOPs, alarm code handling guidelines, and equipment maintenance manuals of energy storage power stations, totaling approximately 5.1 million expert-annotated samples.

[0080] Inference acceleration: The matrix operation is optimized by using the Quantization GEMM (General Matrix Multiplication) algorithm, achieving a single-round inference latency of about 600ms on a 10W edge GPU (NVIDIA Jetson Orin Nano), which meets the requirements for real-time response;

[0081] The implementation details of Quantitative GEMM (QGEMM) are as follows:

[0082] 1. Quantitative representation and mathematical form;

[0083] This embodiment employs weighted Q4 (4-bit) quantization with mixed precision of activated INT8 / FP16 (preferably, FP16 is activated to reduce zero-point compensation overhead). Let the real-valued weight matrix W∈R^(K×N), its quantization representation is as follows:

[0084] ;

[0085] Among them, W q ∈{0,…,15}^(K×N), s w z is a scaling factor vector by column (or by group, group size g=64). w For zero point (z in symmetric quantization) w =8). Given input activations A∈R^(M×K), the fully connected / projective GEMM is approximately:

[0086] ;

[0087] Where ⊙ indicates column scaling. If INT8 quantization is activated (A... q ,s a ,z a ),but

[0088] ;

[0089] Through compensation items Zero-point correction is performed, which is finally completed on the INT32 accumulator, and then weighted to FP16 / INT8 as the input for subsequent layers.

[0090] 2. Data layout and 4-bit aligned storage;

[0091] To reduce bandwidth and improve parallelism, the weights are packaged using nibble and pre-transposed layout:

[0092] Packing and Alignment: Two 4-bit weights are packed into 1 byte; the K dimension is filled in multiples of 8, and the N dimension is filled in multiples of 32 or 64; the row start address is aligned to 128 bits, and the block alignment is 32B / 64B, ensuring full-thread bundle merging and loading.

[0093] Pre-transpose / interleave: W q Pre-transpose to column-major order and interleave in K-dimensional blocks (e.g., interleaving in groups of 32 in the K direction) to match thread bundle access with tensor core (WMMA) fragment loading, reducing runtime transpose overhead.

[0094] Metadata is stored locally: each column block (or group) )of , Stored in the constant area adjacent to the weight block or in global memory, with the reciprocal pre-stored. Multiplication is used instead of division.

[0095] 3. Tiling and shared memory optimization;

[0096] On OrinNano (Ampere architecture), a CUTLASS-style hierarchical partitioning is adopted:

[0097] CTA tiles: ;

[0098] Warp tiles: ;

[0099] Thread-level sharding: Accumulate the sub-block into the INT32 register.

[0100] Will With the packaged The data is moved to shared memory (SMEM) in blocks, using a double-buffered pipeline (Stage0 / 1) with cp.async:

[0101] Stage :calculate Current block of axis;

[0102] Stage Asynchronous prefetch next Block to SMEM;

[0103] By using cp.async.commit / wait and __syncthreads(), we ensure that memory access and computation overlap, thus hiding global memory latency.

[0104] activation It is recommended to use vectorized loading in FP16 / INT8 (e.g., ldg.128). The data is batch-transferred to SMEM in byte stream format and then unpacked in parallel.

[0105] 4. Parallel unpacking and in-situ dequantization within SMEM;

[0106] To avoid additional bandwidth, unpacking and dequantization are performed within shared memory:

[0107] (1) Nibble unpacking: for each byte Parallel extraction of high / low nibbles , This yields two 4-bit weights;

[0108] (2) Zero point calibration: ; Symmetric quantization ;

[0109] (3) Scaling by column: To improve throughput, it is preferable to use group scaling (…). Broadcast in register and to Perform FMA on the vector.

[0110] Two computation paths are available:

[0111] Path A (TensorCoreINT8): will After expanding to INT8, scale by column (or postpone to the epilogue), use the WMMA / IMMAINT8 tensor core for multiplication and addition, accumulate with INT32, and then multiply uniformly in the epilogue. And integrate Bias / activation.

[0112] Path B (CUDACoreINT32): Maintains INT8 multiplication and addition, uses element-wise dot product instructions such as __dp4a to perform batch MAC, and accumulates to INT32; suitable for small batch or limited usage scenarios.

[0113] 5. Epilogue integration and weighting;

[0114] Finish After reduction, it is executed all at once in the epilogue:

[0115]

[0116] Then, as needed:

[0117] (1) Convert to FP16 (when the subsequent layer is FP16);

[0118] (2) Or weighted to INT8 (given) (RNE rounding and saturation clipping are used).

[0119] Simultaneously can be fused / The approximate polynomial reduces the need for secondary memory accesses.

[0120] 6. Parallelism and resource consumption optimization;

[0121] Occupancy: Controls the use of registers per CTA with SMEM (e.g., SMEM) KB / CTA), guaranteeing the number of concurrent CTAs on SM ;

[0122] Vectorized loading: Uses 128-bit / 256-bit width global loading to ensure thread bundle alignment and merged memory access;

[0123] Persistent kernel: Autoregressive decoding keeps weights resident in SM / register tiles, reducing the repeated loading of multiple tokens.

[0124] 7. Memory and bandwidth strategy;

[0125] Weighted compression ratio: Q4 will reduce weighted bandwidth to FP16. INT8 This significantly alleviates bandwidth bottlenecks;

[0126] Prefetching and double buffering: The K-axis prefetches the next block as it advances one block, and the computation and memory access pipelines overlap.

[0127] Constant cache: Place it in __constant__ or read-only cache (LDG) to improve hit rate.

[0128] 8. Algorithm flow (pseudocode);

[0129] enter: (FP16 / INT8, Shape) Packing weight (Q4, ), , Output: (FP16 / INT8, Shape) ) (INT32 accumulator) cp.async prefetch Synchronize with SMEM (double buffer) and Parallel unpacking in SMEM for INT8: (Path A) Loads the WMMA fragment and executes it using a tensor kernel. (Path B) Use __dp4a / INT8MAC to implement vector point accumulation. Epilogue: and integrate activation / weighting

[0130] 9. Engineering hyperparameters and recommended configurations;

[0131] Segment size: (CTA) (Warp);

[0132] Group quantification: Symmetric quantization ; Store by column / group;

[0133] Vectorized memory access: global load width B (e.g., ldg.128); row / column starting address B / B alignment;

[0134] Double buffering: cp.async two-stage pipeline (Stage0 / 1). ;

[0135] Epilogue fusion: Bias + activation (GELU approximate polynomial) + weighting (if needed).

[0136] After optimization, the single-round inference latency of the 7B parameter model can be controlled to approximately [value missing] on a 10W edge GPU platform (NVIDIA Jetson Orin Nano). The scale is sufficient to meet real-time response requirements.

[0137] 4. Security and Trust Layer;

[0138] This layer employs multiple mechanisms to filter out incorrect answers and unauthorized commands, ensuring the security of operational and maintenance processes. The specific implementation method is as follows:

[0139] Confidence assessment module: Calculates the average probability of the LLM output answer as a confidence index to judge the reliability of the answer;

[0140] Whitelist module: Predefines legal operation and maintenance commands (such as "adjust charging current" and "start cooling system"), and only allows commands that match the whitelist to be output;

[0141] Intent classification module: Uses a pre-trained text classification model to identify whether the answer involves sensitive / dangerous operations (such as "force disconnecting the power grid" or "skipping insulation detection"). If it is identified as a dangerous intent, the output is blocked directly.

[0142] 5. Human-computer interaction layer;

[0143] This layer supports dual-channel interaction of text and voice, balancing privacy and low bandwidth requirements. The specific implementation method is as follows:

[0144] Automatic Speech Recognition (ASR): An ASR module runs locally on the edge device, converting a 16kHz speech stream into a digital format. t Convert to text question u t To avoid privacy leaks caused by uploading voice data to the cloud, the word error rate (WER) is kept below 6%.

[0145] Text-to-Speech (TTS): Supports adaptive selection between local and cloud. When the on-site network is stable, the cloud TTS service can be called. When the network is unstable, it switches to local TTS. The Opus encoder is used to control the voice bandwidth to ensure that the total bandwidth does not exceed 128kb / s.

[0146] Text interaction: The operation and maintenance terminal UI displays user text questions and system text answers, making it convenient for engineers to view detailed steps.

[0147] III. Implementation of Key Technical Steps;

[0148] Step A: Multimodal semantic fusion;

[0149] This step encodes multi-source data into a unified feature vector, providing context for dynamic prompts. The specific implementation is as follows:

[0150] 1. Sensor Summary Construction

[0151] At time t, the raw data collected by the BMS is compressed to construct a sensor summary vector s. t The formula is as follows:

[0152]

[0153] Where c is the current alarm code (e.g., 0x12AF in this embodiment), V is the average voltage of a single cluster of cells (V=4.28V in this embodiment), and T max The highest system temperature (T in this embodiment) max = This summary vector can efficiently reflect the core numerical characteristics of the field conditions, avoiding redundancy in the original data.

[0154] 2. Visual encoding;

[0155] The current frame image captured by the camera I t Inputting a pre-trained ResNet-50 convolutional neural network, the network outputs a 2048-dimensional visual feature vector z through its fully connected layers. t The formula is as follows:

[0156]

[0157] This vector can characterize visual features such as battery appearance (e.g., no bulging, no leakage), cooling fan status (e.g., standby), and indicator light information (e.g., green light is normal), providing auxiliary evidence of operating conditions for the model (in this embodiment, z). t The battery compartment was deemed to have no abnormalities in appearance.

[0158] 3. Fusion vector generation;

[0159] The semantic vector e(u) of the user's text question t ), sensor summary t Visual features z t Weighted concatenation is performed to generate the final fused feature vector f. t The formula is as follows:

[0160]

[0161] Wherein, α is the sensor summary injection coefficient, which is used to balance the contributions of numerical conditions (weight 0.4) and visual features (weight 0.6), ensuring that the model prioritizes key numerical data (such as voltage and alarm codes), while combining visual information to correct judgments and avoid the limitations of single-modal data.

[0162] Step B: Dynamic Prompt Construction;

[0163] This step, based on the fusion of features and knowledge base retrieval results, constructs a Prompt that fits the current working conditions, thereby improving the accuracy of LLM inference. The specific implementation is as follows:

[0164] 1. Knowledge base similarity retrieval;

[0165] In the vector database, calculate the fused feature vector f. t With each knowledge document embedding vector Ki The cosine similarity is calculated using the following formula:

[0166]

[0167] Among them, K i Let R be the embedding representation of the i-th knowledge document (such as an SOP section or an alarm code guide). KB (i) Sort in descending order and take the top-3 documents (denoted as K1, K2, K3) as the knowledge basis (in this embodiment, the "Overcharge Alarm Handling Process" in Section 3.2 of the "Operation and Maintenance Manual" is retrieved).

[0168] 2. Prompt generation;

[0169] Top-3 document summaries and fusion features f t And system-level instructions are merged to construct a dynamic Prompt, in the following form:

[0170] ;

[0171] This Prompt has three major advantages: First, by injecting real-time operating characteristics (such as "unit voltage 4.28V, temperature 32℃"), it ensures that the LLM response matches the current equipment status; second, by retrieving document fragments to supplement accurate knowledge, it avoids the LLM outputting incorrect content based solely on memory; and third, by constraining the response format and security boundaries through system-level instructions, it ensures that the output conforms to operation and maintenance standards.

[0172] Step C: Marginal LLM Inference;

[0173] This step quantizes the dynamic Prompt input into an LLM, enabling localized inference and outputting the answer. The specific implementation is as follows:

[0174] 1. Conditional probability prediction, let h t−1 Let w be the hidden state of the LLM (recording the context of the preceding dialogue), and w be the candidate word. The LLM predicts the conditional probability distribution of the next token based on the Prompt and the historical state, as shown in the following formula:

[0175] ;

[0176] Among them, P L The language modeling probability distribution for quantizing LLM is obtained by Q4 quantization and LoRA fine-tuning of the 7B parameter model.

[0177] 2. Answer generation;

[0178] The maximum likelihood decoding algorithm is used to select the candidate word with the highest probability as the prediction token, and the complete answer is generated step by step. The formula is as follows:

[0179] ;

[0180] On a 10W edge GPU, matrix multiplication is optimized by quantizing the GEMM algorithm, with a single-round inference latency of about 600ms, and the answer that conforms to the working condition can be output quickly (in this embodiment, the output is: "This alarm code indicates that the single cell is slightly overcharged. Please perform the following in sequence: (1) reduce the charging current to 0.2C; (2) start the air cooling system with a target temperature of 25℃; (3) observe the voltage drop before normal charging can be resumed").

[0181] Step D: Confidence assessment and security filtering;

[0182] This step performs reliability and security checks on the LLM output answer, as detailed below:

[0183] 1. Confidence level calculation;

[0184] Let the generated answer be a. t Contains |a t | Tokens, define the average probability of all tokens as the confidence level Conf, as shown in the following formula:

[0185]

[0186] Among them, a t,j Let PL(a) be the j-th token of the answer. t,j ∣⋅) represents the predicted probability of this token.

[0187] 2. Confidence level decision;

[0188] A three-level processing strategy is adopted based on the confidence level:

[0189] ;

[0190] In this embodiment, the confidence level of the answer Conf=0.92≥0.70, which satisfies the "direct broadcast" condition.

[0191] 3. Dual safety filtration;

[0192] Whitelist filtering: The operation instructions in the answer are compared with the predefined whitelist. In this embodiment, "reduce the charging current to 0.2C" and "start the air cooling system" are both in the whitelist and are allowed to pass.

[0193] Intent classification filtering: The intent of the answer is identified by the intent classification model. In this embodiment, the intent of the answer is "alarm handling", which is a non-sensitive / dangerous operation and will not be blocked.

[0194] By employing dual filtering, the risk of erroneous instructions caused by LLM "illusion" can be significantly reduced, while unauthorized operations can be avoided.

[0195] Step E: Voice Link Model;

[0196] This step achieves low-bandwidth, high-quality voice interaction, and the specific implementation is as follows:

[0197] 1. Voice encoding configuration;

[0198] The speech data is encoded using the Opus encoder, with the parameters set as follows:

[0199] Bit rate: 24kb / s; Frame interval: 20ms; Encoding latency: <50ms; Upload bandwidth for voice data R up The calculation formula is as follows:

[0200]

[0201] Calculations show that R under this configuration up =24kb / s≤128kb / s, meeting the low bandwidth requirements of the operation and maintenance site, while ensuring voice clarity (word error rate WER=5.8%).

[0202] 2. Voice interaction process;

[0203] Voice input: The microphone captures the voice stream → the local ASR converts it to text (latency ≈ 300ms);

[0204] Voice output: LLM answer → TTS synthesized speech (local TTS latency ≈ 900ms, cloud TTS latency is adjusted according to network) → Opus encoding → speaker playback, ensuring smooth voice interaction.

[0205] Step F: Overall delay control;

[0206] The end-to-end interaction latency of this system is composed of the sum of the latencies of each module, as shown in the following formula:

[0207]

[0208] Typical latency parameters for each module (based on 10W edge GPU) are as follows:

[0209] T ASR (Speech recognition latency) ≈ 300ms; T encode (Multimodal coding delay) ≈ 50ms; T LLM (LLM inference latency) ≈ 600ms; T filter (Security filtering delay) ≈ 50ms; T TTS (Speech synthesis latency) ≈ 900ms;

[0210] According to calculations, T total≈300+50+600+50+900=1900ms=1.9s, which meets the design goal of "first response less than 2 seconds" and can quickly assist operation and maintenance engineers in handling emergency situations.

[0211] IV. Verification of Application Results;

[0212] 1. Specific scenario verification: Taking "alarm code 0x12AF handling" as an example, fully verify the system's effectiveness:

[0213] Input: The maintenance engineer asks via voice, "Alarm code 0x12AF has appeared, what should I do?", and the BMS simultaneously returns "Individual cell voltage 4.28V, temperature 32℃", and the camera returns "Battery compartment appearance normal, cooling fan in standby mode";

[0214] Processing flow: Local ASR-converted text (WER=5.7%) → Multimodal fusion to generate f t →Retrieve from Section 3.2 of the Operations and Maintenance Manual → Construct a dynamic Prompt → Output the answer through LLM inference → Confidence assessment (0.92) → Double security filtering → TTS synthesized speech;

[0215] Output: After 1.8 seconds, the speaker announces the answer and displays the text on the UI. After the engineer follows the steps, the alarm code is cleared, verifying that the system can effectively guide operation and maintenance.

[0216] 2. Batch testing results: 200 common energy storage alarm codes were tested, and the results are as follows:

[0217]

[0218] Test results show that compared with traditional manual retrieval, this system has significantly improved accuracy and response speed. At the same time, it ensures operational safety through security mechanisms and its low-bandwidth design is suitable for the field network environment, which can effectively improve the operation and maintenance efficiency of energy storage power stations.

[0219] In summary, this invention, through the technical solution of "edge LLM quantization + multimodal operating condition fusion + dual security filtering," achieves a low-power, high-real-time, and high-security energy storage operation and maintenance dialogue assistant. It can operate stably on 10W edge devices, with an initial response time of ≤1.9s, an answer accuracy rate of ≥92.4%, and a voice bandwidth of ≤128kb / s, fully meeting the on-site operation and maintenance needs of energy storage power stations. Moreover, it does not rely on the cloud and takes into account both privacy protection and network stability.

[0220] It should be noted that, in this invention, although the specification describes the embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A large language model-based energy storage operation and maintenance multi-modal dialogue system, characterized in that, The system comprises a data acquisition and multi-modal perception module, a semantic fusion and dynamic Prompt module, an edge inference module, a security and trust module, and a human-computer interaction module. The data acquisition and multi-modal perception module is configured to acquire BMS data, image data, and voice data of the energy storage power station. The semantic fusion and dynamic Prompt module is configured to associate the multi-modal data with an operation and maintenance knowledge base to construct a context Prompt containing real-time working conditions. The edge inference module adopts a large language model that has been quantized and fine-tuned for the scene to realize localized inference on a low-power edge GPU device. The security and trust module is configured to evaluate the confidence of the operation and maintenance question and answer results generated by inference, filter them through a whitelist, and classify their intentions to block unauthorized or incorrect instructions. The human-computer interaction module supports voice and text dual-mode interaction, and the voice interaction bandwidth does not exceed a preset threshold.

2. The energy storage operation and maintenance multi-modal dialogue system based on a large language model according to claim 1, characterized in that, The BMS data includes battery cell voltage, battery cluster total current, system maximum temperature, and alarm code. The image data is collected by a camera deployed in the battery compartment, with a frame rate of 30 frames per second and an image resolution of 1280x720. The voice data is collected by a microphone, with a sampling rate of 16 kHz and a sampling accuracy of 16 bits.

3. The energy storage operation and maintenance multi-modal dialogue system based on a large language model according to claim 1, characterized in that, The semantic fusion and dynamic Prompt module further comprises a vector database, in which the operation and maintenance knowledge base is stored in vector form after being converted by a text embedding model. The module calculates the similarity between the multi-modal data fusion feature vector and the knowledge base vector to retrieve Top-3 relevant knowledge documents for constructing the context Prompt.

4. The energy storage operation and maintenance multi-modal dialogue system based on a large language model according to claim 1, characterized in that, The large language model in the edge inference module is a 7B parameter model that has been quantized to Q4 precision and fine-tuned using LoRA technology. The low-power edge GPU device has a power consumption of 10W and a memory of not less than 8GB, and supports a single-round inference delay of the large language model of not more than 600ms.

5. The energy storage operation and maintenance multi-modal dialogue system based on a large language model according to claim 1, characterized in that, The confidence evaluation of the security and trust module obtains a confidence value by calculating the average probability of all tokens in the question and answer results. When the confidence value is greater than or equal to 0.7, the question and answer results are directly output; when 0.5 is less than the confidence value and less than 0.7, the question and answer results need to be output after expert review; when the confidence value is less than 0.5, the output is rejected and a prompt is given.

6. A large language model-based energy storage operation and maintenance multi-modal dialogue method, characterized in that, The method comprises the following steps: (1) acquiring BMS data, image data, and voice data of the energy storage power station through a data acquisition device; (2) performing semantic fusion on the multi-modal data and the operation and maintenance knowledge base to construct a dynamic Prompt containing real-time working conditions; (3) inputting the dynamic Prompt into a large language model that has been quantized and fine-tuned for the scene to perform localized inference on a low-power edge GPU device and generate operation and maintenance question and answer results; (4) evaluating the confidence of the operation and maintenance question and answer results, filtering them through a whitelist, and classifying their intentions to block unauthorized or incorrect instructions; (5) outputting the security-filtered question and answer results in the form of voice or text through a human-computer interaction device, and the voice interaction bandwidth is less than or equal to 128 kb / s.

7. The energy storage operation and maintenance multi-modal dialogue method based on a large language model according to claim 6, characterized in that, In step (1), the BMS data is collected in real time through the Modbus protocol; the image data is collected at a frame rate of 30 frames per second and a resolution of 1280*720; and the voice data is sampled at a rate of 16 kHz and a sampling accuracy of 16 bits.

8. The energy storage operation and maintenance multi-modal dialogue method based on a large language model according to claim 6, characterized in that, In step (2), the semantic fusion includes: converting the BMS data into a sensor summary vector, converting the image data into a visual feature vector, and converting the voice data into a semantic vector. The dynamic Prompt is constructed by merging the top-3 retrieved knowledge documents, the multi-modal fusion feature vector, and the system-level instruction.

9. The energy storage operation and maintenance multi-modal dialogue method based on a large language model according to claim 6, characterized in that, In step (3), the large language model adopts Q4 precision quantization and LoRA fine-tuning technology, and the inference process optimizes matrix operations through the quantized GEMM algorithm. The low-power edge GPU device has a power consumption of 10 W and supports a single round of inference delay of not more than 600 ms.

10. The energy storage operation and maintenance multi-modal dialogue method based on a large language model according to claim 6, characterized in that, In step (4), the whitelist filtering is realized by comparing the operation instructions in the question and answer results with the pre-defined legal operation and maintenance instructions; and the intent classification identifies whether the question and answer results involve sensitive or dangerous operations through a pre-trained text classification model, and if it is identified as a dangerous operation, the output is directly blocked.

Images