MLLM-based energy storage battery data automatic retrieval analysis method and system

Abstract

The invention provides an automatic retrieval analysis method and system for energy storage battery data based on MLLM, and relates to the technical field of battery data management.The method mainly comprises the steps that multi-modal data of an energy storage battery is encoded into unified modal features, then an MLLM model of the incidence relation between the features captured through a multi-head attention mechanism is constructed, and the unified modal features are obtained; and finally, generating a database in combination with the multi-modal data of the energy storage battery, retrieving the received user question in the database, inputting a retrieval result and the user question into an MLLM model for processing and analysis, and generating an answer response. Semantic level communication between a battery physical world and a digital system is achieved through an intelligent MLLM model, an operator can directly obtain a deep analysis result through natural language questioning, operation guidance is output through natural language interaction, professional experience does not need to be relied on, the operation and maintenance threshold is lowered, efficiency is improved, traditional manual statistics and shallow monitoring are replaced, and the method is suitable for large-scale popularization and application. Reusable analysis logic is automatically generated, and subjective misjudgment is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of battery data management, and particularly relates to a method and system for automatically retrieving and analyzing energy storage battery data based on MLLM. Background Art

[0002] The safe and stable operation of energy storage batteries is one of the important guarantees for ensuring energy supply and economic development. In recent years, with the continuous progress of new energy large model technology. Many shortcomings have emerged in the monitoring method of battery packs that used to rely on manual operations in the past. For example, for the massive data generated by various important indicators such as the temperature, voltage, and internal resistance of the storage battery, relying solely on manual monitoring and statistics is inefficient, which further leads to the inability of manual operations to quickly and accurately analyze based on the monitored and statistical data, such as predicting the remaining life of the storage battery.

[0003] Therefore, in order to ensure that the health status of the battery packs in the substation can be strongly monitored and managed, and to improve the overall operation and maintenance level of the substation, an energy storage battery data monitoring system has emerged. However, the existing battery monitoring systems are not intelligent at present. They can only provide online real-time data monitoring and statistics. Although it solves the problem of low efficiency of manual monitoring and statistics to a certain extent, there are still certain limitations. For example, the existing monitoring systems do not have a retrieval and analysis function, and the degree of intelligence is low. Operators can only obtain superficial data, and further data processing and analysis still rely on manual professional knowledge and rich experience. However, at present, the detection and operation and maintenance of storage batteries still require further processing and analysis of data. Therefore, the existing energy storage battery data monitoring systems still have problems such as high operation and maintenance thresholds, low efficiency, and strong professional dependence. Summary of the Invention

[0004] To solve the problems of high operation and maintenance thresholds, low efficiency, and strong professional dependence existing in the prior art, the purpose of the present invention is to provide a method and system for automatically retrieving and analyzing energy storage battery data based on MLLM, which realizes the semantic-level connection between the physical world of the battery and the digital system through an intelligent multi-modal large language model (MLLM). Operators can obtain in-depth analysis through natural language questions and answers, reducing the operation and maintenance threshold and improving efficiency.

[0005] To achieve the above purpose, the present invention provides the following technical solutions: The present invention provides a method for automatic retrieval and analysis of energy storage battery data based on MLLM in a first aspect, including: multimodal encoding: receiving multimodal data of an energy storage battery and encoding it into unified modal features; MLLM model construction and training: constructing an MLLM model that captures the correlation relationships between features through a multi-head attention mechanism, and inputting the obtained unified modal features into the MLLM model for training; automatic retrieval and analysis: generating a database in combination with the multimodal data of the energy storage battery, retrieving the received user questions in the database, and inputting the retrieval results and the user questions into the trained MLLM model for processing and analysis to generate an answer response as a prediction result.

[0006] The present invention provides a preferred solution in a first aspect. The multimodal data includes: text data, structured data, image data, and video data of the energy storage battery; the multimodal encoding specifically includes: mapping the text data, structured data, image data, and video data to the same semantic space to obtain a unified modal text vector.

[0007] Further, based on the above-mentioned preferred solution, the multimodal encoding further includes: text vector tokenization and token embedding processing to output token embeddings for input into the MLLM model for training.

[0008] The present invention provides another preferred solution in a first aspect. The processing process of the MLLM model includes: input embedding, inputting the obtained token embeddings; multi-head attention fusion, calculating the correlation weights between token embeddings through a multi-head attention mechanism, and performing linear projection fusion after concatenating multiple attention heads; layer normalization and residual connection, adding and performing layer normalization processing on the fused results; output embedding, outputting the results of the layer normalization processing in the form of blocks to output block embeddings.

[0009] Further, based on the MLLM model in the above-mentioned preferred solution, the method further includes: output projection, flattening and linearizing the block embeddings to obtain a prediction result.

[0010] In a more preferred solution, the token embedding is a Token sequence, and the multimodal encoding further includes: performing positional encoding on the Token sequence, and inputting the Token sequence after positional encoding into the MLLM model.

[0011] The present invention provides a preferred solution in a first aspect. When data passes through the MLLM model, forward propagation is performed, and the parameters of the MLLM model are optimized through backpropagation.

[0012] In a more preferred solution, the multimodal encoding further includes: batch reprogramming: obtaining a reprogrammed block embedding through block embedding, linearization, text prototype processing, and introducing a multi-head attention mechanism to optimize the embedding representation.

[0013] In a first aspect, the present invention provides a preferred solution, and the automated retrieval and analysis steps specifically include: Multimodal vectorized storage: segmenting the multimodal data of the energy storage battery into text segments and vectorizing and storing them in a vector database; Problem vectorization and similarity search: receiving a user problem, vectorizing it, and retrieving similar text segments in the vector database; MLLM analysis and prediction: inputting the retrieved similar text segments and the vectorized user problem into a trained MLLM model for processing and analysis, and generating an answer response as the prediction result.

[0014] In a second aspect, the present invention provides an automated retrieval and analysis system for energy storage battery data based on MLLM to implement the above method, including: a multimodal encoding module: used to receive the multimodal data of the energy storage battery and encode it into a unified modal feature; an MLLM model construction and training module: used to construct an MLLM model that captures the correlation between features through a multi-head attention mechanism, and input the unified modal feature obtained by encoding into the MLLM model for training; an automated retrieval and analysis module: used to generate a database in combination with the multimodal data of the energy storage battery, retrieve the received user problem in the database, input the retrieval result and the user problem into a trained MLLM model for processing and analysis, and generate an answer response as the prediction result.

[0015] Compared with the prior art, the present invention has the following advantages: The present invention realizes the semantic-level connection between the battery physical world and the digital system through an intelligent MLLM (Multimodal Large Language Model). Operators can directly obtain in-depth analysis results by asking questions in natural language and output operation guidance through natural language interaction, without relying on professional experience, reducing the operation and maintenance threshold and improving efficiency, replacing traditional manual statistics and shallow monitoring, automatically generating reusable analysis logic, and avoiding subjective misjudgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0017] Figure 1 The step diagram of the method for automatically retrieving and analyzing energy storage battery data based on MLLM provided by an embodiment of the present invention; Figure 2 The module diagram of the system for automatically retrieving and analyzing energy storage battery data based on MLLM provided by an embodiment of the present invention; Figure 3 The step diagram of multi-modal encoding in the method for automatically retrieving and analyzing energy storage battery data based on MLLM provided by an embodiment of the present invention; Figure 4 The processing process of the MLLM model in the method for automatically retrieving and analyzing energy storage battery data based on MLLM provided by another embodiment of the present invention; Figure 5 The architecture diagram of processing data by means of a multi-modal large language model in the system for automatically retrieving and analyzing energy storage battery data based on MLLM provided by another embodiment of the present invention; Figure 6 The step diagram of automatic retrieval and analysis in the method for automatically retrieving and analyzing energy storage battery data based on MLLM provided by another embodiment of the present invention; Figure 7 The data flow diagram during the automatic retrieval and analysis process in the method for automatically retrieving and analyzing energy storage battery data based on MLLM provided by another embodiment of the present invention.

[0018] The reference signs are as follows: multi-modal encoding module 1, MLLM model construction and training module 2, automatic retrieval and analysis module 3. Detailed implementation manners

[0019] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0020] Example 1 Please refer to Figure 1 , in one embodiment, a method for automatically retrieving and analyzing energy storage battery data based on MLLM is given, which is mainly implemented through the following steps: S1. Multi-modal encoding: Receive the multi-modal data of the energy storage battery and encode it into unified modal features; S2. MLLM model construction and training: Construct an MLLM model that captures the correlation between features through a multi-head attention mechanism, and input the unified modal features obtained by encoding into the MLLM model for training; S3. Automated Retrieval and Analysis: Generate a database by combining multimodal data of energy storage batteries, retrieve the received user questions in the database, and input the retrieval results and user questions into the trained MLLM model for processing and analysis to generate an answer response as the prediction result.

[0021] Please refer to Figure 2 , in one implementation manner, correspondingly, an automated retrieval and analysis system for energy storage battery data based on MLLM is provided to implement the method of the above implementation manner, which mainly consists of the following parts: Multimodal Encoding Module 1: Used to receive multimodal data of energy storage batteries and encode them into unified modal features; MLLM Model Construction and Training Module 2: Used to construct an MLLM model that captures the correlation between features through the multi-head attention mechanism, and input the encoded unified modal features into the MLLM model for training; Automated Retrieval and Analysis Module 3: Used to generate a database by combining multimodal data of energy storage batteries, retrieve the received user questions in the database, and input the retrieval results and user questions into the trained MLLM model for processing and analysis to generate an answer response as the prediction result.

[0022] The present invention realizes the semantic-level connection between the physical world of the battery and the digital system through an intelligent multimodal large language model (MLLM). Operators can directly obtain in-depth analysis results by asking questions in natural language, output operation guidance through natural language interaction, without relying on professional experience, reduce the operation and maintenance threshold and improve efficiency, replace traditional manual statistics and shallow monitoring, automatically generate reusable analysis logic, and avoid subjective misjudgment.

[0023] Example 2 On the basis of Example 1, a more detailed and preferred implementation manner is given: First, an automated retrieval and analysis method for energy storage battery data based on MLLM is provided, which is mainly realized through the following steps: S1. Multimodal Encoding: Receive multimodal data of energy storage batteries and encode them into unified modal features. The multimodal data in this embodiment mainly includes the following data types: text data, structured data, image data, and video data of energy storage batteries.

[0024] Please refer to Figure 3 , for these multimodal data, the multimodal encoding in this embodiment is specifically realized through the following steps: S11. Map text data, structured data, image data, and video data to the same semantic space to obtain text vectors of a unified modality. Further, in this step, for the unified modality, the text data can be converted into text word vectors through a tokenizer, the image feature vectors of the image data and video data can be extracted through a visual encoder, and the structured data can be mapped into numerical vectors through a linear transformation. The numerical vectors, text word vectors, and image feature vectors are mapped to the same semantic space through a connector, thereby obtaining text vectors of a unified modality.

[0025] S12. Tokenize and token-embed the text vectors to output token embeddings, where the token embeddings are Token sequences for input into the MLLM model for training.

[0026] S13. Perform positional encoding on the Token sequence, and input the Token sequence after positional encoding into the MLLM model.

[0027] S2. Construction and training of the MLLM model: Construct an MLLM model that captures the correlation relationships between features through a multi-head attention mechanism, and input the unified modality features obtained through encoding into the MLLM model for training.

[0028] S3. Automated retrieval and analysis: Combine the multi-modal data of the energy storage battery to generate a database, retrieve the received user questions in the database, and input the retrieval results and the user questions into the trained MLLM model for processing and analysis to generate an answer response as the prediction result.

[0029] Please refer to Figure 4 , the processing process of the MLLM model in this embodiment is as follows: S3a. Input embedding, input the obtained token embeddings; S3b. Multi-head attention fusion, calculate the correlation weights between the token embeddings through the multi-head attention mechanism, and perform linear projection fusion after concatenating multiple attention heads; S3c. Layer normalization and residual connection, perform addition and layer normalization processing on the fused result; S3d. Output embedding, output the result after layer normalization processing in the form of blocks, outputting block embeddings.

[0030] Please refer to Figure 5 , in a preferred embodiment, after the above steps are completed within the MLLM model, the automated retrieval and analysis method for energy storage battery data based on MLLM further performs: output projection, flatten and linearize the block embeddings to obtain the prediction result.

[0031] Please refer to Figure 5, in a more preferred embodiment, based on the above embodiment, the multi-modal encoding further includes the following steps: Batch reprogramming: Obtain reprogrammed block embeddings through block embedding, linearization, text prototyping, and introducing a multi-head attention mechanism to optimize the embedding representation.

[0032] Please refer to Figure 5 , when the data passes through the MLLM model, forward propagation is performed, and the MLLM model parameters are optimized through backpropagation.

[0033] Secondly, the core part of an energy storage battery data automatic retrieval and analysis system based on MLLM is given, that is, an architecture and workflow for processing data with the MLLM model, mainly used to execute steps S1 and S2 of the above embodiments. Please refer to Figure 5 , mainly composed of the following parts: (1) Frozen part and training part: The frozen part (blue snowflakes) and the training part (orange flames) are marked at the bottom of the figure, indicating that some parts of the model are frozen (parameters are not updated), while some parts participate in training (parameters are updated).

[0034] (2) Input and output parts, that is, the pre-trained large language model (embedding layer): Input part, steps S11 to S13 described above can be executed.

[0035] The system can process text input and perform tokenization processing. The final output includes prompt embedding and chunk embedding.

[0036] Word embedding (i.e., chunk embedding) generation: The input text generates a word vector x through a linear transformation embed : ; where W e is the pre-trained embedding matrix, x token is the index vector after word segmentation, and b e is the bias term of the chunk embedding.

[0037] Positional encoding: Inject sequence position information through sine / cosine functions to enhance the model's perception of the input order.

[0038] Output part, the "output projection" step described above can be executed. Through the output projection layer, linear transformation and flattening operations are performed on the output of the model.

[0039] (3) MLLM model, that is, the pre-trained large prediction model (ontology): The "Pre-trained Large Language Model (the ontology)" in the center of the figure is the core module. Above it is the output projection layer, which performs linear transformation and flattening operations on the output of the model. This module is also connected to the output embedding module, which processes data through addition and layer normalization, as well as forward propagation. The multi-head attention mechanism is also related to this module and is used to capture long-range dependencies in the input data. The MLLM model, that is, the pre-trained large prediction model (the ontology), can execute the above steps S3a to S3d. Among them, Attention mechanism stage: Calculate the inter-word correlation weights in the Transformer layer to support parallel semantic modeling.

[0040] ; Among them, Attention(Q, K, V) represents the "attention calculation" operation, which takes three groups of vectors: input query (Query), key (Key), and value (Value), and outputs the result weighted by attention, reflecting the "attention degree" of different positions to the current position. QK T is the matrix multiplication of the transpose of Query and Key, calculating the similarity (inner product) between Query and each Key. Each element in the result matrix reflects the correlation degree between the corresponding position Query and Key. d k is the scaling factor, and d k is the dimension of the QK vector. Since the inner product calculation will cause the numerical value to expand as the dimension increases, dividing by this factor for scaling makes the Softmax more stable (avoiding problems such as gradient disappearance). Softmax is a normalization function that converts the similarity scores into a probability distribution (weights) between 0 and 1, making large scores more prominent and small scores closer to 0, reflecting "attention focus".

[0041] Multi-head attention fusion: Concatenate multiple attention heads and perform linear projection to enhance the ability to capture different semantic dimensions.

[0042] Layer normalization and residual connection: That is, the addition and layer normalization operations in the figure, which standardize the output and retain the original information to obtain the output y, where SubLayer can be self-attention or a feed-forward network.

[0043] ; Among them, LayerNorm is layer normalization. It normalizes the input vector (or matrix) along the layer dimension, calculates the mean and variance, and scales and adjusts the data distribution. Sublayer is the calculation of the sublayer of self-attention. First, perform self-attention on x, and then pass through the feed-forward network to output the sublayer result; the purpose is to perform feature transformation and extraction on the input x.

[0044] Loss calculation stage: Calculate the prediction loss. (4)Batch reprogramming and instance normalization: On the right side of the figure is the reprogramming block embedding mechanism. Through multi-head attention and some text prototype processing, block embeddings are generated. The operations involved include time series block processing and pre-trained word embeddings. The embedding representation is further optimized through batch reprogramming and instance normalization.

[0045] (6)Forward propagation and backpropagation: When data passes through the model, forward propagation (Forward Pass) is performed, as shown by the black arrows in the figure, and backpropagation (Backpropagation) is performed, as shown by the red arrows in the figure, to optimize the model parameters.

[0046] Gradient clipping: Limit the gradient norm to prevent numerical instability, such as Figure 5 , which is performed on the frozen input embedding part. ; Among them, θ is the preset threshold. g is the gradient, θ is the threshold for gradient clipping, ||g|| is the norm of the gradient g, which is used to measure the "length" or "scale" of the gradient, and min: represents the minimum value operation. The overall formula is the gradient update operation, multiplying the original gradient g by the above minimum value to obtain the clipped gradient. When the gradient norm exceeds the threshold θ, the gradient is scaled to limit its magnitude; when it does not exceed, the gradient remains unchanged.

[0047] During the model training process, the AdamW optimizer is introduced in the multi-head attention mechanism to update and combine momentum with adaptive learning rate to adjust parameters, balancing the gradient direction and parameter scale.

[0048]

[0049] Among them, m t is the first-order momentum at the t-th iteration step, which is the exponential moving average of the gradient, used to accumulate historical gradient information and reflect the "trend" of the gradient. v t is the second-order momentum at the t-th iteration step, which is the exponential moving average of the gradient square, used to accumulate historical gradient square information and reflect the "fluctuation degree" of the gradient. β1 and β2 are the momentum decay coefficients, and g tis the gradient calculated at the t-th iteration step, and θ t is the value of the model parameters after the t-th iteration step. η is the learning rate, which controls the "step size" of parameter updates and determines the degree to which each gradient affects the parameters. λ is the weight decay coefficient used for regularization to prevent the model from overfitting. ϵ is a very small positive number (usually taking values such as 10 −8 etc.) to prevent division-by-zero errors during the calculation process.

[0050] Output stage ( Figure 5 Output embedding in freezing): Generate the probability distribution, and generate the output probability through Softmax, where h is the last hidden state.

[0051]

[0052] Among them, P(w i ∣h) is the conditional probability, which represents the probability that the output is the word w i given the context representation h. In natural language processing tasks, it is used to measure the likelihood of a certain word appearing in a specific context environment. exp is the exponential function, which is used to map the input value to the positive real number domain to make the numerical distinction more obvious and prepare for subsequent normalization. W o is the output weight matrix, which is a learnable parameter of the model; b o is the output bias term; W o h + b o performs a linear transformation on the context representation h. The superscript (i) is used to identify the parameters related to the word w i represents the row vector in the output weight matrix W o corresponding to the word w i . The superscript (j) is a summation index used to iterate over all words in the vocabulary. represents the row vector in the output weight matrix W o corresponding to the j-th word in the vocabulary. represents the element in the output bias term b o corresponding to the word w i represents the element in the output bias term b o corresponding to the j-th word.

[0053] Please refer to Figure 6 , and the specific steps of automated retrieval and analysis include: S31. Multimodal vectorized storage: Segment the multimodal data of the energy storage battery into text segments and store them vectorized in the vector database; S32. Question vectorization and similarity search: Receive the user's question and vectorize it, and retrieve similar text segments in the vector database;​​ S33. MLLM Analysis and Prediction: The retrieved similar text segments and the vectorized user question are processed and analyzed in the trained MLLM model to generate an answer response as the prediction result.

[0054] For more specific details, please refer to Figure 7 , the process of automated retrieval-augmented generation (RAG): The energy storage battery data in unstructured or structured form will be converted into vector form as text segments and stored in the vector database. The vector database is mainly used for similarity search and retrieval. The user inputs a question, and this question will also be embedded into the vector database. The vector question is retrieved in the vector database to find the text segment most similar to the question. The retrieved text segment and the question are provided to the MLLM model, and the MLLM model generates a response based on the question and the context. The context refers to the previous one / several questions input into the model. When asking questions subsequently, the results are output in combination with the previous one / several questions asked. Figure 7 The "prompt words" in are used to prompt whether there are the same or similar words in the question input by the user in the database. Simply put, the new data generated by the energy storage battery will be stored in an unstructured form and then sent to the vector database for retrieval. After waiting for the user to input a question, the user's question is retrieved in the vector database, and then the question is answered. Two application examples are given in this embodiment.

[0055] Example 1: After entering the system, an icon for using the large model will appear in the lower right corner. Then, input the question "How to perform capacity verification operation", and the generated answer response is: "Step 1: Find the site details platform. Step 2: Select the power type. Step 3: Perform the capacity verification operation." Each of the three steps can correspond one by one to the navigation bar in the system, realizing semi-automated system management.

[0056] Example 2: The user inputs the question: "Briefly describe the relevant situation of the state of health (SOH) of the energy storage battery", and the generated answer response is: "The current state of health (SOH) of the battery is as follows: According to the data in Document 1, the SOH value of the battery is currently between 0.9465 and 0.9780, and the specific values fluctuate slightly with the increase in the number of cycles. These values indicate that the battery is in good health and is close to the state of a new battery. Generally, the closer the SOH value is to 1, the better the health state of the battery and the higher the capacity retention rate. In addition, the document also provides the remaining useful life (RUL) value, indicating the number of cycles that the battery can still perform. The current RUL value is between 395 and 419 cycles, indicating that the battery still has a relatively long service life. In summary, the current SOH situation of the battery is good, the health state is high, and the remaining useful life is sufficient."

[0057] Based on the above embodiments, the present invention can achieve the following beneficial technical effects: (1) The present invention realizes the semantic-level connection between the physical world of the battery and the digital system through an intelligent multi-modal large language model (MLLM). Operators can directly obtain in-depth analysis results by asking questions in natural language and output operation guidance through natural language interaction, without relying on professional experience, reducing the operation and maintenance threshold and improving efficiency, replacing traditional manual statistics and shallow monitoring, automatically generating reusable analysis logic, and avoiding subjective misjudgment. It can achieve a closed-loop from data monitoring → multi-modal analysis → decision-making response, providing predictive maintenance for energy storage batteries, such as core functions like remaining useful life (RUL) prediction and root cause analysis of faults.

[0058] (2) Through multi-modal unified encoding technology, the present invention maps the text data, structured data, image data, and video data of the energy storage battery to the same semantic space, generates text vectors of a unified modality, and then based on the MLLM model, captures the correlation between cross-modal features through the multi-head attention mechanism, thereby solving the pain point that text logs, numerical indicators, and image and video data in traditional systems cannot be jointly analyzed, realizing semantic-level fusion analysis of multi-modal data, breaking cross-modal information islands, and improving the accuracy of analysis and prediction results.

[0059] (3) Through optimization techniques, namely batch reprogramming, the present invention optimizes the embedding representation through linearization and multi-head attention, improves the few-shot learning ability, optimizes the embedding representation of rare fault modes, and improves the generalization ability for scenarios with insufficient historical samples.

[0060] (4) Through an automated retrieval-augmented generation (RAG) architecture, the present invention can segment and vectorize battery data for storage in a vector database, achieving multi-modal vectorized storage. Vector database retrieval + MLLM generation realizes an integrated response of "retrieval - analysis".

[0061] In summary, through the technical combination of MLLM + RAG + multi-modal encoding, the present invention solves the three core problems of data fragmentation, high analysis threshold, and poor generalization ability in the field of energy storage batteries, and finally realizes the semantic-level connection of "physical world - digital system", providing intelligent operation and maintenance support for new energy infrastructure.

[0062] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), or the like.

[0063] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification. And the above embodiments only express several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but they cannot be understood as a limitation on the scope of the present invention. For those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. An automated retrieval and analysis method for energy storage battery data based on MLLM, characterized in that Including: Multimodal encoding: Receiving multimodal data of the energy storage battery and encoding it into unified modal features; MLLM model construction and training: Constructing an MLLM model that captures the correlation relationships between features through the multi-head attention mechanism, and inputting the unified modal features obtained by encoding into the MLLM model for training; Automated retrieval and analysis: Combining the multimodal data of the energy storage battery to generate a database, retrieving the received user questions in the database, and inputting the retrieval results and the user questions into the trained MLLM model for processing and analysis to generate an answer response as the prediction result.

2. The automated retrieval and analysis method for energy storage battery data based on MLLM according to claim 1, characterized in that, The multimodal data includes: text data, structured data, image data, and video data of the energy storage battery; the multimodal encoding specifically includes: mapping the text data, structured data, image data, and video data to the same semantic space to obtain a unified modal text vector.

3. The method for automatically retrieving and analyzing energy storage battery data based on MLLM according to claim 2, wherein The multimodal encoding further includes: text vector tokenization and token embedding processing to output token embeddings for input into the MLLM model for training.

4. The method for automatically retrieving and analyzing energy storage battery data based on MLLM according to claim 3, wherein, The processing process of the MLLM model includes: input embedding, inputting the obtained token embeddings; multi-head attention fusion, calculating the correlation weights between token embeddings through the multi-head attention mechanism, and performing linear projection fusion after splicing multiple attention heads; layer normalization and residual connection, adding and performing layer normalization processing on the fused results; output embedding, outputting the results of the layer normalization processing in the form of blocks to output block embeddings.

5. The automated retrieval and analysis method for energy storage battery data based on MLLM according to claim 4, wherein Also including: Output projection, flattening and linearizing the block embeddings to obtain the prediction result.

6. The method for automated retrieval and analysis of energy storage battery data based on MLLM according to claim 3, wherein, The token embeddings are Token sequences, and the multimodal encoding further includes: performing positional encoding on the Token sequences, and inputting the Token sequences after positional encoding into the MLLM model.

7. The method for automatically retrieving and analyzing energy storage battery data based on MLLM according to claim 4, wherein When data passes through the MLLM model, forward propagation is performed, and the parameters of the MLLM model are optimized through backpropagation.

8. The method for automatically retrieving and analyzing energy storage battery data based on MLLM according to claim 3, wherein The multimodal encoding further includes: batch reprogramming: obtaining reprogrammed block embeddings through block embeddings, linearization, and text prototyping processing and introducing the multi-head attention mechanism to optimize the embedding representation.

9. The method for automatically retrieving and analyzing energy storage battery data based on MLLM according to claim 1, wherein The specific steps of the automated retrieval and analysis include: Multimodal vectorized storage: Segmenting the multimodal data of the energy storage battery into text segments and vectorized storing them in a vector database; Question vectorization and similarity search: Receiving the user question and vectorizing it, and retrieving similar text segments in the vector database; MLLM analysis and prediction: Inputting the retrieved similar text segments and the vectorized user question into the trained MLLM model for processing and analysis to generate an answer response as the prediction result.

10. A storage battery data automatic retrieval and analysis system based on MLLM, which is used to implement the method described in any one of the above claims 1 to 9, and is characterized in that, Including: Multimodal encoding module: Used to receive the multimodal data of the energy storage battery and encode it into unified modal features; MLLM model construction and training module: Used to construct an MLLM model that captures the correlation relationships between features through the multi-head attention mechanism, and input the unified modal features obtained by encoding into the MLLM model for training; Automated Retrieval and Analysis Module: It is used to generate a database by combining multimodal data of energy storage batteries, retrieve the received user questions in the database, and input the retrieval results and user questions into a trained MLLM model for processing and analysis to generate an answer response as the prediction result.

Images