Power document intelligent generation method based on multi-modal memory fusion

Abstract

The application relates to a power document intelligent generation method based on multimodal memory fusion, and comprises the following steps: S1: obtaining power business original data and preprocessing to obtain preprocessed power multimodal data; S2: according to the preprocessed power multimodal data, performing multimodal representation and alignment to obtain a power multimodal vector; S3: according to the power multimodal vector, an original document pointer and metadata, constructing a power document multimodal memory library; S4: according to a user intention or instruction, performing intention analysis and slot filling to obtain a candidate evidence set strongly related to a task; and S5: according to the candidate evidence set, performing deep context correlation analysis, and through a cooperative screening mechanism, obtaining a final power document. The application significantly improves the comprehensiveness and accuracy of document content.

Description

Technical Field

[0001] This invention relates to the field of intelligent data processing, and in particular to an intelligent generation method for power documents based on multimodal memory fusion. Background Technology

[0002] With the intelligent and digital transformation of power systems, power companies have accumulated a large amount of multi-source, heterogeneous data in their operations, including equipment operation and maintenance, repair, dispatching, and safe production. This data encompasses text reports, statistical tables, on-site photos, engineering drawings, and even voice recordings, exhibiting significant multimodal characteristics. Traditional power documentation relies heavily on manual compilation and a single data source, resulting in cumbersome documentation processes, low information integration efficiency, and difficulty in fully mining and utilizing the deep knowledge contained within multimodal data.

[0003] In recent years, the rapid development of artificial intelligence, especially cutting-edge technologies such as natural language processing and multimodal learning, has provided new technological possibilities for intelligent data processing and automatic document generation in the power industry. Existing automatic document generation systems mostly rely on structured data or plain text, making it difficult to effectively and uniformly model and integrate multimodal information such as images, drawings, and audio. Furthermore, existing solutions generally lack comprehensive support for data traceability, enhanced retrieval, and traceable evidence chains for documents, limiting the widespread application of intelligent document generation systems in the complex scenarios of the power industry. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide an intelligent power document generation method based on multimodal memory fusion, which significantly improves the comprehensiveness and accuracy of document content.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The intelligent generation method for power documents based on multimodal memory fusion includes the following steps:

[0007] S1: Obtain raw power business data and preprocess it to obtain preprocessed multimodal power data;

[0008] S2: Based on the preprocessed power multimodal data, perform multimodal characterization and alignment to obtain the power multimodal vector;

[0009] S3: Construct a power document multimodal memory based on the power multimodal vector, the original document pointer, and metadata;

[0010] S4: Based on user intent or instructions, perform intent parsing and slot filling to obtain a set of candidate evidence strongly related to the task;

[0011] S5: Based on the candidate evidence set, conduct in-depth contextual relevance analysis and obtain the final power documents through a collaborative screening mechanism.

[0012] Furthermore, based on the preprocessed power multimodal data, multimodal characterization is performed, as follows:

[0013] Using a pre-trained BERT model as the base encoder, and through expansion of the power industry terminology dictionary and fine-tuning of the domain corpus, the model's ability to understand professional vocabulary and grammatical structures in the power industry is enhanced, resulting in the power encoder BERT. power ;

[0014] Based on the power encoder, a hierarchical encoding strategy is adopted for encoding the preprocessed power text data: operation tickets and work tickets retain their form field information; defect records and maintenance reports are encoded after being segmented into paragraphs; and equipment ledgers are encoded in combination with row and column structures.

[0015] ;

[0016] Where, X text This represents the preprocessed power text data; H text The changed power text data;

[0017] Capture semantic dependencies within text using attention mechanisms:

[0018] ;

[0019] ;

[0020] ;

[0021] Where, α i,j e represents the attention weight between position i and position j; i,j Let be the attention energy score from position i to position j. For multilayer perceptron; h i ,h j H represents power text data. text The hidden state vector representation of the i-th and j-th tokens; W a Here is the learnable attention parameter matrix; T is the total sequence length; c i This is the final aggregated context vector at position i; Represents vector h i The transpose of;

[0022] Finally, each text fragment is mapped to a dense vector representation of a fixed dimension, forming a feature vector set for the text modality:

[0023] ;

[0024] Among them, v text This represents the global feature vector of the text modality; It is a feedforward neural network; For layer normalization operation;

[0025] For the preprocessed power image data, a convolutional neural network is used for multi-scale feature extraction:

[0026] ;

[0027] v image =GlobalPool(ResNet / F multi );

[0028] Where I is the input image; Pool k (I) Pooling the input image I at scale k to obtain the downsampled image features; Conv k×k A k×k convolution operator is used to perform convolution operations on the pooled image features to extract spatial features at that scale; F scalek The feature map obtained at scale k; To merge feature maps at different scales along the channel dimension and achieve multi-scale feature fusion; F scale1 ,F scale2 ,F scale3 F represents the convolutional feature maps at different scales; multi This represents the fused multi-scale feature representation; GlobalPool represents the pooling function; ResNet is the convolutional feature extractor.

[0029] Each image is ultimately encoded into a high-dimensional feature vector v. image This effectively preserves its visual semantic information;

[0030] For the preprocessed tabular data, the Transformer-based tabular encoding model Tapas is used to jointly model the row headers, column headers, and cell contents of the table:

[0031] ;

[0032] ;

[0033] Among them, T content The text content of the table cell; T header For table column headers, specify the semantic labels or field names for each column; P row For row position encoding;

[0034] Pcol Encoding for column positions; H table The set of encoded vectors for all cells in the table output by the Tapas model represents the serialized vector representation of the entire table; TimeEmbedding(t) represents the embedding vector at time t; H t The table feature vector at each time step; As an attention mechanism, different weights are assigned to the cell vectors of the input sequence Ht, and weighted aggregation is performed to obtain the overall table features; v table This is the final global feature vector for the table modality.

[0035] Furthermore, multimodal representation alignment is performed as follows:

[0036] A multimodal pairing dataset in the power sector was constructed, including text-image modality pairs, text-table modality pairs, and image-table modality pairs. A loss function was learned through contrastive learning. This makes semantically related data from different modalities closer together in the vector space, and semantically unrelated data farther apart:

[0037] ;

[0038] Where N is the number of samples in the batch; Let i be the feature vector of the i-th sample in mode a; Let i be the feature vector of the i-th sample in mode b; Let j be the feature vector of the j-th sample in mode b; τ is the temperature coefficient;

[0039] MLP is used as the mapping network between modalities to project the feature vectors of different modalities into a unified semantic space;

[0040]

[0041] ;

[0042] Among them, z text z is the projection vector of the text modality; image z is the projection vector of the image modality; table The projection vector of the table modality; MLP text Multilayer perceptron projector for text modalities; MLP image MLP projection head for image modality; MLP table is the MLP projection head for the tabular modality; u is the fused unified multimodal representation vector, serving as the final multimodal semantic representation; W ub is the weight matrix of the linear layer, used to linearly transform the concatenated vector; u For bias terms; Presentation layer normalization operation;

[0043] Simultaneously, by leveraging knowledge distillation techniques, teachers can guide students in online learning to achieve better cross-modal representations, thereby improving alignment quality and efficiency.

[0044] ;

[0045] in, For knowledge distillation loss; The Kullback-Leibler divergence; z student ,z teacher These are the output vectors of the student network and the teacher network, respectively. is the softmax function normalized to a probability distribution; T is the distillation temperature parameter;

[0046] The total loss function is:

[0047] ;

[0048] Among them, L total Let λ be the overall loss objective for multimodal training; λ1 is the weight controlling the contribution of knowledge distillation loss to the total loss; λ2 is the weight controlling the contribution of regularization loss to the total loss; L reg This is the loss due to regularization.

[0049] Furthermore, based on the power multimodal vector, the original document pointer, and metadata, a power document multimodal memory is constructed, as follows:

[0050] Based on the standardized power multimodal vectors obtained in step S2, a vector database is constructed as the core storage layer of the memory, adopting a distributed vector database architecture. A multi-dimensional indexing system is built for the vector database. First, an approximate nearest neighbor index based on vector similarity is established, using IVF, HNSW, or LSH algorithms, selecting the optimal indexing strategy based on the vector dimension and data scale. Simultaneously, traditional metadata-based indexes are established, including time indexes, device indexes, business type indexes, and keyword inverted indexes. Composite indexes are constructed to support multi-condition combined queries. The indexes employ an incremental update mechanism, asynchronously updating relevant indexes when new data is written, ensuring the real-time nature and consistency of query results.

[0051] Based on the vector database, a knowledge graph for the power industry is constructed to enhance the semantic understanding capability of the memory. Each record in the vector database is mapped to the corresponding node in the knowledge graph through entity linking technology, enabling collaborative work between vector retrieval and graph reasoning. A cross-modal association mapping mechanism is established, which establishes associations for different modal data from the same business scenario through business logic, spatiotemporal relationships, and content semantic dimensions. A multimodal association graph is constructed, with business events as the central node, connecting related text, images, tables, and other data of different modalities as child nodes to form a complete event knowledge network.

[0052] Furthermore, based on user intent or instructions, intent parsing and slot filling are performed, as follows:

[0053] When a user inputs a request to generate an electricity document, the system first performs deep semantic analysis and intent recognition on the user's natural language command, and then adopts an intent classification model based on BERT. The intent classification model uses a multilayer perceptron architecture, and outputs the probability distribution of each intent category through a softmax function, selecting the category with the highest probability as the primary intent.

[0054] Based on the identified user intent, sequence labeling technology is used for slot filling to extract key structured information from user instructions; BiLSTM-CRF model is used for named entity recognition and slot labeling, and knowledge graphs in the power industry are combined for entity linking and disambiguation to ensure the accuracy and standardization of extracted information.

[0055] Furthermore, a set of candidate evidence strongly relevant to the task is obtained, specifically as follows: Based on the parsed intent and slot information, a retrieval strategy based on vector semantic similarity encodes the user query into a vector representation and performs an approximate nearest neighbor search in the vector database to recall semantically relevant candidate documents; secondly, a traditional retrieval strategy based on keyword matching uses the BM25 algorithm to perform precise matching retrieval based on keywords in the slots; a reasoning retrieval strategy based on knowledge graphs performs multi-hop reasoning through entity relationships in the graph to discover indirectly relevant evidence; and a structured retrieval based on metadata filtering performs precise filtering based on metadata conditions such as time, device, and business type. The above retrieval results are then merged using a weighted fusion algorithm to obtain a set of candidate evidence strongly relevant to the task.

[0056] Furthermore, based on the candidate evidence set, an in-depth contextual relevance analysis is performed, as follows:

[0057] Based on the user's query intent Q and the candidate evidence set, a relevance analysis is performed, including semantic relevance, business logic relevance, and spatiotemporal relevance analysis.

[0058] The semantic relevance analysis is based on the query vector q and the evidence modality vector z. m (d) Weighted max pooling:

[0059] ;

[0060] Where d represents candidate evidence; m represents modality type; ω m The weights of mode m; s(q,z) m (d) is the similarity function, z m (d) is the vector representation of evidence d in mode m; The maximum weighted similarity across all modalities is used to represent the final semantic relevance score; SemRel(Q, d) represents the semantic relevance.

[0061] Enhanced slot alignment:

[0062] ;

[0063] in, This is a collection of slot types, containing key information categories in the power sector; Total number of slot types; δ f The importance weight of slot f; f Q To query the value of slot f; f d The value of slot f in the document; For standardized functions; II Here, SlotRel(Q,d) is the indicator function; SlotRel(Q,d) is the slot correlation.

[0064] The aforementioned business logic relevance is based on the power knowledge graph G=(V,E), which maps queries and evidence to a subgraph G. Q G d Graph matching and causal chain consistency scoring are used:

[0065] ;

[0066] Among them, G Q G is the business logic subgraph extracted from the query; d The business logic subgraph extracted from the document; η1 is the graph structure similarity weight; η2 is the path alignment weight; GED(G Q G d ) represents the graph edit distance; PathAlign(G) Q G d ) represents the alignment of critical business paths; BizRel(Q,d) represents the business logic relevance;

[0067] Based on the above correlation analysis, obtain the comprehensive contextual relevance Rel(Q,d):

[0068] Rel(Q,d)=α1SemRel(Q,d)+α2SlotRel(Q,d)+α3BizRel(Q,d);

[0069] Where α1, α2, and α3 are weighting coefficients.

[0070] Furthermore, a collaborative filtering mechanism is used to obtain the final power documents, as follows:

[0071] Based on the candidate evidence set after correlation analysis, a heuristic greedy algorithm is used to obtain the optimal evidence subset; then, the evidence is mapped to chapters according to the target document template; statistical tables and graphs are generated for structured data and time series curves; the graphs are presented in the document as placeholder titles and explanatory text, and bidirectional links are formed with the original tables or graphs;

[0072] During the content generation phase, industry terminology and corporate writing standards are followed, and consistency checks are performed on indicators, time, and equipment numbers across paragraphs to avoid inconsistencies.

[0073] After generation, the quality gate is jointly implemented by the rules and the model. If any missing items or conflicts are found, the process is backed up to the candidate pool for targeted supplementation and replacement. If a high-risk judgment is found, a manual review process is triggered. The final output is a complete, traceable, and modal power document.

[0074] The present invention has the following beneficial effects:

[0075] 1. This invention achieves semantic-level fusion of information from different modalities through domain-adapted pre-trained models and cross-modal alignment technology, enabling the generated documents to comprehensively utilize rich information sources such as on-site photos, equipment parameter tables, and historical trend charts, significantly improving the comprehensiveness and accuracy of the document content;

[0076] 2. This invention performs in-depth intent parsing and automatic key slot filling based on complex user business instructions. It can flexibly understand the differences in various power task scenarios and dynamically organize information points. In the downstream generation process, it adopts in-depth contextual relevance analysis and multi-objective collaborative screening to accurately match user needs with actual evidence, and realize end-to-end unmanned intelligent generation from evidence to structured documents.

[0077] 3. Construct a multimodal memory library and a full-process evidence chain tracking system. All document content is supported by original data and evidence, and every step of the operation is traceable. Through in-depth correlation analysis and multi-dimensional quality control, we ensure that the output power documents have a clear structure. Attached Figure Description

[0078] Figure 1This is a flowchart of the method of the present invention. Detailed Implementation

[0079] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0080] refer to Figure 1 In this embodiment, a method for intelligent generation of power documents based on multimodal memory fusion is provided, including the following steps:

[0081] S1: Obtain raw power business data and preprocess it to obtain preprocessed multimodal power data;

[0082] S2: Based on the preprocessed power multimodal data, perform multimodal characterization and alignment to obtain the power multimodal vector;

[0083] S3: Construct a power document multimodal memory based on the power multimodal vector, the original document pointer, and metadata;

[0084] S4: Based on user intent or instructions (natural language or forms), perform intent parsing and slot filling to obtain a set of candidate evidence strongly related to the task;

[0085] S5: Based on the candidate evidence set, conduct in-depth contextual relevance analysis to ensure that the evidence is highly matched with user needs, and obtain the final power document through a collaborative screening mechanism.

[0086] In this embodiment, the raw data of the power business includes text data, image data, and tabular data. The text data includes operation tickets, work tickets, defect records, maintenance reports, accident analyses, duty logs, and equipment ledgers. The image data includes on-site inspection photos, equipment nameplate photos, infrared thermal images, drone aerial images, and key frames of monitoring videos. The tabular data is extracted from historical databases and real-time databases and includes equipment operating parameters, power consumption statistics, fault statistics, and maintenance plans.

[0087] In this embodiment, preprocessing is performed as follows: The collected raw data undergoes quality assessment and cleaning. For text data, invalid characters are removed, encoding errors are corrected, formatting is standardized, and sensitive information involving trade secrets and personal privacy is identified and de-identified. For image data, noise reduction, distortion correction, and resolution standardization are performed, and blurry or overexposed images are removed. For tabular data, missing value processing, outlier detection, and data type standardization are performed. A unified multimodal data storage format and metadata standard are established, converting text data into a UTF-8 encoded structured format while preserving the original document's chapter hierarchy and semantic tags. Image data is uniformly compressed into JPEG format, and thumbnails are generated for quick preview. Tabular data is standardized into CSV format, clearly defining field meanings, data types, and units of measurement. Complete metadata information is added to each piece of multimodal data, including data source, creation time, modification time, associated device number, and business scenario tag. Data lineage is established, recording the data derivation and transformation process to ensure data traceability.

[0088] In this embodiment, multimodal characterization is performed based on the preprocessed power multimodal data, as follows:

[0089] Using a pre-trained BERT model as the base encoder, and through expansion of the power industry terminology dictionary and fine-tuning of the domain corpus, the model's ability to understand professional vocabulary and grammatical structures in the power industry is enhanced, resulting in the power encoder BERT. power ;

[0090] Based on the power encoder, a hierarchical encoding strategy is adopted for encoding the preprocessed power text data: operation tickets and work tickets retain their form field information; defect records and maintenance reports are encoded after being segmented into paragraphs; and equipment ledgers are encoded in combination with row and column structures.

[0091] ;

[0092] Where, X text This represents the preprocessed power text data; H text The changed power text data;

[0093] Capture semantic dependencies within text using attention mechanisms:

[0094] ;

[0095] ;

[0096] ;

[0097] Where, α i,j e represents the attention weight between position i and position j;i,j Let be the attention energy score from position i to position j. For multilayer perceptron; h i ,h j H represents power text data. text The hidden state vector representation of the i-th and j-th tokens; W a Here is the learnable attention parameter matrix; T is the total sequence length; c i This is the final aggregated context vector at position i; Represents vector h i The transpose of;

[0098] Finally, each text fragment is mapped to a dense vector representation of a fixed dimension, forming a feature vector set for the text modality:

[0099] ;

[0100] Among them, v text This represents the global feature vector of the text modality; It is a feedforward neural network; For layer normalization operation;

[0101] For the preprocessed power image data, a convolutional neural network is used for multi-scale feature extraction:

[0102] ;

[0103] v image =GlobalPool(ResNet / F multi );

[0104] Where I is the input image; Pool k (I) Pooling the input image I at scale k to obtain the downsampled image features; Conv k×k A k×k convolution operator is used to perform convolution operations on the pooled image features to extract spatial features at that scale; F scalek The feature map obtained at scale k; To merge feature maps at different scales along the channel dimension and achieve multi-scale feature fusion; F scale1 ,F scale2 ,F scale3 These are convolutional feature maps at different scales (e.g., 1, 2, 3); F multi This represents the fused multi-scale feature representation; GlobalPool represents the pooling function; ResNet is the convolutional feature extractor.

[0105] Each image is ultimately encoded into a high-dimensional feature vector v. imageThis effectively preserves its visual semantic information;

[0106] For the preprocessed tabular data, the Transformer-based tabular encoding model Tapas is used to jointly model the row headers, column headers, and cell contents of the table:

[0107] ;

[0108] ;

[0109] Among them, T content The text content of the table cell; T header For table column headers, specify the semantic labels or field names for each column; P row This is a row position code, describing the row number information of the cell in the table;

[0110] P col Column position coding describes which column the cell belongs to; H table This is the set of encoded vectors for all cells in the table output by the Tapas model, representing the serialized vector representation of the entire table; TimeEmbedding(t) represents the embedding vector at time t, reflecting the specific moment or periodicity of the data occurrence; H t Each table feature vector is a combination of its content and corresponding time information at each time point. As an attention mechanism, different weights are assigned to the cell vectors of the input sequence Ht, and weighted aggregation is performed to obtain the overall table features; v table This is the final global feature vector for the table modality.

[0111] In this embodiment, multimodal representation alignment is performed as follows:

[0112] A multimodal pairing dataset in the power industry is constructed, including text-image pairs (such as maintenance reports and on-site photos), text-table pairs (such as fault descriptions and statistical data), and image-table pairs (such as equipment status diagrams and operating parameters). A loss function is learned through comparison. This makes semantically related data from different modalities closer together in the vector space, and semantically unrelated data farther apart:

[0113] ;

[0114] Where N is the number of samples in the batch; Let be the feature vector of the i-th sample in modality a (such as text); Let i be the feature vector of the i-th sample in modality b (such as an image or table); Let j be the feature vector of the j-th sample in mode b; τ is a temperature coefficient used to adjust the smoothness of the distribution, and is usually a positive number (such as 0.07).

[0115] MLP is used as the mapping network between modalities to project the feature vectors of different modalities into a unified semantic space;

[0116]

[0117] ;

[0118] Among them, z text z is the projection vector of the text modality; image z is the projection vector of the image modality; table The projection vector of the table modality; MLP text Multilayer perceptron projector for text modalities; MLP image MLP projection head for image modality; MLP table is the MLP projection head for the tabular modality; u is the fused unified multimodal representation vector, serving as the final multimodal semantic representation; W u b is the weight matrix of the linear layer, used to linearly transform the concatenated vector; u For bias terms; Presentation layer normalization operation;

[0119] Simultaneously, by leveraging knowledge distillation techniques, teachers can guide students in online learning to achieve better cross-modal representations, thereby improving alignment quality and efficiency.

[0120] ;

[0121] in, For knowledge distillation loss; The Kullback-Leibler divergence measures the difference between two probability distributions; z student ,z teacher These are the output vectors of the student network and the teacher network, respectively. is the softmax function normalized to a probability distribution; T is the distillation temperature parameter, which controls the smoothness of the softmax output distribution;

[0122] The total loss function is:

[0123] ;

[0124] Among them, L total Let λ be the overall loss objective for multimodal training; λ1 is the weight controlling the contribution of knowledge distillation loss to the total loss; λ2 is the weight controlling the contribution of regularization loss to the total loss; Lreg This is a regularization loss, such as L2 regularization, used to control the parameter size and enhance the model's generalization ability.

[0125] In this embodiment, the feature vectors of each modality, after alignment learning, are mapped to a unified high-dimensional vector space, ensuring that vectors from different modalities have the same dimension and comparable semantic expressive power. A progressive training strategy is adopted, first training the encoders of each modality separately, then performing cross-modal alignment, and finally performing end-to-end joint optimization. Regularization techniques are introduced to prevent overfitting, and methods such as dropout and weight decay are used to improve the model's generalization ability. Simultaneously, vector quality evaluation metrics are designed, including intramodal consistency, cross-modal similarity, and semantic fidelity, to ensure that the generated power multimodal vectors retain the key information of the original data while possessing good cross-modal correlation capabilities. Finally, a standardized set of power multimodal vectors is output, with each vector carrying a pointer to the original data and metadata information, providing a high-quality feature representation foundation for subsequent memory construction and retrieval.

[0126] In this embodiment, a power document multimodal memory is constructed based on the power multimodal vector, the original document pointer, and metadata, as detailed below:

[0127] Based on the standardized power multimodal vectors obtained in step S2, a vector database is constructed as the core storage layer of the memory. A distributed vector database architecture, such as Milvus, Pinecone, or Faiss, is adopted. Each vector record contains a fixed-length feature vector (typically 512 or 768 dimensions), a file path pointer to the original data, complete metadata information, and a version control identifier, ensuring a strong correlation between the vectors and the original data and traceability of data lineage. A multi-dimensional indexing system is built for the vector database. First, an approximate nearest neighbor index based on vector similarity is established, using IVF (Inverted File System), HNSW (Hierarchical Navigable Small World Graph), or LSH (Locality Sensitive Hash) algorithms, selecting the optimal indexing strategy based on the vector dimension and data scale. Simultaneously, traditional metadata-based indexes are established, including a time index (supporting time range queries), a device index (for quick location by device number), a business type index (for document category classification), and a keyword inverted index (supporting text retrieval). A composite index is constructed to support multi-condition combined queries, such as complex query scenarios like "infrared detection reports of a certain substation within a specific time period." The indexes adopt an incremental update mechanism, asynchronously updating relevant indexes when new data is written, ensuring the real-time nature and consistency of query results.

[0128] Based on a vector database, a knowledge graph for the power industry is constructed to enhance the semantic understanding capabilities of the memory. Through entity recognition and relation extraction technologies, equipment entities, location entities, personnel entities, and event entities are extracted from power documents, and semantic relationships between them are established, including equipment-location, equipment-status, event-cause, and maintenance-equipment relationships. The knowledge graph is stored in RDF triple format and managed using Neo4j or other graph databases. Each record in the vector database is mapped to the corresponding node in the knowledge graph through entity linking technology, enabling collaborative work between vector retrieval and graph reasoning. To fully utilize the complementarity of multimodal data, a cross-modal association mapping mechanism is established. For different modal data originating from the same business scenario (such as text descriptions in maintenance reports, on-site photos, and equipment parameter tables), associations are established through business logic, spatiotemporal relationships, and content semantic dimensions. A multimodal association graph is constructed, with business events as the central node, connecting related text, images, and tables of different modalities as child nodes to form a complete event knowledge network.

[0129] In this embodiment, intent parsing and slot filling are performed based on user intent or instructions, as detailed below:

[0130] When a user inputs a request to generate power documents, the system first performs deep semantic analysis and intent recognition on the user's natural language command. It then employs a BERT-based intent classification model, pre-trained and fine-tuned on a professional corpus of power-related language, to accurately identify the user's core intent types, such as "generating fault analysis reports," "formulating equipment maintenance plans," "writing safe operating procedures," "writing technical upgrade plans," and "formulating emergency plans." The intent classification model uses a multilayer perceptron architecture, outputting the probability distribution of each intent category through a softmax function, and selecting the category with the highest probability as the primary intent.

[0131] Based on the identified user intent, sequence labeling technology is used for slot filling to extract key structured information from user instructions. A slot system specifically designed for the power industry is presented, including equipment slots (such as "220kV transformer", "switchgear", "busbar"), location slots (such as "500kV substation", "line section"), time slots (such as "January 2024", "last three months"), fault type slots (such as "grounding fault", "overload trip"), operation type slots (such as "power outage maintenance", "live work"), and document type slots (such as "technical report", "operation ticket", "work ticket"). BiLSTM-CRF model is used for named entity recognition and slot labeling, and a knowledge graph of the power industry is combined for entity linking and disambiguation to ensure the accuracy and standardization of extracted information.

[0132] In this embodiment, a set of candidate evidence strongly related to the task is obtained as follows: Based on the parsed intent and slot information, a retrieval strategy based on vector semantic similarity encodes the user query into a vector representation and performs an approximate nearest neighbor search in the vector database to recall semantically relevant candidate documents; secondly, a traditional retrieval strategy based on keyword matching uses the BM25 algorithm to perform precise matching retrieval based on keywords in the slots; a reasoning retrieval strategy based on knowledge graphs performs multi-hop reasoning through entity relationships in the graph to discover indirectly relevant evidence; and a structured retrieval based on metadata filtering performs precise filtering based on metadata conditions such as time, device, and business type, and the above retrieval results are merged through a weighted fusion algorithm to obtain a set of candidate evidence strongly related to the task.

[0133] In this embodiment, a deep contextual relevance analysis is performed based on the candidate evidence set, as detailed below:

[0134] Based on the user's query intent Q and the candidate evidence set (text, images, tables, etc.), a correlation analysis is performed, including semantic correlation, business logic correlation and spatiotemporal correlation analysis.

[0135] The semantic relevance analysis is based on the query vector q and the evidence modality vector z. m (d) Weighted max pooling:

[0136] ;

[0137] Where d represents candidate evidence; m represents modality type; ω m The weights of mode m; s(q,z) m (d) is the similarity function, z m (d) is the vector representation of evidence d in mode m; The maximum weighted similarity across all modalities is used to represent the final semantic relevance score; SemRel(Q, d) represents the semantic relevance.

[0138] Enhanced slot alignment (slot set including device, location, time, fault type, etc.) F ):

[0139] ;

[0140] in, This is a collection of slot types, containing key information categories in the power sector; Equipment slots: transformer model, switch type, line number, etc.; Location slots: substation name, geographical location, voltage level, etc.; Time slots: fault time, maintenance plan time, etc.; Fault slots: fault type, abnormal phenomenon, etc. Total number of slot types; δ fThe importance weight of slot f; core equipment slot: δ f =1.0

[0141] Auxiliary information slot: δ f =0.5-0.8; f Q To query the value of slot f; f d The value of slot f in the document; For standardized functions; II Here, SlotRel(Q,d) is the indicator function; SlotRel(Q,d) is the slot correlation.

[0142] The aforementioned business logic relevance is based on the power knowledge graph G=(V,E), which maps queries and evidence to a subgraph G. Q G d Graph matching and causal chain consistency scoring are used:

[0143] ;

[0144] Among them, G Q G is the business logic subgraph extracted from the query; d The graph represents the business logic subgraph extracted from the document; η1 is the graph structure similarity weight; η2 is the path alignment weight; GED(GQ,Gd) is the graph edit distance.

[0145] Measure the graph G Q Convert to G d Minimum number of edit operations required, including node / edge insertion, deletion, and replacement operations; PathAlign(G Q G d ) represents the alignment of critical business paths; BizRel(Q,d) represents the business logic relevance;

[0146] Based on the above correlation analysis, obtain the comprehensive contextual relevance Rel(Q,d):

[0147] Rel(Q,d)=α1SemRel(Q,d)+α2SlotRel(Q,d)+α3BizRel(Q,d);

[0148] Where α1, α2, and α3 are weighting coefficients.

[0149] In this embodiment, a collaborative filtering mechanism is used to obtain the final power document, as detailed below:

[0150] Based on the candidate evidence set obtained from the correlation analysis, a heuristic greedy algorithm is used to obtain the optimal evidence subset. Then, the evidence is mapped to chapters according to the target document template (e.g., fault analysis report / maintenance plan / safety briefing): the sections for background and object, fault phenomena, monitoring and data, analysis and inference, handling measures and risks, and conclusions and recommendations are filled in one by one. During filling, the reference pointer and evidence number are maintained, and a summary of the evidence source and a timestamp are given in parentheses at the end of each paragraph to ensure readability and verifiability. Statistical tables and graphs (e.g., load rate trends, temperature rise curves, fault frequency distribution) are generated for structured data and time-series curves. The graphs and graphs are presented in the document as placeholder titles and explanatory text, and are bidirectionally linked to the original tables or graphs.

[0151] The content generation stage follows industry terminology and corporate writing standards: standardized terminology, unified units (including conversion and precision), and objective and robust sentence structure. Consistency checks are performed on indicators, time, and equipment numbers across paragraphs to avoid inconsistencies.

[0152] After generation, a quality gatekeeping system combining rules and models is implemented: fact-checking back scan, logical coherence detection, format and layout checks, and review of security-related wording. If any missing or conflicting items are found, the system reverts to the candidate pool for targeted supplementation and replacement. If a high-risk judgment is found, a manual review process is triggered. The final output is a complete, traceable, and modally comprehensive power document. The document includes: a cover and abstract; chapter text; a list of figures and tables; and a list of cited evidence (including source, time, hash, and credibility).

[0153] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for intelligent generation of power documents based on multimodal memory fusion, characterized in that, Includes the following steps: S1: Obtain raw power business data and preprocess it to obtain preprocessed multimodal power data; S2: Based on the preprocessed power multimodal data, perform multimodal characterization and alignment to obtain the power multimodal vector; S3: Construct a power document multimodal memory based on the power multimodal vector, the original document pointer, and metadata; S4: Based on user intent or instructions, perform intent parsing and slot filling to obtain a set of candidate evidence strongly related to the task; S5: Based on the candidate evidence set, conduct in-depth contextual relevance analysis and obtain the final power documents through a collaborative screening mechanism; The in-depth contextual relevance analysis based on the candidate evidence set is as follows: Based on the user's query intent Q and the candidate evidence set, a relevance analysis is performed, including semantic relevance, business logic relevance, and spatiotemporal relevance analysis. The semantic relevance analysis is based on the query vector q and the evidence modality vector z. m (d) Weighted max pooling: ； Where d represents candidate evidence; m represents modality type; ω m The weights of mode m; s(q,z) m (d) is the similarity function, z m (d) is the vector representation of evidence d in mode m; The maximum weighted similarity across all modalities is used to represent the final semantic relevance score; SemRel(Q, d) represents the semantic relevance. Enhanced slot alignment: ； in, This is a collection of slot types, containing key information categories in the power sector; Total number of slot types; δ f The importance weight of slot f; f Q To query the value of slot f; f d Let f be the value of slot f in the document; normalize(·) is the normalization function; II[·] is the indicator function; SlotRel(Q,d) is the slot correlation; The aforementioned business logic relevance is based on the power knowledge graph G=(V,E), which maps queries and evidence to a subgraph G. Q G d Graph matching and causal chain consistency scoring are used: ； Among them, G Q G is the business logic subgraph extracted from the query; d The business logic subgraph extracted from the document; η1 is the graph structure similarity weight; η2 is the path alignment weight; GED(G Q G d ) represents the graph edit distance; PathAlign(G) Q G d ) represents the alignment of critical business paths; BizRel(Q,d) represents the business logic relevance; Based on the above correlation analysis, obtain the comprehensive contextual relevance Rel(Q,d): Rel(Q,d)=α1SemRel(Q,d)+α2SlotRel(Q,d)+α3BizRel(Q,d); Where α1, α2, and α3 are weighting coefficients.

2. The intelligent power document generation method based on multimodal memory fusion according to claim 1, characterized in that, The multimodal characterization is performed based on the preprocessed power multimodal data, as follows: Using a pre-trained BERT model as the base encoder, and through expansion of the power industry terminology dictionary and fine-tuning of the domain corpus, the model's ability to understand professional vocabulary and grammatical structures in the power industry is enhanced, resulting in the power encoder BERT. power ; According to the power encoder, a layered encoding strategy is adopted for encoding the preprocessed power text data: operation tickets and work tickets retain their form field information; defect records and maintenance reports are encoded after being segmented into paragraphs. Equipment ledgers are coded using a row and column structure; ; Where, X text This represents the preprocessed power text data; H text The changed power text data; Capture the context vector within the electricity text data using an attention mechanism: ; ； ； Where, α i,j e represents the attention weight between position i and position j; i,j Let h be the attention energy score from position i to position j; MLP(·) is a multilayer perceptron; i ,h j H represents power text data. text The hidden state vector representation of the i-th and j-th tokens; W a Here is the learnable attention parameter matrix; T is the total sequence length; c i This is the final aggregated context vector at position i; Represents vector h i The transpose of; Finally, each text fragment is mapped to a dense vector representation of a fixed dimension, forming a feature vector set for the text modality: ; Among them, v text is the global feature vector of the text modality; FFN(·) is the feedforward neural network; LayerNorm(·) is the layer normalization operation; For the preprocessed power image data, a convolutional neural network is used for multi-scale feature extraction: ； v image =GlobalPool(ResNet / F multi ); Where I is the input image; Pool k (I) Pooling the input image I at scale k to obtain the downsampled image features; Conv k×k For k×k convolution operators, convolution operations are performed on the pooled image features to extract spatial features at that scale; Here is the feature map obtained at scale k; Concat([·]) is used to merge feature maps at different scales according to the channel dimension, thereby achieving the fusion of multi-scale features; These are convolutional feature maps at different scales; F multi This represents the fused multi-scale feature representation; GlobalPool represents the pooling function; ResNet is the convolutional feature extractor. Each image is ultimately encoded into a high-dimensional feature vector v. image This effectively preserves its visual semantic information; For the preprocessed tabular data, the Transformer-based tabular encoding model Tapas is used to jointly model the row headers, column headers, and cell contents of the table: ； ； Among them, T content The text content of the table cell; T header For table column headers, specify the semantic labels or field names for each column; P row For row position encoding; P col Encoding for column positions; H table The set of encoded vectors for all cells in the table output by the Tapas model represents the serialized vector representation of the entire table; TimeEmbedding(t) represents the embedding vector at time t; H t For each time step, there is a table feature vector; Attention(·) is the attention mechanism that assigns different weights to the cell vectors of the input sequence Ht, and performs weighted aggregation to obtain the overall table features; v table This is the final global feature vector for the table modality.

3. The intelligent power document generation method based on multimodal memory fusion according to claim 2, characterized in that, The multimodal representation and alignment are as follows: Construct a multimodal pairing dataset in the power field, including text modality-image modality pairs, text modality-table modality pairs, and image modality-table modality pairs; By comparing and learning loss functions This makes semantically related data from different modalities closer together in the vector space, and semantically unrelated data farther apart: ； Where N is the number of samples in the batch; Let i be the feature vector of the i-th sample in mode a; Let i be the feature vector of the i-th sample in mode b; Let be the feature vector of the j-th sample in mode b; sim(·,·) is the cosine similarity; τ is the temperature coefficient; MLP is used as the mapping network between modalities to project the feature vectors of different modalities into a unified semantic space; ； ； Among them, z text z is the projection vector of the text modality; image z is the projection vector of the image modality; table The projection vector of the table modality; MLP text (·) represents the multilayer perceptron projection head corresponding to the text modality; MLP image MLP projection head for image modality; MLP table (·) represents the MLP projection head for the tabular modality; u is the fused unified multimodal representation vector, serving as the final multimodal semantic representation; W u b is the weight matrix of the linear layer, used to linearly transform the concatenated vector; u For bias terms; LayerNorm(·) represents the layer normalization operation; Simultaneously, by leveraging knowledge distillation techniques, teachers can guide students in online learning to achieve better cross-modal representations, thereby improving alignment quality and efficiency. ； in, For knowledge distillation loss; KL(·,·) is the Kullback-Leibler divergence; z student ,z teacher , respectively, are the output vectors of the student network and the teacher network; softmax(·) is the softmax function normalized to a probability distribution; These are distillation temperature parameters; The total loss function is: ； Among them, L total Let λ be the overall loss objective for multimodal training; λ1 is the weight controlling the contribution of knowledge distillation loss to the total loss; λ2 is the weight controlling the contribution of regularization loss to the total loss; L reg This is the loss due to regularization.

4. The intelligent power document generation method based on multimodal memory fusion according to claim 1, characterized in that, The process of constructing a power document multimodal memory based on power multimodal vectors, original document pointers, and metadata is as follows: Based on the standardized power multimodal vectors obtained in step S2, a vector database is constructed as the core storage layer of the memory, adopting a distributed vector database architecture. A multi-dimensional indexing system is built for the vector database. First, an approximate nearest neighbor index based on vector similarity is established, using the IVF algorithm, and the optimal indexing strategy is selected according to the vector dimension and data scale. At the same time, traditional indexes based on metadata are established, including time index, device index, business type index, and keyword inverted index. Based on the vector database, a knowledge graph for the power industry is constructed to enhance the semantic understanding capability of the memory. Each record in the vector database is mapped to the corresponding node in the knowledge graph through entity linking technology, enabling collaborative work between vector retrieval and graph reasoning. A cross-modal association mapping mechanism is established, which establishes associations for different modal data from the same business scenario through business logic, spatiotemporal relationships, and content semantic dimensions. A multimodal association graph is constructed, with business events as the central node, connecting related text, images, tables, and other data of different modalities as child nodes to form a complete event knowledge network.

5. The intelligent power document generation method based on multimodal memory fusion according to claim 1, characterized in that, The process of parsing intent and filling slots based on user intent or instructions is as follows: When a user inputs a request to generate an electricity document, the system first performs deep semantic analysis and intent recognition on the user's natural language command, and then adopts an intent classification model based on BERT. The intent classification model uses a multilayer perceptron architecture, and outputs the probability distribution of each intent category through a softmax function, selecting the category with the highest probability as the primary intent. Based on the identified user intent, sequence labeling technology is used to fill slots and extract key structured information from user commands; The BiLSTM-CRF model is used for named entity recognition and slot labeling, and combined with the power industry knowledge graph for entity linking and disambiguation to ensure the accuracy and standardization of extracted information.

6. The intelligent generation method for power documents based on multimodal memory fusion according to claim 5, characterized in that, The acquisition of a set of candidate evidence strongly related to the task is specifically as follows: Based on the parsed intent and slot information, a retrieval strategy based on vector semantic similarity encodes the user query into a vector representation and performs an approximate nearest neighbor search in the vector database to recall semantically relevant candidate documents; secondly, a traditional retrieval strategy based on keyword matching uses the BM25 algorithm to perform precise matching retrieval based on keywords in the slots; and thirdly, a reasoning retrieval strategy based on knowledge graphs performs multi-hop reasoning through entity relationships in the graph to discover indirectly relevant evidence. Structured retrieval based on metadata filtering performs precise filtering based on metadata conditions such as time, device, and business type, and merges the retrieval results through a weighted fusion algorithm to obtain a set of candidate evidence strongly related to the task.

7. The intelligent power document generation method based on multimodal memory fusion according to claim 6, characterized in that, The process of obtaining the final power documents through a collaborative filtering mechanism is as follows: Based on the candidate evidence set after correlation analysis, a heuristic greedy algorithm is used to obtain the optimal evidence subset; then, the evidence is mapped to chapters according to the target document template; statistical tables and graphs are generated for structured data and time series curves; the graphs are presented in the document as placeholder titles and explanatory text, and bidirectional links are formed with the original tables or graphs; During the content generation phase, industry terminology and corporate writing standards are followed, and consistency checks are performed on indicators, time, and equipment numbers across paragraphs to avoid inconsistencies. After generation, the quality gate is jointly implemented by the rules and the model. If any missing items or conflicts are found, the system will fall back to the candidate pool for targeted supplementation and replacement. If a high-risk judgment is found, a manual review process will be triggered. The final output is a complete, traceable, and modal power document.

Images