Large model-based multi-modal database dynamic fusion optimization method and system
By collaborating with a large language model and a knowledge base in the power sector, efficient cleaning and fusion of multimodal power data are achieved. This solves the problems of modal fragmentation and reliance on manual rules for cleaning, improves the processing efficiency and quality of power data, and supports the stable operation of the power system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA ELECTRIC POWER RESEARCH INSTITUTE CO LTD
- Filing Date
- 2025-06-12
- Publication Date
- 2026-07-21
Smart Images
Figure CN120670414B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of power data processing technology, and specifically relates to a dynamic fusion optimization method and system for multimodal databases based on large models. Background Technology
[0002] In the field of power industry text data processing technology, with the acceleration of the digitalization process in the power industry, the demand for efficient processing of power industry text data is becoming increasingly urgent. Existing power industry text databases have exposed many problems that urgently need to be solved.
[0003] Power sector data is multi-source, encompassing various formats such as text, images, and numerical data, and exhibits heterogeneity. However, current technologies mostly employ independent processing modules to handle different modalities of data, lacking a unified semantic alignment mechanism. This makes effective integration and collaborative processing between different data types difficult, significantly limiting the comprehensive mining and utilization of power data.
[0004] In the power data cleaning process, current methods primarily rely on rule engines or simple machine learning models. However, these methods struggle to handle ambiguities in domain-specific terminology. Furthermore, their accuracy is limited when dealing with noisy data such as redundant records in logs. Manually developing rules is time-consuming and labor-intensive, making it difficult to efficiently and accurately complete cleaning tasks when faced with massive and complex power data, significantly increasing both labor and time costs.
[0005] In summary, the shortcomings of existing power data text databases in areas such as data cleaning and modality fusion severely restrict the efficiency of power data processing. Summary of the Invention
[0006] The purpose of this invention is to provide a dynamic fusion optimization method and system for multimodal databases based on large models, so as to solve the problems of incomplete power data mining and high processing costs in the existing technology.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] In a first aspect, the present invention provides a dynamic fusion and optimization method for multimodal databases based on large models, including:
[0009] The power multimodal data is cleaned based on a large language model to obtain cleaned power multimodal data;
[0010] Based on the cleaned multimodal power data, cross-modal feature extraction is performed to obtain the features of each mode, and the features of each mode are aligned using a cross-modal alignment loss function;
[0011] Based on the characteristics of each aligned modality, as well as data quality indicators, business tags, and device lifecycle parameters, the weight of each modality is calculated.
[0012] The optimization is achieved by dynamically adjusting the weight allocation of each modality through preset emergency event keyword detection and sensor mutation response strategies.
[0013] Furthermore, the step of cleaning the power multimodal data based on the large language model to obtain cleaned power multimodal data includes:
[0014] Power multimodal data includes text, images, and sensor data. First, based on a large model, various types of raw power text are segmented and semantically parsed to identify potential errors such as grammatical errors, terminological deviations, and logical contradictions. Then, the generative capabilities of the large model are used for semantic error correction and enhancement. Next, the error correction results are compared with a power domain knowledge base to perform rule verification in terms of entity matching and relational constraints. At the same time, consistency verification is performed by combining image and sensor data.
[0015] Furthermore, during data cleaning, the cleaning rules are updated by mining error patterns, the large model is fine-tuned using the cleaned data, the cleaning results are scored for confidence, low-confidence data is pushed to manual review, and the review results are fed back to the knowledge base.
[0016] Furthermore, based on the cleaned multimodal power data, cross-modal feature extraction is performed to obtain the features of each mode, including:
[0017] For power text data, a large language model is used for encoding, and domain-adaptive fine-tuning or LoRA (Local Algorithm for Refining and Optimizing Pre-trained Models) is used for efficient fine-tuning to extract specialized vocabulary, thus obtaining text features. For image data, a large visual model is used to extract features, and vectors are generated by combining the topology diagram of power equipment to obtain image features, with weights adjusted through a position-sensitive attention mechanism. For sensor data, a Long Short-Term Memory (LSTM) network or a Transformer encoder based on a self-attention mechanism is used to extract features, thus obtaining sensor temporal features.
[0018] Furthermore, the step of aligning the features of each modality using a cross-modal alignment loss function includes:
[0019] By adding a Granger causality constraint term to the cross-modal alignment loss, a cross-modal Granger causality loss function is obtained. The semantic alignment of sensor temporal features and text features is achieved through the cross-modal Granger causality loss function.
[0020] The formula is ,in S represents the sensor temporal features, and T represents the text features.
[0021] Furthermore, the calculation of modal weights based on the aligned features of each modality, as well as data quality metrics, service tags, and device lifecycle parameters, includes:
[0022] Multimodal features and text features Image features Sensor timing characteristics Data quality indicators Q text For text integrity, Q image Image resolution, Q sensor Sensor sampling rate, service tag and equipment lifecycle parameters As the input to a lightweight gating network, through and Calculate the initial weights for each mode; in the formula... This is a learnable weight matrix used to map input features to the hidden space. For multimodal text features, For image features, For sensor timing characteristics, For data quality indicators, For business tags, For equipment lifecycle parameters, For bias terms; This is a learnable weight matrix used to map the gated vectors to the weight space. This is a bias term.
[0023] Furthermore, the optimization is achieved by dynamically adjusting the weight allocation of each modality through a preset sudden event keyword detection and sensor mutation response strategy, including:
[0024] When keywords related to sudden events are detected, based on Calculated initial weights , , Make dynamic adjustments; Used to record text logs, containing information such as device operating status descriptions. If it contains keywords related to emergencies, then If the sensor changes abruptly... ,but For topology embedding correction, for critical hub equipment, through Increase topology weights, For device nodes in the graph Connectivity in; through fusion formula Obtain fusion features The fused features are used for downstream tasks, in the formula For multimodal text features, For image features, For sensor timing characteristics; through Perform classification prediction, where, This is the weight matrix for the classification task, used to map the fused features to the class space. This represents the bias term for the classification task; the weight network parameters are updated using a sliding window, with the formula being... This allows the model to continuously optimize weight allocation based on new data. In the formula, This represents the weight matrix at the current time step t. for The amount of weight change used to map input features to the hidden space.
[0025] Secondly, this invention provides a dynamic fusion and optimization system for multimodal databases based on large models, comprising:
[0026] The data cleaning module is used to clean the power multimodal data based on the large language model to obtain the cleaned power multimodal data.
[0027] The feature extraction module is used to perform cross-modal feature extraction based on the cleaned power multimodal data, obtain the features of each modality, and align the features of each modality through the cross-modal alignment loss function;
[0028] The weight calculation module is used to calculate the weight of each modality based on the features of each aligned modality, as well as data quality indicators, business tags, and device lifecycle parameters.
[0029] The allocation optimization module is used to dynamically adjust the weight allocation of each modality through preset sudden event keyword detection and sensor mutation response strategies to complete the optimization.
[0030] Furthermore, in the data cleaning module, the step of cleaning the power multimodal data based on a large language model to obtain cleaned power multimodal data includes:
[0031] Power multimodal data includes text, images, and sensor data. First, based on a large model, various types of raw power text are segmented and semantically parsed to identify potential errors such as grammatical errors, terminological deviations, and logical contradictions. Then, the generative capabilities of the large model are used for semantic error correction and enhancement. Next, the error correction results are compared with a power domain knowledge base to perform rule verification in terms of entity matching and relational constraints. At the same time, consistency verification is performed by combining image and sensor data.
[0032] Furthermore, during data cleaning, the cleaning rules are updated by mining error patterns, the large model is fine-tuned using the cleaned data, the cleaning results are scored for confidence, low-confidence data is pushed to manual review, and the review results are fed back to the knowledge base.
[0033] Furthermore, in the feature extraction module, cross-modal feature extraction is performed based on the cleaned power multimodal data to obtain features for each modality, including:
[0034] For power text data, a large language model is used for encoding, and domain-adaptive fine-tuning or LoRA (Local Algorithm for Refining and Optimizing Pre-trained Models) is used for efficient fine-tuning to extract specialized vocabulary, thus obtaining text features. For image data, a large visual model is used to extract features, and vectors are generated by combining the topology diagram of power equipment to obtain image features, with weights adjusted through a position-sensitive attention mechanism. For sensor data, a Long Short-Term Memory (LSTM) network or a Transformer encoder based on a self-attention mechanism is used to extract features, thus obtaining sensor temporal features.
[0035] Furthermore, the step of aligning the features of each modality using a cross-modal alignment loss function includes:
[0036] By adding a Granger causality constraint term to the cross-modal alignment loss, a cross-modal Granger causality loss function is obtained. The semantic alignment of sensor temporal features and text features is achieved through the cross-modal Granger causality loss function.
[0037] The formula is ,in S represents the sensor temporal features, and T represents the text features.
[0038] Furthermore, in the weight calculation module, the calculation of each modality weight based on the aligned features of each modality, as well as data quality indicators, business tags, and device lifecycle parameters, includes:
[0039] Multimodal features and text features Image features Sensor timing characteristics Data quality indicators Q text For text integrity, Q image Image resolution, Q sensor Sensor sampling rate, service tag and equipment lifecycle parameters As the input to a lightweight gating network, through and Calculate the initial weights for each mode; in the formula... This is a learnable weight matrix used to map input features to the hidden space. For multimodal text features, For image features, For sensor timing characteristics, For data quality indicators, For business tags, For equipment lifecycle parameters, For bias terms; This is a learnable weight matrix used to map the gated vectors to the weight space. This is a bias term.
[0040] Furthermore, in the allocation optimization module, the optimization is achieved by dynamically adjusting the weight allocation of each modality through a preset sudden event keyword detection and sensor mutation response strategy, including:
[0041] When keywords related to sudden events are detected, based on Calculated initial weights , , Make dynamic adjustments; Used to record text logs, containing information such as device operating status descriptions. If it contains keywords related to emergencies, then If the sensor changes abruptly... ,but For topology embedding correction, for critical hub equipment, through Increase topology weights, For device nodes in the graph Connectivity in; through fusion formula Obtain fusion features The fused features are used for downstream tasks, in the formula For multimodal text features, For image features, For sensor timing characteristics; through Perform classification prediction, where, This is the weight matrix for the classification task, used to map the fused features to the class space. This represents the bias term for the classification task; the weight network parameters are updated using a sliding window, with the formula being... This allows the model to continuously optimize weight allocation based on new data. In the formula, This represents the weight matrix at the current time step t. for The amount of weight change used to map input features to the hidden space.
[0042] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the dynamic fusion optimization method for a multimodal database based on a large model.
[0043] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the dynamic fusion optimization method for multimodal databases based on a large model.
[0044] Compared with the prior art, the present invention has the following technical effects:
[0045] This invention, through the collaboration of a large language model and a domain knowledge base, accurately identifies and corrects text errors, reducing noise and erroneous information in the data. Consistency verification of multimodal data, combined with image and sensor data, ensures the reliability of text descriptions. Dynamic optimization and self-learning mechanisms continuously improve the cleaning rules and optimize the LLM model, guaranteeing the accuracy and efficiency of data cleaning and providing a solid foundation for subsequent power data analysis and decision-making.
[0046] Cross-modal feature extraction methods address the semantic alignment and unified representation issues of multimodal information, enabling the effective fusion of different data types. A dynamic weight allocation mechanism adjusts the fusion weights in real time based on data quality and business scenarios, optimizing database query and analysis efficiency. This improves the utilization rate of multimodal power data, reduces manual cleaning costs, and enhances the efficiency and quality of power text data processing, providing stronger data support for the operation, maintenance, and management of power systems. Attached Figure Description
[0047] Figure 1 This is a flowchart of the present invention.
[0048] Figure 2 This is a diagram illustrating the working principle of the data cleaning process in this invention.
[0049] Figure 3 This is a flowchart of the multimodal data dynamic fusion process of the present invention. Detailed Implementation
[0050] The present invention will be further described below with reference to the accompanying drawings:
[0051] Example 1, please refer to Figure 1 This invention provides a dynamic fusion optimization method for multimodal databases based on large models, including:
[0052] The power multimodal data is cleaned based on a large language model to obtain cleaned power multimodal data;
[0053] Based on the cleaned multimodal power data, cross-modal feature extraction is performed to obtain the features of each mode, and the features of each mode are aligned using a cross-modal alignment loss function;
[0054] Based on the characteristics of each aligned modality, as well as data quality indicators, business tags, and device lifecycle parameters, the weight of each modality is calculated.
[0055] The optimization is achieved by dynamically adjusting the weight allocation of each modality through preset emergency event keyword detection and sensor mutation response strategies.
[0056] This invention aims to solve the problems of modal fragmentation and reliance on manual rules for cleaning in existing power text databases, improve the utilization rate of multimodal power data, reduce manual cleaning costs, improve the efficiency and quality of power text data processing, and provide stronger data support for the operation, maintenance and management of power systems.
[0057] Example 2, please refer to Figure 2 and Figure 2 This invention provides a dynamic fusion and optimization method for multimodal databases based on large models, specifically including:
[0058] S1. A multimodal data cleaning process based on the collaboration of a large language model (LLM) and a knowledge base in the power field is adopted to clean the power text data. Through operations such as semantic error correction, cross-modal verification, and dynamic optimization, the power text database is efficiently purified, providing high-quality data support for subsequent power data processing and applications.
[0059] S2. A multimodal dynamic fusion mechanism is adopted to fuse multimodal information such as power text, images, and sensor data. The semantic alignment and unified representation of multimodal information are solved by cross-modal feature extraction method. A dynamic weight allocation mechanism is used to adjust the fusion weight in real time according to data quality and business scenario, optimize database query and analysis efficiency, and improve the comprehensive utilization value of power multimodal data.
[0060] Furthermore, in step S1, various raw power texts, such as equipment logs and work order records, are first processed based on LLM. Through word segmentation and semantic parsing, potential errors such as grammatical errors, terminological discrepancies, and logical contradictions are identified. Then, the semantic correction enhancement is performed using the generation capabilities of LLM. Next, the correction results are compared with a power domain knowledge base, and rule verification is performed from the aspects of entity matching and relational constraints. Simultaneously, consistency verification is performed using multimodal data such as images and sensors. In addition, this method also has a dynamic optimization and self-learning mechanism. It updates the cleaning rules by mining error patterns, fine-tunes the LLM model using the cleaned data, scores the confidence level of the cleaning results, and pushes low-confidence data for manual review. The review results are fed back to the knowledge base, thereby continuously improving the effectiveness and accuracy of data cleaning.
[0061] Furthermore, in step S2, different methods are employed for different types of power data in terms of cross-modal feature extraction. For power text, a large language model is used for encoding, and domain-adaptive fine-tuning and LoRA technology are used to enhance the capture of specialized vocabulary. For images, a large visual model is used to extract features, combined with a power equipment topology diagram to generate vectors, and a position-sensitive attention mechanism is designed to adjust the weights. For sensor time-series data, LSTM or Transformer encoders are used to extract features, and causal constraint alignment methods are used to guide the alignment of these features with text and image features in the shared latent space. Regarding the dynamic weight allocation mechanism, a lightweight scoring network is constructed, using data quality indicators, business requirement labels, and real-time context as inputs to calculate the fusion weight coefficients for each modality. Simultaneously, online learning mechanisms and emergency response strategies are used to adjust the weights in real time, and specific designs are implemented based on the characteristics of the power sector, focusing on equipment lifecycle perception and grid topology embedding, to ensure the scientific validity and effectiveness of multimodal data fusion.
[0062] Example 3: This invention provides a dynamic fusion and optimization method for multimodal databases based on large models, specifically including:
[0063] I. Implementation of Adaptive Data Cleaning
[0064] In the power sector, the processing of multimodal data (text, images, sensor data) is crucial for ensuring the stable operation of the power system. However, existing data cleaning methods have many limitations, such as difficulty in adapting to dynamic data changes, insufficient multimodal collaborative processing capabilities, and high computational overhead. To address these challenges, this invention proposes a comprehensive adaptive data cleaning implementation method tailored to the characteristics of multimodal data in the power sector. This method encompasses multiple approaches, including confidence scoring and multimodal consistency verification, incremental cleaning based on dynamic clustering, and adaptive threshold adjustment based on reinforcement learning. The specific implementation steps, algorithms, and formulas will be detailed below.
[0065] For different types of data, preprocessing and feature extraction are performed first. When processing power equipment log text, a Large Language Model (LLM) is used for word segmentation and semantic analysis. For infrared images, features are extracted using ResNet-50. And reduced to dimensionality by t-SNE For text, description vectors are extracted using LLM. Dimensionality reduction is performed simultaneously. Sensor data is then used to calculate the mean through a sliding window. and standard deviation This is to prepare for subsequent calculations of anomaly probabilities.
[0066] Confidence level for generating descriptive text of power equipment logs The calculation formula is: ,in For the text segmentation results, This is a terminology matching function for the power industry (1 for a successful match, 0 for otherwise). Sensor data is used to calculate the anomaly probability using the Z-score. .
[0067] Calculate the semantic consistency score between text descriptions and sensor data. The formula is ,in This is a text embedding vector (extracted via LLM). This refers to the sensor data feature vector (such as the spectral features after Fourier transform). This is the weighting coefficient (initial value 0.6, dynamically adjusted). If This data is identified as dirty data, triggering the cleaning rules. When, the rule engine is invoked to supplement missing fields (such as device model); when Furthermore, the text has high confidence, so the sensor data is marked as abnormal, triggering manual review.
[0068] The online K-means algorithm is used to cluster the dimensionality-reduced multimodal data (image and text features), and the cluster centers are updated. The formula is ,in The learning rate decreases as the amount of data increases. , (Number of processed samples). Calculate the Mahalanobis distance between the sample and the cluster center. ,like , marked as noise.
[0069] Local cleaning is initiated for noisy samples. If the matching degree between keywords in the text description (such as "insulation failure") and image features (the proportion of cracks detected at the edge) is lower than a threshold, the record is deleted. The cleaned data is added to the training set, and the image description generation module of LLM is fine-tuned.
[0070] For power work order text, define the status:
[0071] ;
[0072] in , For dynamic weights, initial values .
[0073] Action space The severity of the cleaning rules is adjusted accordingly. The reward function is... Data utility is evaluated through downstream tasks (such as fault prediction accuracy). Adjust dynamically according to business needs.
[0074] The optimal threshold adjustment policy is learned using DQN (Deep Q-Network), with the network input being... Output .use - Greedy Strategy Balance Exploration and Utilization (Initial) (5% decay per round).
[0075] To better implement the adaptive data cleaning method of this invention, the following formulas and parameter optimization suggestions are provided. For t-SNE dimensionality reduction, it is recommended to set the perplexity to 30 and the number of iterations to 1000. For the Z-score window, a 5-minute sliding window (i.e., 300 sampling points) is recommended for power sensor data. The DQN hyperparameter is set as: discount factor. Batch size 32, learning rate 0.001.
[0076] This comprehensive embodiment boasts significant technical advantages and differentiated features. It introduces feedback mechanisms (such as reinforcement learning and online clustering), breaking through the limitations of traditional static rules and enabling adaptive adjustments to the cleaning strategy based on dynamic data changes. Through cross-validation of image, text, and sensor data, multimodal collaboration is achieved, effectively improving cleaning accuracy. The use of t-SNE dimensionality reduction and online learning reduces computational overhead, offering the advantage of lightweight computation.
[0077] II. Implementation of Dynamic Fusion of Multimodal Power Data
[0078] In the power sector, multimodal data (text, images, and sensor time-series data) contains rich information about equipment operation. However, existing data fusion methods struggle to meet core requirements such as domain adaptation, topology embedding, and causal alignment, leading to insufficient data utilization and limited decision-making accuracy. To address these issues, this invention proposes a dynamic fusion implementation method for power multimodal data, the specific implementation steps of which will be detailed below.
[0079] Text feature encoding:
[0080] To improve the adaptability of text feature encoding to the power industry, this embodiment uses ELECTRA as the base model and integrates LoRA (Low-Rank Adaptation) for domain-adaptive fine-tuning. (The formula is used to...) Encode, where , A low-rank matrix ( Training only , Parameters. Simultaneously, secondary pre-training on a power terminology database can significantly improve the model's ability to capture specialized vocabulary, enabling text features to more accurately reflect information in the power sector.
[0081] Image feature extraction:
[0082] Constructing a power grid topology diagram , where nodes For equipment, side This establishes connection relationships. Topological feature vectors are generated using a Graph Attention Network (GAT). The formula is Next, the device's GPS coordinates will be... Convert to position encoding , with image features After concatenation, the data is input into a multi-head attention module, and then... Obtain enhanced image features .
[0083] Sensor timing processing:
[0084] Extracting temporal features using a Transformer encoder And add a causal mask to prevent future information leakage, i.e. A Granger causality constraint term is added to the cross-modal alignment loss, as shown in the formula: ,in Granger causality ensures that the causal relationship between sensor data and text data is effectively reflected at the feature level through time series lag correlation calculation.
[0085] The input parameters of the scoring network include multimodal features. , , Data quality indicators (e.g., text integrity, image resolution, sensor sampling rate), business tags and equipment lifecycle parameters A lightweight gating network (parameter count < 1MB) is used, through... and The initial weights of each mode are calculated, and the weight allocation takes into account multiple factors while ensuring computational efficiency.
[0086] In terms of emergency response, the weights are dynamically adjusted when keywords (such as "lightning strike" or "overload") are detected. If it includes "lightning strike", then If the sensor changes abruptly... ,but For topology embedding correction, for critical hub equipment (such as substation main transformers), through... Increase topology weights, For device nodes in the graph The connectivity in the data makes the weight allocation more in line with the actual situation in the power sector.
[0087] Through the fusion formula Obtain fusion features This fusion feature can be applied to downstream tasks, such as fault classification, through... It can perform classification prediction; it can also optimize retrieval through Approximate Nearest Neighbor (ANN) indexes to improve data retrieval efficiency.
[0088] For online learning, a sliding window is used to update the weight network parameters (window size = 1000 samples), and the formula is as follows: This allows the model to continuously optimize weight allocation based on new data. For consistency checks, text-image conflict detection is performed; if the image description generated by the LLM conflicts with the log semantics, it is resolved through... Manual review is triggered to ensure data consistency and accuracy.
[0089] In summary, this embodiment achieves dynamic fusion of multimodal power data through cross-modal feature extraction, dynamic weight allocation, and a closed-loop feedback mechanism for dynamic fusion and self-optimization. This satisfies core requirements such as domain adaptation, topology embedding, and causal alignment, thereby improving the utilization value of power data and the accuracy of decision-making.
[0090] This invention first utilizes a Large Language Model (LLM) and a power domain knowledge base to collaboratively clean power text data. Through semantic parsing, error correction, multimodal verification, and dynamic optimization, data quality is improved. Then, a multimodal dynamic fusion mechanism is employed. For different modalities, cross-modal feature extraction is performed using techniques such as domain-adaptive fine-tuning, topological relationship-based attention mechanisms, and causal constraint alignment. Dynamic weight allocation is achieved by constructing a lightweight scoring network. This solution addresses the problems of modal fragmentation and reliance on manual rules for cleaning in existing power text databases, improving the utilization rate of multimodal power data, reducing manual cleaning costs, and enhancing data processing efficiency and quality, thus providing strong support for the operation, maintenance, and management of power systems.
[0091] In another embodiment of the present invention, a dynamic fusion optimization system for multimodal databases based on large models is provided, which can be used to implement the above-mentioned dynamic fusion optimization method for multimodal databases based on large models. Specifically, the system includes:
[0092] The data cleaning module is used to clean the power multimodal data based on the large language model to obtain the cleaned power multimodal data.
[0093] The feature extraction module is used to perform cross-modal feature extraction based on the cleaned power multimodal data, obtain the features of each modality, and align the features of each modality through the cross-modal alignment loss function;
[0094] The weight calculation module is used to calculate the weight of each modality based on the features of each aligned modality, as well as data quality indicators, business tags, and device lifecycle parameters.
[0095] The allocation optimization module is used to dynamically adjust the weight allocation of each modality through preset sudden event keyword detection and sensor mutation response strategies to complete the optimization.
[0096] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0097] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a dynamic fusion optimization method for multimodal databases based on large models.
[0098] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the dynamic fusion optimization method for multimodal databases based on large models in the above embodiments.
[0099] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0100] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A dynamic fusion and optimization method for multimodal databases based on large models, characterized in that, include: The power multimodal data is cleaned based on a large language model to obtain cleaned power multimodal data; Based on the cleaned multimodal power data, cross-modal feature extraction is performed to obtain the features of each mode, and the features of each mode are aligned using a cross-modal alignment loss function; Based on the characteristics of each aligned modality, as well as data quality indicators, business tags, and device lifecycle parameters, the weight of each modality is calculated. The optimization is achieved by dynamically adjusting the weight allocation of each modality through preset emergency event keyword detection and sensor mutation response strategies. The calculation of modal weights based on the aligned features of each modality, as well as data quality indicators, service tags, and device lifecycle parameters, includes: Multimodal features and text features Image features Sensor timing characteristics Data quality indicators Q text For text integrity, Q image Image resolution, Q sensor Sensor sampling rate, service tag and equipment lifecycle parameters As the input to a lightweight gating network, through and Calculate the initial weights for each mode; in the formula... This is a learnable weight matrix used to map input features to the hidden space. For multimodal text features, For image features, For sensor timing characteristics, For data quality indicators, For business tags, For equipment lifecycle parameters, For bias terms; This is a learnable weight matrix used to map the gated vectors to the weight space. For bias terms; The optimization is achieved by dynamically adjusting the modal weight allocation through a preset sudden event keyword detection and sensor mutation response strategy, including: When keywords related to sudden events are detected, based on Calculated initial weights , , Make dynamic adjustments; Used to record text logs, containing information such as device operating status descriptions. If it contains keywords related to emergencies, then If the sensor changes abruptly... ,but For topology embedding correction, for critical hub equipment, through Increase topology weights, For device nodes in the graph Connectivity in; through fusion formula Obtain fusion features The fused features are used for downstream tasks, in the formula For multimodal text features, For image features, For sensor timing characteristics; through Perform classification prediction, where, This is the weight matrix for the classification task, used to map the fused features to the class space. This represents the bias term for the classification task; the weight network parameters are updated using a sliding window, with the formula being... This allows the model to continuously optimize weight allocation based on new data. In the formula, This represents the weight matrix at the current time step t. for The amount of weight change used to map input features to the hidden space.
2. The dynamic fusion and optimization method for multimodal databases based on a large model according to claim 1, characterized in that, The process of cleaning the power multimodal data based on a large language model yields cleaned power multimodal data, including: Power multimodal data includes text, images, and sensor data. First, based on a large model, various types of raw power text are segmented and semantically parsed to identify potential errors such as grammatical errors, terminological deviations, and logical contradictions. Then, the generative capabilities of the large model are used for semantic error correction and enhancement. Next, the error correction results are compared with a power domain knowledge base to perform rule verification in terms of entity matching and relational constraints. At the same time, consistency verification is performed by combining image and sensor data.
3. The dynamic fusion and optimization method for multimodal databases based on a large model according to claim 2, characterized in that, During data cleaning, the cleaning rules are updated by identifying error patterns, the large model is fine-tuned using the cleaned data, the cleaning results are scored for confidence, low-confidence data is pushed for manual review, and the review results are fed back to the knowledge base.
4. The dynamic fusion and optimization method for multimodal databases based on a large model according to claim 1, characterized in that, Based on the cleaned multimodal power data, cross-modal feature extraction is performed to obtain the features of each mode, including: For power text data, a large language model is used for encoding, and domain-adaptive fine-tuning or LoRA (Local Algorithm for Refining and Optimizing Pre-trained Models) is used for efficient fine-tuning to extract specialized vocabulary, thus obtaining text features. For image data, a large visual model is used to extract features, and vectors are generated by combining the topology diagram of power equipment to obtain image features, with weights adjusted through a position-sensitive attention mechanism. For sensor data, a Long Short-Term Memory (LSTM) network or a Transformer encoder based on a self-attention mechanism is used to extract features, thus obtaining sensor temporal features.
5. The dynamic fusion and optimization method for multimodal databases based on a large model according to claim 4, characterized in that, The step of aligning features of each modality using a cross-modal alignment loss function includes: By adding a Granger causality constraint term to the cross-modal alignment loss, a cross-modal Granger causality loss function is obtained. The semantic alignment of sensor temporal features and text features is achieved through the cross-modal Granger causality loss function. The formula is ,in S represents the sensor temporal features, and T represents the multimodal text features.
6. A dynamic fusion and optimization system for multimodal databases based on large models, characterized in that, include: The data cleaning module is used to clean the power multimodal data based on the large language model to obtain the cleaned power multimodal data. The feature extraction module is used to perform cross-modal feature extraction based on the cleaned power multimodal data, obtain the features of each modality, and align the features of each modality through the cross-modal alignment loss function; The weight calculation module is used to calculate the weight of each modality based on the features of each aligned modality, as well as data quality indicators, business tags, and device lifecycle parameters. The allocation optimization module is used to dynamically adjust the weight allocation of each modality through preset sudden event keyword detection and sensor mutation response strategies to complete the optimization; The calculation of modal weights based on the aligned features of each modality, as well as data quality indicators, service tags, and device lifecycle parameters, includes: Multimodal features and text features Image features Sensor timing characteristics Data quality indicators Q text For text integrity, Q image Image resolution, Q sensor Sensor sampling rate, service tag and equipment lifecycle parameters As the input to a lightweight gating network, through and Calculate the initial weights for each mode; in the formula... This is a learnable weight matrix used to map input features to the hidden space. For multimodal text features, For image features, For sensor timing characteristics, For data quality indicators, For business tags, For equipment lifecycle parameters, For bias terms; This is a learnable weight matrix used to map the gated vectors to the weight space. For bias terms; The optimization is achieved by dynamically adjusting the modal weight allocation through a preset sudden event keyword detection and sensor mutation response strategy, including: When keywords related to sudden events are detected, based on Calculated initial weights , , Make dynamic adjustments; Used to record text logs, containing information such as device operating status descriptions. If it contains keywords related to emergencies, then If the sensor changes abruptly... ,but For topology embedding correction, for critical hub equipment, through Increase topology weights, For device nodes in the graph Connectivity in; through fusion formula Obtain fusion features The fused features are used for downstream tasks, in the formula For multimodal text features, For image features, For sensor timing characteristics; through Perform classification prediction, where, This is the weight matrix for the classification task, used to map the fused features to the class space. This represents the bias term for the classification task; the weight network parameters are updated using a sliding window, with the formula being... This allows the model to continuously optimize weight allocation based on new data. In the formula, This represents the weight matrix at the current time step t. for The amount of weight change used to map input features to the hidden space.
7. The dynamic fusion optimization system for multimodal databases based on a large model according to claim 6, characterized in that, In the data cleaning module, the step of cleaning the power multimodal data based on a large language model to obtain cleaned power multimodal data includes: Power multimodal data includes text, images, and sensor data. First, based on a large model, various types of raw power text are segmented and semantically parsed to identify potential errors such as grammatical errors, terminological deviations, and logical contradictions. Then, the generative capabilities of the large model are used for semantic error correction and enhancement. Next, the error correction results are compared with a power domain knowledge base to perform rule verification in terms of entity matching and relational constraints. At the same time, consistency verification is performed by combining image and sensor data.
8. The dynamic fusion and optimization system for multimodal databases based on a large model according to claim 7, characterized in that, During data cleaning, the cleaning rules are updated by identifying error patterns, the large model is fine-tuned using the cleaned data, the cleaning results are scored for confidence, low-confidence data is pushed for manual review, and the review results are fed back to the knowledge base.
9. The dynamic fusion optimization system for multimodal databases based on a large model according to claim 6, characterized in that, In the feature extraction module, cross-modal feature extraction is performed based on the cleaned multimodal power data to obtain features for each modality, including: For power text data, a large language model is used for encoding, and domain-adaptive fine-tuning or LoRA (Local Algorithm for Refining and Optimizing Pre-trained Models) is used for efficient fine-tuning to extract specialized vocabulary, thus obtaining text features. For image data, a large visual model is used to extract features, and vectors are generated by combining the topology diagram of power equipment to obtain image features, with weights adjusted through a position-sensitive attention mechanism. For sensor data, a Long Short-Term Memory (LSTM) network or a Transformer encoder based on a self-attention mechanism is used to extract features, thus obtaining sensor temporal features.
10. The dynamic fusion optimization system for multimodal databases based on a large model according to claim 9, characterized in that, The step of aligning features of each modality using a cross-modal alignment loss function includes: By adding a Granger causality constraint term to the cross-modal alignment loss, a cross-modal Granger causality loss function is obtained. The semantic alignment of sensor temporal features and text features is achieved through the cross-modal Granger causality loss function. The formula is ,in S represents the sensor temporal features, and T represents the multimodal text features.
11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the dynamic fusion optimization method for multimodal databases based on large models as described in any one of claims 1 to 5.
12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the dynamic fusion optimization method for multimodal databases based on large models as described in any one of claims 1 to 5.