Small sample learning data expansion method and system based on multi-modal information fusion

Abstract

The invention discloses a small sample learning data expansion method and system based on multi-modal information fusion, and relates to the technical field of machine learning, and the method comprises the steps: obtaining multi-modal data in a power operation environment, and constructing a multi-modal information data set, environment features and equipment state features are extracted from the multi-modal information data set to construct a multi-modal feature set; preprocessing the multi-modal feature set, and performing weighted fusion on the preprocessed multi-modal features to generate a multi-modal feature vector; constructing a multi-modal data expansion model, generating a virtual sample, verifying and fusing the virtual sample, and expanding a multi-modal information data set; and balancing the expanded data set by adopting a data balance strategy, acquiring power operation data in real time, and regularly updating the multi-modal data expansion model through an online learning mechanism. According to the method, various features can be comprehensively extracted by combining the sensor data and the image data of the electric power working environment, and the diversity of the data set is greatly enriched.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a small sample learning data expansion method, system, device and medium based on multimodal information fusion. Background Art

[0002] With the rapid development of the power system, power companies have put forward higher requirements for stability, reliability and efficiency. The traditional manual inspection and maintenance method can no longer meet the increasingly complex power equipment management needs, especially in key links such as power transmission and transformation equipment and on-site operation safety monitoring. Manual inspection is not only labor-intensive and inefficient, but also easily affected by human factors, making it difficult to achieve real-time and accurate fault diagnosis and processing. In order to improve the operating efficiency and stability of the power system, artificial intelligence technology has gradually become an inevitable choice for power companies to inspect and maintain, especially efficient, accurate and explainable artificial intelligence detection technology, which can improve the monitoring ability and fault identification accuracy of equipment. In the process of intelligent upgrading of the power system, artificial intelligence technologies such as multimodal information fusion, sample expansion, and large model construction can achieve efficient fault identification and safety monitoring, break through the limitations of traditional manual inspection mode, and promote the power industry to develop in a more intelligent and automated direction.

[0003] Although the application of artificial intelligence in the field of power inspection and intelligent detection has made significant progress, current artificial intelligence technology still faces many challenges in practical applications. Most existing artificial intelligence models rely on single-modal information, only rely on images or only rely on sensor data for analysis, and lack effective fusion of multi-modal information, which limits the recognition ability of the model and cannot fully utilize the potential of multi-source data in power inspection. Existing artificial intelligence models perform poorly when faced with small sample data. The scarcity of data samples makes model training prone to overfitting, reducing its generalization ability. At the same time, due to the problem of sample imbalance, existing methods are often prone to high false alarm rates and missed alarm rates, affecting the effect of the model in practical applications. Although the small sample problem can be alleviated to a certain extent through technologies such as transfer learning, due to the diversity and complexity of data, existing transfer methods still find it difficult to achieve efficient knowledge transfer and model optimization. Summary of the invention

[0004] The purpose of the present invention is to provide a small sample learning data expansion method, system, device and medium based on multimodal information fusion to solve the technical problems existing in the prior art.

[0005] The present invention is achieved through the following technical solutions: In a first aspect, an embodiment of the present invention provides a method for expanding data for small sample learning based on multimodal information fusion, comprising the following steps: Acquire multimodal data in the power operation environment to construct a multimodal information data set, and extract environmental features and equipment status features from the multimodal information data set to construct a multimodal feature set; Preprocessing the multimodal feature set to obtain preprocessed multimodal features, comprehensively evaluating the importance of multimodal data, and using an attention mechanism network to weightedly fuse the preprocessed multimodal features to generate a multimodal feature vector; Inputting the multimodal feature vector into a pre-built generative adversarial network model to construct a multimodal data expansion model, performing high-dimensional feature space analysis, identifying correlation patterns between multimodal features, and generating virtual samples, verifying and fusing the virtual samples, and expanding the multimodal information data set to obtain an expanded data set; A data balancing strategy is used to balance the expanded data set, power operation data is acquired in real time, and the multimodal data expansion model is regularly updated through an online learning mechanism.

[0006] Furthermore, the multimodal data includes: power operation environment data, power equipment operation data and power equipment image data. The specific method of acquiring the multimodal data in the power operation environment and constructing the multimodal information data set includes: Time-align the multimodal data, fuse the power operation environment data, power equipment operation data and power equipment image data to obtain a multimodal information data set; The specific method of extracting environmental features and device status features from a multimodal information data set to construct a multimodal feature set includes: Long short-term memory network is used to extract time series features of power operation environment data, capture the dynamic change law of environmental data over time, and extract environmental change characteristics and periodic information; A convolutional neural network is used to extract deep features of the power equipment image data, wherein the deep features of the power equipment image data include equipment appearance features and texture features; Extract equipment health status characteristics based on the operating data of power equipment; The environmental change characteristics and periodic information, the deep characteristics of the power equipment image data and the equipment health status characteristics are integrated to form a comprehensive multimodal feature set.

[0007] Furthermore, the multimodal feature set is preprocessed to obtain preprocessed multimodal features, and a specific method for comprehensively evaluating the importance of multimodal data includes: Clean the multimodal features to remove missing values, outliers, and noise data; The Pearson correlation coefficient is used to evaluate the correlation and mutual information between different modalities and identify the modal data that is important to the task; Information gain is used to calculate the contribution of each mode to the target variable, evaluate the importance of each mode, generate the weight value of each mode, and obtain the evaluation result; Based on the evaluation results, the chi-square test was used to remove redundant or irrelevant features and retain the features with important information content.

[0008] Furthermore, the specific method of using the attention mechanism network to perform weighted fusion on the preprocessed multimodal features to generate a multimodal feature vector includes: Input the preprocessed multimodal features into the attention mechanism network and initialize the weight value of each modality; The attention mechanism is used to dynamically adjust the weight of each modality according to the contextual information of the modality, and the weighted modal features are merged according to the weights to generate a multimodal feature vector.

[0009] Furthermore, the generative adversarial network model includes a power operation environment data subnetwork, a power equipment operation data subnetwork, and a power equipment image data subnetwork; The multimodal feature vector is input into a pre-built generative adversarial network model to build a multimodal data expansion model. The specific method of generating virtual samples includes: The multimodal feature vector is input as an input to a plurality of sub-generators, each of which is responsible for generating a virtual sample of a corresponding modality to obtain generated data; During the training process, the virtual samples of different modalities are evaluated by the discriminator, the similarity between the virtual samples and the real samples is judged, and the discrimination results are obtained; Combining the generated data and discrimination results of each modality, an adversarial mechanism is used to jointly optimize the generator and discriminator. During the optimization process, an adversarial learning framework is used to optimize the performance of each modality generator, and the loss function is adjusted to improve the quality of virtual samples generated by the discriminator.

[0010] Furthermore, the specific method of verifying and fusing the virtual samples and expanding the multimodal information data set to obtain the expanded data set includes: The generated virtual samples are preliminarily compared with the original multimodal information dataset to evaluate the rationality of the virtual samples in each modality; Reconstruction error analysis is used to test the matching degree between the generated virtual samples and the original data in terms of pattern; The verified virtual samples are combined with the original multimodal information dataset, and a weighted fusion strategy is adopted to mix the generated virtual samples with the actual samples to obtain the expanded multimodal information dataset; The expanded multimodal information data set is deduplicated and cleaned to obtain the expanded data set.

[0011] Furthermore, the specific method of balancing the expanded data set using the data balancing strategy includes: Analyze the expanded data set, identify data with uneven category distribution, and use oversampling and undersampling to adjust the data distribution with uneven category distribution to obtain balanced data; Cross-validation was used to check the quality of the balanced data.

[0012] In a second aspect, another embodiment of the present invention provides a small sample learning data expansion system based on multimodal information fusion, comprising: a feature extraction module, a weighted fusion module, a multimodal data expansion module and a data balancing module; The feature extraction module is used to acquire multimodal data in the power operation environment to construct a multimodal information data set, and extract environmental features and equipment status features from the multimodal information data set to construct a multimodal feature set; The weighted fusion module is used to preprocess the multimodal feature set to obtain preprocessed multimodal features, comprehensively evaluate the importance of multimodal data, and use an attention mechanism network to weightedly fuse the preprocessed multimodal features to generate a multimodal feature vector; The multimodal data expansion module is used to input the multimodal feature vector into the pre-built generative adversarial network model to construct a multimodal data expansion model, perform high-dimensional feature space analysis, identify the correlation pattern between multimodal features, and generate virtual samples, verify and fuse the virtual samples, and expand the multimodal information data set to obtain an expanded data set; The data balancing module uses a data balancing strategy to balance the expanded data set, acquires power operation data in real time, and uses an online learning mechanism to regularly update the multimodal data expansion model.

[0013] In a third aspect, another embodiment of the present invention provides an electronic device, comprising a processor, an input device, an output device and a memory, wherein the processor is connected to the input device, the output device and the memory, respectively, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method described in the above embodiment.

[0014] In a fourth aspect, another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method described in the above embodiment.

[0015] The small sample learning data expansion method based on multimodal information fusion provided by the embodiment of the present invention has the following advantages: (1) By combining sensor data and image data of the power operation environment, a variety of features can be fully extracted, including environmental features and equipment status features, greatly enriching the diversity of the data set. The attention mechanism network is used to perform weighted fusion of different modal data, ensuring the effective integration of various feature information and improving the accuracy of feature representation. This method effectively improves the performance of small sample learning models in data-scarce conditions, so that relatively accurate and robust prediction results can still be obtained under limited samples.

[0016] (2) The data is expanded and optimized by generating adversarial networks and data balancing strategies, solving the problems of class imbalance and insufficient samples. By generating virtual samples and verifying and integrating the generated results, the representativeness of the data set is enhanced and the generalization ability of the model is improved. At the same time, combined with the online learning mechanism, new data can be acquired in real time and the model can be updated regularly, so that high adaptability and accuracy can be maintained in the dynamic power operation environment, which can provide strong data support for intelligent decision-making and prediction in the power operation environment.

[0017] A small sample learning data expansion system, terminal and medium based on multimodal information fusion provided in an embodiment of the present invention have the same inventive concept and the same beneficial effects as a small sample learning data expansion method based on multimodal information fusion provided in an embodiment of the present invention, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 A flowchart of a small sample learning data expansion method based on multimodal information fusion provided in the first embodiment of the present invention; Figure 2 A structural block diagram of a small sample learning data expansion system based on multimodal information fusion provided in the second embodiment of the present invention; Figure 3 This is a structural block diagram of an electronic device provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0020] See also Figure 1 The first embodiment of the present invention provides a small sample learning data expansion method based on multimodal information fusion, comprising the following steps:

[0021] S1. Acquire multimodal data in the power operation environment to construct a multimodal information data set, extract environmental features and equipment status features from the multimodal information data set to construct a multimodal feature set; S2. Preprocess the multimodal feature set to obtain preprocessed multimodal features, comprehensively evaluate the importance of multimodal data, perform weighted fusion on the preprocessed multimodal features through an attention mechanism network, and generate a multimodal feature vector; S3. Input the multimodal feature vector into the pre-built generative adversarial network model, build a multimodal data expansion model, perform high-dimensional feature space analysis, identify the association pattern of multimodal features in the power operation environment, generate virtual samples, verify and fuse the generated virtual samples, and expand the multimodal information data set to obtain an expanded data set; S4. A data balancing strategy is used to balance the expanded data set, and power operation data is obtained in real time. The multimodal data expansion model is regularly updated through an online learning mechanism.

[0022] In this embodiment, step S1: multimodal data in the power operation environment is obtained through sensor equipment and image acquisition devices. Among them, the multimodal data includes power operation environment data, power equipment operation data and power equipment image data. Power operation environment data: environmental data collected by environmental monitoring equipment (temperature, humidity, air pressure and other sensors), reflecting the physical conditions of the power operation environment. Power equipment operation data: data obtained by monitoring the operating status of power equipment (current, voltage, power load), used to evaluate the working condition of the equipment. Power equipment image data: equipment appearance data obtained by an image acquisition device (camera), used to identify the status and faults of the equipment. These data are constructed into a multimodal information data set through multimodal information fusion, and then the environmental characteristics of the power operation environment and the status characteristics of the power equipment are extracted. Specifically, environmental characteristics may include changes in temperature and humidity, and power equipment status characteristics may include the operating status and health indicators of power equipment. The extraction of these characteristics lays the foundation for subsequent data processing and analysis.

[0023] Step S2: Preprocess the constructed multimodal feature set to obtain preprocessed multimodal features. The purpose of preprocessing is to clean data, remove noise, fill missing values, etc. After that, by comprehensively evaluating the importance of different modal data, the attention mechanism network (Attention Mechanism) is used to perform weighted fusion on the preprocessed multimodal features. The core of this step is to use the attention mechanism to dynamically assign different weights to each modality, so that when synthesizing the final feature vector, the features that contribute significantly to the task can be more prominent, while ignoring irrelevant or redundant parts. Finally, a comprehensive multimodal feature vector is obtained, which represents the fusion result of various types of information in the power operation environment.

[0024] Step S3: Input the generated multimodal feature vector into the pre-built Generative Adversarial Network (GAN) model. The model consists of two main parts: the generator and the discriminator. The generator generates virtual samples based on the input feature vector, while the discriminator is responsible for evaluating the authenticity of the generated virtual samples and judging whether they are similar to real samples. Through high-dimensional feature space analysis, the GAN model can identify the association patterns between multimodal data and generate new virtual samples that do not appear in the original data set. After the generated virtual samples are verified and fused by the discriminator, the original multimodal information data set is further expanded.

[0025] Step S4: Apply data balancing strategy on the expanded data set to solve the training problem caused by class imbalance. Specifically, class imbalance will lead to poor prediction accuracy of the model on some categories, so oversampling or undersampling methods are used to adjust the category distribution so that the amount of data in each category is relatively balanced. After that, the model is regularly updated through the online learning mechanism in combination with the real-time power operation data. This means that over time, the model can automatically adjust according to new data, thereby improving its adaptability and accuracy in practical applications.

[0026] In this embodiment, multimodal data refers to different forms of data collected from different types of sensors and devices. Time alignment: Since data from different sensors are usually collected at different time points, it is necessary to time align the data so that the multimodal data at each moment can correctly correspond to the power operation status at the same moment. This is the basis of data fusion and ensures that the model can correctly process various types of data during training. Attention mechanism network: This is a deep learning algorithm that automatically adjusts the weights of input data according to different features so that the model can focus on the most important information for the task. Through the attention mechanism, the model can identify more critical information in data of multiple modes and then perform weighted fusion of the data. Multimodal feature vector: The feature vector obtained after preprocessing and weighted fusion of the attention mechanism contains comprehensive information from data of different modes (such as environmental data, equipment operation data, and image data). The feature vector is the input for model learning and data expansion. Generative adversarial network model: A deep learning model based on game theory, which includes two networks: a generator and a discriminator. The generator is used to generate virtual samples, and the discriminator is used to determine whether these samples are similar to real samples. During the training process, the generator and the discriminator are optimized in a confrontational manner, and finally the generator can generate high-quality virtual samples that are similar to real data. High-dimensional feature space analysis: refers to analyzing the relationships and patterns of data in a multidimensional feature space. For multimodal data, the potential relationships and rules between different modal data can be mined through the representation of high-dimensional space, thereby helping to generate more accurate virtual samples. Virtual samples: sample data generated by generative adversarial networks, which are similar to real data in features and are used to expand the original data set. This helps to solve the problem of insufficient data in small sample learning and provide more training samples. Data balancing strategy: When dealing with classification tasks, there may be imbalances in the data of different categories in the data set (for example, one class of samples is much more than other classes). Through the data balancing strategy, the data set can be oversampled (increasing the number of minority class samples) or undersampled (reducing the number of majority class samples) to make the data set more balanced, thereby improving the generalization ability of the model. Online learning mechanism: Online learning is a machine learning method. The multimodal data augmentation model can obtain new data in real time during actual operation and continuously update and optimize the model. Through online learning, the multimodal data augmentation model can adapt to the ever-changing conditions in the power operation environment and ensure the long-term effectiveness of the system.

[0027] Specifically, multimodal data includes power operation environment data, power equipment operation data and power equipment image data. The specific process of obtaining multimodal data in the power operation environment and constructing a multimodal information data set is as follows: time align the multimodal data to ensure the synchronization of different modal data on the time axis, and fuse the power operation environment data and power equipment image data to form a multimodal information data set.

[0028] In this embodiment, the three types of data, namely, power operation environment data, power equipment operation data and power equipment image data, respectively provide information from different angles: the power operation environment data reflects the environmental conditions (temperature, humidity, air pressure); the power equipment operation data provides the real-time operation status of the equipment (current, voltage, load); and the power equipment image data provides visual information for the appearance and status of the equipment (equipment appearance, fault indication). Time alignment: Multimodal data comes from different types of sensors and devices, and their data acquisition times may be different, so the data needs to be time aligned. This step ensures the synchronization of timestamps from different data sources, so that at the same time, different modal data correspond to the status under the same power operation environment. For example, the environmental data and equipment image data at a certain moment need to be synchronized to accurately reflect the actual status of the equipment at that moment. Data fusion: Fusion of power operation environment data and power equipment image data. The purpose of data fusion is to merge data of different modalities into a unified data set, thereby providing more comprehensive and accurate information. For example, power equipment image data can provide the appearance status of the equipment, while environmental data can provide external conditions that affect the operation of the equipment. Combining them can help better understand the operating status of the equipment and its relationship with environmental factors. Forming a multimodal information dataset: After time alignment and data fusion, a complete multimodal information dataset is finally formed. This dataset contains information from different data sources and can provide rich input data for subsequent analysis, modeling, and learning.

[0029] Specifically, the specific process of extracting environmental features and device status features from the multimodal information data set to construct a multimodal feature set is as follows: Use convolutional neural networks to extract deep features of power equipment image data, including equipment appearance features and texture features; Long short-term memory network is used to extract time series features of power operation environment data, capture the dynamic change law of environmental data over time, and extract environmental change characteristics and periodic information; Extract equipment health status characteristics based on the operating data of power equipment; The extracted deep features of power equipment image data, environmental change features and equipment health status features are fused to form a comprehensive multimodal feature set.

[0030] In this embodiment, a convolutional neural network (CNN) extracts deep features of power equipment image data. The power equipment image obtained by an image acquisition device (such as a camera) contains information such as equipment appearance, surface texture, and component wear. A convolutional neural network (CNN) is a deep learning network commonly used for image data analysis, which can automatically extract meaningful features from the original image. In an embodiment of the present invention, CNN is used to extract deep features from power equipment image data, mainly including: equipment appearance features and texture features. Equipment appearance features, such as whether there are visible damages such as cracks and corrosion on the surface of the equipment, can help detect potential failures of the equipment. Texture features: refers to texture information in an image, which can reveal changes in details on the surface of the equipment, which may indicate damage or aging of the equipment. A long short-term memory network (LSTM) extracts time series features of power operation environment data, which includes environmental data such as temperature, humidity, and air pressure, which are usually time series data that change over time. A long short-term memory network (LSTM) is a recurrent neural network (RNN) specifically used to process and predict time series data. In an embodiment of the present invention, LSTM is used to capture the dynamic change rules in power operation environment data, especially long-term dependencies in the time dimension.

[0031] Features extracted by LSTM can include: Environmental change characteristics: The changing trends of environmental conditions (such as temperature, humidity, etc.) can reflect whether the environment has undergone drastic changes.

[0032] Periodic information: For example, certain environmental factors (such as temperature) may show periodic fluctuations throughout the day or year, and LSTM is able to identify these periodic patterns.

[0033] Extraction of equipment health status features: The operating data of power equipment includes the equipment's working status data, such as current, voltage, load, etc. These data can reflect the equipment's operating efficiency and health status.

[0034] Equipment health status characteristics: Based on the equipment operation data, the equipment health status characteristics can be extracted, which usually include: Equipment working stability: such as whether the equipment is in an overload or underload state for a long time. Abnormal detection: such as current fluctuations, sudden decreases in equipment operating efficiency, etc., which may indicate early signs of equipment failure.

[0035] Fusion of multimodal features: Fusion of data from different modalities: The various features extracted from power equipment image data, environmental data, and equipment operation data are independent, but they each contain important information about the power operation environment and equipment status. In order to build a comprehensive feature set, it is necessary to fuse the data features from these different sources. Comprehensive multimodal feature set: The extracted deep features (from equipment image data), environmental change features (from environmental data), and equipment health status features (from equipment operation data) are fused to form a comprehensive feature vector. This feature set contains data from different modalities, which can fully reflect the various factors in the power operation environment and provide sufficient information for subsequent data analysis and model training.

[0036] Specifically, the specific process of preprocessing the multimodal feature set and comprehensively evaluating the importance of multimodal data is as follows: The extracted multimodal features are cleaned to remove missing values, outliers and noise data; The Pearson correlation coefficient is used to evaluate the correlation and mutual information between different modalities and identify the modal data that are important to the task; Information gain is used to calculate the contribution of each mode to the target variable, evaluate the importance of each mode, and generate the weight value of each mode; Based on the evaluation results, the chi-square test was used to remove redundant or irrelevant features and retain the important informative features.

[0037] In this embodiment, the preprocessed multimodal feature data is input: data of different modalities (such as power equipment image data, environmental data, and equipment operation data) have been extracted and preprocessed by specific neural networks (such as CNN, LSTM, etc.). These data contain the data features of each modality. These preprocessed features will be sent as input to the attention mechanism network for further processing. The feature information of each modality will form a set of input data respectively. Initialize the weight value of each modality: In the attention mechanism, the data features of each input modality will be assigned an initial weight value. Usually these weights are randomly initialized and represent the contribution of each modality to the final feature in the initial state. Use the attention mechanism to dynamically adjust the weight: The attention mechanism is a technology inspired by the human visual system, which can dynamically adjust the importance of each part according to the contextual information of the input data. In an embodiment of the present invention, the attention mechanism adjusts their weights according to the features of each modality and their contextual relationship in the current task. Simply put, the attention mechanism evaluates the contribution of each modality to the final prediction or target task and adjusts its weight accordingly. When processing multimodal data, some modalities may be more important than other modalities in some tasks. For example, when analyzing the health status of a device, the device operation data may be more important than the environmental data, while in some special cases, the environmental data (such as temperature, humidity, etc.) may become particularly critical. The attention mechanism can dynamically assign and adjust the weight of each modality based on this information. Merge the features of each modality by weight: Once the weight of each modality is adjusted, the next step is to weight the features of the data of different modalities according to these weights. Weighted merging means multiplying the features of each modality by its corresponding weight value, and then adding the weighted feature values ​​of all modalities to obtain a fused multimodal feature vector. This feature vector integrates the information of all modalities and is weighted according to the importance of different modalities in the current task. Generate a multimodal feature vector: Finally, the weighted feature vector processed by the attention mechanism is the multimodal feature vector. This vector represents the comprehensive information of the multimodal data and contains the most important features of all modalities in the current task. This vector can be used for subsequent model training, virtual sample generation, or other data analysis tasks.

[0038] Specifically, the attention mechanism network is used to perform weighted fusion on the preprocessed multimodal features to generate a multimodal feature vector. The specific process is as follows: Input the preprocessed multimodal features into the attention mechanism network and initialize the weight value of each modality; The attention mechanism is used to dynamically adjust the weight of each modality according to the contextual information of the modality, and the weighted modal features are merged according to the weights to generate a multimodal feature vector.

[0039] In this embodiment, the multimodal features after preprocessing are input: the multimodal features processed in the previous steps have been extracted and preprocessed, and contain features of different modes (such as power equipment image data, environmental data, and equipment operation data). These features will be passed into the attention mechanism network as input. The data of each modality will be converted into a specific feature vector, representing the information under the modality, such as the deep features of the image or the temporal features of the environmental data. Initialize the weight value of each modality: In the attention mechanism network, each modality will be assigned an initial weight value. The initialization of the weight value provides a starting point for subsequent dynamic adjustment. Use the attention mechanism to dynamically adjust the weight of each modality: The core of the attention mechanism is to dynamically evaluate and adjust the weight of each modality so that the model can determine its importance based on the contextual information of each modality. Specifically, the model adjusts the weight of the modality by analyzing the contribution of each modality in the current task. For example, in a power operation environment, the operation data of the power equipment may be more important than the environmental data in some tasks, so the model will assign a higher weight to the equipment operation data. At the same time, contextual information is a key factor in the attention mechanism: it refers to the contribution of a modality's data to the final task at a certain moment or in a certain situation. For example, when a device fails, the device operation data may have a higher weight than the environmental data. Through this mechanism, the model can automatically determine which modalities need more attention in a specific situation, thereby adjusting their weights and ultimately determining the degree of influence of each modality on the generated feature vector. Merge features of each modality by weight: After dynamically adjusting the weights, the model performs a weighted merge of the feature vectors of each modality. The feature vector of each modality is multiplied by its weight value (i.e., the adjusted weight), and then the weighted features of all modalities are summed. For example, if the weight of modality 1 is 0.7 and the weight of modality 2 is 0.3, the feature vector of modality 1 will be multiplied by 0.7, the feature vector of modality 2 will be multiplied by 0.3, and then the weighted results of the two will be added to form a fused feature vector. This weighted merge ensures that the importance of each modality is appropriately reflected according to its context, and the generated feature vector can better express the information of all modalities. Generate multimodal feature vector: Generate a fused multimodal feature vector by weighted merging of feature vectors of all modalities. This vector contains information of all input modalities and adjusts the contribution of each part according to the importance of different modalities. This feature vector can effectively capture and represent the key information in multimodal data, providing strong support for subsequent model training, virtual sample generation and other tasks.

[0040] Specifically, the specific process of constructing a multimodal data augmentation model and generating virtual samples is as follows: A multimodal generative adversarial network is constructed based on the multimodal feature vector, including multiple sub-networks, each of which focuses on the generation of data in different modalities. The generative adversarial network includes a power operation environment data sub-network, a power equipment operation data sub-network, and a power equipment image data sub-network. The preprocessed multimodal feature vector is used as input to multiple sub-generators, each of which is responsible for generating virtual samples of the corresponding modality; During the training process, the discriminator evaluates the generated samples of different modalities and determines the similarity between the generated samples and the real samples; Combining the generated data and discrimination results of each modality, the generator and discriminator are jointly optimized through an adversarial mechanism. During the optimization process, the adversarial learning framework is used to adjust the loss function to optimize the performance of each modality generator and generate virtual samples for each modality.

[0041] In this embodiment, a multimodal generative adversarial network is constructed based on a multimodal feature vector: a generative adversarial network (GAN) is a method of training a model through an adversarial learning mechanism, including a generator and a discriminator. In an embodiment of the present invention, a multimodal generative adversarial network is used for data of different modes, that is, a generative adversarial network including multiple subnetworks. Each subnetwork focuses on data generation of different modes. Specifically: Power operation environment data subnetwork: responsible for generating virtual samples related to the power operation environment. Power equipment operation data subnetwork: responsible for generating virtual data related to the operation status of power equipment. Power equipment image data subnetwork: responsible for generating virtual samples related to power equipment images. The multimodal feature vector is used as input and input into multiple subgenerators: the preprocessed multimodal feature vector will be used as input data and sent to multiple subgenerators. These generators are trained for virtual sample generation tasks of different modes respectively. For example, the image feature vector of the power equipment will be sent to the power equipment image data subnetwork, and the operation status data vector of the power equipment will be sent to the power equipment operation data subnetwork. The discriminator evaluates the generated samples of different modalities: During the training process, the generator of each subnetwork generates virtual samples of the corresponding modality. The role of the discriminator is to evaluate the similarity between the generated virtual samples and the real samples, and to determine whether the generated samples can be "indistinguishable from the real". The discriminator is usually a binary classification network, which outputs a similarity score by comparing the distribution difference between the real samples and the generated virtual samples. If the generated virtual sample is highly similar to the real sample, the discriminator will output a "real" signal; otherwise, it will output a "fake" signal. Combine the generated data and the discriminant results of each modality for joint optimization: The generator and the discriminator perform "adversarial" optimization during the training process: The goal of the generator is to improve the quality of the generated virtual samples, making it difficult for the discriminator to distinguish between virtual samples and real samples; the goal of the discriminator is to enhance its discrimination ability and accurately distinguish between virtual samples and real samples. By combining the data generated by the generator and the feedback results of the discriminator, the generator continuously adjusts its parameters during the training process to optimize the quality of the generated samples. The discriminator improves the accuracy of judging the generated virtual samples by optimizing its own loss function. Optimizing the performance of each modality generator through an adversarial learning framework: The adversarial learning framework refers to the game process between the generator and the discriminator. The generator loss function aims to train the generator so that the data it generates is as real as possible for the discriminator. The generator improves the quality of generated virtual samples by minimizing this loss function.

[0042] The calculation formula of the generator loss function is: , in, is the generator loss function, which represents the optimization objective of the generator. is the input of the generator, which is a multimodal feature vector. This input vector is usually distributed Generation, used to generate virtual samples in high-dimensional space. The virtual samples generated by the generator are obtained by inputting the multimodal feature vector To generate. Virtual samples generated by the discriminator The judgment result, this value represents the generated sample Whether it is judged as a real sample by the discriminator, the output is a value between [0, 1], 1 means "real" and 0 means "false". is the expected symbol, indicating the generated virtual sample Take the expected value and calculate the probability that the generated virtual sample makes the discriminator think it is a real sample. The goal of the generator is to maximize the output of the discriminator for the virtual sample ( ), that is, the generated virtual samples are judged as real samples by the discriminator. To this end, the loss function of the generator improves the quality of the generated samples by minimizing the prediction error of the discriminator on the generated samples.

[0043] The discriminator loss function is used to evaluate the ability of the discriminator, which needs to distinguish between real samples and generated samples. The goal of the discriminator is to maximize its correct judgment of real samples and judge the falsity of generated samples as accurately as possible.

[0044] The calculation formula of the discriminator loss function is: , in, is the discriminator loss function, which represents the optimization objective of the discriminator. is a real sample from the data distribution ,These are real power operation environment data, equipment operation data or equipment image data. For the discriminator to the real sample The judgment result indicates the probability that the sample is considered true by the discriminator. For real samples, the output of the discriminator should be close to 1. Virtual samples generated by the generator, based on random noise of the input . Generate virtual samples for the discriminator pair The judgment result indicates that the generated sample The probability of being judged as real by the discriminator. For the generated virtual samples, the output of the discriminator should be close to 0. is the expected symbol, indicating the expected value calculation of the loss function for all real samples and generated virtual samples. The goal of the discriminator is to maximize its output for real samples. (to make it close to 1), and minimize the output of the generated samples, (make it close to 0). This means that the discriminator needs to distinguish between real and generated virtual samples as correctly as possible. The generator and discriminator perform joint optimization through backpropagation of their respective loss functions until the generator can generate sufficiently realistic virtual samples and the discriminator cannot easily distinguish between virtual samples and real samples. In each round of training, the generator's loss function will be affected by the feedback of the discriminator's output, and by updating the generator's weights, the generator of each modality will perform more and more accurately when generating virtual samples. Loss function optimization: Under the framework of multimodal generative adversarial networks, the loss function of each generator not only takes into account the generation quality of single modality data, but also performs weighted optimization based on the feedback of the discriminator, so that the data generation quality of each modality is improved. Generate virtual samples for each modality: Through multiple rounds of adversarial training and optimization, the generator of each modality will be able to generate high-quality virtual samples. These virtual samples will form an expanded dataset together with real samples for subsequent model training, evaluation and application.

[0045] Specifically, the specific process of verifying and fusing the generated virtual samples and expanding the multimodal information data set to obtain the expanded data is as follows: The generated virtual samples are preliminarily compared with the original multimodal dataset to evaluate the rationality of the virtual samples in each modality; The matching degree between the generated virtual samples and the original data in terms of mode is checked through reconstruction error analysis. The verified virtual samples are combined with the original data set. Through the weighted fusion strategy, the generated virtual samples are mixed with the actual samples to form an expanded multimodal data set. The fused data set is deduplicated and cleaned to expand the multimodal information data set.

[0046] In this embodiment, the generated virtual sample is preliminarily compared with the original multimodal data set to evaluate the rationality of the virtual sample: In this process, the generated virtual sample is preliminarily compared with the original multimodal data set to verify whether the virtual sample conforms to the expected pattern and characteristics. The comparison method can be based on the statistical characteristics, distribution, and performance of the sample in a specific task (such as classification or regression). For example, the statistical characteristics of the power equipment image data, operation data, and environmental data are compared to ensure that the generated sample is similar to the real data in the feature space. The matching degree of the generated virtual sample and the original data in the pattern is checked by reconstruction error analysis: The reconstruction error refers to the error when the generated sample is reconstructed by a specific model (such as an autoencoder). If the generated sample can be accurately reconstructed, it indicates that it has a high similarity with the original data in terms of pattern and characteristics. By calculating the error between the generated sample and the original data during the reconstruction process, the quality of the generated sample can be quantitatively evaluated. A lower reconstruction error indicates that the generated sample is more realistic and can well match the distribution of the original data. Combine the verified virtual samples with the original data set: After the virtual samples pass the preliminary verification and reconstruction error analysis, they will be combined with the original data set. This step ensures that only high-quality and reasonable virtual samples are used to augment the dataset. The combined dataset contains both original data and virtually generated data, which expands the size and diversity of the dataset and enhances the data coverage when training the model. The generated virtual samples are mixed with real samples through a weighted fusion strategy: In this process, a weighted fusion strategy is used to merge virtual samples and real samples. Weighted fusion means that samples from different sources are assigned different weights according to their importance or credibility when mixed. Generally speaking, real samples have higher weights and virtual samples have lower weights, ensuring that the model relies mainly on real data when training. Weighted fusion can be achieved through weighted averaging, weighted summation, or other methods to ensure the balance between the generated virtual samples and real samples and the quality of the data. De-duplication and data cleaning of the fused dataset: After the augmented multimodal dataset is formed, de-duplication and data cleaning must be performed to ensure the quality of the data. The de-duplication step can remove duplicate virtual samples or samples with high similarity to avoid data redundancy leading to model overfitting. Data cleaning is to further improve the quality of the dataset by removing noisy data, inconsistent data, or erroneous data. Expanded multimodal information dataset: After deduplication and data cleaning, the final expanded multimodal information dataset will have higher quality and more diverse samples. These data will be used in subsequent model training, evaluation, testing and other processes to improve the performance and generalization ability of the model when processing different types of data in the power operation environment.

[0047] Specifically, the specific process of balancing the expanded data set using the data balancing strategy is as follows: Analyze the expanded data set, identify data with uneven category distribution, and use oversampling and undersampling to adjust the imbalanced data distribution; Cross-validation was used to check the quality of the balanced data.

[0048] In this embodiment, the expanded data set is analyzed to identify data with uneven category distribution: At this stage, it is first necessary to classify and analyze the expanded data set to understand the proportion of each category (such as device status category, environmental feature category, etc.) in the data set. Through statistical analysis or visualization methods (such as histograms, pie charts, etc.), it can be clearly seen whether there is category imbalance in the data set. If the number of samples in some categories is significantly larger than that in other categories, the model will prefer larger categories during training, affecting the generalization ability and accuracy of the model. Therefore, it is very important to identify these category imbalance problems. Oversampling and undersampling are used to adjust the data distribution of category imbalance: Oversampling: For categories with fewer samples, the number of minority class samples is increased by copying samples of these categories or using data enhancement techniques (such as SMOTE algorithm) to generate new virtual samples. This helps to alleviate category imbalance and prevent the model from ignoring minority classes during training. Undersampling: For categories with more samples, the number of samples in these categories can be reduced by randomly removing some samples, thereby preventing the model from being too biased towards the majority class. In some cases, intelligent undersampling methods (such as cluster sampling) can be used to maintain sample diversity and reduce information loss. Through the reasonable combination of oversampling and undersampling strategies, the category distribution in the data set can be adjusted to make it more balanced, thereby improving the training effect of the model. In order to evaluate whether the balanced data set has a positive impact on model training, the cross-validation method can be used to test the quality of the balanced data. Cross-validation is usually performed by dividing the data set into multiple subsets (such as K-fold cross-validation), using one subset as the validation set each time and the remaining as the training set. In this way, the performance of the model can be tested on different data partitions to ensure that the effect of the balanced data in model training and evaluation is stable and reliable. Cross-validation can not only test the effectiveness of the data balancing strategy, but also help discover potential overfitting problems and ensure that the data balancing strategy can effectively improve the generalization ability of the model.

[0049] An embodiment of the present invention provides a small sample learning data expansion method based on multimodal information fusion, which solves the problem of insufficient performance and poor generalization ability of small sample learning models during training due to data scarcity and category imbalance in power operation environments. By fusing sensor data and image data, and combining the importance assessment and weighted fusion technology of multimodal data, this method can effectively expand the training data set, enhance the diversity and representativeness of the data, and thus improve the accuracy and robustness of the model in the power operation environment. At the same time, by verifying and fusing the generated virtual samples and adopting a data balancing strategy, the distribution of samples is further optimized to ensure that the model can obtain more stable and accurate prediction results when facing unbalanced data.

[0050] See also Figure 2 The second embodiment of the present invention provides a small sample learning data expansion system based on multimodal information fusion, including: a feature extraction module, a weighted fusion module, a multimodal data expansion module and a data balancing module; the feature extraction module is used to obtain multimodal data in the power operation environment through sensor equipment and image acquisition devices, construct a multimodal information data set, extract environmental features, equipment status features and abnormal behavior features from the multimodal information data set, and construct a multimodal feature set; the weighted fusion module is used to pre-process the multimodal feature set, comprehensively evaluate the importance of multimodal data, and use an attention mechanism network to pre-process the multimodal feature set. The processed multimodal features are weightedly fused to generate a multimodal feature vector; the multimodal data expansion module is used to input the multimodal feature vector into the pre-built adversarial network model, build a multimodal data expansion model, perform high-dimensional feature space analysis, identify the correlation pattern between multimodal features, and generate virtual samples, verify and fuse the generated virtual samples, expand the multimodal information data set to obtain the expanded data set; the data balancing module is used to balance the expanded data set through the data balancing strategy, obtain the power operation data in real time, and regularly update the multimodal data expansion model through the online learning mechanism.

[0051] In this implementation scheme, feature extraction module: This module obtains various data in the power operation environment through sensors and image acquisition devices, including environmental features, equipment status features and abnormal behavior features. Then, these features are used to construct a comprehensive multimodal feature set. Weighted fusion module: This module preprocesses the extracted multimodal features and evaluates the importance of each modal data. By using the attention mechanism network, the modal data is dynamically weighted and fused to generate a comprehensive multimodal feature vector. Multimodal data expansion module: This module inputs the generated multimodal feature vector into the pre-built adversarial network model, generates virtual samples by analyzing the correlation patterns between the data, and verifies and fuses these virtual samples to expand the data set. Data balancing module: This module balances the expanded data set through a data balancing strategy to solve the problem of class imbalance. At the same time, it can also obtain new power operation data in real time and regularly update the model through an online learning mechanism.

[0052] The embodiment of the present invention provides a small sample learning data expansion system based on multimodal information fusion. Through a method based on multimodal information fusion, the data set is effectively expanded, the data shortage problem in small sample learning is overcome, and the learning ability and generalization ability of the model in the power operation environment are improved. By fusing sensor data, equipment status data and image data, virtual samples are generated and verified to ensure the quality of the expanded data set and enhance the diversity and accuracy of the training data. By using the attention mechanism and adversarial network model, different modal data are effectively weighted and fused, and the correlation pattern between multimodal data is identified, thereby improving the accuracy and robustness of the model. Through the data balancing strategy, the problem of category imbalance in the expanded data set is solved, the model is prevented from being biased towards certain categories, and the balance of the data set and the fairness of the model are ensured. Through the online learning mechanism, the system can obtain new data in real time and regularly update the model, so that the model can adapt to the dynamic changes in the power operation environment and maintain high efficiency and continuous optimization.

[0053] like Figure 3 As shown, a structural block diagram of an electronic device of the third embodiment of the present invention, the device includes a processor, an input device, an output device and a memory, the processor is connected to the input device, the output device and the memory respectively, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method described in the first embodiment above.

[0054] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0055] Input devices may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and output devices may include a display (LCD, etc.), a speaker, etc.

[0056] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0057] In a specific implementation, the processor, input device, and output device described in the embodiments of the present invention may execute the implementation described in the method embodiment provided in the embodiments of the present invention, or may execute the implementation described in the system embodiment described in the embodiments of the present invention, which will not be described in detail here.

[0058] A fourth embodiment of the present invention further provides an embodiment of a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method described in the first embodiment.

[0059] The computer-readable storage medium may be an internal storage unit of the terminal described in the foregoing embodiment, such as a hard disk or memory of the terminal. The computer-readable storage medium may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0060] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0061] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the terminals and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0062] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0063] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A small sample learning data expansion method based on multimodal information fusion, characterized in that: The following steps are involved: Acquire multimodal data in the power operation environment to construct a multimodal information data set, and extract environmental features and equipment status features from the multimodal information data set to construct a multimodal feature set; Preprocessing the multimodal feature set to obtain preprocessed multimodal features, comprehensively evaluating the importance of multimodal data, and using an attention mechanism network to weightedly fuse the preprocessed multimodal features to generate a multimodal feature vector; Inputting the multimodal feature vector into a pre-built generative adversarial network model to construct a multimodal data expansion model, performing high-dimensional feature space analysis, identifying correlation patterns between multimodal features, generating virtual samples, verifying and fusing the virtual samples, and expanding the multimodal information data set to obtain an expanded data set; A data balancing strategy is used to balance the expanded data set, power operation data is acquired in real time, and the multimodal data expansion model is regularly updated through an online learning mechanism.

2. The small sample learning data expansion method based on multimodal information fusion according to claim 1, characterized in that: The multimodal data includes: power operation environment data, power equipment operation data and power equipment image data. The specific method of acquiring the multimodal data in the power operation environment and constructing a multimodal information data set includes: Time-align the multimodal data, fuse the power operation environment data, power equipment operation data and power equipment image data to obtain a multimodal information data set; The specific method of extracting environmental features and device status features from a multimodal information data set to construct a multimodal feature set includes: The long short-term memory network is used to extract time series features of power operation environment data, capture the dynamic change law of environmental data over time, and extract environmental change characteristics and periodic information; A convolutional neural network is used to extract deep features of the power equipment image data, wherein the deep features of the power equipment image data include equipment appearance features and texture features; Extract equipment health status characteristics based on the operating data of power equipment; The environmental change characteristics and periodic information, the deep characteristics of the power equipment image data and the equipment health status characteristics are integrated to form a comprehensive multimodal feature set.

3. The small sample learning data expansion method based on multimodal information fusion as claimed in claim 2, characterized in that: Preprocessing the multimodal feature set to obtain preprocessed multimodal features, and a specific method for comprehensively evaluating the importance of multimodal data includes: Clean the multimodal features to remove missing values, outliers, and noise data; The Pearson correlation coefficient is used to evaluate the correlation and mutual information between different modalities and identify the modal data that is important to the task; Information gain is used to calculate the contribution of each mode to the target variable, evaluate the importance of each mode, generate the weight value of each mode, and obtain the evaluation result; Based on the evaluation results, the chi-square test was used to remove redundant or irrelevant features and retain the important informative features.

4. The small sample learning data expansion method based on multimodal information fusion as claimed in claim 3, characterized in that: The specific method of using the attention mechanism network to weightedly fuse the preprocessed multimodal features to generate a multimodal feature vector includes: Input the preprocessed multimodal features into the attention mechanism network and initialize the weight value of each modality; The attention mechanism is used to dynamically adjust the weight of each modality according to the contextual information of the modality, and the weighted modal features are merged according to the weights to generate a multimodal feature vector.

5. The small sample learning data expansion method based on multimodal information fusion as claimed in claim 4, characterized in that: The generative adversarial network model includes a power operation environment data sub-network, a power equipment operation data sub-network and a power equipment image data sub-network; The multimodal feature vector is input into a pre-built generative adversarial network model to build a multimodal data expansion model. The specific method of generating virtual samples includes: The multimodal feature vector is input as an input to a plurality of sub-generators, each of which is responsible for generating a virtual sample of a corresponding modality to obtain generated data; During the training process, the virtual samples of different modalities are evaluated by the discriminator, the similarity between the virtual samples and the real samples is judged, and the discrimination results are obtained; Combining the generated data and discrimination results of each modality, an adversarial mechanism is used to jointly optimize the generator and discriminator. During the optimization process, an adversarial learning framework is used to optimize the performance of each modality generator, and the loss function is adjusted to improve the quality of virtual samples generated by the discriminator.

6. The small sample learning data expansion method based on multimodal information fusion as claimed in claim 5, characterized in that: The specific method of verifying and fusing the virtual samples and expanding the multimodal information data set to obtain the expanded data set includes: The generated virtual samples are preliminarily compared with the original multimodal information dataset to evaluate the rationality of the virtual samples in each modality; Reconstruction error analysis is used to test the matching degree between the generated virtual samples and the original data in terms of pattern; The verified virtual samples are combined with the original multimodal information dataset, and a weighted fusion strategy is adopted to mix the generated virtual samples with the actual samples to obtain the expanded multimodal information dataset; The expanded multimodal information data set is deduplicated and cleaned to obtain the expanded data set.

7. The method for expanding data for small sample learning based on multimodal information fusion according to claim 6, characterized in that: The specific method of using the data balancing strategy to balance the expanded data set includes: Analyze the expanded data set, identify data with uneven category distribution, and use oversampling and undersampling to adjust the data distribution with uneven category distribution to obtain balanced data; Cross-validation was used to check the quality of the balanced data.

8. A small sample learning data expansion system based on multimodal information fusion, characterized in that: include: Feature extraction module, weighted fusion module, multimodal data expansion module and data balancing module; The feature extraction module is used to acquire multimodal data in the power operation environment to construct a multimodal information data set, and extract environmental features and equipment status features from the multimodal information data set to construct a multimodal feature set; The weighted fusion module is used to preprocess the multimodal feature set to obtain preprocessed multimodal features, comprehensively evaluate the importance of multimodal data, and use an attention mechanism network to weightedly fuse the preprocessed multimodal features to generate a multimodal feature vector; The multimodal data expansion module is used to input the multimodal feature vector into the pre-built generative adversarial network model to construct a multimodal data expansion model, perform high-dimensional feature space analysis, identify the correlation pattern between multimodal features, and generate virtual samples, verify and fuse the virtual samples, and expand the multimodal information data set to obtain an expanded data set; The data balancing module uses a data balancing strategy to balance the expanded data set, acquires power operation data in real time, and uses an online learning mechanism to regularly update the multimodal data expansion model.

9. An electronic device, comprising a processor, an input device, an output device and a memory, wherein the processor is connected to the input device, the output device and the memory respectively, and the memory is used to store a computer program, wherein the computer program comprises program instructions, and wherein: The processor is configured to call the program instructions to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 7.

Images