Power multi-modal data hierarchical routing feature processing and fusion method and system

Abstract

The invention provides an electric power multi-modal data hierarchical routing feature processing and fusion method and system, and relates to the technical field of electric power multi-modal data process.The method comprises the steps that multi-modal data are obtained and preprocessed; performing feature extraction on the preprocessed multi-modal data; selecting the most suitable expert model through a hybrid expert model, and carrying out feature extraction on the preprocessed multi-modal data; performing feature alignment and fusion on features of different modes after feature extraction; aligning the features of each modal with the features of other modals through a cross-modal attention mechanism, and fusing the features of other modals during alignment to obtain middle-level features of the corresponding modals; and carrying out weighted fusion on the middle-level features of each mode to obtain a fused global feature. According to the invention, efficient processing and fusion of multi-modal data can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of power multimodal data processing, and in particular to a method and system for processing and fusing hierarchical routing features of power multimodal data. Background Art

[0002] With the increasing demand for digital transformation and intelligence in the power industry, more and more multimodal data will be involved in the power system, including not only sensor data, monitoring videos, and image data during equipment operation, but also a large amount of text data, such as operation manuals, fault records, and audio data. These data modalities are complex and diverse, and traditional data processing methods can only target a single modality, making it difficult to process and integrate multimodal data, resulting in low information utilization efficiency and difficulty in forming global intelligent decisions.

[0003] In this context, machine learning and deep learning technologies are gradually being applied in the power industry, especially the development of multimodal data processing technology, which makes it possible to process and analyze multi-source data in the power industry. The Mixture of Experts (MoE) is a model architecture that effectively responds to the challenges of multimodal data processing. By introducing specialized expert models for different data modalities, deep feature extraction and adaptive fusion of different modal data can be achieved, significantly improving data processing efficiency and fusion accuracy.

[0004] However, in the existing multimodal data processing, there is a lack of solutions for effective processing and fusion of multimodal data. When fusing multimodal data, the existing technology is difficult to fully utilize the advantages of each modal data, which easily leads to feature loss or incomplete information, thereby affecting the accuracy of intelligent decision-making. In addition, with the increase in model complexity, the traditional model has extremely high computational costs when processing large-scale multimodal data, and it is difficult to meet the real-time requirements in practical applications. Summary of the invention

[0005] In order to solve the problems existing in the prior art that feature loss or incomplete information is easy to occur when processing multimodal data, as well as low processing efficiency, the purpose of the present invention is to provide a method and system for hierarchical routing feature processing and fusion of power multimodal data, so as to realize efficient processing and fusion of multimodal data.

[0006] In order to achieve the above objectives, according to some embodiments, a first aspect of the present invention provides a method for processing and fusing hierarchical routing features of electric power multimodal data, comprising: Acquire multimodal data and preprocess them; Extract features from the preprocessed multimodal data; select the most suitable expert model through the hybrid expert model to extract features from the preprocessed multimodal data; After feature extraction, the features of different modalities are aligned and fused. Through the cross-modal attention mechanism, the features of each modality are aligned with the features of other modalities, and the features of other modalities are fused during alignment to obtain the middle-level features of the corresponding modality. The middle-level features of each modality are weightedly fused to obtain the fused global features.

[0007] A second aspect of the present invention provides a power multi-modal data hierarchical routing feature processing and fusion system, comprising: A data acquisition module, configured to acquire multimodal data and perform preprocessing; The feature extraction module is configured to extract features from the preprocessed multimodal data; by mixing expert models, the most suitable expert model is selected to extract features from the preprocessed multimodal data; The feature alignment module is configured to align and fuse the features of different modalities after feature extraction; through the cross-modal attention mechanism, the features of each modality are aligned with the features of other modalities, and the features of other modalities are fused during alignment to obtain the middle-level features of the corresponding modality; The weighted fusion module is configured to perform weighted fusion on the middle-level features of each modality to obtain the fused global features.

[0008] A third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the above-mentioned power multimodal data hierarchical routing feature processing and fusion method.

[0009] According to a fourth aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, complete the steps of the above-mentioned method for processing and fusing hierarchical routing features of electric power multimodal data.

[0010] A fifth aspect of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned method for processing and fusion of hierarchical routing features of electric power multimodal data.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The present invention innovatively proposes a hierarchical routing feature processing and fusion method and system for electric power multimodal data. Aiming at the problem that the prior art lacks effective fusion of multimodal data, a feature processing and fusion method based on a hierarchical routing strategy is proposed. Through the hierarchical routing of the bottom layer, middle layer and top layer, data of different modes are gradually processed and fused; wherein, the bottom layer routing is based on a mixed expert model (MoE), and the most suitable expert model is selected to perform preliminary feature extraction on the input multimodal data. The middle layer routing combines the intermediate features of different modal data for further fusion and alignment. The top layer routing comprehensively processes the global features, and finally realizes accurate multimodal feature fusion. The hierarchical routing strategy ensures the effective flow of data in the entire processing flow, avoids the problem of information loss or incomplete features, and significantly improves the accuracy and reliability of feature fusion.

[0012] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0014] Figure 1 is a schematic diagram of the method in Embodiment 1 of the present invention; Figure 2 The figure is a schematic diagram of the data processing flow in the first embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0016] Embodiment 1 Embodiment 1 of the present invention, as Figure 1-Figure 2 As shown, a method for processing and fusing hierarchical routing features of electric power multimodal data is provided, comprising: S1, obtain multimodal data and preprocess it; S2. Extracting features from the preprocessed multimodal data; selecting the most suitable expert model through a hybrid expert model to extract features from the preprocessed multimodal data; S3, align and fuse the features of different modalities after feature extraction; align the features of each modality with the features of other modalities through the cross-modal attention mechanism, and fuse the features of other modalities during alignment to obtain the middle-level features of the corresponding modality; S4. Perform weighted fusion on the middle-level features of each modality to obtain the fused global features.

[0017] In order to realize intelligent decision-making in complex scenarios in the power industry, such as equipment fault diagnosis, intelligent monitoring, state prediction, etc., and improve the accuracy of decision-making, it is necessary to process and fuse multimodal data in the power system, such as images, text, audio, etc., and then use the fusion features after global fusion to make global intelligent decisions. In view of the problems of low efficiency and insufficient precision in multimodal data processing and fusion in the prior art, this embodiment provides a method for hierarchical routing feature processing and fusion of power multimodal data. Through a dynamic expert selection mechanism and a hierarchical routing strategy, multimodal data is adaptively extracted, processed and fused to improve the accuracy and efficiency of data processing.

[0018] The hierarchical routing strategy optimizes the processing path of multimodal data. Through the hierarchical decision tree structure, multimodal data is gradually processed and fused at the bottom, middle and top levels. The bottom-level routing introduces a hybrid expert model and extracts features by screening appropriate expert models. The middle-level routing combines feature alignment between modalities, and the top-level routing comprehensively processes global features to ensure efficient fusion of multimodal data.

[0019] The dynamic expert selection mechanism is implemented based on the Mixture of Experts (MoE) model. The model dynamically selects the expert model that is most suitable for processing the modal data for feature extraction based on the modal characteristics of the input data (such as images, text, audio, etc.). Each expert model performs in-depth processing on data of a specific modality, such as the visual expert model for image data and the text expert model for semantic understanding of text data.

[0020] Specifically, in step S1, multimodal data from the power industry are first collected and preprocessed. The multimodal data generated by the power system, including images, text, and audio, needs to be standardized before entering the model to ensure consistency.

[0021] For image data, the image is first normalized to the same size and resolution. After preprocessing, the data is uniformly divided into small patches of fixed size (patches), for example, each image is divided into patches of size 16x16 pixels. These patches are flattened and converted into vector representations as input to VisionTransformer (ViT). Assume that the input image I is divided into N patches, each patch I i is linearly projected into a high-dimensional vector space, resulting in a sequence of small blocks: , the embedding of each small block is expressed as , forming the input sequence of ViT , which is used in the subsequent Vision Transformer (ViT) for image feature extraction and analysis.

[0022] For text data, word segmentation and word vectorization are required to convert the original text into a vector representation that can be used for model processing. The purpose of word segmentation is to break down long text into individual words or phrases, while word vectorization represents each word as a high-dimensional embedding vector. For example, given a text sequence , each word Ti is embedded into a high-dimensional vector space , which can be expressed as:

[0023] in, Represents word T i Through this process, the text data is converted into a high-dimensional vector representation that the model can process.

[0024] For audio data, Fourier transform is used to convert the time domain signal s(t) into the frequency domain signal S(f):

[0025] This formula is used to convert audio data from the time domain to the frequency domain so that the spectral features in the audio can be captured by the system and used for subsequent processing. A common processing method is to divide the sound data into short time windows of fixed size, and convert the audio signal of each time window into frequency domain features through Fourier transform. Then, the Fourier transform result is converted into a Mel-spectrogram to better conform to human auditory perception. The spectrum of each time window is further divided into smaller "spectral blocks" (similar to "patches" in images), and then the embedded vectors of the spectrum blocks are further processed by Audio Transformer for feature extraction.

[0026] Through these preprocessing steps, multimodal data is converted into a standardized format that the model can process, laying the foundation for subsequent feature extraction and analysis.

[0027] In steps S2-S4, the data flow path is optimized by introducing a hierarchical routing strategy. The hierarchical routing strategy is divided into bottom-level routing, middle-level routing, and top-level routing, which correspond to the initial processing of features, the fusion and alignment of modalities, and the global comprehensive decision-making. The core goal of this strategy is to maximize the processing capabilities of each expert model while ensuring the efficient flow and fusion of multimodal data in the processing process.

[0028] The advantage of adopting a hierarchical routing strategy is that it can give full play to the strengths of each expert model and ensure the efficient flow of data in the entire processing chain. The bottom-level routing is responsible for the extraction of basic features, the middle-level routing focuses on the fusion and alignment of different modal features, and the top-level routing makes global decisions and outputs. Through this hierarchical optimization path design, the system can ensure the accuracy of feature fusion and the efficiency of data processing when dealing with complex power scenarios.

[0029] Step S2 corresponds to the underlying routing stage. After completing the preliminary processing of multimodal features, the underlying routing stage is entered. The underlying routing strategy assigns the input data to the corresponding primary expert model according to the modality of the input data for basic feature processing. For example, image data is processed by Vision Transformer (ViT) in the visual expert to extract global features and complex long-range dependencies in the image. This process ensures that the basic information of each data modality is fully extracted, laying a solid foundation for subsequent processing.

[0030] The feature extraction in the underlying routing is implemented based on a hybrid expert model. The hybrid expert model selects the expert model that best suits the current input data modality type based on a dynamic routing mechanism to further process the features of each modality. For example, image data will be processed by a visual expert, text data by a text expert, and audio data by an expert who specializes in processing audio features. And each modality corresponds to multiple experts, for example, the image modality includes a visual expert No. 1, a visual expert No. 2, and so on.

[0031] Multiple experts within the modality process the data of each modality from different perspectives. For example, the Vision Expert 1 focuses on the extraction of high-level semantic features, such as object detection, facial recognition, or specific scene analysis. This expert may pay more attention to the global information and semantic content in the image. The Vision Expert 2 specializes in processing the spatiotemporal relationship of images, which is suitable for video analysis or time series-related tasks. Similarly, the Text Expert 1 focuses on the understanding of text, such as sentiment analysis of a single sentence, question-and-answer tasks, etc. This expert may focus on processing simple grammatical structures and common expressions. The Text Expert 2 focuses on the deep relationship between words, such as the recognition of synonyms and antonyms, or the understanding of adversarially generated sentences.

[0032] Each expert is not pre-trained for a certain type of data, but is dynamically selected during training based on the input features through a gate mechanism. Different inputs will activate different experts. For some complex tasks, a modality may require multiple experts to participate in processing at the same time. The gate network generates a set of weights for each modality of data, which are used to determine which expert model is most suitable for processing the current input. For example: in the visual modality, for a complex landscape image, multiple experts may be activated to process different levels of image features (semantic features and spatial relationships).

[0033] Specifically, the data of each modality It is assigned to multiple experts of the corresponding modality for processing. The experts directly act on the data of the modality to generate specific expert features. The output of each expert is:

[0034] in, is the processing function of the i-th expert in modality m. The image expert model uses the VisionTransformer (ViT) model, which captures the dependencies between small blocks through a global self-attention mechanism; the text expert model uses a Transformer-based architecture model, which captures the semantic relationship between words in a sentence through a self-attention mechanism; the audio expert model uses the Audio Transformer model, which can capture the long-term dependencies between spectrum blocks by modeling the spectrum blocks of the Mel-spectrogram using a self-attention mechanism. In the process of image and text analysis, the global audio features output by the Audio Transformer can be used for complex audio analysis tasks.

[0035] The underlying routing generates dynamic intra-modality expert weights for each modality’s input data through a gating network. , and use these weights to perform weighted summation on multiple expert outputs of the modality to generate a modality-specific underlying feature representation .

[0036] For a certain modality m (such as text modality T), the system calculates the weight of each expert according to the following formula: :

[0037] in, corresponds to the weight of the i-th expert of modality m; σ corresponds to the activation function (such as Sigmoid); and The weight matrix and bias term corresponding to the i-th expert of modality m, used to learn the rules selected by the expert; Input data features corresponding to mode m.

[0038] The data of each modality will be processed by multiple experts, and the output of the experts will be weighted. Weighted summation to obtain the final underlying routing features :

[0039] in, The underlying routing features corresponding to mode m; The number of experts corresponding to modality m; The i-th expert weight corresponding to modality m; Corresponding to the features of modality m after processing by the i-th expert.

[0040] For each modality (taking text T, image I, and audio A as examples), the underlying routing features are expressed as:

[0041]

[0042]

[0043] These features are based on the data of each modality and incorporate the characteristics of each expert in the modality. The final features are used for modality alignment and feature fusion in the subsequent middle-level routing. The features of each modality are generated independently, and there is no direct interaction between different modalities. This design ensures the independence of modality feature processing and provides pure modality features for subsequent middle-level routing.

[0044] Through the dynamic routing mechanism, each modal data is processed by the most appropriate expert model, and the extracted features can retain their original information to the greatest extent. This dynamic selection and deep feature extraction achieved through the hybrid expert model not only improves the accuracy of feature processing, but also enables the system to flexibly respond to various complex multimodal data inputs.

[0045] Step S3 corresponds to the middle-level routing stage, which is responsible for further fusing and aligning features of different modalities, especially in scenarios where multimodal information needs to be combined, such as combining image features with text features to understand the status or fault information of power equipment. The middle-level routing determines how to fuse by comparing the intermediate features of different modalities.

[0046] Mid-level features The computation of is generated by the cross-modal attention mechanism during the modality alignment process. For a modality m (such as image, text, or audio), It represents the middle-level features generated by the alignment of the modality m in the middle-level routing. It is obtained by aligning attention with features of other modalities. The formula is:

[0047] in, , represented as the query matrix of modality m, which is used to capture the key requirements of modality m; , represented as a bond matrix from all modes, indicating the alignment relationship between modes; , a matrix of values ​​from all modalities, used to provide information after feature alignment. Q , W K , W V is the weight matrix of the attention mechanism, obtained through training.

[0048] Among them, the core of the attention mechanism is based on the query matrix The dot product with the key matrix K calculates the relationship between mode m and other modes, and then uses these relationship weights to perform a weighted summation on the value matrix V:

[0049] in, , K, V are the query matrix of mode m, the key matrix of all modes, and the value matrix of all modes, respectively. k is the dimension of the key, which is used to scale the dot product to stabilize the gradient. Through the above formula, the system is able to calculate the alignment weights between modality m and other modalities and generate the mid-level features of modality m. . Middle level features Focus on the modality and integrate some features of other modalities. This approach ensures that the core features of each modality are retained, while adding supplementary information from other modalities in necessary contexts, thereby achieving information complementarity and enhancement in the modality alignment process. For example, the text modality can obtain more intuitive contextual information through the image modality, while the audio modality can obtain clearer label guidance through the text modality.

[0050] Mid-level features The generation process is through the query matrix of modality m To drive. Query Matrix It reflects the part of the modal feature that needs to be further strengthened, so the calculation result of the intermediate layer feature is centered on the characteristics of mode m. Specifically, the query matrix Designed to be the underlying routing feature from modality m This allows the attention mechanism to focus on selecting information that is useful to this modality from the features of other modalities when calculating, rather than completely integrating all modal features. The main mode is the current mode, but it incorporates supplementary information from other modes through the attention mechanism. This supplementary information comes from the value matrix of other modes. The attention mechanism is based on and key matrix The dot product of is used to calculate the weights, thereby selectively extracting the most relevant information for modality m from other modalities.

[0051] Step S4 corresponds to the top-level routing stage, which is mainly responsible for the comprehensive processing and decision-making of global features. The top-level routing performs global feature aggregation and decision-making by integrating the intermediate features of each modality. For example, when the system processes power equipment failures, it may need to combine the features of the three modalities of image, text, and audio to generate the final fault report.

[0052] The key to feature fusion is that the system should not only retain the unique information of each modality, but also integrate the complementary information between these modalities to maximize the utilization of the overall information. Feature fusion is performed by weighted summation. The features of each modality are assigned different weights. The size of the weight depends on the importance of the modality or the relevance of the task. The weight represents the contribution of each modality to the global decision. Feature fusion can integrate the data features of each modality to generate a unified high-dimensional feature representation. At this point, feature fusion not only retains the uniqueness of a single modality, but also provides richer contextual information through the combination of complementary features, thereby providing support for the subsequent decision-making process.

[0053] In the top-level routing, receive the middle-level features from the middle-level routing , where m represents the type of modality (such as image, text, audio). The top-level routing dynamically learns the weight of each modality in the global task. , weighted fusion of these modal features is performed to generate the final global feature The specific formula is:

[0054] Where M is the total number of modalities (e.g., image, text, and audio modalities, M=3); represents the middle-level features of mode m; is the weight of modality m, indicating the importance of modality m to global feature fusion; weight Can be obtained through learning, satisfying .

[0055] Weight The weight of each modality is dynamically generated through a small neural network or attention mechanism, and the generation formula is:

[0056] Among them, W α is the parameter of the weight generation network, which can be optimized through training. Therefore, the system uses weights For each modality, the mid-level features Weighted, calculate the weighted features of each mode. Sum the weighted features of all modes to generate the final global feature .

[0057] In this way, the system is able to integrate the features of each modality in a global context and generate a unified output for subsequent tasks, such as fault identification reporting of power equipment, anomaly detection, or intelligent decision-making for other tasks.

[0058] In some embodiments, after generating a unified multimodal feature representation (i.e., the final global feature ), further intelligent decisions are made according to the task objectives. The specific form of task decision-making depends on the application scenario of the system. For example, in the fault detection task of power equipment, intelligent identification and diagnosis of equipment faults are made based on the fused multimodal features. By analyzing the final fused features, the key information is extracted and a decision output is generated. For some specific tasks, such as fault detection, classification algorithms can be used to determine whether the equipment has a fault or predict the type of fault.

[0059] Specifically, a decision function D(F final ) to generate fault reports or alarm prompts. For example, when using the classification decision function to classify the fault category, it can be expressed as:

[0060] Among them, y is the final output result, such as the fault category of the equipment, W and b are the weight and bias of the decision model respectively, and the softmax function ensures that the output is a probability distribution. In this framework, a fault identification report can be generated based on the fused features, or an alarm prompt can be generated when the equipment status is abnormal, thereby helping operation and maintenance personnel to discover and handle problems in a timely manner.

[0061] In addition, in this embodiment, the accuracy and efficiency of multimodal data processing are further improved through an adaptive optimization mechanism. The core goal of this process is to continuously adjust and optimize the expert selection strategy and hierarchical routing strategy through feedback mechanisms and online learning, so that the system can dynamically adapt to different task requirements and scene changes, thereby maintaining efficient and accurate performance.

[0062] First, in the actual operation process, feedback optimization is performed based on the execution effect of the task. Through the feedback mechanism, the difference between the decision output of the current task and the actual result is analyzed, and these differences are fed back to the model to adjust the expert selection and routing strategy in the next round of processing. For example, in a fault detection task, if it is found that the features of certain modalities (such as images or text) do not contribute as much to the task as expected, the weight of the modality is reduced or the corresponding expert model is adjusted through the feedback signal. This feedback mechanism can be implemented by gradient updating, and the formula is as follows:

[0063] Among them, θ t+1 is the updated model parameter, θ t is the current model parameter, η is the learning rate, L(θ t ) is the loss function, and the model parameters are updated by minimizing the loss function. This process ensures that the model can be adaptively adjusted after each task execution, continuously optimizing the processing accuracy of multimodal features and the efficiency of routing strategies.

[0064] During the training process, the goal of the model is to minimize the total loss function L(θ t ), the loss function is usually a weighted sum of multiple tasks:

[0065] Among them, L t is the loss of task t, is the weight of task t. Each optimization step updates the expert weights according to these task losses , Bias , weight matrix , , And the routing weight , thereby continuously improving the overall performance. Task t can be a classification task or a regression task.

[0066] For example, in the multimodal anomaly classification task, the goal of the model is to predict a certain anomaly category label based on data of different modalities (such as images, text, audio, etc.). To this end, we usually use the cross-entropy loss function (Cross-EntropyLoss) as the optimization target. The cross-entropy loss function evaluates the performance of the model by measuring the difference between the predicted distribution and the true label:

[0067] Where C is the number of categories, y i is the probability distribution of the actual labels (usually one-hot encoded labels). iis the predicted probability distribution of the model output. Through the cross entropy loss function, the system updates each weight matrix and related parameters to minimize the gap between the predicted category probability and the actual category.

[0068] In regression tasks, the goal is to predict a continuous value (such as device temperature, device load, etc.) based on multimodal data. The mean squared error loss function is usually used to measure the error between the model output and the actual value:

[0069] Where N is the number of samples, y i is the actual value, is the model prediction value. The mean square error loss function makes the model optimization process tend to minimize the difference between the predicted value and the actual value. Through back propagation, the network updates the weight matrix and related parameters to reduce the error.

[0070] On the basis of the feedback mechanism, the ability to adapt to new data and new scenarios is continuously improved through the online learning mechanism. The core of online learning is that the system can update its model parameters and routing strategies in real time as it continuously receives new multimodal data, ensuring that the processing capacity is always maintained at the optimal level. For example, when new equipment or new operating environments appear in the power system, the system can quickly adjust through online learning to adapt to these new data features.

[0071] The mathematical expression of online learning can be realized through the incremental gradient update formula:

[0072] Among them, x t+1 The system adjusts the model parameters through this update formula every time it receives new data, ensuring that the system's ability to process new data is gradually improved. Through this real-time learning mechanism, the system can adapt to complex changes in power scenarios, such as new fault types, new equipment installations, or changes in the operating environment.

[0073] This embodiment introduces a hybrid expert model in feature extraction, and can adaptively select the most appropriate expert model for processing based on the input multimodal data (such as images, text, audio, etc.). This dynamic expert selection mechanism effectively improves the efficiency of multimodal data processing and avoids the redundancy and inefficiency of a single model in traditional methods when processing data of different modalities. Each expert model specializes in processing data of a specific modality, is highly targeted, and can perform deep feature extraction on the data, thereby greatly improving the accuracy and efficiency of feature processing.

[0074] Secondly, the introduction of hierarchical routing strategy greatly optimizes the fusion process of multimodal data. Through the hierarchical routing of the bottom, middle and top layers, the system can gradually process and fuse data of different modalities. The bottom-level routing first selects a suitable expert model for preliminary feature extraction, the middle-level routing combines the intermediate features of different modal data for further fusion and alignment, and the top-level routing comprehensively processes the global features, and finally realizes accurate multimodal feature fusion. This hierarchical optimization strategy ensures the effective flow of data in the entire processing process, avoids the problem of information loss or incomplete features, and significantly improves the accuracy and reliability of feature fusion.

[0075] In addition, the method of the present invention has extremely high computational efficiency. The hybrid expert model uses a dynamic selection mechanism to only activate the expert model related to the current input mode, avoiding redundant calculations of irrelevant modes, reducing computational overhead, and ensuring that the system can maintain low computational resource consumption when processing large-scale multimodal data. This has important practical value for data scenarios that require real-time processing in the power industry, especially in tasks that require rapid response, such as fault detection and equipment monitoring.

[0076] Finally, the method of the present invention has strong flexibility and scalability. Whether facing newly added modal types or data processing requirements in different scenarios, the system can adaptively respond to various complex power application scenarios by dynamically adjusting the selection and routing strategies of expert models. This flexibility enables the method to be widely used in various data analysis and intelligent decision-making tasks in power systems, especially in improving the application effects in the fields of power equipment fault diagnosis, intelligent monitoring and status prediction, etc., with significant advantages.

[0077] Embodiment 2 This embodiment provides a power multi-modal data hierarchical routing feature processing and fusion system, including: A data acquisition module, configured to acquire multimodal data and perform preprocessing; The feature extraction module is configured to extract features from the preprocessed multimodal data; by mixing expert models, the most suitable expert model is selected to extract features from the preprocessed multimodal data; The feature alignment module is configured to align and fuse the features of different modalities after feature extraction; through the cross-modal attention mechanism, the features of each modality are aligned with the features of other modalities, and the features of other modalities are fused during alignment to obtain the middle-level features of the corresponding modality; The weighted fusion module is configured to perform weighted fusion on the middle-level features of each modality to obtain the fused global features.

[0078] It should be noted here that the various modules in this embodiment correspond one-to-one to the steps of the method in Example 1, and the specific implementation process is the same, which will not be repeated here.

[0079] Embodiment 3 This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the method in the first embodiment.

[0080] Embodiment 4 This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the method in the first embodiment are completed.

[0081] Embodiment 5 This embodiment provides a computer program product, including a computer program / instruction, which implements the steps of the method in Embodiment 1 when executed by a processor.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for processing and fusing hierarchical routing features of electric power multimodal data, characterized in that: include: Acquire multimodal data and preprocess them; Perform feature extraction on the preprocessed multimodal data; By mixing expert models, the most suitable expert model is selected to extract features from the preprocessed multimodal data; After feature extraction, the features of different modalities are aligned and fused. Through the cross-modal attention mechanism, the features of each modality are aligned with the features of other modalities, and the features of other modalities are fused during alignment to obtain the middle-level features of the corresponding modality. The middle-level features of each modality are weightedly fused to obtain the fused global features.

2. The method for processing and fusing hierarchical routing features of electric power multimodal data according to claim 1, characterized in that: The multimodal data includes image data, text data, and audio data; the hybrid expert model includes multiple image experts, multiple text experts, and multiple audio experts; and a gating mechanism is used to select multiple experts of corresponding modalities for processing the multimodal data.

3. The method for processing and fusing hierarchical routing features of electric power multimodal data according to claim 2, characterized in that: The output features of the hybrid expert model are obtained by weighted summing the features processed by each expert and the expert weights corresponding to each expert.

4. The method for processing and fusing hierarchical routing features of electric power multimodal data according to claim 1, characterized in that: The middle-level features of each modality are obtained through attention calculation. In the attention calculation, the query matrix is ​​calculated based on the features of the corresponding modality, the key matrix is ​​calculated based on the key matrices of all modalities, and the value matrix is ​​calculated based on the value matrices of all modalities.

5. The method for processing and fusing hierarchical routing features of electric power multimodal data according to claim 1, characterized in that: It also includes updating the parameters of the hybrid expert model by minimizing the loss function using a gradient update method.

6. The method for processing and fusing hierarchical routing features of electric power multimodal data according to claim 1, characterized in that: It also includes adjusting the parameters of the hybrid expert model according to new data input using an incremental gradient update method.

7. The power multi-modal data hierarchical routing feature processing and fusion system is characterized by: include: A data acquisition module, configured to acquire multimodal data and perform preprocessing; A feature extraction module is configured to extract features from the preprocessed multimodal data; By mixing expert models, the most suitable expert model is selected to extract features from the preprocessed multimodal data; The feature alignment module is configured to align and fuse the features of different modalities after feature extraction; through the cross-modal attention mechanism, the features of each modality are aligned with the features of other modalities, and the features of other modalities are fused during alignment to obtain the middle-level features of the corresponding modality; The weighted fusion module is configured to perform weighted fusion on the middle-level features of each modality to obtain the fused global features.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images