Adjustable multimodal data aggregation method and system based on multimodal data fusion

Abstract

The present invention discloses an adjustable multimodal data aggregation method and system based on multimodal data fusion, which relates to the technical field of multimodal data fusion, and comprises the following steps: constructing a knowledge graph, and according to a multimodal data fluctuation database, obtaining a first number of convolution kernels of different scales, and performing data matching on the database to obtain a first range of convolution kernel scales; dynamically extracting features from multimodal data according to the optimal scale of each convolution kernel; performing correlation analysis on the extracted features in combination with the knowledge graph, and according to the correlation analysis results, aggregating the information of each power equipment node; the present application solves the problems of insufficient precision and depth in multimodal data correlation analysis, poor load aggregation effect, and lack of efficiency in power system equipment information aggregation by comprehensively using deep learning technology, multimodal data correlation analysis technology, and advanced aggregation algorithms.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data fusion, and more specifically, to an adjustable multimodal data aggregation method and system based on multimodal data fusion. Background Art

[0002] As the power system continues to evolve towards intelligence, the complexity of its operation and management is increasing day by day. On the demand side, the complexity, variability and uncertainty of users' electricity consumption behavior bring many challenges to the stable and reliable operation of the power grid. In order to effectively deal with these situations, the power system urgently needs more accurate and intelligent load management strategies to achieve accurate control and optimized scheduling of adjustable multi-modal data, thereby improving the operating efficiency and stability of the entire power system.

[0003] There have been many attempts in the field of research and application of adjustable multimodal data. For example, in order to explore the adjustable load potential of different industries, different processes, and different power-consuming equipment on the customer side during the peak period of power load, some researchers have built an industry adjustable load calculation model that is convenient for engineering implementation. It processes data by establishing a survey data warehouse and building an adaptive analysis model for load adjustment. At the same time, in response to the multi-level aggregation of large-scale regional adjustable resources to participate in the interactive response needs of the power grid, a hierarchical and graded control system architecture for load-side resources and an interactive fusion method for adjustable load information are proposed.

[0004] The above disclosed technical solutions have at least the following technical problems:

[0005] However, existing adjustable multimodal data management and scheduling technologies still have obvious limitations. At the data level, traditional multimodal data correlation analysis is difficult to fully explore the correlation between different sources and different types of data. As a result, multimodal information cannot be effectively integrated during data processing, making the analysis results inaccurate.

[0006] In terms of load aggregation, existing algorithms are deficient in the aggregation calculation of new adjustable multimodal data. Traditional algorithms are difficult to adapt to the complex load structure and diverse load characteristics in modern power systems, and cannot accurately aggregate and analyze multimodal data, resulting in a lack of effective data support in the process of power resource allocation and optimization. In view of the above problems, the present invention proposes a solution. Summary of the invention

[0007] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present invention provide an adjustable multimodal data aggregation method and system based on multimodal data fusion, which solves the problems in the above-mentioned background technology by comprehensively applying deep learning technology, multimodal data correlation analysis technology and advanced aggregation algorithms.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] An adjustable multimodal data aggregation method based on multimodal data fusion includes constructing a knowledge graph, and according to a multimodal data fluctuation database, performing data matching on the database to obtain a first number of convolution kernels of different scales and a first range of convolution kernel scales; performing attribute limitation on the convolution kernel of the convolution layer to obtain the optimal scale of each convolution kernel, and dynamically extracting features from the multimodal data according to the optimal scale of each convolution kernel; performing correlation analysis on the extracted features in combination with the knowledge graph, and aggregating information on each power equipment node according to the correlation analysis results.

[0010] In a preferred embodiment, the multimodal data fluctuation database is used to perform data matching on the database to obtain a first number of convolution kernels of different scales and a first range of convolution kernel scales, specifically as follows: a multimodal data fluctuation database is constructed, and a first number of convolution kernels of different scales is obtained according to the multimodal data fluctuation database; and data matching is performed on the multimodal data fluctuation data to be analyzed and the database to obtain a first range of convolution kernel scales.

[0011] In a preferred embodiment, the dynamic feature extraction of multimodal data is performed according to the optimal scale of each convolution kernel, as follows: based on the K-means clustering algorithm combined with the variance analysis method, the distribution of a first number of convolution kernels within a first range is limited to obtain the optimal scale of each convolution kernel; according to the multimodal data fluctuation database, an initial sensitivity value is assigned to each convolution kernel, and forward propagation is performed to obtain the activation frequency of each convolution kernel; according to the activation frequency, the sensitivity of each convolution kernel is obtained, and the convolution kernel for feature map processing is switched according to the sensitivity and the optimal scale of each convolution kernel, so as to dynamically extract features of the multimodal data fluctuation data.

[0012] In a preferred embodiment, the convolution kernel for processing the feature map is switched according to the sensitivity and the optimal scale of each convolution kernel, specifically as follows: the sensitivity of the feature map is evaluated based on the sensitivity analysis algorithm of eigenvalue decomposition; the multimodal database is matched according to the sensitivity of the feature map and the sensitivity of each convolution kernel, the convolution kernel for processing the feature map is switched, the features of the multimodal data are dynamically extracted, and feature map fragments are generated; the feature map fragments are spliced ​​to generate a feature map.

[0013] In a preferred embodiment, the feature map fragments are spliced ​​as follows: the feature map fragments output by the convolution kernels of each scale are spliced ​​to fuse the feature information extracted at different scales.

[0014] In a preferred embodiment, the extracted features are combined with the knowledge graph for correlation analysis, specifically as follows: modal data is obtained, and the modal data is integrated in a preset order to obtain a dimensional tensor; the dimensional tensor includes the number of modes, the number of features of each modality, and the feature vector after feature extraction; each modal data is spliced ​​along the feature dimension to obtain the processed data dimension; the query matrix and the key matrix are obtained by linearly mapping the processed data dimension, and the weight matrix dimensions of the query matrix and the key matrix are obtained; according to the query matrix, the key matrix, and the weight matrix dimensions of the query matrix and the key matrix, an attention score matrix is ​​obtained based on a preset attention score matrix formula; according to the attention score matrix, the relationship category between the corresponding entities is found in the knowledge graph, and the relationship category includes a direct physical connection relationship and a functional dependency relationship; if there is a direct physical connection relationship and a functional dependency relationship between each pair of features, a stability weight is added to the feature; according to the attention score matrix and the stability weight, an attention score matrix is ​​calculated based on a preset attention score matrix formula; the attention score matrix is ​​normalized to obtain a normalized attention score matrix, that is, the correlation between different features.

[0015] In a preferred embodiment, the information of each power equipment node is aggregated according to the correlation analysis results, as follows: a matrix of edge weights is constructed by setting all attention scores below a preset value to 0; an output value is calculated by multiplying a learnable weight matrix, a normalized form of a degree matrix, feature nodes of the current layer, and a matrix for representing edge weights; the output value is nonlinearly transformed through an activation function to obtain node features of the next layer, and feature information of neighboring nodes is aggregated; in each information interaction process, deep feature transformation and update operations are performed on the aggregated information through the synergistic effect of the learnable weight matrix and the activation function.

[0016] A system of an adjustable multimodal data aggregation method based on multimodal data fusion includes a database matching module, a dynamic feature extraction module and an aggregation module, and there are connections between the modules; the database matching module is used to construct a knowledge graph, and according to the multimodal data fluctuation database, the database is matched with data to obtain a first number of convolution kernels of different scales and a first range of convolution kernel scales; the dynamic feature extraction module is used to limit the attributes of the convolution kernel of the convolution layer, obtain the optimal scale of each convolution kernel, and dynamically extract features from the multimodal data according to the optimal scale of each convolution kernel; the aggregation module is used to perform correlation analysis on the extracted features in combination with the knowledge graph, and aggregate the information of each power equipment node according to the correlation analysis results.

[0017] The technical effects and advantages of the adjustable multimodal data aggregation method and system based on multimodal data fusion of the present invention are as follows:

[0018] 1. The technical problem addressed by the present invention focuses on how to efficiently aggregate multimodal data to achieve accurate management and optimized scheduling of adjustable multimodal data in the power system. Most of the existing load aggregation methods are based on traditional machine learning techniques. When processing multimodal data, these methods often find it difficult to capture the deep-level features and complex relationships of the data, limiting the adaptability of the power system to complex power consumption behaviors and the improvement of its intelligence level.

[0019] 2. The present invention solves the above problems by adopting deep learning technology. The powerful ability of deep learning in feature extraction and representation learning enables the present invention to effectively analyze data from different sources and types. By using an AWGCN-based aggregation method combined with the advantages of deep learning automatic feature extraction, the present invention improves the accuracy and efficiency of load aggregation, while enhancing the adaptability of the power system to complex power consumption behaviors and the overall intelligence level of operation.

[0020] 3. The technical solution of the present invention demonstrates extremely significant results in comprehensively improving the level of intelligent management of power systems. The in-depth application of deep learning technology fundamentally optimizes the multimodal data correlation analysis process. In the feature extraction stage, MSCNN extracts feature vectors rich in information. These feature vectors not only cover local detail changes, but also encompass the overall macro trends, providing high-dimensional, multi-level and highly representative information for subsequent analysis. Subsequently, with the help of RelAtten, the multimodal data is subjected to correlation analysis to further explore the complex hidden correlations between the data, so that the features extracted from different sources and types of data can more accurately and comprehensively map the actual operating status of the power system. This data fusion and analysis method driven by deep learning has successfully broken through the inherent limitations of traditional machine learning methods in processing multimodal data, and deeply captured the deep-level characteristics and complex interrelationships of the data. After deep processing by the AWGCN aggregation module, the aggregation results that can accurately reflect the operating status of the power grid and the load change trend are further refined. These aggregation results have broad and in-depth application prospects in many key areas of the power system, including but not limited to real-time monitoring of power grids; optimizing the dispatching link, formulating more scientific and reasonable dispatching strategies based on the aggregation results, and realizing efficient allocation of power resources in different regions and at different times; in the process of formulating demand response strategies, providing key basis for accurately grasping changes in user-side demand, and promoting the balance and coordination of power supply and demand. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a structural schematic diagram of the adjustable multimodal data aggregation method based on multimodal data fusion of the present invention.

[0022] Figure 2It is a schematic diagram of the system structure of the adjustable multimodal data aggregation method based on multimodal data fusion of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Embodiment 1, Figure 1 The present invention provides an adjustable multimodal data aggregation method based on multimodal data fusion, comprising the following steps:

[0025] S1, construct a knowledge graph, and according to the multimodal data fluctuation database, perform data matching on the database to obtain a first number of convolution kernels of different scales and a first range of convolution kernel scales.

[0026] In this embodiment, a knowledge graph is constructed as follows:

[0027] Identify key entities by identifying power system operation data, including substation monitoring data, grid line data, and user power consumption behavior data;

[0028] The key entities include substations, transformers, busbars, circuit breakers and grid lines in the power system operation data;

[0029] Obtain the connection topology data of the device under test and extract the relationship of the device under test. If the bus provides input power to the transformer, and the bus output voltage is consistent with the transformer input voltage, and there is a power transmission relationship, then it is a power supply relationship;

[0030] The identified and extracted entities, relationships, and attribute information are converted into a format that can be received by the graph database through data conversion tools.

[0031] According to the multimodal data fluctuation database, data matching is performed on the database to obtain a first number of convolution kernels of different scales and a first range of convolution kernel scales, as follows:

[0032] Building a multimodal data fluctuation database, and obtaining a first number of convolution kernels of different scales according to the multimodal data fluctuation database;

[0033] The multimodal data fluctuation data to be analyzed is matched with the database to obtain the first range of the convolution kernel scale.

[0034] Furthermore, it is explained what data is included in the multimodal data fluctuation database:

[0035] Transient overvoltage, a spike signal in a very short time, requires a minimum-scale convolution kernel (such as 3×3) for high-resolution extraction;

[0036] Harmonic fluctuations, high-frequency but relatively periodic fluctuations, are suitable for small-scale convolution kernels (such as 5×5) to extract details;

[0037] Tidal current oscillation, medium-frequency periodic fluctuations, the fluctuations are relatively periodic, but the frequency is lower than the harmonics, suitable for medium-scale convolution kernels (such as 7×7) to track the oscillation pattern;

[0038] Low-frequency trend changes (daily load curve): Slowly changing signals that change slowly over time require a larger convolution kernel (such as 9×9) to capture long-term trends;

[0039] Frequency drift (influence of wind power and photovoltaic fluctuations): ultra-low frequency trend, which changes slowly but has a great impact on grid stability, requires ultra-large-scale convolution kernels (such as 11×11) for long-term window analysis;

[0040] The size and corresponding sensitivity of each convolution kernel, as well as the sensitivity of the feature map to be processed.

[0041] According to the matching results, the convolution kernel scale range corresponding to the fluctuation type is extracted from the database, as follows:

[0042] If a transient overvoltage is detected, a small scale range (3×3 to 5×5) is selected;

[0043] If a low-frequency trend is detected, a large scale range (9×11 to 11×11) is selected.

[0044] S2, based on the K-means clustering algorithm and variance analysis method, limit the distribution of the first number of convolution kernels within the first range, obtain the optimal scale of each convolution kernel, and perform dynamic feature extraction on the multimodal data according to the optimal scale of each convolution kernel.

[0045] The dynamic feature extraction of multimodal data is performed according to the optimal scale of each convolution kernel, as follows:

[0046] Based on the K-means clustering algorithm combined with the variance analysis method, the distribution of the first number of convolution kernels within the first range is limited to obtain the optimal scale of each convolution kernel;

[0047] According to the multimodal data fluctuation database, an initial sensitivity value is assigned to each convolution kernel, and forward propagation is performed to obtain the activation frequency of each convolution kernel;

[0048] According to the activation frequency, the sensitivity of each convolution kernel is obtained, and the convolution kernel of the feature map is switched according to the sensitivity and the optimal scale of each convolution kernel, and dynamic feature extraction is performed on the multimodal data fluctuation data.

[0049] The convolution kernel for feature map processing is switched according to the sensitivity and the optimal scale of each convolution kernel, as follows:

[0050] The sensitivity analysis algorithm based on eigenvalue decomposition is used to evaluate the sensitivity of the feature map;

[0051] The multimodal database is matched according to the sensitivity of the feature map and the sensitivity of each convolution kernel, the convolution kernel for processing the feature map is switched, the features of the multimodal data are dynamically extracted, and feature map fragments are generated;

[0052] The feature map fragments are spliced ​​together to generate a feature map.

[0053] The feature map fragments are spliced ​​as follows:

[0054] Before concatenation, some preprocessing of the feature maps may be required to ensure their consistency in spatial resolution and number of channels. When the number of channels is inconsistent, they can be adjusted by duplicating channels, adding zero channels, or applying 1x1 convolutions.

[0055] The adjusted feature maps are concatenated in the channel dimension to ensure that the information of each feature map is preserved and can be further processed and fused in subsequent network layers;

[0056] The concatenated feature map can be used as the input of subsequent network layers for further convolution, pooling, and full connection.

[0057] It should be noted that the activation frequency refers to the activity of each convolution kernel when processing data; the initial sensitivity value is in the range of [0, 1], and the sensitivity of the convolution kernel refers to the response intensity of the convolution kernel to the characteristics of the input data. If a convolution kernel has a higher activation frequency, it means that it responds more strongly to the current input data. Conversely, a convolution kernel with a low activation frequency has a lower sensitivity because it responds less strongly to the current data.

[0058] The calculation formula based on the K-means clustering algorithm combined with variance analysis is as follows:

[0059]

[0060]

[0061] In the formula, is the within-cluster scale variance, represents the average value of the scale within each cluster, represents the value of the scale within each cluster, is the number of cluster sizes, is the global mean of all scales, It is a cluster The mean of It is a cluster The number of points, is the between-cluster scale variance, represents the kth cluster scale.

[0062] It should be noted that K-means clustering is performed on the convolution kernel scales within the first range, and convolution kernels of similar scales are classified into the same cluster. The optimal scale of each convolution kernel is determined by selecting the clustering result with the smallest scale variance within the cluster and the largest scale variance between clusters.

[0063] The calculation formula of the sensitivity is as follows:

[0064]

[0065] In the formula, is the sensitivity value, is the initial sensitivity value, is the learning rate, which controls the speed of sensitivity update, is the activation frequency.

[0066] Prepare multiple groups of convolutional layers of different scales and equip them with convolution kernels of different sizes to dynamically extract features from data at different granularities. Small-scale convolution kernels focus on local fine features, while large-scale convolution kernels cover more macroscopic information.

[0067] The local fine features include small fluctuations in power in a short period of time and local detail changes in spike signals that appear instantly;

[0068] The more macro information includes the periodic change rules and long-term trend trends in the time series;

[0069] The feature maps output by the convolutional layers of each scale are subsequently spliced ​​to fuse the feature information extracted at different scales, ensuring that the final extracted feature vector can not only reflect the dynamic changes of local details, but also comprehensively and macroscopically reflect the overall rules.

[0070] S3, conducts correlation analysis on the extracted features combined with the knowledge graph, and aggregates the information of each power equipment node based on the correlation analysis results.

[0071] In this embodiment, the extracted features are combined with the knowledge graph to perform correlation analysis, as follows:

[0072] When there is When there are different modes, the modal data are integrated in a preset order to obtain a dimension of A tensor of , where N is the number of modes, B is the number of features of each mode, and T represents the feature vector after feature extraction;

[0073] The data of each mode are spliced ​​along the feature dimension to obtain the processed data dimension ;

[0074] By performing linear mapping on the processed data dimensions to obtain the query matrix and the key matrix, the weight matrix dimensions of the query matrix and the key matrix are obtained;

[0075] According to the query matrix, the key matrix, and the weight matrix dimensions of the query matrix and the key matrix, an attention score matrix is ​​obtained based on a preset attention score matrix formula;

[0076] Find the relationship categories between corresponding entities in the knowledge graph according to the attention score matrix, where the relationship categories include direct physical connection relationships and functional dependency relationships;

[0077] If there is a direct physical connection and functional dependency between each pair of features, a stability weight is added to the feature;

[0078] The attention score matrix is ​​calculated based on the preset attention score matrix formula according to the attention score matrix and the stability weight to obtain the attention score matrix;

[0079] The attention score matrix is ​​normalized to obtain the normalized attention score matrix, that is, the correlation between different features.

[0080] The calculation formula of the attention score matrix is ​​as follows:

[0081]

[0082] The calculation formula of the attention score matrix is ​​as follows:

[0083]

[0084] The calculation formula for normalizing the attention score matrix is ​​as follows:

[0085]

[0086] In the formula, is the attention score matrix, is the key matrix, is the query matrix, and is the weight matrix dimension of the query matrix and key matrix, is the attention score matrix, indicating Sample characteristics and The correlation score between the features, is the attention score matrix between feature i and feature j, is the stability weight, represents the normalized attention score matrix, is the number of modes, is the number of features for each mode, represents the kth feature of the ith mode.

[0087] It should be noted that the normalized attention score matrix has a sum of 1 in each row, and the elements in the matrix can be understood as The feature pair The attention probability of each feature is calculated; through the calculation process based on the attention mechanism, the attention score matrix is ​​first obtained, and then the matrix is ​​obtained after normalization, which can comprehensively reflect the complex correlation relationship between different modal features in a quantitative form. The improved attention mechanism based on the knowledge graph can more comprehensively and accurately analyze the intrinsic correlation between multimodal data, and explore the complex correlation laws hidden behind the data, providing more valuable correlation information support for multimodal data fusion and load aggregation.

[0088] S5, according to the correlation analysis result, the information of each power equipment node is aggregated.

[0089] In this embodiment, the information of each power equipment node is aggregated according to the correlation analysis result, as follows:

[0090] By setting all attention scores below the preset value to 0, a matrix for representing edge weights is constructed;

[0091] Get the learnable weight matrix, the normalized form of the degree matrix, the feature nodes of the current layer, and the matrix used to represent the edge weights, and multiply them to get an output value;

[0092] The output value is transformed nonlinearly through the activation function to obtain the node features of the next layer, thereby aggregating the feature information from neighboring nodes;

[0093] In each information interaction process, the aggregated information is deeply transformed and updated through the synergy of the learnable weight matrix and activation function.

[0094] The calculation formula of the node features of the next layer is as follows:

[0095]

[0096] In the formula, is the node feature of the next layer, is the activation function, is the degree matrix, represents the normalized form of the degree matrix, is a matrix used to represent edge weights, For the The feature nodes of the layer, is the learnable weight matrix.

[0097] It should be noted that as the number of AWGCN layers continues to increase, each node can gradually break through the limitations of local information and gradually integrate more global information from other areas in the graph, thereby successfully completing the complex aggregation process from local feature perception to overall feature fusion. Before the entire AWGCN-based aggregation module runs, the number of aggregations is set in advance according to the actual needs of the power system and the analysis objectives. For example, if the number of different power system equipment such as busbars and main transformers is determined , can be determined based on a certain proportion of the number of devices or a specific distribution law; if it is based on the operating status of the power system Such as normal operation, peak load, fault status, etc. Determine the number of categories , can be judged comprehensively based on the complexity of the state and the amount of data, such as under normal operating conditions The value is relatively small. In order to analyze the impact range of the fault in more detail under the fault state The value is large. During the dynamic calculation process of AWGCN, its internal algorithm continuously analyzes and reorganizes the nodes. Once the calculated subgraphs, the training process is stopped immediately. There are different aggregation results, each of which can comprehensively reflect the characteristic information of the data subset it covers related to power, electricity, load change trends, etc. during the operation of the power system.

[0098] The aggregation results are combined with the power system structure and user distribution information in the knowledge graph to generate a visualization chart, as follows:

[0099] The aggregated data is integrated with the knowledge graph to establish a mapping relationship between the aggregation results and the entities in the knowledge graph;

[0100] Import the aggregation results and the information in the knowledge graph into the visualization tool to generate a visualization chart;

[0101] Analyze the relationship between load distribution and power system operation status based on the generated visualization charts, and formulate load adjustment plans based on the analysis results, including optimizing power dispatch, increasing and decreasing power generation capacity;

[0102] According to the adjustment plan, the power system is actually operated and the adjusted power system is continuously monitored.

[0103] Embodiment 2, Figure 2 A system of an adjustable multimodal data aggregation method based on multimodal data fusion of the present invention is provided, comprising a database matching module, a dynamic feature extraction module and an aggregation module, and there are connections between the modules;

[0104] A database matching module is used to construct a knowledge graph and, according to the multimodal data fluctuation database, perform data matching on the database to obtain a first number of convolution kernels of different scales and a first range of convolution kernel scales;

[0105] The dynamic feature extraction module is used to limit the attributes of the convolution kernels of the convolution layer, obtain the optimal scale of each convolution kernel, and perform dynamic feature extraction on multimodal data according to the optimal scale of each convolution kernel;

[0106] The aggregation module is used to perform correlation analysis on the extracted features combined with the knowledge graph, and aggregate the information of each power equipment node based on the correlation analysis results.

[0107] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0108] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0109] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0110] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0111] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0112] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An adjustable multimodal data aggregation method based on multimodal data fusion, characterized in that: The steps include: Construct a knowledge graph, and perform data matching on the database according to the multimodal data fluctuation database to obtain a first number of convolution kernels of different scales and a first range of convolution kernel scales; Based on the K-means clustering algorithm and the variance analysis method, the distribution of the first number of convolution kernels within the first range is limited to obtain the optimal scale of each convolution kernel, and dynamic feature extraction is performed on the multimodal data according to the optimal scale of each convolution kernel; Obtain multimodal data and concatenate them to generate dimensional tensors, and linearly map them to construct query and key matrices. Combine the physical connections and functional dependencies between entities in the knowledge graph to introduce stability weights, correct the attention score matrix and normalize it to obtain the normalized attention score matrix. Based on the normalized attention score matrix, the edge weight matrix is ​​constructed, and the learnable weight matrix is ​​obtained and combined with the activation function for nonlinear transformation to obtain the node features of the next layer and the features of the aggregated neighbor nodes; The first number of convolution kernel distributions is defined as follows: In the formula, is the intra-cluster scale variance, is the average value of the scale within each cluster, is the value of the scale within each cluster, is the number of cluster sizes, is the global mean of all scales, It is a cluster The mean of Cluster The number of points, is the inter-cluster scale variance, is the kth cluster size.

2. The adjustable multimodal data aggregation method based on multimodal data fusion according to claim 1, characterized in that: According to the multimodal data fluctuation database, data matching is performed on the database to obtain a first number of convolution kernels of different scales and a first range of convolution kernel scales, as follows: Building a multimodal data fluctuation database, and obtaining a first number of convolution kernels of different scales according to the multimodal data fluctuation database; The multimodal data fluctuation data to be analyzed is matched with the database to obtain the first range of the convolution kernel scale.

3. The adjustable multimodal data aggregation method based on multimodal data fusion according to claim 2 is characterized in that: The dynamic feature extraction of multimodal data is performed according to the optimal scale of each convolution kernel, as follows: According to the multimodal data fluctuation database, an initial sensitivity value is assigned to each convolution kernel, and forward propagation is performed to obtain the activation frequency of each convolution kernel; According to the activation frequency, the sensitivity of each convolution kernel is obtained, and the convolution kernel for feature map processing is switched according to the sensitivity and the optimal scale of each convolution kernel, and dynamic feature extraction is performed on the multimodal data fluctuation data.

4. The adjustable multimodal data aggregation method based on multimodal data fusion according to claim 3 is characterized in that: The convolution kernel for feature map processing is switched according to the sensitivity and the optimal scale of each convolution kernel, as follows: The sensitivity analysis algorithm based on eigenvalue decomposition is used to evaluate the sensitivity of the feature map; The multimodal database is matched according to the sensitivity of the feature map and the sensitivity of each convolution kernel, and the convolution kernel for processing the feature map is switched to dynamically extract the features of the multimodal data and generate feature map fragments; The feature map fragments are spliced ​​together to generate a feature map.

5. The adjustable multimodal data aggregation method based on multimodal data fusion according to claim 4 is characterized in that: The feature map fragments are spliced ​​as follows: The feature map fragments output by the convolution kernels of each scale are spliced ​​to fuse the feature information extracted at different scales.

6. The adjustable multimodal data aggregation method based on multimodal data fusion according to claim 5, characterized in that: The calculation formula of the attention score matrix is ​​as follows: The calculation formula of the attention score matrix is ​​as follows: The calculation formula for normalizing the attention score matrix is ​​as follows: The calculation formula of the node features of the next layer is as follows: In the formula, is the node feature of the next layer, is the activation function, is the degree matrix, is the normalized form of the degree matrix, is the matrix of edge weights, For the The feature nodes of the layer, is the learnable weight matrix, is the attention score matrix, is the key matrix, is the query matrix, and is the weight matrix dimension of the query matrix and key matrix, is the attention score matrix, indicating the Sample characteristics and The correlation score between the features, is the attention score matrix between feature i and feature j, is the stability weight, is the normalized attention score matrix, is the number of modes, is the number of features for each mode, is the kth feature of the ith mode.

7. A system using the adjustable multimodal data aggregation method based on multimodal data fusion as claimed in any one of claims 1 to 6, comprising a database matching module, a dynamic feature extraction module and an aggregation module, wherein the modules are connected; A database matching module is used to construct a knowledge graph and, according to the multimodal data fluctuation database, perform data matching on the database to obtain a first number of convolution kernels of different scales and a first range of convolution kernel scales; The dynamic feature extraction module is used to limit the attributes of the convolution kernels of the convolution layer, obtain the optimal scale of each convolution kernel, and perform dynamic feature extraction on multimodal data according to the optimal scale of each convolution kernel; The aggregation module is used to perform correlation analysis on the extracted features combined with the knowledge graph, and aggregate the information of each power equipment node based on the correlation analysis results.

Images